Skip to content

Add Cosmos3-Edge to the TensorRT-LLM Reasoner cookbook - #383

Open
ConstBob wants to merge 3 commits into
NVIDIA:mainfrom
ConstBob:docs/cosmos3-edge-reasoner-trtllm
Open

ConstBob wants to merge 3 commits into
NVIDIA:mainfrom
ConstBob:docs/cosmos3-edge-reasoner-trtllm

Conversation

@ConstBob

@ConstBob ConstBob commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

The cookbook content has moved out of this repo.
Please review the migrated changes here instead:
NVIDIA/cosmos-framework#299

Summary

Add Cosmos3-Edge to the TensorRT-LLM Reasoner cookbook.

Cosmos3-Nano and Cosmos3-Super stay on the pinned nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22 image. That image does not include Cosmos3EdgeForConditionalGeneration, so Cosmos3-Edge is documented as a single-GPU server started from a TensorRT-LLM source build that includes NVIDIA/TensorRT-LLM#20307.

Changes

  • cookbooks/cosmos3/README.md
    • Separates the Nano/Super release-image path from the Edge source-build path.
    • Adds the single-GPU Cosmos3-Edge trtllm-serve command on port 8001.
  • cookbooks/cosmos3/reasoner/README.md
    • Lists Cosmos3-Edge in the TensorRT-LLM quickstart and notebook walkthrough.
  • cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb
    • Keeps the Nano and Super Docker cells on release:1.3.0rc22.
    • Adds an Edge cell that starts trtllm-serve nvidia/Cosmos3-Edge from the current environment and exits with a clear message when trtllm-serve is unavailable.
    • Updates the readiness cell to follow either the Docker container log or the Edge server log, and to ignore a missing Docker daemon when Edge is running outside Docker.

Testing

  • Validated the notebook JSON and ran bash -n on every bash cell.
  • Exercised the readiness cell locally with a stub /health server:
    • the Docker container path reaches ready;
    • the Edge process path reaches ready;
    • an exited Edge process fails instead of waiting until timeout.
  • Ran the Edge server and readiness cells on one L40S with the nvidia/Cosmos3-Edge checkpoint, using a TensorRT-LLM environment that includes the Edge architecture.
    • trtllm-serve nvidia/Cosmos3-Edge --host 0.0.0.0 --port 8001 --max_num_tokens 32768 started successfully.
    • /health became ready.
    • /v1/models returned Cosmos3-Edge.
    • An OpenAI-compatible image request with assets/robot_153.jpg returned a non-empty caption.

The run used max_tokens=128 for the image request; the cookbook example keeps max_tokens=4096. Nano and Super launch commands are unchanged and were not rerun.

Keep the Edge Reasoner server commands on their own branch so they stay out of the TensorRT-Edge-LLM policy docs.
Keep Nano and Super on the pinned release image, and start Edge only from a TensorRT-LLM checkout that includes Cosmos3EdgeForConditionalGeneration.
…adiness cell.

Edge starts without a container, so a docker CLI with no daemon should not spam the wait loop.
@akushonkamen

Copy link
Copy Markdown

Hi! I'd like to work on adding Cosmos3-Edge to the TensorRT-LLM Reasoner cookbook. Will submit a PR shortly.

@akushonkamen

Copy link
Copy Markdown

Update: I found that this work is already in flight as NVIDIA/cosmos-framework#299 (the cookbook content has migrated to cosmos-framework). Withdrawing to avoid duplication — apologies for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants