> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sglang.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Cosmos3

> Serve NVIDIA Cosmos3 image, video, sound, and action generation with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation.md) for your GPU platform.

Choose a [generation or policy request](#4-openai-compatible-requests) matching the checkpoint; the Request tab queries model metadata.

> **Installation and prerequisites**
>
> Follow the [SGLang Diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation.md). Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment.
>
> Cosmos3 guardrails are enabled by default when the package is available:
>
> ```bash Command theme={null}
> pip install "cosmos-guardrail==0.3.1"
> ```
>
> `cosmos-guardrail` downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` before starting the server.
>
> There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the *\_pickle.UnpicklingError* error occurs during startup, you should change `weight_only=True` to `weights_only=False` parameter in *cosmos\_guardrail/cosmos\_utils.py*:
>
> ```
> #!/usr/bin/env bash
> COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")"
> sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py"
> ```

## 2. Model capabilities

[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler.

Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository.

| Model | Status | Notes |
| - | - | - |
| `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V, V2V, joint sound, and action |
| `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint |
| `nvidia/Cosmos3-Super-Text2Image` | Supported | T2I-specialized checkpoint |
| `nvidia/Cosmos3-Super-Image2Video` | Supported | I2V-specialized checkpoint |
| `nvidia/Cosmos3-Nano-Policy-DROID` | Supported | DROID policy action generation |
| `nvidia/Cosmos3-Edge` | Supported | 4B dense model for T2I, T2V, I2V, V2V, and action generation |
| `nvidia/Cosmos3-Edge-Policy-DROID` | Supported | 4B DROID policy action generation |
| `nvidia/Cosmos3-Super-Text2Image-4Step` | Supported | 64B T2I checkpoint distilled to a fixed 4-step schedule |
| `nvidia/Cosmos3-Super-Image2Video-4Step` | Supported | 64B I2V checkpoint distilled to a fixed 4-step schedule |

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native `FlowUniPCMultistepScheduler`; distilled checkpoints use the fixed sigma schedule stored in the checkpoint.

The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all action modes, and `3.0` for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level `flow_shift`.

## 3. Serve Cosmos3

For Intel Xeon CPU deployment, use `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` with the selected Cosmos3 checkpoint.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path nvidia/Cosmos3-Nano --port 30010
```

Serve `Cosmos3-Nano` directly from the Hugging Face model ID:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1
```

With `--performance-mode auto`, Cosmos3 Nano keeps its DiT and VAE resident
when every selected GPU has at least 90 GiB available at startup. Other
Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold,
auto mode retains the conservative DiT component-offload policy. Cosmos3 runs
one DiT per pipeline, so component offload above the threshold only pays to
copy the weights out to host memory and back on every request. Serve
`Cosmos3-Super` across multiple GPUs as shown below so each rank holds a shard
of the weights.

For `Cosmos3-Super`, split the model across multiple GPUs:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4
```

The server also accepts the specialized `nvidia/Cosmos3-Super-Text2Image` and `nvidia/Cosmos3-Super-Image2Video` checkpoint IDs.

### Edge checkpoints

`Cosmos3-Edge` is a 4B dense model and can be served on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1
```

Edge is trained for 256p and 480p generation. Its default video configuration is `832x480` with `guidance_scale=5.0`; its default image configuration is `640x640` with `guidance_scale=7.0`. Supported sizes are `832x480`, `480x832`, `640x480`, `480x640`, `480x480`, `640x640`, `448x256`, `256x448`, and `256x256`.

Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with `nvidia/Cosmos3-Edge-Policy-DROID`.

### Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4
```

For distilled I2V, replace the model path with `nvidia/Cosmos3-Super-Image2Video-4Step`. SGLang detects both checkpoints from `scheduler/scheduler_config.json`, uses the checkpoint's fixed four-step sigma schedule, and forces `guidance_scale=1.0`. Do not tune `num_inference_steps` or `flow_shift` for these checkpoints.

### Simulation and multiview checkpoints

These checkpoints use native pipelines with checkpoint-specific conditioning:

| Checkpoint | Inputs | Rollout |
| - | - | - |
| `nvidia/Cosmos3-Nano-Sim-Bimanual` | Image, prompt, camera or robot actions | Causal chunks with committed KV history |
| `nvidia/Cosmos3-Nano-Sim-Transfer` | Control video and prompt | Causal control-video transfer |
| `nvidia/Cosmos3-Nano-Transfer-Auto` | Per-camera WSM clips and captions; optional LiDAR | Bidirectional multiview generation |
| `nvidia/Cosmos3-Nano-Sim-Auto` | Per-camera WSM clips and captions; optional LiDAR | Causal multiview generation |

For example, start the Bimanual server:

```bash theme={null}
sglang serve --model-path nvidia/Cosmos3-Nano-Sim-Bimanual
```

Submit image-conditioned video requests through `/v1/videos`. Supply `action`
rows according to the checkpoint's embodiment contract, selected by `domain_name`
or `domain_id`. Native contracts accept raw rows with `action_normalization="contract"`;
unified contracts require normalized 59-dimensional rows and default to
`action_normalization="none"`. Without actions, Bimanual uses null-action conditioning.
These pipelines predict video from actions, not actions from video; use the policy
checkpoints above for action prediction.

For realtime Bimanual sessions, add
`--pipeline-class-name CosmosDreamsRealtimePipeline` and use
[`/v1/realtime_video`](https://docs.sglang.io/docs/sglang-diffusion/realtime_models).
Each tick consumes one action block and preserves causal KV history.

Transfer-Auto and Sim-Auto take a `multiview` object with `views`, each containing
`camera_key`, `control_path`, optional `vision_path`, and optional `prompt`. Joint checkpoints
also accept `lidar`; set `lidar.decode=true` to return decoded range maps and previews.
Outputs pack all frames of each camera consecutively, in request camera order.

LiDAR uses built-in compiled FlexAttention by default. The optional NATTEN path
requires a wheel matching your PyTorch/CUDA versions and
`SGLANG_DIFFUSION_COSMOS3_LIDAR_USE_NATTEN=1`; it is not installed by default.

Bimanual and Sim-Transfer support TP, but not SP or CFG parallelism. Transfer-Auto
supports Ulysses without TP or Ring; Sim-Auto supports TP and Ulysses subject to
the checkpoint's query/KV head divisibility, but not Ring or CFG parallelism.
Causal pipelines use the exported fixed-step schedule and require guidance `1.0`.

Use `--component-residency` for memory-constrained deployments. Joint checkpoints
register `lidar_encoder` and `lidar_decoder` separately from the camera `vae`.
Bimanual overlaps causal VAE decode only when the VAE is resident; offloaded VAEs
use the final serial decode stage. History-window reduction and frustum retrieval
change the attention context and are not lossless acceleration settings.

## 4. OpenAI-Compatible Requests

### Text to image

Cosmos3 text-to-image uses `/v1/images/generations`. The default Cosmos3 image response is `b64_json`, matching vLLM-Omni's examples.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

With a server running `nvidia/Cosmos3-Super-Text2Image-4Step`, omit the scheduler controls and use `guidance_scale=1.0`:

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

### Text to video with sound

Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4
```

### Image to video

This mirrors the official `nvidia/Cosmos3-Nano` Hugging Face image-to-video example:

```python Python theme={null}
import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)
```

For the distilled I2V checkpoint, use the same API with a server running `nvidia/Cosmos3-Super-Image2Video-4Step`. The recommended request is 480p and does not specify scheduler controls:

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Video to video

Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Action generation

For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1
```

Use `nvidia/Cosmos3-Edge-Policy-DROID` in the same command to serve the smaller 4B policy checkpoint.

`policy` and `inverse_dynamics` return actions, so their canonical API is the synchronous `/v1/actions/generations` endpoint. The following request predicts a 16-step action chunk from one observation image. `action_horizon=16` maps to the model's `num_frames=17` convention.

```python Python theme={null}
import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])
```

Use `GET /v1/actions/metadata` to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the `/v1/actions/realtime` websocket use the same action envelope.

To batch policy observations inside one request, opt in with a bounded batch size:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1 \
  --batching-max-size 4
```

Send one image per observation as a list or `[B, H, W, C]` uint8 array in `input.input_reference`, and either one prompt per image or one scalar prompt to broadcast across the batch. Batched prompts must currently tokenize to the same length because Cosmos3 GEN cross-attention does not mask padded text K/V. All items in one request share the domain, resolution, action horizon, and denoise settings. The standard action envelope returns one `data[i]` item per input, each with action shape `[H, D]`. For a compact msgpack response containing one `[B, H, D]` array, set `runtime.response_format="raw"` and read the top-level `actions` field.

For JSON, `input_reference` can be a list of base64 image payloads. For msgpack, it can be a packed uint8 numpy array directly:

```json JSON theme={null}
{
  "input": {
    "prompt": ["pick up the block", "close the drawer"],
    "input_reference": [
      {"b64_json": "<first-image-base64>"},
      {"b64_json": "<second-image-base64>"}
    ]
  },
  "parameters": {
    "action_mode": "policy",
    "domain_name": "droid_lerobot"
  }
}
```

The batch size cannot exceed `--batching-max-size`; this keeps one request from bypassing the server's configured memory limit. Batching applies to `action_mode="policy"` only. A request seed controls the random stream for the whole batch, so a batched result is deterministic for that request but is not expected to be bit-exact with separately seeded B=1 requests.

`inverse_dynamics` also uses `/v1/actions/generations`; set `action_mode="inverse_dynamics"` and pass an observation video URL or server-local path as `input.observation.video`. Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name.

`forward_dynamics` is intentionally different: it consumes an action array and predicts video, so it remains on `/v1/videos`. Action-producing modes submitted to `/v1/videos` return HTTP 400 with the canonical action endpoint in the error message.

## 5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`. For distilled checkpoints, SGLang replaces `num_inference_steps` with the checkpoint's fixed four-step schedule and forces `guidance_scale=1.0`; negative-prompt CFG and request-level `flow_shift` do not apply.

Top-level Cosmos3 request fields:

* `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer.
* `flow_shift`: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I, `10.0` for non-Edge video and all action modes, or `3.0` for Edge video).
* `guidance_interval`: optional `[start, end]` noise interval for CFG. Non-distilled T2I defaults to `[400, 1000]`; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

* `generate_sound`: generate a sound track whose duration follows `num_frames / fps`.
* `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration.
* `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`.
* `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning.
* `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`.
* `domain_name` / `domain_id`: select the action embodiment head.
* `raw_action_dim`: number of active action dimensions; inferred for known domain names.
* `action`: action array with shape `[T, D]`, required by `forward_dynamics`.
* `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS.
* `action_view_point`: viewpoint used in the structured action caption.
* `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`.

Pass model-specific controls through `extra_body` with the OpenAI Python SDK.
Raw JSON may keep them at the top level; multipart video requests should put
them in the `extra_params` JSON object. The legacy image `extra_args` container
remains accepted for compatibility, but new clients should use `extra_body`:

* `use_duration_template`: whether to append SGLang's generated duration suffix to video prompts.
* `use_resolution_template`: accepted for vLLM-Omni request compatibility.
* `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template.
* `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.