Repository navigation
Cosmos3 Nano & Super-I2V IQ-Verified Score Reproduction Discrepancies #75
Description
Hi,
I'm from the Cosmos team. I was trying to reproduce the Physics-IQ Verified scores for Cosmos3-Nano and Super-I2V displayed on the leaderboard (30.3 +/- 0.6 and 39.5 +/- 0.8 respectively).
However, my scores (below) for these two models are consistently significantly higher than what is reported on the leaderboard. For reference, I am using the same config (link) and upsampling the bpp prompts + switch-frame (per our docs) via Opus-4.8. Note, we generate 121 frames and drop the first frame as documented in our repo. In fact, for Nano, I found that even passing only the raw/original bpp directly into the model (without any prompt upsampling) outperforms the published score. For inference, I am using vllm-omni for Super-I2V and cosmos-framework for Nano as per our publicly released documentation (note Super-I2V wasn't supported via cosmos-framework at time of writing). All jobs were run on NVIDIA GB200 (NVL72) gpus.
| Model | Prompts | Resolution | Physics-IQ Verified (n=4) | Per-seed (0 / 1 / 2 / 3) | Published | Δ |
|---|---|---|---|---|---|---|
| Cosmos3-Super-Image2Video | upsampled bpp | 1280×720 | 42.68 ± 0.76 | 43.39 / 41.66 / 42.53 / 43.12 | 39.5 ± 0.8 | +3.18 |
| Cosmos3-Nano | upsampled bpp | 1280×720 | 37.25 ± 0.89 | 36.57 / 37.74 / 36.43 / 38.24 | 30.3 ± 0.6 | +6.95 |
| Cosmos3-Nano | raw bpp | 1280×720 | 34.39 ± 1.20 | 35.47 / 33.25 / 33.46 / 35.38 | 30.3 ± 0.6 | +4.09 |
Here's the subscores
| Model | Prompts | Verified Score | Verified Spatial IoU | Verified Weighted Spatial IoU | Verified Spatiotemporal IoU | Verified MSE |
|---|---|---|---|---|---|---|
| Cosmos3-Super-Image2Video | upsampled bpp | 42.68 ± 0.76 | 57.41 ± 1.51 | 42.98 ± 1.47 | 32.48 ± 0.97 | 37.84 ± 0.41 |
| Cosmos3-Nano | upsampled bpp | 37.25 ± 0.89 | 51.91 ± 2.26 | 36.30 ± 1.13 | 25.21 ± 1.33 | 35.56 ± 0.78 |
| Cosmos3-Nano | raw bpp | 34.39 ± 1.20 | 45.01 ± 2.29 | 29.82 ± 1.40 | 24.58 ± 3.25 | 38.14 ± 1.07 |
I can provide any additional artifacts, if needed.
Given this discrepancy, should I create an MR to update the leaderboard scores?
Activity
tuliti commentedon Aug 31, 2026
Hi @akashgokul,
thank you for running this on your end as well!
Will take a detailed look tomorrow and let you know, how to proceed.
blahner commentedon Aug 31, 2026
Hi, +1 on this issue. I've been getting better than published results on I2V cosmos3-nano with no upsampling for both op and bpp. I also use seeds 0, 1, 2, and 3 for 4 runs. I'm using cosmos-framework, and my config is slightly different than @akashgokul, possibly accounting for our differences:
Guidance: 6.0
num_steps: 35
shift: 10.0
sigma_max: 80.0
tuliti commentedon Sep 1, 2026
Hi @akashgokul,
@blahner thank you for your input.
A related older issue seems to have had an even lower score (see link).
We are currently running some sanity checks and will add the best ranked version for bpp and upsampled bpp to the repo.
tuliti commentedon Sep 7, 2026
added to the leaderboard and nvidia is uploading the videos.
can be closed. @rgeirhos
rgeirhos commentedon Sep 7, 2026
Closing; please feel free to re-open if necessary!