Repository navigation
Add Cosmos3-Super and Cosmos3-Nano I2V/V2V scores with prompt upsampling - #87
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
Thanks for the PR, and congrats on the results! @tuliti is this PR LGTM on your end? Seems like the upsampled prompts are available at https://huggingface.co/datasets/deepkyu/cosmos3-physicsiq-verified-v2v-prompt-upsampling/raw/main/cosmos3-super-v2v/descriptions.csv and submission cards are provided as well. |
|
thank you for the pr and the results! We have reached out via mail to clarify remaining questions prior acceptance. |
52c932a to
75cc75c
Compare
|
Just wanted to share that we added best-of-n with WMReward + MBR consensus result as well. Thanks. |
|
we thank the team for their thoughtful responses. |
priyankjaini
left a comment
priyankjaini
left a comment
There was a problem hiding this comment.
Thanks for the submission and congrats on the results. LGTM!
Summary
Add Physics-IQ Verified results for Cosmos3-Super and Cosmos3-Nano with GPT-6-Astra prompt upsampling, including best-of-8 selection.
Results shown as mean ± sample standard deviation are calculated across four generation seeds. best-of-8 uses WMReward + MBR Consensus for candidate selection, and those results represent one complete selection run.
Baseline Component Scores
Official Verified component scores (%), reported as mean ± sample standard deviation across four generation seeds. Higher is better for every component, including the normalized MSE score.
Evaluation Protocol
All results cover the full 198 benchmark cases: 66 scenarios * three perspectives.
Prompt upsampling. We use GPT-6-Astra with the official base best-practice descriptions and conditioning inputs, making open-loop API calls with no tool use or web access. Within each input mode, both models (Cosmos3-Super and Cosmos3-Nano) and all seeds use the same frozen set of 198 upsampled prompts.
Baseline. Mean ± sample standard deviation across four generation seeds, with one video per case per seed and no best-of-N selection. These statistics measure generation variation with fixed prompts.
best-of-8. We generate candidates per case using eight seeds, giving 1,584 candidates per model and input mode. One candidate per case is selected using a fixed rank fusion of WMReward and generated-video motion-mask MBR consensus. The same selector configuration is used across models and input modes.
The selector does not access either ground-truth take, ground-truth masks, or benchmark labels. best-of-8 results aggregate the selected candidates' per-case scores.
Generation uses UniPC with 50 steps, guidance 3.0, and 1280×720 output at 24 FPS.
Evidence
Existing baseline videos, exact generation prompts, and submission cards are available at:
I2V submission evidence
V2V submission evidence
best-of-8 uses the same generation prompts documented in the submission cards.