Skip to content

Add Physis-Lang (Cosmos3 Nano) and Physis-Lang (Cosmos3 Super) to Physics-IQ Verified - #83

Merged
rgeirhos merged 5 commits into
google-deepmind:mainfrom
xianzhengma:codex/dense-physics2-a0-eval
Sep 28, 2026
Merged

rgeirhos merged 5 commits into
google-deepmind:mainfrom
xianzhengma:codex/dense-physics2-a0-eval

Conversation

@xianzhengma

@xianzhengma xianzhengma commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

We propose two entries for the Physics-IQ Verified I2V leaderboard:

Model Input type Physics-IQ Verified
Physis-Lang (Cosmos3 Nano) i2v 43.29 +/- 1.52
Physis-Lang (Cosmos3 Super) i2v 48.23 +/- 1.43

These are fine-tuned Cosmos-based models, not the unmodified Cosmos baselines. Results are self-reported and submitted for verification. Values are percentages, reported as mean +/- sample standard deviation across four independent generation runs.

Evaluation protocol

  • Physics-IQ Verified, with I2V conditioning using the Verified switch frames.
  • Four runs with seeds 0, 1, 2 and 3; 198 videos per run, 792 per model.
  • Validated scoring-video settings: 1280x720, 24 FPS, 120 frames (five seconds), for all 792 videos per model.
  • Verified final_score_view, converted to a percentage. Aggregation uses unrounded per-run results.
  • No Best-of-N sampling or score-based selection among the four runs.
  • Our own custom prompts, consisting of base captions plus physical reasoning, frozen across runs, with the Cosmos default negative prompt. These are not the official original descriptions or unmodified official best-practice prompts. Offline prompt preprocessing is part of the method.
Seed Physis-Lang (Cosmos3 Nano) Physis-Lang (Cosmos3 Super)
0 43.73 49.83
1 43.13 48.44
2 41.31 46.36
3 44.97 48.30
Verified submetric Nano mean +/- SD Super mean +/- SD
Spatial IoU 56.17 +/- 1.76 59.87 +/- 1.43
Spatiotemporal IoU 31.01 +/- 2.16 41.57 +/- 3.11
Weighted spatial IoU 43.39 +/- 2.16 48.53 +/- 1.06
MSE score 42.58 +/- 0.85 42.96 +/- 1.29

Method documentation

Paper PDF: https://github.com/Physis-Intelligence/Physis-Lang/blob/main/assets/Physis-Lang_arxiv.pdf

Our manuscript also includes evaluations on PhyGenBench, VideoPhy2 and PhyGround. We will update this PR with the arXiv link when available.

Inference hardware and timing

Model Generation hardware GPUs per sample Mean logged wall seconds/video GPU-seconds/video
Physis-Lang (Cosmos3 Nano) NVIDIA A100 4 215.18 860.71
Physis-Lang (Cosmos3 Super) NVIDIA GB300 4 203.93 815.70

Hardware models are confirmed by the submitter. Timings were recovered from all 792 accepted samples per model, using the interval from saved sample arguments to saved sample outputs. They include output work, exclude process/model startup, and are not CUDA-kernel profiling measurements. GPU-seconds equal the unrounded mean logged interval multiplied by four; the cards explicitly document this unit.

Prompt upsampling used GPT-5.5 through an API, not local upsampling GPUs. Measured monetary costs are unavailable; unknown costs are not reported as zero.

Review materials and submission status

We request the private upload route described in README section H for the actual prompt CSVs, generated videos, evaluation outputs, submission cards and current manuscript. Please provide a run ID and scoped upload credentials for each model. We will upload the materials to the destination you provide; they have not yet been uploaded.

Both models have passed the official CSV/video component checks for all four runs (792 videos each). Runtime prompt consistency and archive read-back hash checks have also passed. The Google CLA check has passed.

The submitter has confirmed acceptance of submission terms version 1.0 and all five card warranties on behalf of NVIDIA; both cards record that confirmation at 2026-09-26T02:00:33Z. The complete official --validate-only check has not yet passed because the current validator rejects the three null GPU/count/time fields for API-based prompt upsampling. Media validation is not a claim of final submission acceptance. Private artifacts have not been uploaded, and we will update delivery and full-validation status before requesting final acceptance.

Two points we would appreciate guidance on:

  1. For a privately available video model using an API-based prompt upsampler, the template permits null upsampling GPU/count/time fields, but the current validator requires them based on video-model availability. What representation should we use? Our own model-specific descriptions are recorded as bpp, with the actual CSV provided privately.
  2. Can unknown monetary costs be represented as NaN, with generation hardware and measured timing provided for a maintainer estimate? We do not have measured billing data.

Proposed repository change

Add two rows to the Physics-IQ Verified table in README.md. The Original leaderboard and existing model scores are unchanged. The proposed rows are marked pending review; final numbering, dates and ranking annotations will be reconciled with the live table before merge.

Please also advise on the corresponding website update after acceptance.

@google-cla

google-cla Bot commented Sep 25, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@xianzhengma
xianzhengma marked this pull request as ready for review September 25, 2026 04:45
@rgeirhos

rgeirhos commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks for the PR! Happy to merge; if you don't mind could you do the following changes:

  1. sync to latest README version
  2. update PR with correct leaderboard numbering and dates (replacing the 'pending review' entries)
  3. following the pattern in the current README (<model> reported <here>) with links to paper & results

Thank you!

@xianzhengma
xianzhengma force-pushed the codex/dense-physics2-a0-eval branch from 534e0ae to 4cd4eca Compare September 28, 2026 13:00
@xianzhengma

Copy link
Copy Markdown
Contributor Author

Thanks for the guidance. I’ve synced the README with the latest version and updated the leaderboard positions and dates to match the live leaderboard. The model names link directly to the paper PDF, and each “reported here” link points to this PR, following the existing README format. Please let me know if any further changes are needed.

@rgeirhos rgeirhos left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rgeirhos
rgeirhos merged commit 7862050 into google-deepmind:main Sep 28, 2026
6 checks passed
@rgeirhos

Copy link
Copy Markdown
Collaborator

Thanks @xianzhengma, much appreciated. The PR is now merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants