take-2 is a byte-identical copy of take-1 in 40 of the 198 test scenario-views #84

In the public bucket, 40 of the 198 take-2 test videos have the same md5 as their take-1, both in split-videos/testing/8FPS/ and in split-videos/testing/30FPS/. Physics-IQ Verified on Hugging Face shows the same pattern: 32 to 35 of its 198 pairs (depending on the FPS folder) point to the same LFS object, and all of them are among those 40.

In #28, take-2 is described as the same scenarios recorded a second time, used to estimate the physical variance that scores are normalized against. For the 35 byte-identical Verified pairs at 24 FPS, the public evaluation CSVs have zero take-to-take MSE.

Affected scenarios in Original (all three views unless noted)

ball-behind-rotating-paper, duck-static, glass-stays-same, light-on-block, light-on-mug, light-on-mug-block, light-on-statue, magnet-domino, match-blows-balloon, mirror-ball-rotate, mirror-teapot-rotate, smiley-ball-rotates, teapot-rotates, and duck-falls-in-box (center view only).

Several of them are dynamic scenes (magnet-domino, match-blows-balloon, ball-behind-rotating-paper), where two separate recordings would not produce identical files.

Effect on a published score

The calculation uses IQTable from physiq/calculate_iq_score_stable.py at 8c5fcdf on the public per-view results of Kandinsky-WM 1.0 (the four seeds in eval_results/ of Messimm/Kandinsky-WM-1.0-Physics-IQ-Verified):

On the 35 copied scenario-views, the MSE component rounds to 0.000 in all four runs; the individual values range from about 1.70e-7 to 2.61e-5. With zero take-to-take MSE, the component is clip(1e-8 / model_mse, 0, 1) for positive model MSE (calculate_iq_score_stable.py:226 and :262). It is therefore approximately zero for these generations, but not for every possible one: a model MSE of zero receives full credit after clipping.

The table reports the mean and the sample standard deviation across the four seeds, computed before rounding. Excluding these views changes the evaluation set and raises the mean by 3.27 points. This is only a sensitivity check. It does not estimate the score after replacing the copied takes with independent recordings, and it does not establish a change in leaderboard ordering.

One related detail: the CSV reader rounds every per-frame value to four decimals (calculate_iq_score.py:24-36). For duck-falls-in-box (center view) the Verified files differ, but all its take-to-take MSE values round to 0.0000, so 36 scenario-views end up with zero physical-variance MSE in the Kandinsky CSVs.

Reproduce (no video download, standard library only)

import json, re, urllib.parse, urllib.request
from collections import defaultdict


def listing(prefix):
    items, tok = [], None
    while True:
        q = {"prefix": prefix, "fields": "items(name,md5Hash),nextPageToken"}
        if tok:
            q["pageToken"] = tok
        url = "https://storage.googleapis.com/storage/v1/b/physics-iq-benchmark/o?" + urllib.parse.urlencode(q)
        d = json.load(urllib.request.urlopen(url))
        items += d.get("items", [])
        tok = d.get("nextPageToken")
        if not tok:
            return items


pairs = defaultdict(dict)
for o in listing("split-videos/testing/8FPS/"):
    m = re.search(r"(perspective-\w+)_take-(\d)_(.+)\.mp4$", o["name"])
    if m:
        view, take, scenario = m.groups()
        pairs[(scenario, view)][take] = o["md5Hash"]
same = sorted(k for k, v in pairs.items() if v.get("1") and v.get("1") == v.get("2"))
print(len(pairs), "pairs,", len(same), "with take-2 identical to take-1")
for scenario, view in same:
    print(scenario, view)

For Physics-IQ Verified, the check compares the Git-object oid of each LFS pointer returned by the Hugging Face tree API at dataset revision b65f567dd8446a4076c7f72446335b3c217a8464. The content SHA-256 itself is masked in the public response; matching pointer objects are evidence of identical referenced content, and no gated video was downloaded. The score comparison runs IQTable.compute_scores_scenario_by_view() on the public CSVs (dataset revision 5e1b90622e45b7b3b85c8418f26886a069b1ea6c) and averages either all views or only the remaining ones. Different file hashes alone do not establish that the remaining videos are independent recordings.

Question

Were these identical take pairs intentional, for example because a second recording was unavailable? If so, could the exceptions and their intended treatment in the physical-variance normalization be documented? If independent recordings were intended, could the released take-2 videos and their corresponding masks be checked together?

The investigation used AI coding assistants, including scripted checks and numerical review.

This isn't my job, it's more of a passion. I enjoy digging into how benchmarks are actually scored, and this one seemed worth asking about.