feature/adm/cuda: read the contrast masking neighbours of scales 1 to 3 as the CPU does - #1647

integer_adm_scale1 to integer_adm_scale3 of adm_cuda differ from the CPU on every frame that is small in either dimension: up to 1.0e-2 at scale 1, 5.6e-2 at scale 2 and 2.7e-1 at scale 3 on frames of 17 to 64 pixels, and 1.7e-3 on a real 160x90 clip. This is bug 1 of #1564; the other bugs of that issue are separate changes.

Cause

i4_adm_cm_line_kernel() (libvmaf/src/feature/cuda/integer_adm/adm_cm.cu) steps the three neighbours of a sample with offset_i and offset_j. At i == 0 (j == 0) with top <= 0 (left <= 0) it reads the rows {1, 2, 3} (columns likewise) and takes the centre sample from row 2. The CPU's I4_ADM_CM_THRESH_S_* macros (integer_adm.c) mirror around row 0: {1, 0, 1}, centre in row 0. At the bottom and right borders both sides replicate, {n - 2, n - 1, n - 1}.

top and left are 0 whenever a band has 14 samples or fewer in that dimension. At scale 3 that is a frame of 224 pixels or fewer, at scale 2 of 112 or fewer, at scale 1 of 56 or fewer, so the border branches are live on small frames.

Reproducer

Master 8e7a1ac4e, -Denable_cuda=true, RTX 4090. The CLI prints six digits. Real clip, the 160x90 pair of the Netflix test resources (ref_test_0_1_src01_hrc00_576x324_576x324_vs_src01_hrc01_576x324_576x324_q_160x90.yuv, 48 frames), CPU scalar against adm_cuda:

vmaf -r ref_q_160x90.yuv -d dis_q_160x90.yuv -w 160 -h 90 -p 420 -b 8 --feature adm --cpumask -1 --no_prediction --json -o cpu.json
vmaf -r ref_q_160x90.yuv -d dis_q_160x90.yuv -w 160 -h 90 -p 420 -b 8 --feature adm_cuda --no_prediction --json -o cuda.json

Synthetic frames, 17x17 to 64x64 and fifteen other sizes up to 320x180, 8 and 10 bits, random and structured content, two frames each (212 configurations; the generator is the one in the new test). adm_cuda against the scalar CPU, compared at %.17g through the C API. Largest absolute difference over the configurations of the size group:

For frames with a side of 32 pixels or fewer the scalar CPU on master is itself wrong at scale 3 (#1599); those rows use the scalar CPU with the fixes of #1599, #1600, #1602 and #1633 merged locally as the reference. Both dimensions of 33 or more give the same scalar CPU scores on master and with those fixes (checked on all 212 configurations).

Fix

The three positions of a neighbourhood are taken as min(abs(p - 1), n - 1), p and min(p + 1, n - 1). That is {1, 0, 1} at the top and left border, {n - 2, n - 1, n - 1} at the bottom and right border, and no position leaves the band. The interior is unchanged. This is the index scheme #1564 proposes for bug 1.

Tests

test_cuda_adm_cm_border (new, libvmaf/test/, built with enable_cuda; it reports a skip when no CUDA device can be opened) compares integer_adm_scale1 to integer_adm_scale3 of adm_cuda with the scalar CPU adm on frames of 33x33, 48x48, 66x34, 100x33 and 130x40 at 8 and 10 bits, random and structured content, with a tolerance of 1e-9. Unpatched master fails it (33x33, 8 bit, random content: 1.83e-2); this branch passes it. The sizes keep both dimensions at 33 or more, so that the test does not depend on the CPU fixes.

Validation

Master 8e7a1ac4e, 2026-10-01. x86-64 Linux, GCC 16.2.1, nvcc 13.4, RTX 4090.

Rebased on master 9e48141b (2026-10-02). On that base, with -Denable_cuda=true -Denable_float=true -Denable_checkasm=true, meson test gives 28 of 29 (the failure is test_cuda_pic_preallocation, which also fails on unpatched 9e48141b, 27 of 28 there), and the CPU -Db_sanitize=address,undefined -Db_lto=false build gives 22 of 25 on master and here (test_predict and test_pic_preallocation abort under LeakSanitizer and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603); all three also abort on unpatched 9e48141b). The CLI output of the three Netflix pairs (--gpumask 0, vmaf_v0.6.1) against unpatched 9e48141b: identical on all three pairs (src01 76.668905, checkerboard 1 px 35.068667, 10 px 7.985899). The sweeps and the other measurements below were taken on 8e7a1ac4e and not repeated; the files and x86 code paths they depend on are unchanged since (the one upstream change in between is an arm64-only ADM kernel).

• Merging this branch with the four other CUDA ADM branches of the series gives a clean tree except for libvmaf/test/meson.build, where every branch adds its test at the same place.
• This branch alone against the scalar CPU, 212 configurations of the sweep above: scales 1 to 3 identical in all of them. Scale 0 is unchanged from master (it still differs on 88 of the 106 size / bit depth rows; see the companion changes).
• Real pairs, this branch against master, adm_cuda at %.17g (integer_adm2, scales 0 to 3 and the numerators and denominators): src01 576x324 8 bit (48 frames) and 10 bit (3 frames), checkerboard 1 px and 10 px 1920x1080 (3 frames each), KristenAndSara 1280x720, akiyo 352x288, sparks 480x270 10 bit (5 frames): identical to master and to the scalar CPU. No golden assertion changes.
• compute-sanitizer --tool memcheck reports no error for master and for the merge of the five branches at 24x24, 20x36, 33x33 and 48x48: the rows past the band that the old scheme reads are inside the allocation.
• Full meson test on the merge of the five branches: 28 of 29 pass. test_cuda_pic_preallocation aborts with SIGSEGV in test_cuda_picture_preallocation_method_host_pinned; it does the same on unpatched master.

The workflow run on this PR needs a maintainer's approval.