feature/adm: reject frames below 17 pixels in either dimension at init - #1642

Integer ADM crashes on any frame whose width or height is 16 or below. This makes init() refuse those frames with -EINVAL and a message that names the limit. Fixes #1607.

Why refuse instead of compute

#1607 asked whether these sizes should work or be refused, and nobody has answered. The history points at refusing: #1035 reported this crash in 2022, #1046 fixed it by rejecting w <= 32 || h <= 32 in init() ("ADM probably doesn't make sense as a metric at these very small patch sizes"), and 966be8d (#1485) dropped that check.

float_adm is the reference definition, and it does not define a score for these frames either. Same synthetic 4:2:0 content (3 frames, pseudo-random luma/chroma with a distortion of up to +-12), float_adm at frame index 2:

ADM is a ratio of retained to original detail, so a score of 1.4 (and 3.1 for scale 3 at 16x16) is not a measurement. Making the integer path return numbers at these sizes would give it values its own reference does not have, so it refuses them.

Cause

Two defects behind the crash, both from #1607 and re-checked on master 8e7a1ac4:

1. adm_cm() computes (uint32_t)ceil(log2(w) - 4) for shift_xhcub and shift_xvcub (- 3 for shift_xdcub). For a scale 0 band of 8 samples or fewer, that is a frame dimension of 16 or less, the double is negative and converting it to uint32_t is undefined; every >> shift_xhcub after it shifts by 4294967295. The same expression is in adm_avx2.c (adm_cm_avx2), adm_avx512.c (adm_cm_avx512) and in the CUDA host code (integer_adm_cuda.c).
2. dwt2_src_indices_filt() loops for (i = 1; i < h_half - 2; ++i) with an unsigned h_half. A 16-pixel dimension reaches scale 3 with a 2-sample input (16 to 8 to 4 to 2), so h_half == 1, the bound wraps to 4294967295 and the loop writes past the index table. Same for w_half.

Where the boundary is: a dimension d reaches scale 3 as a DWT input of ceil(d / 8) samples and the transform needs 3, which is d >= 17. That is also exactly where ceil(log2(band) - 4) stops being negative.

Reproducer

Master 9e48141b, release build, x86-64, 16x16 4:2:0 clips cut from the src01 pair (3 frames):

head -c 1152 python/test/resource/yuv/src01_hrc00_576x324.yuv > ref_16x16.yuv
head -c 1152 python/test/resource/yuv/src01_hrc01_576x324.yuv > dis_16x16.yuv
build/tools/vmaf -r ref_16x16.yuv -d dis_16x16.yuv -w 16 -h 16 -p 420 -b 8 \
  --feature adm --no_prediction -q -o /dev/null

With --model version=vmaf_v0.6.1 instead of --feature adm, this branch refuses the 16x16 clip with the same message and exit 234.

Fix

init() returns -EINVAL and logs the size and the limit when w or h is below ADM_MIN_DIM (17, defined in integer_adm.h with the derivation). Frames of 17 and above take the same path as before. A second commit applies the same check at the top of init_fex_cuda(), because adm_cuda carries the same host expression and is picked for the same frames when a CUDA context exists; drop that commit if you would rather keep this CPU-only.

The process exit status for the refused run changes from 139 (SIGSEGV) to 234 (-EINVAL as an unsigned byte).

What this does not touch

• 17 to 32 pixels. Those sizes are covered by feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 (scale 3 index) and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 (rounding constant at band widths 9 to 16); this change is independent of both. They merge with this branch without conflicts (git merge-tree).
• checkasm. post_dwt_sizes in check_adm.c has a 16x16 entry, and check_adm_cm passes half of it, an 8-sample band, to adm_cm() and its SIMD twins directly, not through init(). A UBSan build of checkasm --test=adm with -fsanitize=float-cast-overflow still reports -1 is outside the range of representable values of type 'unsigned int' at integer_adm.c:1567 and the matching lines in adm_avx2.c and adm_avx512.c (153 reports with this branch and feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 merged). Removing the sub-17 entries from that table, or clamping the shift in the helpers, is a separate decision; this change does not make either.

Tests

New test_adm_minimum_dimension in libvmaf/test/test_feature_extractor.c: init() fails with -EINVAL for 1x1, 8x8, 16x16, 64x16 and 16x64 and succeeds for 17x17, 64x17, 17x64 and 64x64. With master's integer_adm.c and the new test, the case fails ("adm init should reject a frame below 17 pixels in either dimension") in a release build, no sanitizer needed; with this change test_feature_extractor passes 5/5.

Validation

x86-64 Linux (AVX-512 host), GCC 16.2.1, on master 9e48141b. The sanitizer builds use -Db_sanitize=address,undefined -Db_lto=false and add -fsanitize=float-cast-overflow, which GCC's undefined group does not include.

Rebased on master 9e48141b (2026-10-02). The size sweep and the CUDA run below were taken on 8e7a1ac4 and not repeated (integer_adm.c, adm_avx2.c, adm_avx512.c and integer_adm_cuda.c are unchanged between the two, and integer_adm.c differs only by one NEON dispatch line in init(), after the lines this change touches); the reproducer table, the meson test counts and the Netflix pair comparison were re-run on 9e48141b. On the CUDA build, meson test is 24 of 25 on master and here (test_cuda_pic_preallocation fails on both), and the CLI output with --gpumask 0 on the three Netflix pairs is identical to master.

• Size sweep with a small driver on the C API (odd sizes included, so it is not confounded by the raw reader of tools: read frames with an odd width or height #1604): every square size 1 to 34 and 24 non-square sizes (64x1, 1x64, 64x2, 2x64, 64x8, 8x64, 64x15, 15x64, 64x16, 16x64, 64x17, 17x64, 64x32, 32x64, 576x16, 16x576, 100x17, 17x100, 33x16, 16x33, 20x16, 16x20, 18x35, 35x18) at the default dispatch, --cpumask 16 and --cpumask -1: 174 runs per build.
  • Master, release: 78 score, 96 exit with SIGSEGV (every size with a dimension of 16 or less).
  • Master, -Db_sanitize=address,undefined plus -fsanitize=float-cast-overflow: 17 runs finish without a report, 157 report.
  • This branch, release: 78 score, 96 refused with the message, no crash.
  • This branch, sanitizer: 17 clean, 96 refused, 61 report. Every one of the 61 is a size from 17 to 32 (and 64x17, 17x64, 17x100, 18x35, 32x64), the range feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 fix; none has a dimension below 17.
  • This branch with feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 merged, sanitizer: 77 clean, 96 refused, 1 report: 64x17 at --cpumask 16, shift exponent -5 is negative in get_best15_from32() (adm_avx2.c:1350), the known AVX2 defect of feature/adm: keep get_best15_from32() inside its defined range on AVX2 #1635, content dependent and not size related.
  • feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 merged without this branch, sanitizer: 77 clean, 96 crash. They do not reach the sizes below 17, as the issue said.
• meson test (-Denable_float=true -Denable_checkasm=true): 25/25 on master, 25/25 here. -Db_sanitize=address,undefined -Db_lto=false build: 22 pass and 3 fail on master, 22 pass and 3 fail here; test_predict and test_pic_preallocation fail on LeakSanitizer reports and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603 on master), on both, and this change does not touch them. With feature/adm: fix out-of-bounds read at scale 3 for frame dimensions 17 to 32 #1599 and feature/adm: fix undefined rounding constant in adm_cm for frame widths 17 to 32 #1600 also merged (measured on 8e7a1ac4, without checkasm, not repeated): 25/25 release, 23/25 sanitizer, same two aborts.
• Scores unchanged: the three Netflix reference pairs, run with --feature psnr --feature float_ssim, give identical per-frame and pooled output on master and on this branch with the default CPU dispatch and with SIMD masked off (also against the three-way merge). No golden assertion changes.
• CUDA commit: built with CUDA 13.4 and run on an RTX 4090 with --feature adm_cuda: 16x16 and 64x16 are refused with adm_cuda: invalid size ... and exit 234, 24x24 scores. The behaviour of adm_cuda below 17 on master was not measured, and it was not compared with the CPU at any size.

The workflow run on this PR needs a maintainer's approval.