Controllable Generative Video Temporal Grounding

Generate once. Rank, select, or reject.

Overview · Results · Models · Quick Start · Evaluation · Training · Citation

Official implementation of Grounding with Confidence: Controllable Generative Video Temporal Grounding.

Inference, evaluation, and training code are included. CTVG-4B weights are available with the matched confidence head. Original training videos and offline verifier labels are not bundled. See the release guide for validation scope and upstream terms; public-wrapper GPU validation remains pending.

Overview

Generating candidate intervals and deciding which ones to return are distinct tasks. Grounding with Confidence equips generative video temporal grounding with an interval-level confidence head, allowing the same decoded candidates to serve different application requirements.

• Score in the original grounding pass. A lightweight head reads cached decoder states, without adding confidence tokens or invoking an external verifier at inference.
• Choose what to return. Use confidence for ranking, threshold-based selection, or rejection, without generating the candidates again.
• Learn generation and selection. Distinct supervision trains the generator and confidence head, while set-level reinforcement learning rewards candidate coverage and the quality of the selected set.

Results

Learning what to return increases the value of a single generation. On OMTG-Bench, confidence improves selection from the same candidate pool, with the largest gains under tight return budgets.

Official query-macro [email protected] (%) on all 320 queries, using the same 1,314 post-NMS candidates:

Global return budget	Generation order	Token likelihood	External verifier	Confidence (ours)
10%	9.95	9.41	11.00	14.42
25%	26.48	22.37	26.31	31.12
50%	51.34	43.05	47.72	52.82
75%	64.76	61.00	62.45	65.30

The return budget is shared across queries; each selector returns the same total number of intervals. All candidate intervals remain fixed.

Models

Model	Base model	Hugging Face
CTVG-4B (Confidence-based TVG)	TimeLens2-4B	Alibaba-VELLDEPTH/CTVG-4B

The release includes the complete merged BF16 generator and its matched confidence head (generator step 200, 150 online head updates), plus tokenizer, processor, and pairing metadata. Download the whole repository; loading the generator alone does not apply confidence scoring. See model details and download instructions.

Verified release revision: f732bd4e21fa62e63029374b35a36123bc6f8a11.

Quick Start

Installation

Use Python 3.10+ and a CUDA GPU. From the repository root:

python -m pip install -e .
python -m pip install -r requirements.txt

Inference

Prepare a JSONL file following examples/requests.jsonl, using your own video path, query, and video duration. The command below downloads the pinned HF release automatically; alternatively, pass a complete local release directory to --model.

python scripts/infer.py \
  --model Alibaba-VELLDEPTH/CTVG-4B \
  --revision f732bd4e21fa62e63029374b35a36123bc6f8a11 \
  --input examples/requests.jsonl \
  --output-dir outputs/example \
  --threshold 0.5

The output contains candidate intervals, their confidence scores, and the selected intervals. Changing the threshold changes the accepted set; it does not require another decoding pass. The example uses a cutoff of 0.5; choose a threshold on validation data for your application.

See the inference guide for input formats, decoding settings, and Hugging Face model loading.

Evaluation

Evaluate saved candidates at a fixed confidence threshold (CPU, no model needed):

python -m pip install -r requirements-eval.txt
python scripts/evaluate.py \
  --input examples/predictions.jsonl \
  --threshold 0.5

The included synthetic example runs on CPU. For benchmark evaluation, provide predictions with ground-truth intervals. The evaluator reports query-macro precision, recall, and tF1 at multiple IoU thresholds, together with count accuracy, temporal overlap, and EtF1. See the evaluation guide for metric definitions.

To run inference and evaluation together, include gt_intervals in every request:

python scripts/infer.py \
  --model Alibaba-VELLDEPTH/CTVG-4B \
  --revision f732bd4e21fa62e63029374b35a36123bc6f8a11 \
  --input /path/to/benchmark-requests.jsonl \
  --output-dir outputs/benchmark --threshold 0.5 --evaluate

This writes predictions.jsonl and evaluation.json, retaining failed queries in the denominator. Add --check-only for input/artifact validation without loading a GPU model. For saved predictions, evaluate.py --requests checks the full original query set; it never tunes the threshold on evaluation labels.

Training and Data

Download the public training data from the official Hugging Face repositories:

• TimeLens2: TimeLens2-93K.
• OMTG: OMTG-56K, with SFT and RL subsets.

After downloading the videos and annotations, normalize the annotations into the documented input schemas, then use our preparation scripts to convert them into our SFT manifests and RL Parquet format. Confidence SFT additionally requires offline candidate replay and verifier-v2 soft scores; these are not supplied by the two upstream datasets. See the training workflow for SFT → RL → inference/evaluation.

Entry	Purpose
scripts/prepare_sft.py	Build checksum-verified v1.3 targets/manifests from authentic caller-supplied replay and verifier soft labels
scripts/train_sft.py	LoRA generator SFT and detached span-confidence head training
scripts/merge_sft.py	Merge the matched SFT adapter into the pinned base for RL
scripts/prepare_rl.py	Convert caller-supplied train/monitor requests to RL Parquet
scripts/train_rl.py	Preview or execute the three-phase, eight-GPU RL recipe
scripts/export_rl.py	Export trusted self-produced distributed checkpoints for inference
scripts/infer.py --sft-checkpoint ... --evaluate	Evaluate a locally trained SFT adapter and matching head without merging

Use separate environments for SFT/inference (requirements.txt, Torch 2.6) and RL (requirements-rl.txt, Torch 2.8). Detailed schemas, dependencies and commands are in the SFT guide, RL guide, and data guide. Paper-specific replay, verifier scores, and split manifests are not bundled; downloading the public datasets alone does not reproduce the exact paper weights.

CPU checks:

python -m pip install -r requirements-dev.txt
python -m unittest discover -s tests -v

Acknowledgements

We thank the authors of TimeLens2 and OMTG for releasing their code and training datasets: TimeLens2-93K and OMTG-56K. Their resources provide essential foundations for this work. Please also credit and cite these upstream projects when using their data or code.

Our implementation additionally builds on OMTG's vendored verl and Hugging Face Transformers; we thank these communities for their open research infrastructure.

Citation

If you find this work useful, please cite:

@article{chen2026grounding,
  title={Grounding with Confidence: Controllable Generative Video Temporal Grounding},
  author={Chen, Jinhao and Cui, Benlei and Jia, Ruijian and Wang, Ziheng and Wo, Tianyu and Sun, Pengfei and Huang, Longtao and Xue, Hui and Yang, Yitong and Hong, Haiwen},
  journal={arXiv preprint arXiv:2609.39883},
  year={2026}
}