Accepted by NeurIPS 2026
Yechi Ma1 Wei Hua1 Shu Kong2,3
1Zhejiang University 2University of Macau 3IAIBS
Paper · Project Page · Challenge & Leaderboard · Dataset
AutoExpert studies auto-annotation from expert-crafted guidelines: learning from textual class definitions and a few labeled RGB examples to annotate LiDAR data with 3D cuboids. The paper introduces two benchmarks, AutoExpert-nuScenes and AutoExpert-PandaSet, and the auto3D method.
This repository was originally released under the anonymous account annoguide. The personal repository is cc50121/AutoExpert-3D-benchmark.
| AutoExpert-nuScenes | AutoExpert-PandaSet | |
|---|---|---|
| Source dataset | nuScenes | PandaSet |
| Target categories | 18 guideline-defined classes | 25 guideline-defined classes |
| Training supervision | Guideline texts and few-shot, federated 2D annotations | Guideline texts and few-shot, federated 2D annotations |
| Validation | 570 frames sampled from the official nuScenes training split | 8 re-annotated keyframes |
| Test | 6,019 frames from the official nuScenes validation split | 192 re-annotated keyframes, following the split in paper §3 |
| Annotation basis | nuScenes annotations with the AutoExpert evaluation protocol | Our corrected and re-annotated PandaSet subset, comprising 200 non-continuous keyframes and 4,695 cuboids in total |
| Paper results | Main paper, Table 1 | Appendix L, Table 22 |
| Workshop challenge | Development benchmark | Official CVPR 2026 Workshop Challenge benchmark |
The training examples use 4–8 selected RGB images per class from the source training sets to represent guideline examples. Their annotations are federated: only the target class is labeled in each example image. The benchmark does not use the potentially copyrighted example images embedded in the original guidelines.
No 3D annotations are used for training. Validation annotations support prompt/model selection and expert-in-the-loop quality control; they are separate from the held-out test annotations. AutoExpert evaluates all 18 nuScenes guideline classes rather than the standard 10-class nuScenes detection task.
AutoExpert-PandaSet uses a manually re-annotated subset rather than the original PandaSet cuboids unchanged. As described in Appendix L and Figure 12, the original annotations contain misaligned or "ghost" cuboids whose objects are not identifiable in the RGB views. We re-annotated the selected frames with well-aligned 3D boxes for visually identifiable objects. The challenge dataset documentation describes the released 2D examples, validation annotations, and test inputs.
- 3D mAP averages per-class AP over ground-plane center-distance thresholds of 0.5, 1.0, 2.0, and 4.0 meters.
- nuScenes NDS combines mAP with translation, scale, orientation, velocity, and attribute errors.
- PandaSet NDS is adapted to use translation, scale, and orientation errors, excluding velocity and attribute errors (paper Appendix L, Eq. 6). NDS values from the two datasets should be interpreted within their respective protocols.
- 2D mAP is measured at IoU = 0.5.
All mAP and NDS values in the tables below are shown on a 0–100 scale; the challenge server displays scores on a 0–1 scale.
The paper organizes the pipeline into three components:
- 2D detection and segmentation. Refine class names with a VLM using guideline texts and examples, select prompts on the validation set, and finetune GroundingDINO with the few-shot 2D examples. SAM produces instance masks.
- 2D-to-3D association. Use known camera/LiDAR calibration and sensor poses to associate the 2D detections with LiDAR points.
- 3D cuboid generation and refinement. Use VLM-guided Multiple Hypotheses Testing (v-MHT) with instance-specific geometric priors, class-aware LiDAR sweep aggregation, geometry-aided scoring, and tracking-based confidence refinement.
auto3D is the complete method; v-MHT is its cuboid-generation component. On AutoExpert-nuScenes, v-MHT alone reaches 21.9 mAP / 25.2 NDS, while the complete auto3D system reaches 25.4 mAP / 27.2 NDS (paper Table 4).
| Method | 3D mAP ↑ | NDS ↑ |
|---|---|---|
| Oyster | 6.3 | 10.7 |
| LISO | 8.9 | 13.1 |
| UNION | 9.7 | 13.8 |
| CPD | 10.1 | 14.2 |
| Find&Prop | 11.3 | 16.0 |
| OpenSight | 11.8 | 16.4 |
| OpenBox | 10.6 | 15.5 |
| AnnofreeOD | 11.7 | 16.2 |
| CM3D | 12.1 | 16.6 |
| CM3D + ft-GD | 18.2 | 23.1 |
| auto3D (ours) | 25.4 | 27.2 |
Here, ft-GD denotes our GroundingDINO finetuned with refined class names and few-shot guideline examples. Table 1 in the paper also reports enhanced variants of the other baselines.
These results use AutoExpert-PandaSet, not the original PandaSet annotation protocol.
| Method | 3D mAP ↑ | Adapted NDS ↑ |
|---|---|---|
| OpenBox | 10.9 | 15.0 |
| OpenBox + ft-GD | 15.9 | 18.9 |
| AnnofreeOD | 11.5 | 15.4 |
| AnnofreeOD + ft-GD | 16.1 | 19.3 |
| CM3D | 12.3 | 16.0 |
| CM3D + ft-GD | 16.6 | 19.7 |
| auto3D (ours) | 18.4 | 27.6 |
The Automated 3D Cuboid Annotation on LiDAR Data challenge evaluates submissions on AutoExpert-PandaSet with our corrected annotations. The leaderboard ranks submissions by NDS.
The following standings were retrieved from the challenge leaderboard on October 20, 2026. The public and private leaderboard views returned the same standings at that time.
| Leaderboard rank | Team / entry | NDS ↑ | 3D mAP ↑ |
|---|---|---|---|
| 1 | iMerit | 29.36 | 23.24 |
| 2 | AnnoExpert (organizer baseline) | 27.61 | 18.36 |
| 3 | EgoCar | 23.17 | 18.13 |
| 4 | Weaver | 21.03 | 18.07 |
| 5 | LogicCrafters | 20.89 | 17.68 |
iMerit is the first-place entry on the linked leaderboard. The organizer baseline is listed under the name AnnoExpert; its 18.36 mAP / 27.61 NDS round to the paper's PandaSet auto3D result of 18.4 / 27.6. Challenge submissions are listed separately from the paper's baseline comparisons. Visit the live leaderboard for the complete standings.
| Resource | Link |
|---|---|
| Source sensor data | nuScenes official download |
| Few-shot guideline examples | Google Drive |
| Small validation set for prompt/model selection | Google Drive |
| Finetuned GroundingDINO weights | Google Drive |
| Saved 2D detection results (694.6 MB) | Google Drive |
| Saved 3D detection results (396.1 MB) | Google Drive |
The existing Google Drive artifacts belong to the nuScenes workflow. They are not PandaSet challenge predictions. Their availability does not by itself verify reproduction of the final paper's auto3D scores.
| Resource | Link |
|---|---|
| Guideline examples, validation annotations, and test inputs | Hugging Face dataset |
| LiDAR sequences and sensor data | Hugging Face sequences |
| PandaSet loader and calibration utilities | PandaSet devkit |
| Submission format, evaluation server, and leaderboard | Challenge Space |
Consult the challenge dataset card for the current annotation release status and exact submission format. The source datasets remain subject to their original license terms.
Clone the personal repository:
git clone https://github.com/cc50121/AutoExpert-3D-benchmark.git
cd AutoExpert-3D-benchmark
conda create -n autoexpert python=3.10
conda activate autoexpertInstall a compatible PyTorch/CUDA build and follow the bundled MMDetection installation guide. Use compatible MMEngine/MMCV versions for the bundled MMDetection checkout.
pip install -U openmim
mim install "mmengine>=0.7.1,<1.0.0"
mim install "mmcv>=2.0.0,<2.2.0"
cd mmdetection
pip install -v -e .
cd ..
pip install segment_anything numbaModels used in the paper include GroundingDINO, SAM, GPT-4o, and Qwen. Local VLM serving can use vLLM; configure it in an environment compatible with the chosen model and serving version.
Current code coverage. The released scripts primarily implement the nuScenes workflow. The current tree includes class-level shape priors in shape_priors_gpt4.json, but does not include the previously documented src/vlm_geometric_reasoning.py entry point. The commands below describe the available scripts; they do not establish that the complete final v-MHT/auto3D pipeline has been released or reproduced. For PandaSet, use the challenge resources above.
Run commands from the repository root. Before running, review paths, checkpoints, and output locations in the scripts and configs. The workflow expects prepared nuScenes info files, downloaded model weights, and SAM checkpoints. The evaluation script expects the extended 18-class nuScenes evaluation setup (detection_lt3d), which is not the standard 10-class devkit configuration.
Generate 2D labels used by the data preparation scripts:
python mmdetection/tools/GD/make_2D_labels.py --info_path data/nuscenes/nuscenes_infos_train.pkl --output_dir_path data/nuscenes/samples/labels_2D_COCO/CAM_ALL_train
python mmdetection/tools/GD/make_2D_labels.py --info_path data/nuscenes/nuscenes_infos_val.pkl --output_dir_path data/nuscenes/samples/labels_2D_COCO/CAM_ALL_val
python mmdetection/tools/GD/make_2D_annos_train_few_shot.py
python mmdetection/tools/GD/make_few_shot_file_name.py
python mmdetection/tools/GD/make_2D_annos_val.pyThese preparation utilities use existing source annotations to construct the prescribed examples/evaluation files. Training must follow the benchmark's few-shot 2D supervision protocol; do not use dense source 3D labels as training supervision.
python mmdetection/tools/train.py mmdetection/configs/mm_grounding_dino/grounding_dino_swin-l_finetune_8xb4_20e_nuscenes_train.py
python mmdetection/tools/test.py mmdetection/configs/mm_grounding_dino/grounding_dino_swin-l_finetune_8xb4_20e_nuscenes_test.py weights/GD.pthpython mmdetection/tools/GD/add_file_name_in_2D_results.py
python src/gen_masks.py
python src/masks_to_3D_results.pypython src/post_process/json2pkl.py
python src/post_process/average_score_by_tracking.py
python src/eval/eval_3D_results.pySee demo-annoexpert-infer-vis.ipynb for the released inference/visualization demo.
Please use the updated paper title:
@inproceedings{ma2026autoexpert,
title = {{AutoExpert}: Automating 3D {LiDAR} Annotation from Expert-Crafted Guidelines},
author = {Ma, Yechi and Hua, Wei and Kong, Shu},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://arxiv.org/pdf/2506.02914v3}
}Repository code is provided under the MIT License. Please also comply with the applicable source dataset and model licenses.