Skip to content

README.md

Latest commit

 

History

History
247 lines (213 loc) · 44.5 KB

README.md

File metadata and controls

247 lines (213 loc) · 44.5 KB
Raw

Awesome-LLM-3D Awesome Maintenance PR's Welcome arXiv

🏠 About

Here is a curated list of papers about 3D-Related Tasks empowered by Large Language Models (LLMs). It contains various tasks including 3D understanding, reasoning, generation, and embodied agents. Also, we include other Foundation Models (CLIP, SAM) for the whole picture of this area.

This is an active repository, you can watch for following the latest advances. If you find it useful, please kindly star ⭐ this repo and cite the paper.

🔥 News

Table of Contents

3D Unified Understanding and Generation via LLM

Date Keywords Institute (first) Paper Publication Others
2025-11-07 Omni-View PKU Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images ICLR 2026 github
2025-08-16 UniUGG FDU UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding ICLR 2026 github

3D Understanding via LLM

Date Keywords Institute (first) Paper Publication Others
2025-11-27 G2VLM Shanghai AI Lab G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning Arxiv github
2025-11-07 Omni-View PKU Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images ICLR 2026 github
2025-08-16 UniUGG FDU UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding ICLR 2026 github
2025-07-31 3D-R1 PKU 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding Arxiv project
2025-06-11 LEO-VL BIGAI LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning Arxiv project
2025-06-02 3DRS HKU MLLMs Need 3D-Aware Representation Supervision for Scene Understanding Arxiv project
2025-05-30 VG LLM CUHK Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors Arxiv project
2025-05-29 Spatial-MLLM THU Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence Arxiv project
2025-05-28 SeeGround HKUST(GZ) Zero-Shot 3D Visual Grounding from Vision-Language Models CVPRW'25 project
2025-05-28 3DLLM-Mem UCLA/Google 3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model NeurIPS'25 project
2025-04-24 3D-LLaVA U of Adelaide 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer CVPR '25 github
2025-04-03 Ross3D CASIA Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness Arxiv project
2025-03-08 SplatTalk GIT SplatTalk: 3D VQA with Gaussian Splatting Arxiv github
2025-03-01 Inst3D-LMM ZJU Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning CVPR '25 github
2025-02-13 ENEL SH AILab ENEL: Exploring the Potential of Encoder-free Architectures in 3D LMMs Arxiv project
2025-02-02 LSceneLLM SCUT LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences CVPR '25 project
2025-01-02 GPT4Scene HKU GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models Arxiv project
2024-12-05 SeeGround HKUST(GZ) SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding CVPR '25 project
2024-12-03 Video-3D LLM CUHK Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding CVPR '25 project
2024-11-29 PerLA Fondazione Bruno Kessler PerLA: Perceptive 3D Language Assistant CVPR '25 project
2024-10-12 Situation3D UIUC Situational Awareness Matters in 3D Vision Language Reasoning CVPR '24 project
2024-09-30 Robin3D HKU Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning ICCV '25 github
2024-09-28 LLaVA-3D HKU LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Arxiv project
2024-09-08 MSR3D BIGAI Multi-modal Situated Reasoning in 3D Scenes NeurIPS '24 project
2024-08-28 GreenPLM HUST More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding Arxiv github
2024-06-17 LLaNA UniBO LLaNA: Large Language and NeRF Assistant NeurIPS '24 project
2024-06-07 SpatialPIN Oxford SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors NeurIPS '24 project
2024-06-03 SpatialRGPT UCSD SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models NeurIPS '24 github
2024-05-02 MiniGPT-3D HUST MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors ACM MM '24 project
2024-02-27 ShapeLLM XJTU ShapeLLM: Universal 3D Object Understanding for Embodied Interaction Arxiv project
2024-01-22 SpatialVLM Google DeepMind SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities CVPR '24 project
2023-12-21 LiDAR-LLM PKU LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding Arxiv project
2023-12-15 3DAP Shanghai AI Lab 3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V Arxiv project
2023-12-13 Chat-Scene ZJU Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers NeurIPS '24 github
2023-12-5 GPT4Point HKU GPT4Point: A Unified Framework for Point-Language Understanding and Generation Arxiv github
2023-11-30 LL3DA Fudan University LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning Arxiv github
2023-11-26 ZSVG3D CUHK(SZ) Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding Arxiv project
2023-11-18 LEO BIGAI An Embodied Generalist Agent in 3D World ICML '24 github
2023-10-14 JM3D-LLM Xiamen University JM3D & JM3D-LLM: Elevating 3D Representation with Joint Multi-modal Cues ACM MM '23 github
2023-10-10 Uni3D BAAI Uni3D: Exploring Unified 3D Representation at Scale ICLR '24 project
2023-9-27 - KAUST Zero-Shot 3D Shape Correspondence Siggraph Asia '23 -
2023-9-21 LLM-Grounder U-Mich LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent ICRA '24 github
2023-9-1 Point-Bind CUHK Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following Arxiv github
2023-8-31 PointLLM CUHK PointLLM: Empowering Large Language Models to Understand Point Clouds ECCV '24 github
2023-8-17 Chat-3D ZJU Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes Arxiv github
2023-8-8 3D-VisTA BIGAI 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment ICCV '23 github
2023-7-24 3D-LLM UCLA 3D-LLM: Injecting the 3D World into Large Language Models NeurIPS '23 github
2023-3-29 ViewRefer CUHK ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding ICCV '23 github
2022-9-12 - MIT Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding Arxiv github

3D Understanding via other Foundation Models

ID keywords Institute (first) Paper Publication Others
2025-11-20 POMA-3D Imperial POMA-3D: The Point Map Way to 3D Scene Understanding Arxiv project
2025-07-26 OV-3DDet HKUST CoDAv2: Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection TPAMI '25 github
2025-02-20 CrossOver Stanford CrossOver: 3D Scene Cross-Modal Alignment CVPR '25 project
2025-02-05 SAGA SJTU Segment Any 3D Gaussians AAAI '25 project
2024-10-12 Lexicon3D UIUC Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding NeurIPS '24 project
2024-10-07 Diff2Scene CMU Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models ECCV'24 project
2024-07-19 OpenSU3D TUM OpenSU3D: Open World 3D Scene Understanding using Foundation Models ICRA '25 project
2024-04-07 Any2Point Shanghai AI Lab Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding ECCV'24 github
2024-03-16 N2F2 Oxford-VGG N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields Arxiv -
2023-12-17 SAI3D PKU SAI3D: Segment Any Instance in 3D Scenes Arxiv project
2023-12-17 Open3DIS VinAI Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance Arxiv project
2023-11-6 OVIR-3D Rutgers University OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data CoRL '23 github
2023-10-29 OpenMask3D ETH OpenMask3D: Open-Vocabulary 3D Instance Segmentation NeurIPS '23 project
2023-10-5 Open-Fusion - Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation Arxiv github
2023-9-22 OV-3DDet HKUST CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection NeurIPS '23 github
2023-9-19 LAMP - From Language to 3D Worlds: Adapting Language Model for Point Cloud Perception OpenReview -
2023-9-15 OpenNerf - OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views OpenReview github
2023-9-1 OpenIns3D Cambridge OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation Arxiv project
2023-6-7 Contrastive Lift Oxford-VGG Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion NeurIPS '23 github
2023-6-4 Multi-CLIP ETH Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes Arxiv -
2023-5-23 3D-OVS NTU Weakly Supervised 3D Open-vocabulary Segmentation NeurIPS '23 github
2023-5-21 VL-Fields University of Edinburgh VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations ICRA '23 project
2023-5-8 CLIP-FO3D Tsinghua University CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP ICCVW '23 -
2023-4-12 3D-VQA ETH CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes CVPRW '23 github
2023-4-3 RegionPLC HKU RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding Arxiv project
2023-3-20 CG3D JHU CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition Arxiv github
2023-3-16 LERF UC Berkeley LERF: Language Embedded Radiance Fields ICCV '23 github
2023-2-14 ConceptFusion MIT ConceptFusion: Open-set Multimodal 3D Mapping RSS '23 project
2023-1-12 CLIP2Scene HKU CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP CVPR '23 github
2022-12-1 UniT3D TUM UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding ICCV '23 github
2022-11-29 PLA HKU PLA: Language-Driven Open-Vocabulary 3D Scene Understanding CVPR '23 github
2022-11-28 OpenScene ETHz OpenScene: 3D Scene Understanding with Open Vocabularies CVPR '23 github
2022-10-11 CLIP-Fields NYU CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory Arxiv project
2022-7-23 Semantic Abstraction Columbia Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models CoRL '22 project
2022-4-26 ScanNet200 TUM Language-Grounded Indoor 3D Semantic Segmentation in the Wild ECCV '22 project

3D Reasoning

Date keywords Institute (first) Paper Publication Others
2025-12-15 RoboTracer BUAA RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics Arxiv project
2025-06-11 SceneCOT BIGAI SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes Arxiv project
2025-06-04 RoboRefer BUAA RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics Arxiv project
2024-09-08 MSR3D BIGAI Multi-modal Situated Reasoning in 3D Scenes NeurIPS '24 project
2023-5-20 3D-CLR UCLA 3D Concept Learning and Reasoning from Multi-View Images CVPR '23 github
- Transcribe3D TTI, Chicago Transcribe3D: Grounding LLMs Using Transcribed Information for 3D Referential Reasoning with Self-Corrected Finetuning CoRL '23 github

3D Generation

Date keywords Institute Paper Publication Others
2025-11-07 Omni-View PKU Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images ICLR 2026 github
2025-08-16 UniUGG FDU UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding ICLR 2026 github
2024-11-14 LLaMA-Mesh THU LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models Arxiv project
2023-11-29 ShapeGPT Fudan University ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model Arxiv github
2023-11-27 MeshGPT TUM MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers Arxiv project
2023-10-19 3D-GPT ANU 3D-GPT: Procedural 3D Modeling with Large Language Models Arxiv github
2023-9-21 LLMR MIT LLMR: Real-time Prompting of Interactive Worlds using Large Language Models Arxiv -
2023-9-20 DreamLLM MEGVII DreamLLM: Synergistic Multimodal Comprehension and Creation Arxiv github
2023-4-1 ChatAvatar Deemos Tech DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance ACM TOG website

3D Embodied Agent

Date keywords Institute Paper Publication Others
2025-12-15 RoboTracer BUAA RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics Arxiv project
2025-06-04 RoboRefer BUAA RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics Arxiv project
2025-05-30 VeBrain Shanghai AI Lab Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces Arxiv project
2025-05-28 3DLLM-Mem UCLA 3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model Arxiv project
2024-01-22 SpatialVLM Deepmind SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities CVPR '24 project
2023-12-05 NaviLLM CUHK Towards Learning a Generalist Model for Embodied Navigation CVPR '24 project
2023-11-27 Dobb-E NYU On Bringing Robots Home Arxiv github
2023-11-26 STEVE ZJU See and Think: Embodied Agent in Virtual Environment Arxiv github
2023-11-18 LEO BIGAI An Embodied Generalist Agent in 3D World ICML '24 github
2023-9-14 UniHSI Shanghai AI Lab Unified Human-Scene Interaction via Prompted Chain-of-Contacts Arxiv github
2023-7-28 RT-2 Google-DeepMind RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control Arxiv github
2023-7-12 SayPlan QUT Centre for Robotics SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning CoRL '23 github
2023-7-12 VoxPoser Stanford VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models Arxiv github
2022-12-13 RT-1 Google RT-1: Robotics Transformer for Real-World Control at Scale Arxiv github
2022-12-8 LLM-Planner The Ohio State University LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models ICCV '23 github
2022-10-11 CLIP-Fields NYU, Meta CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory RSS '23 github
2022-09-20 NLMap-SayCan Google Open-vocabulary Queryable Scene Representations for Real World Planning ICRA '23 github

3D Benchmarks

Date keywords Institute Paper Publication Others
2026-01-22 Real-3DQA VGG, Oxford Do 3D Large Language Models Really Understand 3D Spatial Relationships? ICLR 2026 project
2025-12-15 RoboTracer BUAA RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics Arxiv project
:-----: :------------------: :--------------: :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- :---------: :---------:
2025-11-28 ORS3D HUST Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution AAAI ‘26 project
2025-06-04 RoboRefer BUAA RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics Arxiv project
2025-06-09 SpaCE-10 SJTU SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence Arxiv project
2025-05-01 Hypo3D Imperial Hypo3D: Exploring Hypothetical Reasoning in 3D ICML'25 project
2025-06-04 Anywhere3D BIGAI From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes NeurIPS '25 project
2025-05-01 SpatialVQA JHU SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models CVPR'25 project
2025-04-03 SPAR Fudan University From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D Arxiv project
2025-03-28 Beacon3D BIGAI Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis CVPR '25 project
2025-03-08 3D-CoT PolyU, EIT Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learning Arxiv dataset
2024-09-08 MSQA / MSNN BIGAI Multi-modal Situated Reasoning in 3D Scenes NeurIPS '24 project
2024-08-29 Space3D-Bench ETHz Space3D-Bench: Spatial 3D Question Answering Benchmark Arxiv project
2024-07-24 City-3DQA HKUST 3D Question Answering for City Scene Understanding ACM MM'24 project
2024-06-13 MMScan Shanghai AI Lab MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations Arxiv github
2024-06-10 3D-GRAND / 3D-POPE UMich 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination Arxiv project
2024-06-03 SpatialRGPT-Bench UCSD SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models NeurIPS '24 github
2024-05-27 Reason3D UC Merced Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model 3DV'25 project
2024-1-18 SceneVerse BIGAI SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding ECCV '24 github
2023-12-26 EmbodiedScan Shanghai AI Lab EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI Arxiv github
2023-12-17 M3DBench Fudan University M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts Arxiv github
2023-11-29 - DeepMind Leveraging VLM-Based Pipelines to Annotate 3D Objects ICML '24 github
2023-09-14 CrossCoherence UniBO Looking at words and points with attention: a benchmark for text-to-shape coherence ICCV '23 github
2022-10-14 SQA3D BIGAI SQA3D: Situated Question Answering in 3D Scenes ICLR '23 github
2022-09-24 FE-3DGQA Beihang University Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline TCSVT github
2021-12-20 ScanQA RIKEN AIP ScanQA: 3D Question Answering for Spatial Scene Understanding CVPR '23 github
2020-12-3 Scan2Cap TUM Scan2Cap: Context-aware Dense Captioning in RGB-D Scans CVPR '21 github
2020-8-23 ReferIt3D Stanford ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes ECCV '20 github
2019-12-18 ScanRefer TUM ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language ECCV '20 github

Contributing

Your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for 3D LLMs, you could vote for them by adding 👍 to them.

If you have any questions about this opinionated list, please get in touch at xianzheng@robots.ox.ac.uk or Wechat ID: mxz1997112.

Star History

Star History Chart

Citation

If you find this repository useful, please consider citing this paper:

@misc{ma2024llmsstep3dworld,
      title={When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models}, 
      author={Xianzheng Ma and Yash Bhalgat and Brandon Smart and Shuai Chen and Xinghui Li and Jian Ding and Jindong Gu and Dave Zhenyu Chen and Songyou Peng and Jia-Wang Bian and Philip H Torr and Marc Pollefeys and Matthias Nießner and Ian D Reid and Angel X. Chang and Iro Laina and Victor Adrian Prisacariu},
      year={2024},
      journal={arXiv preprint arXiv:2405.10255},
}

Acknowledgement

This repo is inspired by Awesome-LLM