[Project Page] [Paper]

by Xiang Li*, Jian Ding*, Zhaoyang Chen, Mohamed Elhoseiny.

We present Uni3DL, a unified model for 3D and Language understanding. Distinct from existing unified vision-language models in 3D which are limited in task variety and predominantly dependent on projected multi-view images, Uni3DL operates directly on point clouds. This approach significantly expands the range of supported tasks in 3D, encompassing both vision and vision-language tasks in 3D. At the core of Uni3DL, a query transformer is designed to learn task-agnostic semantic and mask outputs by attending to 3D visual features, and a task router is employed to selectively generate task-specific outputs required for diverse tasks. With a unified architecture, our Uni3DL model enjoys seamless task decomposition and substantial parameter sharing across tasks. Uni3DL has been rigorously evaluated across diverse 3D vision-language understanding tasks, including semantic segmentation, object detection, instance segmentation, visual grounding, 3D captioning, and text-3D cross-modal retrieval. It demonstrates performance on par with or surpassing state-of-the-art (SOTA) task-specific models. We hope our benchmark and Uni3DL model will serve as a solid step to ease future research in unified models in the realm of 3D and language understanding.

pip3 install torch==1.13.1 torchvision==0.14.1 --extra-index-url https://download.pytorch.org/whl/cu113
python -m pip install 'git+https://github.com/MaureenZOU/detectron2-xyz.git'
python -m pip install -r requirements.txt
sh install_cococapeval.sh

sbatch srun_joint.sh

sbatch srun_s3dis_sem.sh

sbatch srun_s3dis_inst.sh

• We build our work on top of X-Decoder and Mask3D
• We appreciate the contructive dicussion with Xueyan Zou

@inproceedings{li2025uni3dl,
  title={Uni3DL: A Unified Model for 3D Vision-Language Understanding},
  author={Li, Xiang and Ding, Jian and Chen, Zhaoyang and Elhoseiny, Mohamed},
  booktitle={European Conference on Computer Vision},
  pages={74--92},
  year={2025},
  organization={Springer}
}