Language Prompt for Autonomous Driving

Dongming Wu*, Wencheng Han*, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen

This is the official implementation of Language Prompt for Autonomous Driving.

• We propose a new large-scale language prompt set for driving scenes, named NuPrompt. As far as we know, it is the first dataset specializing in multiple 3D objects of interest from video domain.
• We construct a new prompt-based driving perceiving task, which requires using a language prompt as a semantic cue to predict object trajectories.
• We develop a simple end-to-end baseline model, called PromptTrack, which effectively fuses cross-modal features in a newly built prompt reasoning branch to predict referent objects, showing impressive performance.

• [2024.06.27] Data and code are released. Welcome to try it!
• [2023.09.11] Our paper is released at arXiv.

We expand nuScenes dataset with annotating language prompts, named NuPrompt. It is a large-scale dataset for language prompt in driving scenes, which contains 40,147 language prompts for 3D objects. Thanks to nuScenes, our descriptions are closed to real-driving nature and complexity, covering a 3D, multi-view, and multi-frame space.

The data can be downloaded from NuPrompt.

Our model is built upon PF-Track.

Please refer to data.md for preparing data and pre-trained models.

Please refer to environment.md for environment installation.

Please refer to training_inference.md for training and evaluation.

If you find our work useful in your research, please consider citing them.

@article{wu2023language,
  title={Language Prompt for Autonomous Driving},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Liu, Yingfei and Zhang, Xiangyu and Shen, Jianbing},
  journal={arXiv preprint},
  year={2023}
}

@inproceedings{wu2023referring,
  title={Referring multi-object tracking},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Dong, Xingping and Zhang, Xiangyu and Shen, Jianbing},
  booktitle={CVPR},
  year={2023}
}

We thank the authors that open the following projects.

• MMDetection3d
• nuScenes
• PF-Track
• PETR
• MOTR