A collection of VLMs papers, blogs, and projects, with a focus on VLMs in Autonomous Driving and related reasoning techniques.

• https://platform.openai.com/docs/guides/reasoning
• 

• Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving[arxiv]

  • Code

• EMMA: End-to-End Multimodal Model for Autonomous Driving[arxiv]

  • Code ⚠️
  • Blog

• VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions [ECCV2024]

  • Code ⚠️
  • Datasets

• HE-Drive: Human-Like End-to-End Driving with Vision Language Models [arxiv]

  • Code
  • NuScenes-Datasets-Website
  • OpenScene-Datasets-Tutorials

• Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving [ECCV2024]

  • Code
  • Datasets-GoogelDisk
  • Datasets-BaiduDisk

• Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving [CoRL 2024]

  • Code coming soon
  • Datasets coming soon

• MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving [arxiv]

  • Code coming soon
  • Drive-LM-Datasets-Tutorials
  • CODA-LM-Datasets-Website
  • CODA-LM-Datasets-Tutorials

• LMDrive: Closed-Loop End-to-End Driving with Large Language Models [CVPR2024]

  • Code
  • LMDrive-Datasets

• CoDrivingLLM: Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-making Framework [arxiv]

  • Code
  • Datasets ⚠️

• XGEN-MM-VID (BLIP-3-VIDEO): YOU ONLY NEED 32 TOKENS TO REPRESENT A VIDEO EVEN IN VLMS[arxiv]

  • Code ⚠️
  • Datasets ⚠️

• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks[arxiv]

  • Code

• Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training[arxiv]

  • Code ⚠️
  • HuggingFace

• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving [arxiv]

  • Code
  • Datasets coming soon

• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

  • Code

• CLIP:Learning Transferable Visual Models From Natural Language Supervision

  • Code

• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

  • Code