Official implementation of 'Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding'.

The reamining about ModelNet40 will be open-source soon!

[2023.9] We release AAAI2024 'Point-PEFT', adapting 3D pre-trained Models with 1% parameters to downstream tasks .

Large foundation models have recently emerged as a prominent focus of interest, attaining superior performance in widespread scenarios. Due to the scarcity of 3D data, many efforts have been made to adapt pre-trained transformers from vision to 3D domains. However, such 2D-to-3D approaches are still limited, due to the potential loss of spatial geometries and high computation cost. More importantly, their frameworks are mainly designed for 2D models, lacking a general any-to-3D paradigm. In this paper, we introduce Any2Point, a parameter-efficient method to empower any-modality large models (vision, language, audio) for 3D understanding. Given a frozen transformer from any source modality, we propose a 3D-to-any (1D or 2D) virtual projection strategy that correlates the input 3D points to the original 1D or 2D positions within the source modality. This mechanism enables us to assign each 3D token with a positional encoding paired with the pre-trained model, which avoids 3D geometry loss caused by the true projection and better motivates the transformer for 3D learning with 1D/2D positional priors. Then, within each transformer block, we insert an any-to-3D guided adapter module for parameter-efficient fine-tuning. The adapter incorporates prior spatial knowledge from the source modality to guide the local feature aggregation of 3D tokens, compelling the semantic adaption of any-modality transformers. We conduct extensive experiments to showcase the effectiveness and efficiency of our method.

We report the pre-training modality (Pre-train), the number of learnable parameters (#Param) on the "PB-T50-RS" split of ScanObjectNN (SCAN.) and ModelNet40 (MN.). * indicates utilizing the voting strategy.

Real-world shape classification on the PB-T50-RS split of ScanObjectNN:

Synthetic shape classification on the ModelNet40:

Create a conda environment and install basic dependencies:

git clone https://github.com/Ivan-Tang-3D/Any2Point.git
cd Any2Point

conda create -n Any2Point python=3.7
conda activate Any2Point

# Install the according versions of torch and torchvision
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch

conda install -c pyg pytorch-cluster pytorch-scatter pytorch-sparse -y
pip install torch-geometric==2.0

source install.sh

For pre-training and fine-tuning, please follow DATASET.md to install ModelNet40, ScanObjectNN, and ShapeNetPart datasets, referring to Point-BERT. Specially Put the unzip folder under data/. The Language Part Training just occupies 26GB Memory.

The final directory structure should be:

│Any2Point/
├──Any2Point_CLIP_Lang/
├──ckpts/
├──data/
│   ├──ModelNet/
│   ├──ScanObjectNN/
├──...

Please download the CLIP_pre-train.pth, DINOV2_pre-train.pth and ImageBind_audio_pre-train.pth into the ckpts/ folder.

For the PB-T50-RS split of ScanObjectNN, run:

Any2Point_CLIP_Lang

cd Any2Point_CLIP_Lang
sh fine_tune.sh

Any2Point_DINOV2_Vision

cd Any2Point_DINOV2_Vision
sh fine_tune.sh

Any2Point_ImageBind_audio

cd Any2Point_ImageBind_audio
sh fine_tune.sh

For the ModelNet40, run:

Any2Point_CLIP_Lang

cd Any2Point_clip_lang_modelnet
sh fine_tune.sh

Any2Point_DINOV2

cd Any2Point_DINOV2_modelnet
sh fine_tune.sh

Any2Point_ImageBind

cd Any2Point_ImageBind_Modelnet
sh fine_tune.sh

This repo benefits from Pix4Point, Point-NN, PointTransformerV2, Openpoints. Thanks for their wonderful works.