PyTorch implementation of "Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition" [PDF]

• Python >= 3.6
• PyTorch >= 1.5.0
• PyYAML, tqdm, tensorboardX
• We provide the dependency file of the experimental environment, you can install all dependencies by creating a new anaconda virtual environment and running pip install -r requirements.txt
• Run pip install -e .

There are 4 datasets to download:

• NTU RGB+D 60 Skeleton
• NTU RGB+D 120 Skeleton
• NW-UCLA
• Kinetics 400 Skeleton

Please download the raw data from NTU RGB+D 60, NTU RGB+D 120, NW-UCLA, and Kinetics Skeleton 400.

For detailed download instructions, please go to the README.

Generate NTU RGB+D 60, NTU RGB+D 120, NW-UCLA, and Kinetics Skeleton 400 datasets:

cd ./data
python ntu_gendata.py
python ntu120_gendata.py
python ucla_data.py
python kinetics_gendata.py

Meanwhile, PYSKL provides the processed skeleton data for all datasets as pickle files (which can be directly used for training and testing). Please check Data Doc for the download links and descriptions of the annotation format. You can also use the provided script to generate the processed pickle files.

Construct NTU 60-LT, NTU 120-LT, and NW-UCLA-LT：

cd ./data
python imblance_gentxt.py
python imblance_gendata_from_txt.py

You can use following commands for training and testing. Basically, we support distributed training on a single server with multiple GPUs.

# Training
bash tools/dist_train.sh {config_name} {num_gpus} {other_options}
# For example: train BRL on NTURGB+D XSub (Joint Modality) with one GPU, with validation, and test the last and the best (with best validation metric) checkpoint.
bash tools/dist_train.sh configs/BRL/ntu60_xsub_LT/j.py 1 --validate --test-last --test-best

# Testing
bash tools/dist_test.sh {config_name} {checkpoint} {num_gpus} --out {output_file} --eval {optional_arguments}
# For example: test BRL on NTURGB+D XSub (Joint Modality) with metrics `top_k_accuracy`, and dump the result to `result.pkl`.
bash tools/dist_test.sh configs/BRL/ntu60_xsub_LT/j.py checkpoints/SOME_CHECKPOINT.pth 1 --eval top_k_accuracy --out result.pkl

# Ensemble the results
cd ./tools
python ensemble.py

We provide the results trained on NTURGB+D 60 and NTURGB+D 120 in the long-tailed training setting. We also provide checkpoints for six modalities: Joint, Bone, Skip, Joint Motion, Bone Motion, and Skip Motion. The accuracy of each modality links to the weight file.

We release numerous checkpoints for BRL trained with various modalities, annotations on NTURGB+D and NTURGB+D 120. The accuracy of each modality links to the weight file.

• ST-GCN (AAAI 2018) [Model Zoo]
• AAGCN (TIP) [Model Zoo]
• MS-G3D (CVPR 2020 Oral) [Model Zoo]
• CTR-GCN (ICCV 2021) [Model Zoo]

For specific examples and other pre-trained models, please go to the README.

This repo is based on MS-G3D, CTR-GCN, and PYSKL.

Thanks to the original authors for their work!

Please cite this work if you find it useful:

  @article{liu2023balanced,
    title={Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition},
    author={Liu, Hongda and Wang, Yunlong and Ren, Min and Hu, Junxing and Luo, Zhengquan and Hou, Guangqi and Sun, Zhenan},
    journal={arXiv preprint arXiv:2308.14024},
    year={2023}
  }

For any questions, feel free to contact: hongda.liu@cripac.ia.ac.cn