PyTorch implementation of "Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition" [PDF]

• Python >= 3.6
• PyTorch >= 1.5.0
• PyYAML, tqdm, tensorboardX
• We provide the dependency file of the experimental environment, you can install all dependencies by creating a new anaconda virtual environment and running pip install -r requirements.txt
• Run pip install -e .

There are 4 datasets to download:

• NTU RGB+D 60
• NTU RGB+D 120
• NW-UCLA
• Kinetics-Skeleton

Please check Data Download for the download links and descriptions of the annotation format. After finishing the download and placement, the BRL code is now ready to run.

We provide the long-tailed settings for NTU 60-LT and NTU 120-LT, which are generated by imblance_gentxt. Additionally, we use ucla_data to construct NW-UCLA-LT. Through the config, BRL will directly apply the corresponding long-tailed settings during training.

If you would like to construct the experiments with classical methods, a detailed download instructions is provided here. You could follow the official processing of these classical methods for the raw data.

For a fair comparison, we provide the processing script imblance_gendata_from_txt that helps to generate the same long-tailed version of the raw data.

You can use following commands for training and testing. Basically, we support distributed training on a single server with multiple GPUs.

# Training
bash tools/dist_train.sh {config_name} {num_gpus} {other_options}
# For example: train BRL on NTURGB+D XSub (Joint Modality) with one GPU, with validation, and test the last and the best (with best validation metric) checkpoint.
bash tools/dist_train.sh configs/BRL/ntu60_xsub_LT/j.py 1 --validate --test-last --test-best

# Testing
bash tools/dist_test.sh {config_name} {checkpoint} {num_gpus} --out {output_file} --eval {optional_arguments}
# For example: test BRL on NTURGB+D XSub (Joint Modality) with metrics `top_k_accuracy`, and dump the result to `result.pkl`.
bash tools/dist_test.sh configs/BRL/ntu60_xsub_LT/j.py checkpoints/SOME_CHECKPOINT.pth 1 --eval top_k_accuracy --out result.pkl

# Ensemble the results
cd ./tools
python ensemble.py

We present the results trained on NTU RGB+D 60, NTU RGB+D 120 and Kinetics-400 in the long-tailed training setting. We also provide checkpoints for six modalities: Joint, Bone, Skip, Joint Motion, Bone Motion, and Skip Motion.

We release numerous checkpoints for BRL trained with various modalities, annotations on NTU RGB+D 60, NTU RGB+D 120 and Kinetics-400. The accuracy of each modality links to the weight file.

• ST-GCN (AAAI 2018) [Model Zoo]
• AAGCN (TIP) [Model Zoo]
• MS-G3D (CVPR 2020 Oral) [Model Zoo]
• CTR-GCN (ICCV 2021) [Model Zoo]

For specific examples and other pre-trained models, please go to the README.

This repo is based on MS-G3D, CTR-GCN, and PYSKL.

Thanks to the authors for their work!

@article{liu2023balanced,
  title={Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition},
  author={Liu, Hongda and Wang, Yunlong and Ren, Min and Hu, Junxing and Luo, Zhengquan and Hou, Guangqi and Sun, Zhenan},
  journal={arXiv preprint arXiv:2308.14024},
  year={2023}
}

For any questions, feel free to contact: hongda.liu@cripac.ia.ac.cn