Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization (ECCV 2020)
paper |
poster | presentation
We argue that existing methods for weakly-supervised temporal activity localization are not able to sufficiently distinguish background information from activities of interest for each video even though such an ability is critical to strong temporal activity localization. To this end, we propose a novel method named Adversarial and Angular Center Loss with a Pair of Triplets (A2CL-PT). Our method outperforms all the previous state-of-the-art approaches. Specifically, the average mAP of IoU thresholds from 0.1 to 0.9 on THUMOS14 dataset is significantly improved from 27.9% to 30.0%.
| Method \ mAP(%) | @0.1 | @0.2 | @0.3 | @0.4 | @0.5 | @0.6 | @0.7 | @0.8 | @0.9 | AVG |
|---|---|---|---|---|---|---|---|---|---|---|
| UntrimmedNet | 44.4 | 37.7 | 28.2 | 21.1 | 13.7 | - | - | - | - | - |
| STPN | 52.0 | 44.7 | 35.5 | 25.8 | 16.9 | 9.9 | 4.3 | 1.2 | 0.1 | 21.2 |
| W-TALC | 55.2 | 49.6 | 40.1 | 31.1 | 22.8 | - | 7.6 | - | - | - |
| AutoLoc | - | - | 35.8 | 29.0 | 21.2 | 13.4 | 5.8 | - | - | - |
| CleanNet | - | - | 37.0 | 30.9 | 23.9 | 13.9 | 7.1 | - | - | - |
| MAAN | 59.8 | 50.8 | 41.1 | 30.6 | 20.3 | 12.0 | 6.9 | 2.6 | 0.2 | 24.9 |
| BaS-Net | 58.2 | 52.3 | 44.6 | 36.0 | 27.0 | 18.6 | 10.4 | 3.9 | 0.5 | 27.9 |
| A2CL-PT (Ours) | 61.2 | 56.1 | 48.1 | 39.0 | 30.1 | 19.2 | 10.6 | 4.8 | 1.0 | 30.0 |
The main goal of temporal activity localization is to find the start and end times of activities from untrimmed videos. A weakly-supervised version has recently taken foot in the community: here, one assumes that only video-level groundtruth activity labels are available. These video-level activity annotations are easy to collect and already exist across many datasets, thus weakly-supervised methods can be applied to a broader range of situations.
Full example video clip is included in examples folder. You can reproduce the detection results by using run_example.py
First, clone this repository and download these pre-extracted I3D features of the THUMOS14 dataset: feature_train.npy and feature_val.npy.
Then, put these files in the dataset/THUMOS14 folder and just run
$ python main.py --mode val
This will reproduce the results reported in the paper. You can also train the model from scratch by running
$ python main.py --mode train
You can refer to the main.py file to play with the hyperparameters (margins, alpha, beta, gamma, omega, etc.).
-
We performed all the experiments with Python 3.6 and PyTorch 1.3.1 on a single GPU (TITAN Xp).
-
We also provide the pre-extracted features of ActivityNet-1.3 dataset: link. As described in our paper, you also need to add a 1D grouped convolutional layer (k=13, p=12, d=2). Please refer to this discussion.
@inproceedings{min2020adversarial,
title={Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization},
author={Min, Kyle and Corso, Jason J},
booktitle={European Conference on Computer Vision},
pages={283--299},
year={2020},
organization={Springer}
}