Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, Sun Jian
Real-time Object Detection for Streaming Perception, CVPR 2022 (Oral)
[Paper]
| Model | size | velocity | sAP 0.5:0.95 |
sAP50 | sAP75 | weights | COCO pretrained weights |
|---|---|---|---|---|---|---|---|
| StreamYOLO-s | 600×960 | 1x | 29.8 | 50.3 | 29.8 | github | github |
| StreamYOLO-m | 600×960 | 1x | 33.7 | 54.5 | 34.0 | github | github |
| StreamYOLO-l | 600×960 | 1x | 36.9 | 58.1 | 37.5 | github | github |
| StreamYOLO-l | 600×960 | 2x | 34.6 | 56.3 | 34.7 | github | github |
| StreamYOLO-l | 600×960 | still | 39.4 | 60.0 | 40.2 | github | github |
Dataset preparation
You can download Argoverse-1.1 full dataset and annotation from HERE and unzip it.
The folder structure should be organized as follows before our processing.
StreamYOLO
├── exps
├── tools
├── yolox
├── data
│ ├── Argoverse-1.1
│ │ ├── annotations
│ │ ├── tracking
│ │ ├── train
│ │ ├── val
│ │ ├── test
│ ├── Argoverse-HD
│ │ ├── annotations
│ │ ├── test-meta.json
│ │ ├── train.json
│ │ ├── val.jsonThe hash strings represent different video sequences in Argoverse, and ring_front_center is one of the sensors for that sequence. Argoverse-HD annotations correspond to images from this sensor. Information from other sensors (other ring cameras or LiDAR) is not used, but our framework can be also extended to these modalities or to a multi-modality setting.
Installation
# basic python libraries
conda create --name streamyolo python=3.7
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install yolox==0.3
git clone git@github.com:yancie-yjr/StreamYOLO.git
cd StreamYOLO/
# add StreamYOLO to PYTHONPATH and add this line to ~/.bashrc or ~/.zshrc (change the file accordingly)
ADDPATH=$(pwd)
echo export PYTHONPATH=$PYTHONPATH:$ADDPATH >> ~/.bashrc
source ~/.bashrc
# Installing `mmcv` for the official sAP evaluation:
# Please replace `{cu_version}` and ``{torch_version}`` with the versions you are currently using.
# You will get import or runtime errors if the versions are incorrect.
pip install mmcv-full==1.1.5 -f https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html
Reproduce our results on Argoverse-HD
Step1. Prepare COCO dataset
cd <StreamYOLO_HOME>
ln -s /path/to/your/Argoverse-1.1 ./data/Argoverse-1.1
ln -s /path/to/your/Argoverse-HD ./data/Argoverse-HDStep2. Reproduce our results on Argoverse:
python tools/train.py -f cfgs/m_s50_onex_dfp_tal_flip.py -d 8 -b 32 -c [/path/to/your/coco_pretrained_path] -o --fp16- -d: number of gpu devices.
- -b: total batch size, the recommended number for -b is num-gpu * 8.
- --fp16: mixed precision training.
- -c: model checkpoint path.
Offline Evaluation
We support batch testing for fast evaluation:
python tools/eval.py -f cfgs/l_s50_onex_dfp_tal_flip.py -c [/path/to/your/model_path] -b 64 -d 8 --conf 0.01 [--fp16] [--fuse]- --fuse: fuse conv and bn.
- -d: number of GPUs used for evaluation. DEFAULT: All GPUs available will be used.
- -b: total batch size across on all GPUs.
- -c: model checkpoint path.
- --conf: NMS threshold. If using 0.001, the performance will further improve by 0.2~0.3 sAP.
Online Evaluation
We modify the online evaluation from sAP
Please use 1 V100 GPU to test the performance since other GPUs with low computing power will trigger non-real-time results!!!!!!!!
cd sAP/streamyolo
bash streamyolo.shPlease cite the following paper if this repo helps your research:
@inproceedings{streamyolo,
title={Real-time Object Detection for Streaming Perception},
author={Yang, Jinrong and Liu, Songtao and Li, Zeming and Li, Xiaoping and Sun, Jian},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={5385--5395},
year={2022}
}
@article{yang2022streamyolo,
title={StreamYOLO: Real-time Object Detection for Streaming Perception},
author={Yang, Jinrong and Liu, Songtao and Li, Zeming and Li, Xiaoping and Sun, Jian},
journal={arXiv preprint arXiv:2207.10433},
year={2022}
}This repo is released under the Apache 2.0 license. Please see the LICENSE file for more information.