Skip to content

Commit

Permalink
v1.0 release
Browse files Browse the repository at this point in the history
  • Loading branch information
@tianweiy
tianweiy committed Jan 7, 2021
1 parent 604585a commit e26ea57
Show file tree
Hide file tree
Showing 171 changed files with 5,481 additions and 23,448 deletions.
2 changes: 1 addition & 1 deletion LICENSE
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2020 Tianwei Yin and Xingyi Zhou
Copyright (c) 2020-2021 Tianwei Yin and Xingyi Zhou

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
121 changes: 45 additions & 76 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@
year={2020},
}

## Updates

[2020-12-11] **NEW:** 3 out of the top 4 entries in the recent NeurIPS 2020 [nuScenes 3D Detection challenge](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any) used CenterPoint. Congratualations to other participants and please stay tuned for more updates on nuScenes and Waymo soon.
## NEWS

[2020-08-10] We now support vehicle detection on [Waymo](docs/WAYMO.md) with SOTA performance.
[2021-01-06] CenterPoint v1.0 is released. Without bells and whistles, we rank first among all Lidar-only methods on Waymo Open Dataset with a single model that runs at 11 FPS. Check out CenterPoint's model zoo for [Waymo](configs/waymo/README.md) and [nuScenes](configs/nusc/README.md).

[2020-12-11] 3 out of the top 4 entries in the recent NeurIPS 2020 [nuScenes 3D Detection challenge](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any) used CenterPoint. Congratualations to other participants and please stay tuned for more updates on nuScenes and Waymo soon.

## Contact
Any questions or discussion are welcome!
Expand All @@ -30,48 +31,63 @@ Tianwei Yin [yintianwei@utexas.edu](mailto:yintianwei@utexas.edu)
Xingyi Zhou [zhouxy@cs.utexas.edu](mailto:zhouxy@cs.utexas.edu)

## Abstract
Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection, but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objectsas points. We use a keypoint detector to find centers of objects, and simply regress to other attributes, including 3D size, 3D orientation, and velocity. In our center-based framework, 3D object tracking simplifies to greedy closest-point matching.The resulting detection and tracking algorithm is simple, efficient, and effective. On the nuScenes dataset, our point-based representations performs 3-4mAP higher than the box-based counterparts for 3D detection, and 6 AMOTA higher for 3D tracking. Our real-time model runs end-to-end 3D detection and tracking at 30 FPS with 54.2AMOTA and 48.3mAP while the best single model achieves 60.3mAP for 3D detection, and 63.8AMOTA for 3D tracking.
Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-the-art performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, CenterPoint outperforms all previous single model method by a large margin and ranks first among all Lidar-only submissions.

# Highlights

- **Simple:** Two sentences method summary: We use standard 3D point cloud encoder with a few convolutional layers in the head to produce a bird-eye-view heatmap and other dense regression outputs including the offset to centers in the previous frame. Detection is a simple local peak extraction, and tracking is a closest-distance matching.
# Highlights

- **Fast:** Our [PointPillars model](configs/centerpoint/nusc_centerpoint_pp_02voxel_circle_nms.py) runs at *30* FPS with *48.3* AP and *59.1* AMOTA for simultaneous 3D detection and tracking on the nuScenes dataset.
- **Simple:** Two sentences method summary: We use standard 3D point cloud encoder with a few convolutional layers in the head to produce a bird-eye-view heatmap and other dense regression outputs including the offset to centers in the previous frame. Detection is a simple local peak extraction with refinement, and tracking is a closest-distance matching.

- **Accurate**: Our [best single model](configs/centerpoint/nusc_centerpoint_voxelnet_dcn_0075voxel_flip_testset.py) achieves *60.3* mAP and *67.3* NDS on nuScenes detection testset.
- **Fast and Accurate**: Our best single model achieves *71.9* mAPH on Waymo and *65.5* NDS on nuScenes while running at 11FPS+.

- **Extensible**: Simple baseline to switch in your backbone and novel algorithms.
- **Extensible**: Simple replacement for anchor-based detector in your novel algorithms.

## Main results

#### 3D detection
#### 3D detection on Waymo test set

| | #Frame | Veh_L2 | Ped_L2 | Cyc_L2 | MAPH | FPS |
|---------|---------|--------|--------|---------|--------|-------|
|VoxelNet | 1 | 71.9 | 67.0 | 68.2 | 69.0 | 13 |
|VoxelNet | 2 | 73.0 | 71.5 | 71.3 | 71.9 | 11 |

#### 3D detection on Waymo domain adaptation test set

| | #Frame | Veh_L2 | Ped_L2 | Cyc_L2 | MAPH | FPS |
|---------|---------|--------|--------|---------|--------|-------|
|VoxelNet | 2 | 56.1 | 47.8 | 65.2 | 56.3 | 11 |


#### 3D detection on nuScenes test set

| | MAP ↑ | NDS ↑ | PKL ↓ | FPS ↑|
|---------|---------|--------|--------|------|
|VoxelNet | 58.0 | 65.5 | 0.69 | 11 |

| | Split | MAP | NDS | FPS |
|---------|---------|---------|--------|-------|
| PointPillars-512 | Val | 48.3 | 59.1 | 30.3 |
| VoxelNet-1024 | Val | 55.4 | 63.8 | 14.5 |
| VoxelNet-1440_dcn_flip | Val | 59.1 | 67.1 | 2.2 |
| VoxelNet-1440_dcn_flip | Test | 60.3 | 67.3 | 2.2 |

#### 3D Tracking
#### 3D tracking on Waymo test set

| | Split | Tracking time | Total time | AMOTA ↑ | AMOTP ↓ |
|-----------------------|-----------|---------------|--------------|---------|---------|
| CenterPoint_pillar_512| val | 1ms | 34ms | 54.2 | 0.680 |
| CenterPoint_voxel_1024| val | 1ms | 70ms | 62.6 | 0.630 |
| CenterPoint_voxel_1440_dcn_flip | val | 1ms | 451ms | 65.9 | 0.567 |
| CenterPoint_voxel_1440_dcn_flip | test | 1ms | 451ms | 63.8 | 0.555 |
| | #Frame | Veh_L2 | Ped_L2 | Cyc_L2 | MOTA | FPS |
|---------|---------|--------|--------|---------|--------|-------|
| VoxelNet| 2 | 59.4 | 56.6 | 60.0 | 58.7 | 11 |

All results are tested on a Titan Xp GPU with batch size 1. More models and details can be found in [MODEL_ZOO.md](docs/MODEL_ZOO.md).

#### 3D Tracking on nuScenes test set

| | AMOTA ↑ | AMOTP ↓ |
|----------|---------|---------|
| VoxelNet (flip test) | 63.8 | 0.555 |


All results are tested on a Titan RTX GPU with batch size 1.

## Third-party resources

- [AFDet](https://arxiv.org/abs/2006.12671): another work inspired by CenterNet achieves good performance on KITTI/Waymo dataset.
- [mmdetection3d](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/centerpoint): CenterPoint in mmdet framework.

## Use CenterPoint

We provide a demo with PointPillars model for 3D object detection on the nuScenes dataset.

### Basic Installation

```bash
Expand Down Expand Up @@ -100,13 +116,13 @@ For more advanced usage, please refer to [INSTALL](docs/INSTALL.md) to set up mo

## Benchmark Evaluation and Training

Please refer to [GETTING_START](docs/GETTING_START.md) to prepare the data. Then follow the instruction there to reproduce our detection and tracking results. All detection configurations are included in [configs](configs) and we provide the scripts for all tracking experiments in [tracking_scripts](tracking_scripts). The pretrained models, log, and each model's prediction files are provided in the [MODEL_ZOO.md](docs/MODEL_ZOO.md).
Please refer to [GETTING_START](docs/GETTING_START.md) to prepare the data. Then follow the instruction there to reproduce our detection and tracking results. All detection configurations are included in [configs](configs) and we provide the scripts for all tracking experiments in [tracking_scripts](tracking_scripts).


## License

CenterPoint is release under MIT license (see [LICENSE](LICENSE)). It is developed based on a forked version of [det3d](https://github.com/poodarchu/Det3D/tree/56402d4761a5b73acd23080f537599b0888cce07). We also incorperate a large amount of code from [CenterNet](https://github.com/xingyizhou/CenterNet)
and [CenterTrack](https://github.com/xingyizhou/CenterTrack). See the [NOTICE](docs/NOTICE) for details. Note that the nuScenes dataset is free of charge for non-commercial activities. Please contact the [nuScenes team](https://www.nuscenes.org) for commercial usage.
and [CenterTrack](https://github.com/xingyizhou/CenterTrack). See the [NOTICE](docs/NOTICE) for details. Note that both nuScenes and Waymo datasets are under non-commercial licenses.

## Acknowlegement
This project is not possible without multiple great opensourced codebases. We list some notable examples below.
Expand All @@ -117,51 +133,4 @@ This project is not possible without multiple great opensourced codebases. We li
* [CenterNet](https://github.com/xingyizhou/CenterNet)
* [mmcv](https://github.com/open-mmlab/mmcv)
* [mmdetection](https://github.com/open-mmlab/mmdetection)
* [maskrcnn_benchmark](https://github.com/facebookresearch/maskrcnn-benchmark)
* [PCDet](https://github.com/sshaoshuai/PCDet)

**CenterPoint is deeply influenced by the following projects. Please consider citing the relevant papers.**

```
@article{zhu2019classbalanced,
title={Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection},
author={Zhu, Benjin and Jiang, Zhengkai and Zhou, Xiangxin and Li, Zeming and Yu, Gang},
journal={arXiv:1908.09492},
year={2019}
}
@article{lang2019pillar,
title={PointPillars: Fast Encoders for Object Detection From Point Clouds},
journal={CVPR},
author={Lang, Alex H. and Vora, Sourabh and Caesar, Holger and Zhou, Lubing and Yang, Jiong and Beijbom, Oscar},
year={2019},
}
@article{zhou2018voxelnet,
title={VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection},
journal={CVPR},
author={Zhou, Yin and Tuzel, Oncel},
year={2018},
}
@article{yan2018second,
title={Second: Sparsely embedded convolutional detection},
author={Yan, Yan and Mao, Yuxing and Li, Bo},
journal={Sensors},
year={2018},
}
@article{zhou2019objects,
title={Objects as Points},
author={Zhou, Xingyi and Wang, Dequan and Kr{\"a}henb{\"u}hl, Philipp},
journal={arXiv:1904.07850},
year={2019}
}
@article{zhou2020tracking,
title={Tracking Objects as Points},
author={Zhou, Xingyi and Koltun, Vladlen and Kr{\"a}henb{\"u}hl, Philipp},
journal={arXiv:2004.01177},
year={2020}
}
```
* [OpenPCDet](https://github.com/open-mmlab/OpenPCDet)
Loading

0 comments on commit e26ea57

Please sign in to comment.