init

Monica-1133 · Jan 7, 2019 · 9f50190 · 9f50190
1 parent 736af4b
commit 9f50190
Show file tree

Hide file tree

Showing 65 changed files with 6,309 additions and 1 deletion.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,12 @@
+*DS_Store*
+tools/vis*
+cache
+data
+data/
+cache/
+results/
+
+*.swp
+
+*.pyc
+*.o*
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2018, University of Michigan
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/README.md b/README.md
@@ -1 +1,164 @@
-# ExtremeNet
+# ExtremeNet: Training and Evaluation Code
+Code for **bottom-up** object detection by grouping extreme and center points:
+![](readme/teaser.png)
+> [**Bottom-up Object Detection by Grouping Extreme and Center Points**](https://arxiv.org/abs/xxxx.xxxxx),            
+> Xingyi Zhou, Jiacheng Zhuo, Philipp Kr&auml;henb&uuml;hl,        
+> *arXiv technical report*         
+
+This project is developed upon the [CornerNet code](https://github.com/princeton-vl/CornerNet) and contains the code from [Deep Extreme Cut](https://github.com/scaelles/DEXTR-PyTorch). Thanks to the original authors!
+
+Contact: [zhouxy2017@gmail.com](mailto:zhouxy2017@gmail.com). Any questions or discussions are welcomed! 
+
+## Abstract 
+
+With the advent of deep learning, object detection drifted from a bottom-up to a top-down recognition problem. State of the art algorithms enumerate a near-exhaustive list of object locations and classify each into: object or not. In this paper, we show that bottom-up approaches still perform competitively. We detect four extreme points (top-most, left-most, bottom-most, right-most) and one center point of objects using a standard keypoint estimation network. We group the five keypoints into a bounding box if they are geometrically aligned. Object detection is then a purely appearance-based keypoint estimation problem, without region classification or implicit feature learning. The proposed method performs on-par with the state-of-the-art region based detection methods, with a bounding box AP of 43.2% on COCO test-dev. In addition, our estimated extreme points directly span a coarse octagonal mask, with a COCO Mask AP of 18.9%, much better than the Mask AP of vanilla bounding boxes. Extreme point guided segmentation further improves this to 34.1% Mask AP.
+
+## Installation
+The code was tested with [Anaconda](https://www.anaconda.com/download) Python 3.6 and [PyTorch]((http://pytorch.org/)) v0.4.1. After install Anaconda:
+
+1. Clone this repo:
+
+    ~~~
+    ExtremeNet_ROOT=/path/to/clone/ExtremeNet
+    git clone --recursive https://github.com/xingyizhou/ExtremeNet ExtremeNet_ROOT
+    ~~~
+
+
+2. Create an Anaconda environment using the provided package list from [Cornernet](https://github.com/princeton-vl/CornerNet).
+
+    ~~~
+    conda create --name CornerNet --file conda_packagelist.txt
+    source activate CornerNet
+    ~~~
+
+3. Compiling NMS (originally from [Faster R-CNN](https://github.com/rbgirshick/py-faster-rcnn/blob/master/lib/nms/cpu_nms.pyx) and [Soft-NMS](https://github.com/bharatsingh430/soft-nms/blob/master/lib/nms/cpu_nms.pyx)).
+
+    ~~~
+    cd ExtremeNet_ROOT/external
+    make
+    ~~~
+
+## Demo
+
+- Download our [pre-trained model](https://drive.google.com/file/d/1re-A74WRvuhE528X6sWsg1eEbMG8dmE4/view?usp=sharing) and put it in `cache/`.
+- Optionally, if you want to test instance segmentation with [Deep Extreme Cut](https://github.com/scaelles/DEXTR-PyTorch), download their [PASCAL + SBD pertained model](https://data.vision.ee.ethz.ch/kmaninis/share/DEXTR/Downloads/models/dextr_pascal-sbd.pth) and put it in `cache/`. 
+- Run the demo 
+
+    ~~~
+    Python demo.py ExtremeNet [--demo /path/to/image/or/folder] [--show_mask]
+    ~~~
+
+    Contents in `[]` are optional. By default, it runs the sample images provided in `ExtremeNet_ROOT/images/` (from [Detectron](https://github.com/facebookresearch/Detectron/tree/master/demo)). We show the predicted extreme point heatmaps (combined four heatmaps and overlaid on the input image), the predicted center point heatmap, and the detection and octagon mask results. If setup correctly, the output will look like:
+    
+    <img src='readme/extreme.png' align="center" width="400px"> <img src='readme/center.png' align="center" width="400px">
+    
+    <p align="center"> 
+        <img src='readme/octagon.png' align="center" width="400px">
+    </p>
+
+    If `--show_mask` is turned on, it further pipelined with [Deep Extreme Cut](https://github.com/scaelles/DEXTR-PyTorch) for instance segmentation. The output will look like:
+    <p align="center"> 
+        <img src='readme/mask.png' align="center" width="400px">
+    </p>
+
+
+## Data preparation
+
+If you want to reproduce the results in the paper for benchmark evaluation and training, you will need to setup dataset.
+
+### Installing MS COCO APIs
+~~~
+cd ExtremeNet_ROOT/data
+git clone https://github.com/cocodataset/cocoapi.git coco
+cd ExtremeNet_ROOT/data/coco/PythonAPI
+Make
+python setup.up install --user
+~~~
+
+### Downloading MS COCO Data
+- Download the images (2017 Train, 2017 Val, 2017 Test) from [coco website](http://cocodataset.org/#download).
+- Download annotation files (2017 train/val and test image info) from [coco website](http://cocodataset.org/#download). 
+- Place the data (or create symlinks) to make the data folder like:
+
+  ~~~
+  ${ExtremeNet_ROOT}
+  |-- data
+  `-- |-- coco
+      `-- |-- annotations
+          |   |-- instances_train2017.json
+          |   |-- instances_val2017.json
+          |   |-- image_info_test-dev2017.json
+          `-- images
+              |-- train2017
+              |-- val2017
+              |-- test2017
+  ~~~
+- Generate extreme point annotation from segmentation:
+    
+    ~~~
+    cd ExtremeNet_ROOT/tools/
+    python gen_coco_extreme_points.py
+    ~~~
+  It generates `instances_extreme_train2017.json` and `instances_extreme_val2017.json` in `data/coco/annotations/`. 
+
+## Benchmark Evaluation
+
+After downloading our pre-trained model and the dataset,
+
+- Run the following command to evaluate object detection:
+
+  ~~~
+  python test.py ExtremeNet [--suffix multi_scale]
+  ~~~
+  The results on COCO validation set should be `40.3` box AP without `--suffix multi_scale` and `43.3` box AP with `--suffix multi_scale`. 
+
+- After obtaining the detection results, run the following commands for instance segmentation:
+
+  ~~~
+  python eval_dextr_mask.py results/ExtremeNet/250000/validation/multi_scale/results.json
+  ~~~
+  The results on COCO validation set should be `34.6` mask AP (The evaluation will be slow).
+
+- You can test with other hyper-parameters by creating a new config file (`ExtremeNet-<suffix>.json`) in `config/`. 
+
+
+## Training 
+
+You will need 5 12GB GPUs to reproduce our training. Our model is fine-tuned on the 10-GPU pre-trained [CornerNet model](https://drive.google.com/file/d/1UHjVzSG27Ms0VfSFeGYJ2h2AYZ6d4Le_/view?usp=sharing) (training from scratch may result in 2 AP lower based on our preliminary experiments). After downloading the CornerNet model and put it in `cache/`, run
+
+~~~
+python train.py ExtremeNet
+~~~
+
+You can resume a half-trained model by 
+
+~~~
+python train.py ExtremeNet --iter xxxx
+~~~
+
+## Citation
+If you find this model useful for your resesarch, please use the following BibTeX entry.
+
+    @inproceedings{zhou2019bottomup,
+      title={Bottom-up Object Detection by Grouping Extreme and Center Points},
+      author={Zhou, Xingyi and Zhuo, Jiacheng and Kr{\"a}henb{\"u}hl, Philipp},
+      booktitle={arXiv preprint arXiv:xxxx.xxxxx},
+      year={2019}
+    }
+    
+Please also considering citing the CornerNet paper (where this code is heavily borrowed from) and Deep Extreme Cut paper (if you use the instance segmentation part).
+
+    @inproceedings{law2018cornernet,
+      title={CornerNet: Detecting Objects as Paired Keypoints},
+      author={Law, Hei and Deng, Jia},
+      booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
+      pages={734--750},
+      year={2018}
+    }
+
+    @Inproceedings{Man+18,
+      Title          = {Deep Extreme Cut: From Extreme Points to Object Segmentation},
+      Author         = {K.K. Maninis and S. Caelles and J. Pont-Tuset and L. {Van Gool}},
+      Booktitle      = {Computer Vision and Pattern Recognition (CVPR)},
+      Year           = {2018}
+    }
diff --git a/conda_packagelist.txt b/conda_packagelist.txt
@@ -0,0 +1,88 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+@EXPLICIT
+https://repo.continuum.io/pkgs/main/linux-64/blas-1.0-mkl.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/bzip2-1.0.6-h9a117a8_4.tar.bz2
+https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2018.4.16-0.tar.bz2
+https://conda.anaconda.org/caffe2/linux-64/caffe2-cuda8.0-cudnn7-0.8.dev-py36_2018.05.14.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/cairo-1.14.12-h7636065_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/certifi-2018.4.16-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/cffi-1.11.5-py36h9745a5d_0.tar.bz2
+https://repo.continuum.io/pkgs/free/linux-64/cudatoolkit-8.0-3.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/cycler-0.10.0-py36h93f1223_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/dbus-1.13.2-h714fa37_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/expat-2.2.5-he0dffb1_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/ffmpeg-3.4-h7264315_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/fontconfig-2.12.6-h49f89f6_0.tar.bz2
+https://repo.continuum.io/pkgs/free/linux-64/freeglut-2.8.1-0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/freetype-2.8-hab7d2ae_1.tar.bz2
+https://repo.continuum.io/pkgs/free/linux-64/future-0.16.0-py36_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/gflags-2.2.1-hf484d3e_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/glib-2.56.1-h000015b_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/glog-0.3.5-hf484d3e_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/graphite2-1.3.11-hf63cedd_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/gstreamer-1.14.0-hb453b48_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/h5py-2.8.0-py36hca9c191_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/harfbuzz-1.7.6-h5f0a787_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/hdf5-1.8.18-h6792536_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/icu-58.2-h9c2bf20_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/intel-openmp-2018.0.0-8.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/jasper-2.0.14-h07fcdf6_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/kiwisolver-1.0.1-py36h764f252_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libedit-3.1-heed3624_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libffi-3.2.1-hd88cf55_4.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libgcc-ng-7.2.0-hdf63c60_3.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libgfortran-ng-7.2.0-hdf63c60_3.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libglu-9.0.0-h0c0bdc1_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libopus-1.2.1-hb9ed12e_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libpng-1.6.34-hb9fc6fc_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libprotobuf-3.5.2-h6f1eeef_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libstdcxx-ng-7.2.0-hdf63c60_3.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libtiff-4.0.9-h28f6b97_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libvpx-1.6.1-h888fd40_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libxcb-1.13-h1bed415_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/libxml2-2.9.8-hf84eae3_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/matplotlib-2.2.2-py36h0e671d2_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/mkl-2018.0.2-1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/mkl_fft-1.0.1-py36h3010b51_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/mkl_random-1.0.1-py36h629b387_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/ncurses-6.0-h9df7e31_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/ninja-1.8.2-py36h6bb024c_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/numpy-1.14.3-py36hcd700cb_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/numpy-base-1.14.3-py36h9be14a7_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/olefile-0.45.1-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/opencv-3.3.1-py36h9248ab4_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/openssl-1.0.2o-h20670df_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pcre-8.42-h439df22_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pillow-5.1.0-py36h3deb7b8_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pip-10.0.1-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pixman-0.34.0-hceecf20_3.tar.bz2
+https://conda.anaconda.org/conda-forge/linux-64/protobuf-3.5.2-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pycparser-2.18-py36hf9f622e_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pyparsing-2.2.0-py36hee85983_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pyqt-5.9.2-py36h751905a_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/python-3.6.5-hc3d631a_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/python-dateutil-2.7.2-py36_0.tar.bz2
+https://conda.anaconda.org/pytorch/linux-64/pytorch-0.4.1-py36_cuda9.0.176_cudnn7.1.2_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pytz-2018.4-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/pyyaml-3.12-py36hafb9ca4_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/qt-5.9.5-h7e424d6_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/readline-7.0-ha6073c6_4.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/scikit-learn-0.19.1-py36h7aa7ec6_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/scipy-1.1.0-py36hfc37229_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/setuptools-39.1.0-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/sip-4.19.8-py36hf484d3e_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/six-1.11.0-py36h372c433_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/sqlite-3.23.1-he433501_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/tk-8.6.7-hc745277_3.tar.bz2
+https://conda.anaconda.org/pytorch/linux-64/torchvision-0.2.1-py36_1.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/tornado-5.0.2-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/tqdm-4.23.0-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/wheel-0.31.0-py36_0.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/xz-5.2.3-h5e939de_4.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/yaml-0.1.7-had09818_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/zlib-1.2.11-ha838bed_2.tar.bz2
+https://repo.continuum.io/pkgs/main/linux-64/progress-1.4-py36_0.tar.bz2