Jitesh Jain, Jiachen Li^†, MangTik Chiu^†, Ali Hassani, Nikita Orlov, Humphrey Shi

^† Equal Contribution

[Project Page] [arXiv] [pdf] [BibTeX]

This repo contains the code for our paper OneFormer: One Transformer to Rule Universal Image Segmentation.

• OneFormer is a novel task-guided contrastive query-based end-to-end transformer-based universal image segmentation framework.
• OneFormer, when trained only once, outperforms the existing individually trained models on all three image segmentation tasks.
• We propose a task-conditioned joint training strategy, uniformly sampling the ground truth domain (panoptic, semantic, or instance) during training.
• During inference, OneFormer outputs task-dynamic predictions.

1. News
2. Installation Instructions
3. Dataset Preparation
4. Execution Instructions
   • Training
   • Evaluation
   • Demo
5. Results
6. Citation

• Project Page, ArXiv Preprint and GitHub Repo are public!
• OneFormer sets new SOTA on Cityscapes val with single-scale inference on Panoptic Segmentation with 68.5 PQ score and Instance Segmentation with 46.7 AP score!
• OneFormer sets new SOTA on ADE20K val on Panoptic Segmentation with 50.2 PQ score Instance Segmentation with 37.6 AP!
• OneFormer sets new SOTA on COCO val on Panoptic Segmentation with 58.0 PQ score!

• We use PyTorch 3.8.13, PyTorch-1.10.1 and CUDA 11.3.
• We use Detectron2-v0.6.
• For complete installation instructions, please see INSTALL.md.

• We experiment on three major benchmark dataset: ADE20K, Cityscapes and COCO 2017.
• Please see Preparing Datasets for OneFormer for complete instructions for preparing the datasets.

• We train all our models using 8 A6000 (48 GB each) GPUs.
• We use 8 A100 (80 GB each) for training Swin-L^† OneFormer and DiNAT-L^† OneFormer on COCO and all models with ConvNeXt-XL^† backbone. We also train the 896x896 models on ADE20K on 8 A100 GPUs.
• Please see Getting Started with OneFormer for training commands.

• Please see Getting Started with OneFormer for evaluation commands.

• We provide a quick to run demo on Colab .

• Please see OneFormer Demo for command line instructions on running the demo.

• † denotes the backbones were pretrained on ImageNet-22k.
• Pre-trained models can be downloaded following the instructions given under tools.

If you found OneFormer useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!

@article{jain2022oneformer,
      title={OneFormer: One Transformer to Rule Universal Image Segmentation},
      author={Jitesh Jain and Jiachen Li and MangTik Chiu and Ali Hassani and Nikita Orlov and Humphrey Shi},
      journal={arXiv}, 
      year={2022}
    }

We thank the authors Mask2Former and GroupViT for releasing their helpful codebases.