The repository contains the official implementation of MaNo, a training-free method to assess the generalization performance of neural networks under distribution shifts introduced in

MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts.
Renchunzi Xie*, Ambroise Odonnat*, Vasilii Feofanov*, Weijian Deng, Jianfeng Zhang, Bo An.
*Equal contribution.

MaNo is an efficient training-free approach grounded in theory that leverages logits to estimate the generalization performance of a pre-trained neural network under distribution shifts. It makes use of $\sigma$, a novel normalization function to deal with poorly-calibrated scenarios. Given a pre-trained model $f$ and a test set $D_{test} = (x_i)_{i=1}^N$ with K classes, MaNo operates as follows:

• Recover the logits $\theta_i = f(x_i)$
• Normalize the logits $\theta_i \to \sigma(\theta_i) \in \Delta_K$
• Fill a prediction matrix $P \in \mathbb{R}^{N \times K}$ with rows $\sigma(\theta_i)$
• Compute the estimation score $S(f, D_{test})$ as the scaled p-norm of $P$, i.e., $S(f, D_{test}) = \frac{1}{NK} \lVert P \rVert_p$.

We conduct large-scale experiments under several distribution shifts with ResNets, ConvNext, and ViT.

🥇 SOTA performance. Our approach MaNo provides the best and most robust accuracy estimation.

🚀 Qualitative benefits. Our approach is training-free, fast and memory efficient.

1. CIFAR10 & CIFAR-100 can be downloaded in the code.
2. Download TinyImageNet

wget http://cs231n.stanford.edu/tiny-imagenet-200.zip

3. Download ImageNet

https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data

4. Download Office-Home

pip install gdown==4.6.0
gdown https://drive.google.com/uc?id=1JMFEHM46xmgp2RSX6iVgcR5fCpZkeruJ

5. Download PACS

https://www.kaggle.com/datasets/nickfratto/pacs-dataset

6. Download DomainNet

wget http://csr.bu.edu/ftp/visda/2019/multi-source/groundtruth/clipart.zip
wget http://csr.bu.edu/ftp/visda/2019/multi-source/infograph.zip
wget http://csr.bu.edu/ftp/visda/2019/multi-source/groundtruth/painting.zip
wget http://csr.bu.edu/ftp/visda/2019/multi-source/quickdraw.zip
wget http://csr.bu.edu/ftp/visda/2019/multi-source/real.zip
wget http://csr.bu.edu/ftp/visda/2019/multi-source/sketch.zip

1. Download CIFAR-10C & CIFAR-100C

wget https://zenodo.org/record/2535967/files/CIFAR-10-C.tar
wget https://zenodo.org/record/3555552/files/CIFAR-100-C.tar

2. Download TinyImageNet-C

wget http://cs231n.stanford.edu/tiny-imagenet-200.zip

3. Download ImageNet-C

wget https://zenodo.org/record/2235448/files/blur.tar
wget https://zenodo.org/record/2235448/files/digital.tar
wget https://zenodo.org/record/2235448/files/extra.tar
wget https://zenodo.org/record/2235448/files/noise.tar
wget https://zenodo.org/record/2235448/files/weather.tar

Step 1: Pre-train models on CIFAR-10, CIFAR-100 and TinyImageNet using commands in ./bash/init_base_model.sh.

Step 2: Estimate OOD error on CIFAR-10C, CIFAR-100C, and TinyImageNet-C using commands in ./bash/mano.sh.

• Renchunzi Xie
• Ambroise Odonnat
• Vasilii Feofanov

The code is distributed under the MIT license.

If you find this work useful in your research, please cite:

@misc{xie2024manoexploitingmatrixnorm,
      title={MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts}, 
      author={Renchunzi Xie and Ambroise Odonnat and Vasilii Feofanov and Weijian Deng and Jianfeng Zhang and Bo An},
      year={2024},
      eprint={2405.18979},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2405.18979}, 
}