This repository contains accompanying code for the article introducing Meta-Dataset, https://arxiv.org/abs/1903.03096.

This code is provided here in order to give more details on the implementation of the data-providing pipeline, our back-bones and models, as well as the experimental setting.

See below for user instructions, including how to install the software, download and convert the data, and train implemented models.

See this introduction notebook for a demonstration of how to sample data from the pipeline (episodes or batches).

We are currently working on updating the code and improving the instructions to facilitate designing and running new experiments.

This is not an officially supported Google product.

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle this recently, we find the current procedure and datasets that are used to systematically assess progress in this setting lacking. To address this, we propose Meta-Dataset: a new benchmark for training and evaluating few-shot classifiers that is large-scale, consists of multiple datasets, and presents more natural and realistic tasks. The aim is to measure the ability of state-of-the-art models to leverage diverse sources of data to achieve higher generalization, and to evaluate that generalization ability in a more challenging and realistic setting. We additionally measure robustness to variations in the number of available examples and the number of classes. Finally our extensive empirical evaluation leads us to identify weaknesses in Prototypical Networks and MAML, two popular few-shot classification methods, and to propose a new method, Proto-MAML, which achieves improved performance on our benchmark.

Meta-Dataset is now compatible with Python 2 and Python 3, but has mostly been used with Python 2 up to now, so glitches with Python 3 are still possible. The code has not been tested with TensorFlow 2 yet.

• We recommend you follow these instructions to install TensorFlow.
• A list of packages to install is available in requirements.txt, you can install them using pip.
• Clone the meta-dataset repository. Most command lines start with python -m meta_dataset.<something>, and should be typed from within that clone (where a meta_dataset Python module should be visible).

Meta-Dataset uses several established datasets, that are available from different sources. You can find below a summary of these datasets, as well as instructions to download them and convert them into a common format.

For brevity of the command line examples, we assume the following environment variables are defined:

• $DATASRC: root of where the original data is downloaded and potentially extracted from compressed files. This directory does not need to be available after the data conversion is done.
• $SPLITS: directory where *_splits.json files will be created, one per dataset. For instance, $SPLITS/fungi_splits.json contains information about which classes are part of the meta-training, meta-validation, and meta-test set. These files are only used during the dataset conversion phase, but can help troubleshooting later. To re-use the canonical splits instead of re-generating them, you can make it point to meta_dataset/dataset_conversion in your checkout.
• $RECORDS: root directory that will contain the converted datasets (one per sub-directory). This directory needs to be available during training and evaluation.

Experiments are defined via gin configuration files, that are under meta_dataset/learn/gin/:

• setups/ contain generic setups for classes of experiment, for instance which datasets to use (imagenet or all), parameters for sampling the number of ways and shots of episodes.
• models/ define settings for different meta-learning algorithms (baselines, prototypical networks, MAML...)
• default/ contains files that each correspond to one experiment, mostly defining a setup and a model, with default values for training hyperparameters.
• best/ contains files with values for training hyperparameters that achieved the best performance during hyperparameter search.

There are two main architectures, or "backbones": four_layer_convnet (sometimes convnet for short) and resnet, that can be used in the baselines ("k-NN" and "Finetune"), ProtoNet, and MatchingNet. Their layers do not have a trainable bias since it would be negated by the use of batch normalization. For fo-MAML and ProtoMAML, each of the backbones have a version with trainable biases (due to the way batch normalization is handled), resp. four_layer_convnet_maml (or mamlconvnet) and resnet_maml (sometimes mamlresnet); these can also be used by the baseline for pre-training of the MAML models.

See Reproducing best results for instructions to launch training experiments with the values of hyperparameters that were selected in the paper. The hyperparameters (including the backbone, whether to train from scratch or from pre-trained weights, and the number of training updates) were selected using only the validation classes of the ILSVRC 2012 dataset for all experiments. Even when training on "all" datasets, the validation classes of the other datasets were not used.

We tried our best to reproduce the original results using the public code on Google Cloud VMs, but there is inherent noise and variability in the computation. For transparency, we will include the results of the reproduced experiments as well as the original ones, which were reported in the article. We also include the validation errors on ILSVRC.

1. The MSCOCO results reported in the v1 version of the arXiv paper were obtained with a different valid/test split. The "repro" version is consistent with the latest version of the splits.

TODO: Range / distribution of tried hyperparameters