tutorials/classy_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Creating a custom dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will learn how to do the following: \n",
    "\n",
    "1. Create a custom dataset within Classy Vision\n",
    "2. Integrate a dataset with Classy Vision's configuration system\n",
    "3. Iterate over the samples contained in a dataset\n",
    "4. Using transforms with Classy Vision\n",
    "5. Create a ImageNet dataset, using standard transforms, using torchvision\n",
    "\n",
    "If you haven't already read our [Getting started](https://classyvision.ai/tutorials/getting_started) tutorial, we recommend starting there before reading this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create a custom dataset within Classy Vision\n",
    "\n",
    "Creating a dataset for use / using an existing dataset in Classy Vision is as easy as it is in PyTorch, it only requires wrapping the dataset in our dataloading class, ClassyDataset.\n",
    "\n",
    "First, specify a dataset with a `__getitem__` and `__len__` function, the same as required by torch.utils.data.Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.utils.data\n",
    "import torch\n",
    "\n",
    "class MyDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.length = 100\n",
    "        \n",
    "    def __getitem__(self, idx):\n",
    "        assert idx >= 0 and idx < self.length, \\\n",
    "            \"Provided index {} must be in range [0, {}).\".format(idx, self.length)\n",
    "        return torch.rand(3, 100, 100)\n",
    "    \n",
    "    def __len__(self):\n",
    "        return self.length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for most training tasks we want to be able to configure the batchsize on the fly, transform samples, shuffle the dataset, maybe limit the number of samples to shorten a training run, and then construct an iterator for the training loop. ClassyDataset is a simple wrapper that provides this functionality.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.dataset import ClassyDataset\n",
    "\n",
    "class MyClassyDataset(ClassyDataset):\n",
    "    def __init__(self, batchsize_per_replica, shuffle, transform, num_samples):\n",
    "        dataset = MyDataset()\n",
    "        super().__init__(dataset, batchsize_per_replica, shuffle, transform, num_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's that easy! Later in the tutorial we will see how to use the iterator, but before moving on, let's talk about what each of these arguments does.\n",
    "- __batchsize_per_replica__: the batchsize per trainer (so if you have 8 GPUs with 1 trainer processes and a batchsize_per_replica of 32, then your batchsize for single update is 8 * 32 = 256).\n",
    "- __shuffle__: If true, then shuffle the dataset before each epoch.\n",
    "- __transform__: A callable applied to each sample before returning. Note that this can get tricky since many datasets (e.g. torchvision datasets) return complex samples containing both the image / video content and a label and possibly additional metadata. We pass the _whole_ sample to the transform, so it needs to know how to parse the sample...more on this later.\n",
    "- __num_samples__: Not needed in the standard use cases, but this allows a user to adjust the length of samples retrieved in an epoch, can be convenient for debugging via config (e.g. setting num_samples = 10 will speed up training). By default this is set to None and iteration proceeds over the whole dataset.\n",
    "\n",
    "To get started with a basic task just do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.tasks import ClassificationTask\n",
    "\n",
    "my_dataset = MyClassyDataset(\n",
    "    batchsize_per_replica=10, \n",
    "    shuffle=True, \n",
    "    transform=None, \n",
    "    num_samples=None,\n",
    ")\n",
    "\n",
    "# Note, the \"train\" here is the phase type.\n",
    "# It tells the task to set the model in train mode / do a backwards pass, etc, using our new dataset\n",
    "my_task = ClassificationTask().set_dataset(my_dataset, \"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details on training a model, please see our [Getting started](https://classyvision.ai/tutorials/getting_started) tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Integrating a dataset with Classy Vision's configuration system\n",
    "\n",
    "Classy Vision is also able to read a configuration file and instantiate the dataset. This is useful to keep your experiments organized and reproducible. For that, you have to:\n",
    "\n",
    "- Implement a from_config method\n",
    "- Add the register_model decorator to MyClassyDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.dataset import ClassyDataset, register_dataset\n",
    "from classy_vision.dataset.transforms import build_transforms\n",
    "\n",
    "@register_dataset(\"my_dataset\")\n",
    "class MyClassyDataset(ClassyDataset):\n",
    "    def __init__(self, batchsize_per_replica, shuffle, transform, num_samples):\n",
    "        dataset = MyDataset()\n",
    "        super().__init__(dataset, batchsize_per_replica, shuffle, transform, num_samples)\n",
    "        \n",
    "    @classmethod\n",
    "    def from_config(cls, config):\n",
    "        transform = build_transforms(config[\"transforms\"])\n",
    "        return cls(\n",
    "            batchsize_per_replica=config[\"batchsize_per_replica\"],\n",
    "            shuffle=config[\"shuffle\"],\n",
    "            transform=transform,\n",
    "            num_samples=config[\"num_samples\"],\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start using this dataset in our configurations. The string argument passed to the register_dataset is a unique identifier for this model (if you try to register two models with the same name, it will throw an error):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.dataset import build_dataset\n",
    "import torch\n",
    "\n",
    "dataset_config = {\n",
    "    \"name\": \"my_dataset\",\n",
    "    \"batchsize_per_replica\": 10,\n",
    "    \"shuffle\": True,\n",
    "    \"transforms\": [{\"name\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]}],\n",
    "    \"num_samples\": None,\n",
    "}\n",
    "my_dataset = build_dataset(dataset_config)\n",
    "assert isinstance(my_dataset, MyClassyDataset)\n",
    "\n",
    "sample = my_dataset[0]\n",
    "print(sample.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Iterate over the samples contained in a dataset\n",
    "\n",
    "As mentioned above, the ClassyDataset class adds several pieces of basic logic for constructing a torch.utils.data.Dataloader for your dataset. ClassyDataset supports local and distributed training out-of-box by internally using a PyTorch DistributedSampler for sampling the dataset along with the PyTorch Dataloader for batching and parallelizing sample retrieval. To get an iterable for epoch 0, do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.dataset import build_dataset\n",
    "import torch\n",
    "\n",
    "dataset_config = {\n",
    "    \"name\": \"my_dataset\",\n",
    "    \"batchsize_per_replica\": 10,\n",
    "    \"shuffle\": True,\n",
    "    \"transforms\": [],\n",
    "    \"num_samples\": None,\n",
    "}\n",
    "my_dataset = build_dataset(dataset_config)\n",
    "assert isinstance(my_dataset, MyClassyDataset)\n",
    "\n",
    "# multiprocessing_context can be set to \"spawn\", \"forkserver\", \"fork\" or None.\n",
    "# If None is used, then the dataloader inherits the context of the parent thread.\n",
    "# If num_workers is 0, then multiprocessing is not used by the dataloader\n",
    "#\n",
    "# A warning, while fork is fast and simple to get started with, it \n",
    "# is unsafe to use with threading and can lead to difficult to debug errors.\n",
    "# Spawn / forkserver are threadsafe, but they come with additional startup costs.\n",
    "iterator = my_dataset.iterator(\n",
    "    shuffle_seed=0,\n",
    "    epoch=0,\n",
    "    num_workers=0,  # 0 indicates to do dataloading on the master process\n",
    "    pin_memory=False,\n",
    "    multiprocessing_context=None,\n",
    ")\n",
    "assert isinstance(iterator, torch.utils.data.DataLoader)\n",
    "\n",
    "# Iterate over all 100 samples.\n",
    "for sample in iter(iterator):\n",
    "    # Do stuff with sample...\n",
    "    # Note that size now has an extra dimension representing the batchsize\n",
    "    assert sample.size() == torch.Size([10, 3, 100, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also provide a custom iterator function if you would like to return a custom iterator or a custom sampler. Please see the ClassyDataset code for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Using transforms with Classy Vision\n",
    "\n",
    "You may have noticed in the configuration section that we did something slightly more complicated with the transform configuration. In particular, just like our datasets / models etc, we have a registration / build mechanism for transforms so that transforms can be specified via config. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform example using Classy Vision's synthetic image dataset\n",
    "We also automatically register torchvision transforms, so let's start with an example of how to specify torchvision transforms and the synthetic image dataset we provide for testing / proto-typing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.transforms as transforms\n",
    "from classy_vision.dataset import build_dataset\n",
    "from classy_vision.dataset.classy_synthetic_image import SyntheticImageDataset\n",
    "from classy_vision.dataset.transforms import build_transforms\n",
    "\n",
    "# Declarative approach\n",
    "\n",
    "# Transform to be applied to image\n",
    "image_transform = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(224),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
    "])\n",
    "\n",
    "decl_dataset = SyntheticImageDataset(\n",
    "    batchsize_per_replica=10,\n",
    "    shuffle=True,\n",
    "    transform=image_transform,\n",
    "    num_samples=100,\n",
    "    crop_size=320,\n",
    "    class_ratio=4,\n",
    "    seed=0,\n",
    ")\n",
    "\n",
    "# FAILS!!!!\n",
    "# decl_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This fails! Why?\n",
    "\n",
    "It fails because most datasets don't return just an image, they return image or video content data, label data, and (potentially) sample metadata. In Classy Vision, the sample format is specified by the task and our classification_task expects a dict with input / target keys.\n",
    "\n",
    "For example, the sample format for the SyntheticImageDataset looks like:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`{\"input\": <PIL Image>, \"target\": <Target>}`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our transforms to work, we need to specify which key to apply the transform to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.transforms as transforms\n",
    "from classy_vision.dataset import build_dataset\n",
    "from classy_vision.dataset.classy_synthetic_image import SyntheticImageDataset\n",
    "from classy_vision.dataset.transforms import build_transforms, ApplyTransformToKey\n",
    "\n",
    "# Declarative approach\n",
    "\n",
    "# Transform to be applied to image\n",
    "image_transform = transforms.Compose([\n",
    "        transforms.Resize(256),\n",
    "        transforms.CenterCrop(224),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
    "])\n",
    "\n",
    "# Transform wrapper that says which key to apply the transform to\n",
    "transform = ApplyTransformToKey(\n",
    "    transform=image_transform,\n",
    "    key=\"input\",\n",
    ")\n",
    "\n",
    "decl_dataset = SyntheticImageDataset(\n",
    "    batchsize_per_replica=10,\n",
    "    shuffle=True,\n",
    "    transform=transform,\n",
    "    num_samples=100,\n",
    "    crop_size=320,\n",
    "    class_ratio=4,\n",
    "    seed=0,\n",
    ")\n",
    "\n",
    "# Success!!!!\n",
    "decl_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how to do the same thing via a config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that this cell won't work until we fix the synthetic dataset from_config function\n",
    "\n",
    "from classy_vision.dataset import build_dataset\n",
    "\n",
    "# Configuration approach\n",
    "config = {\n",
    "    \"name\": \"synthetic_image\",\n",
    "    \"batchsize_per_replica\": 10,\n",
    "    \"use_shuffle\": True,\n",
    "    \"transforms\": [\n",
    "        {\n",
    "            \"name\": \"apply_transform_to_key\",\n",
    "            \"transforms\": [\n",
    "                {\"name\": \"Resize\", \"size\": 256},\n",
    "                {\"name\": \"CenterCrop\", \"size\": 224},\n",
    "                {\"name\": \"ToTensor\"},\n",
    "                {\"name\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]},\n",
    "            ],\n",
    "            \"key\": \"input\",\n",
    "        },\n",
    "    ],\n",
    "    \"num_samples\": 100,\n",
    "    \"crop_size\": 320,\n",
    "    \"class_ratio\": 4,\n",
    "    \"seed\": 0\n",
    "}\n",
    "\n",
    "config_dataset = build_dataset(config)\n",
    "\n",
    "# Sample should be the same as that provided by the decl_dataset\n",
    "assert torch.allclose(config_dataset[0][\"input\"], decl_dataset[0][\"input\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform example for a torchvision dataset\n",
    "Torchvision has a different sample format using tuples for images: \n",
    "\n",
    "`(<PIL Image>, <Target>)`\n",
    "\n",
    "The ApplyTransformToKey will still work (the key in this case is '0'), but for our classification tasks, we also want a sample that is a dict with \"input\"/\"target\" keys. \n",
    "\n",
    "Because this is a common dataset format, we provide a convenience transform called \"GenericImageTransform\" which applies a specified transform to the torchvision tuple image key and then maps the whole sample to a dict. This is just a convenience transform, we can also do this using raw composable blocks, but it makes things more verbose.\n",
    "\n",
    "All of the transforms in the next cell have the same effect on an image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.transforms import Compose\n",
    "from classy_vision.dataset.transforms import build_transforms\n",
    "from classy_vision.dataset.transforms.util import GenericImageTransform\n",
    "\n",
    "# Declarative\n",
    "image_transform = transforms.Compose([\n",
    "        transforms.Resize(256),\n",
    "        transforms.CenterCrop(224),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
    "])\n",
    "decl_transform = GenericImageTransform(transform=image_transform)\n",
    "\n",
    "# Configuration with helper function\n",
    "transform_config = [{\n",
    "    \"name\": \"generic_image_transform\",\n",
    "    \"transforms\": [\n",
    "        {\"name\": \"Resize\", \"size\": 256},\n",
    "        {\"name\": \"CenterCrop\", \"size\": 224},\n",
    "        {\"name\": \"ToTensor\"},\n",
    "        {\"name\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]},\n",
    "    ], \n",
    "}]\n",
    "config_helper_transform = build_transforms(transform_config)\n",
    "\n",
    "# Configuration using raw, composable functions:\n",
    "transform_config = [\n",
    "    {\"name\": \"tuple_to_map\", \"list_of_map_keys\": [\"input\", \"target\"]},\n",
    "    {\n",
    "        \"name\": \"apply_transform_to_key\",\n",
    "        \"transforms\": [\n",
    "            {\"name\": \"Resize\", \"size\": 256},\n",
    "            {\"name\": \"CenterCrop\", \"size\": 224},\n",
    "            {\"name\": \"ToTensor\"},\n",
    "            {\"name\": \"Normalize\", \"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]},\n",
    "        ], \n",
    "        \"key\": \"input\",\n",
    "    },\n",
    "]\n",
    "config_raw_transform = build_transforms(transform_config)\n",
    "\n",
    "# These transforms are all functionally the same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Create a Classy Imagenet\n",
    "\n",
    "Now, to complete this tutorial, we show our code for creating an ImageNet dataset in classy vision using the pre-existing torchvision dataset. Code very similar to this (+ some typing and helper functions) is in the datasets folder of the base Classy Vision repository.\n",
    "\n",
    "Note, we do not distribute any of the underlying dataset data with Classy Vision. Before this will work, you will need to download a torchvision compatible copy of the Imagenet dataset yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from classy_vision.dataset import ClassyDataset, register_dataset\n",
    "from classy_vision.dataset.transforms import ClassyTransform, build_transforms\n",
    "from torchvision.datasets.imagenet import ImageNet\n",
    "        \n",
    "        \n",
    "@register_dataset(\"example_imagenet\")\n",
    "class ExampleImageNetDataset(ClassyDataset):\n",
    "    def __init__(\n",
    "        self,\n",
    "        split,\n",
    "        batchsize_per_replica,\n",
    "        shuffle,\n",
    "        transform,\n",
    "        num_samples,\n",
    "        root,  # Root directory for your Imagenet dataset\n",
    "    ):  \n",
    "        # Create torchvision dataset\n",
    "        dataset = ImageNet(root=root, split=split)\n",
    "        super().__init__(\n",
    "            dataset, batchsize_per_replica, shuffle, transform, num_samples\n",
    "        )   \n",
    "\n",
    "    @classmethod\n",
    "    def from_config(cls, config):\n",
    "        batchsize_per_replica = config.get(\"batchsize_per_replica\")\n",
    "        shuffle = config.get(\"use_shuffle\")\n",
    "        num_samples = config.get(\"num_samples\")\n",
    "        transform_config = config.get(\"transforms\")\n",
    "        split = config.get(\"split\")\n",
    "        root = config.get(\"root\")\n",
    "        download = config.get(\"download\")\n",
    "        \n",
    "        transform = build_transforms(transform_config)\n",
    "        return cls(\n",
    "            split=split,\n",
    "            batchsize_per_replica=batchsize_per_replica,\n",
    "            shuffle=shuffle,\n",
    "            transform=transform,\n",
    "            num_samples=num_samples,\n",
    "            root=root,\n",
    "            download=download,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this tutorial we have seen how to create a custom dataset using ClassyDataset, how to integrate this dataset with the configuration system, how to iterate over samples / use multiple workers, how to use transforms in the configuration system and finally we showed an example of how to use a torchvision dataset in Classy Vision. \n",
    "\n",
    "For more details on how to use the dataset for training, please see [Getting started](https://classyvision.ai/tutorials/getting_started)."
   ]
  }
 ],
 "metadata": {
  "bento_stylesheets": {
   "bento/extensions/flow/main.css": true,
   "bento/extensions/kernel_selector/main.css": true,
   "bento/extensions/kernel_ui/main.css": true,
   "bento/extensions/new_kernel/main.css": true,
   "bento/extensions/system_usage/main.css": true,
   "bento/extensions/theme/main.css": true
  },
  "disseminate_notebook_id": {
   "notebook_id": "1152520058270736"
  },
  "disseminate_notebook_info": {
   "bento_version": "20191118-000256",
   "description": "Test of Classy Dataset tutorial\n\nv3",
   "hide_code": false,
   "hipster_group": "",
   "kernel_build_info": {
    "deps": [
     "//fblearner/flow/projects/vision/classy_vision:classy_vision_workflow_lib"
    ],
    "external_deps": []
   },
   "no_uii": true,
   "notebook_number": "179685",
   "others_can_edit": true,
   "reviewers": "",
   "revision_id": "833561470396445",
   "tags": "classy_vision",
   "tasks": "",
   "title": "classy_dataset"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}