tutorials/3-adapter_advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adapters in bigger pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we show how to use adapters to create more complicated pipelines in Steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xgboost\n",
    "import traceback\n",
    "\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from steppy.base import Step, BaseTransformer, make_transformer\n",
    "from steppy.adapter import Adapter, E\n",
    "\n",
    "EXPERIMENT_DIR = './ex3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's recreate the pipeline for digits recognition from notebook #1.\n",
    "\n",
    "We start off by fetching the data. In the latter part of this notebook we will create a model ensembling, hence this time we split the data into three parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "# By default pipelines will try to load previously trained models so we delete the cache to ba sure we're starting from scratch\n",
    "shutil.rmtree(EXPERIMENT_DIR, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()\n",
    "X_digits, y_digits = digits.data, digits.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_digits, y_digits, test_size=0.15, stratify=y_digits, random_state=643793)\n",
    "X_train, X_ens, y_train, y_ens = train_test_split(X_train, y_train, test_size=0.35, stratify=y_train, random_state=976542)\n",
    "\n",
    "print('{} samples for training'.format(len(y_train)))\n",
    "print('{} samples for ensembling'.format(len(y_ens)))\n",
    "print('{} samples for test'.format(len(y_test)))\n",
    "\n",
    "data_train = {\n",
    "    'input': {\n",
    "        'images': X_train,\n",
    "        'labels': y_train,\n",
    "    }\n",
    "}\n",
    "\n",
    "data_ensembling = {\n",
    "    'input': {\n",
    "        'images': X_ens,\n",
    "        'labels': y_ens\n",
    "    }\n",
    "}\n",
    "\n",
    "data_test = {\n",
    "    'input': {\n",
    "        'images': X_test,\n",
    "        'labels': y_test\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -r ./cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define `RandomForestTransformer` in similar manner as before. With one difference, though. `Transform` will use RandomForest's `predict_proba` instead of `predict` which will be useful in the latter part of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomForestTransformer(BaseTransformer):\n",
    "    def __init__(self, random_state=None):\n",
    "        self.estimator = RandomForestClassifier(random_state=random_state, n_estimators=100)\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.estimator.fit(X, y)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        y_proba  = self.estimator.predict_proba(X)\n",
    "        return {'y_proba': y_proba}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.estimator, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step = Step(name='random_forest',\n",
    "               transformer=RandomForestTransformer(),\n",
    "               input_data=['input'],\n",
    "               experiment_directory=EXPERIMENT_DIR,\n",
    "               is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph looks just like in notebook #1. Let's try to execute it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    preds_train_rf = rf_step.fit_transform(data_train)\n",
    "except:\n",
    "    traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, something went wrong. The problem is that `input` dictionary in `data_train` contains fields `images` and `labels`, whereas `RandomForestTransformer` expects arguments `X` and `y`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The solution: adapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle such issues, `Step`'s initializer has `adapter` argument. `Adapter` describes how to reshape the data from the input nodes into the form expected by the transformer or further steps. \n",
    "\n",
    "The basic usage is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step = Step(name='random_forest',\n",
    "               transformer=RandomForestTransformer(),\n",
    "               input_data=['input'],\n",
    "               adapter=Adapter({\n",
    "                   'X': E('input', 'images'),\n",
    "                   'y': E('input', 'labels')\n",
    "               }),\n",
    "               experiment_directory=EXPERIMENT_DIR,\n",
    "               is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We created a new step which gets its data from `input` node.\n",
    "\n",
    "When the program flow gets to `rename_step`, first `adapter`-related code is executed. `RandomForestTransformer`'s `fit_transform` and `transform` methods expect arguments `X` and `y`. The `adapter` is basically a dictionary which for each expected argument tells how to get it. For instance `'X': E('input', 'images')` tells the step that value for `X` is stored under `images` key in the dictionary returned by `input` node.\n",
    "\n",
    "Let's try to fit Random Forest again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step.fit_transform(data_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time it worked like charm - we see class probabilites for the train cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline with model ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very often when we have multiple models which perform on the same level it makes sense to combine them. The created model ensembling tends to be more stable and can even improve results a little.\n",
    "\n",
    "To take advantage of that fact, we will train a couple of forests. Thanks to a different random seeds each forest will make somewhat different predictions, and therefore their combination will improve performance of the entire pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NR_OF_FORESTS = 4\n",
    "random_seeds =  [np.random.randint(1000000) for _ in range(NR_OF_FORESTS)]\n",
    "\n",
    "rf_steps = [Step(name='random_forest_{}'.format(i),\n",
    "                 transformer=RandomForestTransformer(random_state=seed),\n",
    "                 input_data=['input'],\n",
    "                 adapter=Adapter({\n",
    "                     'X': E('input', 'images'),\n",
    "                     'y': E('input', 'labels')\n",
    "                 }),    \n",
    "                 experiment_directory=EXPERIMENT_DIR,\n",
    "                 is_fittable=True)\n",
    "            for i, seed in enumerate(random_seeds)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_steps[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ensembling we will use boosting trees. First we need to create a transformer that will wrap XGBoost. What we need to do is really analogous to what we did for Random Forests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class XGBoostTransformer(BaseTransformer):\n",
    "    def __init__(self, xgb_params, num_boost_round):\n",
    "        self.estimator = None\n",
    "        self.xgb_params = xgb_params\n",
    "        self.num_boost_round = num_boost_round\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        tr_mat = xgboost.DMatrix(X, label=y)\n",
    "        evals = [(tr_mat, 'train')]\n",
    "        self.estimator = xgboost.train(self.xgb_params,\n",
    "                                       tr_mat,\n",
    "                                       num_boost_round=self.num_boost_round,\n",
    "                                       verbose_eval=False,\n",
    "                                       evals=evals)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        test_mat = xgboost.DMatrix(X)\n",
    "        y_proba  = self.estimator.predict(test_mat)\n",
    "        return {'y_proba': y_proba}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump({'estimator': self.estimator,\n",
    "                     'xgb_params': self.xgb_params,\n",
    "                     'num_boost_round': self.num_boost_round},\n",
    "                    filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        d = joblib.load(filepath)\n",
    "        self.estimator = d['estimator']\n",
    "        self.xgb_params = d['xgb_params']\n",
    "        self.num_boost_round = d['num_boost_round']\n",
    "        return self\n",
    "    \n",
    "def get_xgb_params():\n",
    "    return {\n",
    "        'objective': 'multi:softprob',\n",
    "        \"num_class\": 10,\n",
    "        'eta': 0.5,\n",
    "        'max_depth': 4,\n",
    "        'silent': True,\n",
    "        'nthread': -1,\n",
    "        'lambda': 2.0,\n",
    "        'eval_metric': [\"mlogloss\", \"merror\"]\n",
    "    }\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To connect ensembling step with random forests we need to do some more advanced adapting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gather_step = Step(\n",
    "    name='gather_step',\n",
    "    transformer=make_transformer(lambda lst, y: {'X': np.hstack(lst), 'y': y}),\n",
    "    input_steps=rf_steps,\n",
    "    input_data=['input'],\n",
    "    adapter=Adapter({\n",
    "        'lst': [E(rf_step.name, 'y_proba') for rf_step in rf_steps],\n",
    "        'y': E('input', 'labels')\n",
    "    }),\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    is_fittable=False)\n",
    "\n",
    "ensemble_step = Step(name='ensembler',\n",
    "                     transformer=XGBoostTransformer(xgb_params=get_xgb_params(), num_boost_round=10),\n",
    "                     input_steps=[gather_step],\n",
    "                     experiment_directory=EXPERIMENT_DIR,\n",
    "                     is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ensemble_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essentially we split the ensembling part into two steps: `gather_step` collects the outputs from individual classifiers and concatenates them. Then `ensemble_step` fits a classifier to the combined features provided by `gather_step`.\n",
    "\n",
    "Looking at `gather_step` in detail, we used more complex syntax in its adapter this time. The recipe for `y` is still a simple extractor `E('input', 'labels')` but for `lst` the recipe is a list of extractors `[E(rf_step.name, 'y_proba') for rf_step in rf_steps]`. This tells the adapter to build a list of items extracted by following the individual recipes. (Note also that generating this list with a list comprehension means that we can combine a dynamic number of inputs!) The transormer of `gather_step` takes the list built by the adapter and concatenates its items.\n",
    "\n",
    "An adapter is actually a description of how to build arguments for `fit_transform` and `transform`. Let _extractor_ mean a pair of node name and key in the dictionary returned by that node. An adapter is a dictionary, where:\n",
    "- keys must agree with transormer's `fit_transform` and `transform` arguments,\n",
    "- values must be either:\n",
    "  1. an extractor,\n",
    "  2. a list of extractors,\n",
    "  3. a tuple of extractors\n",
    "  4. a dictionary with extractors as values\n",
    "  \n",
    "Step with an adapter proceeds like this:\n",
    "1. It gathers results from preceeding nodes.\n",
    "2. It builds a top-level dictionary with the same keys as the adapter and with values built according to descriptions:\n",
    "   - if the key in the adapter maps to a single extractor, an appropriate object is extracted from the results of input nodes,\n",
    "   - if the key maps to a list of extractors, objects are extracted according to extractors and added to a list,\n",
    "   - if the key maps to a tuple, this works like with lists but a tuple is constructed instead\n",
    "   - if the key maps to a dictionary, an analogous dictionary will be built by applying the extractors and replacing them with the extracted objects\n",
    "3. Arguments of `fit_transform` and `transform` are filled using the adapter's top-level dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if our ensembling works. To properly fit the pipeline we have to fit random forests first using the train data, and then fit the ensembler using part of the data for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for rf_step in rf_steps:\n",
    "    rf_step.fit_transform(data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ensemble_step.fit_transform(data_ensembling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks fine! However, often we are interested only in the class with the highest probability. Let's make a step that will find this class for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GuessesTransformer(BaseTransformer):\n",
    "    def transform(self, y_proba):\n",
    "        return {'y_pred': np.argmax(y_proba, axis=1)}\n",
    "\n",
    "guesses_step = Step(name='guesses_maker',\n",
    "                 transformer=GuessesTransformer(),\n",
    "                 input_steps=[ensemble_step],\n",
    "                 experiment_directory=EXPERIMENT_DIR,\n",
    "                 is_fittable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guesses_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be already familiar with everything that happened here. New step, `guesses_maker`, takes its input from `ensembler`. Adapter will create just one element: `y_pred`. List of bricks used to build `y_pred` has only one element:  `y_proba` found in `ensembler`'s result. Function `lambda lst: np.argmax(lst[0], axis=1)` takes this list and performs row-wise `argmax` on its only element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "guesses_step.fit_transform(data_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have created a quite complicated pipeline, so for sure everyone is anxious to see how it performs. Our final step will carry out the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EvaluationTransformer(BaseTransformer):\n",
    "    def transform(self, y_true, y_proba, y_pred):\n",
    "        return {'Log-loss': log_loss(y_pred=y_proba, y_true=y_true),\n",
    "                'Acc:': '{:.2f}'.format(sum(y_true == y_pred) / len(y_pred))\n",
    "               }\n",
    "\n",
    "evaluation_step = Step(name='evaluator',\n",
    "                 transformer=EvaluationTransformer(),\n",
    "                 input_steps=[ensemble_step, guesses_step],\n",
    "                 input_data=['input'],\n",
    "                 adapter=Adapter({\n",
    "                     'y_proba': E(ensemble_step.name, 'y_proba'),\n",
    "                     'y_pred':  E(guesses_step.name, 'y_pred'),\n",
    "                     'y_true': E('input', 'labels')\n",
    "                 }),\n",
    "                 experiment_directory=EXPERIMENT_DIR,\n",
    "                 is_fittable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_step.fit_transform(data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_step.transform(data_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see thanks to ensembling we improved in comparison to a single model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Peek on pipeline predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing images with model's predictions is always a very rewarding feeling. As a last example we show a step that displays a few images with the predicted probability distributions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_names = [rf_step.name for rf_step in rf_steps] + [ensemble_step.name]\n",
    "class LookAtPredictions(BaseTransformer):\n",
    "    def transform(self, probas, images): \n",
    "        pd.options.display.float_format = '{:5.2f}'.format\n",
    "        for img_nr in range(5):\n",
    "            df = pd.DataFrame({model_names[j]: probas[j][img_nr]\n",
    "                               for j in range(len(model_names))\n",
    "                              },\n",
    "                              index=list(range(10)))\n",
    "            df = df[model_names]\n",
    "            plt.figure(figsize=(6,2))\n",
    "            left =  plt.subplot(1, 2, 1)\n",
    "            right = plt.subplot(1, 2, 2)\n",
    "            left.imshow(images[img_nr].reshape(8, 8), cmap='gray')\n",
    "            right.axis('off')\n",
    "            right.text(0, 0.3, str(df.T), fontsize=14, fontname='monospace')\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_step = Step(\n",
    "    name='display',\n",
    "    transformer=LookAtPredictions(),\n",
    "    input_steps=[ensemble_step] + rf_steps,\n",
    "    input_data=['input'],\n",
    "    adapter=Adapter({\n",
    "        'probas': [E(rf_step.name, 'y_proba') for rf_step in rf_steps] +\n",
    "            [E(ensemble_step.name, 'y_proba')],\n",
    "        'images': E('input', 'images')\n",
    "    }),\n",
    "    experiment_directory=EXPERIMENT_DIR,\n",
    "    is_fittable=False\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display_step.fit_transform(data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "display_step.transform(data_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}