tutorials/usage/tutorial09_automl.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Guide Tutorial 09: AutoML\n",
    "\n",
    "TemporAI provides AutoML tools for finding the best model for your use case in `tempor.automl`, these are demonstrated here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Skip the below cell if you are not on Google Colab / already have TemporAI installed:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install temporai\n",
    "\n",
    "# Or from the repo, for the latest version:\n",
    "# %pip install git+https://github.com/vanderschaarlab/temporai.git"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AutoML in TemporAI Overview\n",
    "\n",
    "TemporAI provides two AutoML approaches (\"seekers\") under the `tempor.automl.seekers` module.\n",
    "\n",
    "1. `MethodSeeker`: Search the hyperparameter space of a particular predictive method.\n",
    "2. `PipelineSeeker`: Search the hyperparameter space of a pipeline like `preprocessing steps -> predictive step`.\n",
    "\n",
    "The optimization strategies are facilitated by [`optuna`](https://optuna.readthedocs.io/) and the currently supported strategies are:\n",
    "* [Bayesian, specifically Tree-structured Parzen estimator](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html) (`\"bayesian\"`),\n",
    "* [Random](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.RandomSampler.html) (`\"random\"`),\n",
    "* [CMA-ES](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.CmaEsSampler.html) (`\"cmaes\"`),\n",
    "* [QMC](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.QMCSampler.html) (`\"qmc\"`),\n",
    "* [Grid](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.GridSampler.html) (`\"grid\"`)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `MethodSeeker`\n",
    "\n",
    "Use a `MethodSeeker` to search for best algorithm and hyperparameters parameters for a particular task.\n",
    "No preprocessing (data transformation) steps are carried out in this approach, so preprocess the data using\n",
    "`tempor.methods.preprocessing` first, as needed.\n",
    "\n",
    "A `MethodSeeker` can be initialized as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_set_up_tuners:341 | Setting up estimators and tuners for study my_automl_study.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator cde_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator ode_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator nn_classifier.\n"
     ]
    }
   ],
   "source": [
    "from tempor import plugin_loader\n",
    "from tempor.automl.seeker import MethodSeeker\n",
    "\n",
    "# Load your dataset.\n",
    "dataset = plugin_loader.get(\"prediction.one_off.sine\", plugin_type=\"datasource\").load()\n",
    "\n",
    "seeker = MethodSeeker(\n",
    "    # Name your AutoML study:\n",
    "    study_name=\"my_automl_study\",\n",
    "    # Select the type of task:\n",
    "    task_type=\"prediction.one_off.classification\",\n",
    "    # Choose which predictive methods to use in the search:\n",
    "    estimator_names=[\n",
    "        \"cde_classifier\",\n",
    "        \"ode_classifier\",\n",
    "        \"nn_classifier\",\n",
    "    ],\n",
    "    # Choose a metric. Metric maximization/minimization will be determined automatically.\n",
    "    metric=\"aucroc\",\n",
    "    # Pass in your dataset.\n",
    "    dataset=dataset,\n",
    "    # How many best models to return:\n",
    "    return_top_k=3,\n",
    "    # Number of AutoML iterations:\n",
    "    num_iter=100,\n",
    "    # Type of AutoML tuner to use:\n",
    "    tuner_type=\"bayesian\",\n",
    "    # You can also provide some other options like early stopping patience, number of cross-validation folds etc.\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then run the AutoML search as below.\n",
    "\n",
    "The below example also shows how you can provide a custom hyperparameter space (*override* the default hyperparameter\n",
    "space for a model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_set_up_tuners:341 | Setting up estimators and tuners for study my_automl_study.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator cde_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator ode_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator nn_classifier.\n"
     ]
    }
   ],
   "source": [
    "from tempor.methods.core.params import IntegerParams, CategoricalParams\n",
    "\n",
    "# Provide a custom hyperparameter space to search for each type of model.\n",
    "# NOTE: For the sake of speed of this example, we limit epochs to 2.\n",
    "hp_space = {\n",
    "    \"cde_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_temporal_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "    \"ode_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "    \"nn_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "}\n",
    "\n",
    "# Initialize a `MethodSeeker` and provide `override_hp_space`.\n",
    "seeker = MethodSeeker(\n",
    "    study_name=\"my_automl_study\",\n",
    "    task_type=\"prediction.one_off.classification\",\n",
    "    estimator_names=[\n",
    "        \"cde_classifier\",\n",
    "        \"ode_classifier\",\n",
    "        \"nn_classifier\",\n",
    "    ],\n",
    "    metric=\"aucroc\",\n",
    "    dataset=dataset,\n",
    "    return_top_k=3,\n",
    "    num_iter=3,  # For the sake of speed of this example, only 3 AutoML iterations.\n",
    "    tuner_type=\"bayesian\",\n",
    "    # Override hyperparameter space:\n",
    "    override_hp_space=hp_space,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator 'cde_classifier' 1/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:41 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from CDEClassifier:\n",
      "{'n_iter': 2, 'n_temporal_units_hidden': 13, 'lr': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:48 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from CDEClassifier:\n",
      "{'n_iter': 2, 'n_temporal_units_hidden': 11, 'lr': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:53 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from CDEClassifier:\n",
      "{'n_iter': 2, 'n_temporal_units_hidden': 20, 'lr': 0.001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:58 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator 'ode_classifier' 2/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:58 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:58 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from ODEClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 13, 'lr': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:01 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from ODEClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:05 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from ODEClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 20, 'lr': 0.001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:08 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator 'nn_classifier' 3/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:08 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:08 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from NeuralNetClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 13, 'lr': 0.01}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:09 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919505596160889, validation loss: 0.6882655620574951\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:09 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.691156268119812, validation loss: 0.6865323185920715\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:10 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.692247211933136, validation loss: 0.6861070990562439\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:10 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6928325891494751, validation loss: 0.6825112104415894\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:11 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919053792953491, validation loss: 0.6819757223129272\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:11 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from NeuralNetClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:12 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919505596160889, validation loss: 0.691270649433136\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:12 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.691156268119812, validation loss: 0.6910805106163025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:13 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.692247211933136, validation loss: 0.6900344491004944\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:13 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6928325891494751, validation loss: 0.6906693577766418\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:14 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919053792953491, validation loss: 0.6910874843597412\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:14 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from NeuralNetClassifier:\n",
      "{'n_iter': 2, 'n_units_hidden': 20, 'lr': 0.001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:14 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919505596160889, validation loss: 0.6896671056747437\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:15 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.691156268119812, validation loss: 0.6886401772499084\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:15 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.692247211933136, validation loss: 0.6884845495223999\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6928325891494751, validation loss: 0.6887120604515076\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6919053792953491, validation loss: 0.6893669962882996\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for cde_classifier scores:\n",
      "[0.46300505050505053, 0.48396464646464643, 0.4901515151515151].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for ode_classifier scores:\n",
      "[0.47992424242424236, 0.48409090909090907, 0.484090909090909].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for nn_classifier scores:\n",
      "[0.37064393939393936, 0.5370580808080809, 0.4579545454545454].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:436 | \n",
      "All estimator definitions searched:\n",
      "['cde_classifier', 'ode_classifier', 'nn_classifier']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:437 | \n",
      "Best scores for each estimator searched:\n",
      "[0.4901515151515151, 0.48409090909090907, 0.5370580808080809]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:search:438 | \n",
      "Best hyperparameters for each estimator searched:\n",
      "[{'n_iter': 2, 'n_temporal_units_hidden': 20, 'lr': 0.001}, {'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}, {'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:565 | \n",
      "Selected score 0.5370580808080809 for nn_classifier with hyperparameters:\n",
      "{'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:565 | \n",
      "Selected score 0.4901515151515151 for cde_classifier with hyperparameters:\n",
      "{'n_iter': 2, 'n_temporal_units_hidden': 20, 'lr': 0.001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:565 | \n",
      "Selected score 0.48409090909090907 for ode_classifier with hyperparameters:\n",
      "{'n_iter': 2, 'n_units_hidden': 11, 'lr': 0.0001}\n"
     ]
    }
   ],
   "source": [
    "# Execute the search.\n",
    "\n",
    "best_methods, best_scores = seeker.search()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">NeuralNetClassifier</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'nn_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">plugin_type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'method'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_static_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_static_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_temporal_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_temporal_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'mode'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'RNN'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'window_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mNeuralNetClassifier\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mname\u001b[0m=\u001b[32m'nn_classifier'\u001b[0m,\n",
       "    \u001b[33mcategory\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mplugin_type\u001b[0m=\u001b[32m'method'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'n_static_units_hidden'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "        \u001b[32m'n_static_layers_hidden'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "        \u001b[32m'n_temporal_units_hidden'\u001b[0m: \u001b[1;36m102\u001b[0m,\n",
       "        \u001b[32m'n_temporal_layers_hidden'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "        \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "        \u001b[32m'mode'\u001b[0m: \u001b[32m'RNN'\u001b[0m,\n",
       "        \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "        \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "        \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.0001\u001b[0m,\n",
       "        \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "        \u001b[32m'window_size'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "        \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "        \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "        \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'patience'\u001b[0m: \u001b[1;36m20\u001b[0m,\n",
       "        \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CDEClassifier</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cde_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">plugin_type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'method'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'atol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'rtol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'interpolation'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'cubic'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mCDEClassifier\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mname\u001b[0m=\u001b[32m'cde_classifier'\u001b[0m,\n",
       "    \u001b[33mcategory\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mplugin_type\u001b[0m=\u001b[32m'method'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'n_units_hidden'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "        \u001b[32m'n_layers_hidden'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "        \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'atol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "        \u001b[32m'rtol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "        \u001b[32m'interpolation'\u001b[0m: \u001b[32m'cubic'\u001b[0m,\n",
       "        \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "        \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "        \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "        \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m500\u001b[0m,\n",
       "        \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "        \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'patience'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "        \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n",
       "        \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "        \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ODEClassifier</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">name</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'ode_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">plugin_type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'method'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'atol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'rtol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'interpolation'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'cubic'</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mODEClassifier\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mname\u001b[0m=\u001b[32m'ode_classifier'\u001b[0m,\n",
       "    \u001b[33mcategory\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mplugin_type\u001b[0m=\u001b[32m'method'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'n_units_hidden'\u001b[0m: \u001b[1;36m11\u001b[0m,\n",
       "        \u001b[32m'n_layers_hidden'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "        \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'atol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "        \u001b[32m'rtol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "        \u001b[32m'interpolation'\u001b[0m: \u001b[32m'cubic'\u001b[0m,\n",
       "        \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.0001\u001b[0m,\n",
       "        \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "        \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "        \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m500\u001b[0m,\n",
       "        \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "        \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "        \u001b[32m'patience'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "        \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n",
       "        \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "        \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The best methods are returned, and can be used by calling .predict() and so on.\n",
    "\n",
    "import rich.pretty  # For pretty printing only.\n",
    "\n",
    "for method in best_methods:\n",
    "    rich.pretty.pprint(method, indent_guides=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `PipelineSeeker`\n",
    "\n",
    "Use a `PipelineSeeker` to search for best *pipeline* (`preprocessing steps -> prediction step`) for a particular task.\n",
    "\n",
    "This seeker will create pipelines comprised of:\n",
    "- A static imputer (if at lease one candidate in ``static_imputers`` provided),\n",
    "- A static scaler (if at lease one candidate in ``static_scalers`` provided),\n",
    "- A temporal imputer (if at lease one candidate in ``temporal_imputers`` provided),\n",
    "- A temporal scaler (if at lease one candidate in ``temporal_scalers`` provided),\n",
    "- The final predictor, from the ``estimator_names`` options.\n",
    "\n",
    "The imputer/scaler candidates will be sampled as a categorical hyperparameter. The hyperparameter spaces of these,\n",
    "and of the final predictor, will be sampled.\n",
    "\n",
    "A `PipelineSeeker` uses a very similar interface to `MethodSeeker`, and can be initialized as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_set_up_tuners:341 | Setting up estimators and tuners for study my_automl_study.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with cde_classifier>.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with ode_classifier>.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:16 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with nn_classifier>.\n"
     ]
    }
   ],
   "source": [
    "from tempor.automl.seeker import PipelineSeeker\n",
    "\n",
    "seeker = PipelineSeeker(\n",
    "    study_name=\"my_automl_study\",\n",
    "    task_type=\"prediction.one_off.classification\",\n",
    "    # The estimators here will be the final step of the pipeline:\n",
    "    estimator_names=[\n",
    "        \"cde_classifier\",\n",
    "        \"ode_classifier\",\n",
    "        \"nn_classifier\",\n",
    "    ],\n",
    "    metric=\"aucroc\",\n",
    "    dataset=dataset,\n",
    "    return_top_k=3,\n",
    "    num_iter=100,\n",
    "    tuner_type=\"bayesian\",\n",
    "    # The following arguments specify the candidates of the different preprocessing steps, e.g.:\n",
    "    static_imputers=[\"static_tabular_imputer\"],\n",
    "    static_scalers=[],\n",
    "    temporal_imputers=[\"ffill\", \"bfill\"],\n",
    "    temporal_scalers=[\"ts_minmax_scaler\"],\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the following preprocessing candidates will be used, if you do not specify the argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Static imputer candidates: ['static_tabular_imputer']\n",
      "Static scaler candidates: ['static_minmax_scaler', 'static_standard_scaler']\n",
      "Temporal imputer candidates: ['ffill', 'ts_tabular_imputer', 'bfill']\n",
      "Temporal scaler candidates: ['ts_minmax_scaler', 'ts_standard_scaler']\n"
     ]
    }
   ],
   "source": [
    "from tempor.automl.seeker import (\n",
    "    DEFAULT_STATIC_IMPUTERS,\n",
    "    DEFAULT_STATIC_SCALERS,\n",
    "    DEFAULT_TEMPORAL_IMPUTERS,\n",
    "    DEFAULT_TEMPORAL_SCALERS,\n",
    ")\n",
    "\n",
    "print(\"Static imputer candidates:\", DEFAULT_STATIC_IMPUTERS)\n",
    "print(\"Static scaler candidates:\", DEFAULT_STATIC_SCALERS)\n",
    "print(\"Temporal imputer candidates:\", DEFAULT_TEMPORAL_IMPUTERS)\n",
    "print(\"Temporal scaler candidates:\", DEFAULT_TEMPORAL_SCALERS)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can execute the search as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.seeker:_set_up_tuners:341 | Setting up estimators and tuners for study my_automl_study.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with cde_classifier>.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with ode_classifier>.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.seeker:_init_estimator:733 | Creating estimator <Pipeline with nn_classifier>.\n"
     ]
    }
   ],
   "source": [
    "from tempor.methods.core.params import IntegerParams, CategoricalParams\n",
    "\n",
    "# Provide a custom hyperparameter space to search for each type of model.\n",
    "# These can be provided for the final (predictive) step of the pipeline.\n",
    "# Default hyperparameter space will be sampled for the preprocessing steps.\n",
    "# NOTE: For the sake of speed of this example, we limit epochs to 2.\n",
    "hp_space = {\n",
    "    \"cde_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_temporal_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "    \"ode_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "    \"nn_classifier\": [\n",
    "        IntegerParams(name=\"n_iter\", low=2, high=2),\n",
    "        IntegerParams(name=\"n_units_hidden\", low=5, high=20),\n",
    "        CategoricalParams(name=\"lr\", choices=[1e-2, 1e-3, 1e-4]),\n",
    "    ],\n",
    "}\n",
    "\n",
    "# Initialize a `PipelineSeeker` and provide `override_hp_space`.\n",
    "seeker = PipelineSeeker(\n",
    "    study_name=\"my_automl_study\",\n",
    "    task_type=\"prediction.one_off.classification\",\n",
    "    estimator_names=[\n",
    "        \"cde_classifier\",\n",
    "        \"ode_classifier\",\n",
    "        \"nn_classifier\",\n",
    "    ],\n",
    "    metric=\"aucroc\",\n",
    "    dataset=dataset,\n",
    "    return_top_k=3,\n",
    "    num_iter=3,  # For the sake of speed of this example, only 3 AutoML iterations.\n",
    "    tuner_type=\"bayesian\",\n",
    "    # Override hyperparameter space:\n",
    "    override_hp_space=hp_space,\n",
    "    # Specify preprocessing candidates:\n",
    "    static_imputers=[\"static_tabular_imputer\"],\n",
    "    static_scalers=[\"static_minmax_scaler\", \"static_standard_scaler\"],\n",
    "    temporal_imputers=[],\n",
    "    temporal_scalers=[\"ts_minmax_scaler\", \"ts_standard_scaler\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator '<Pipeline with cde_classifier>' 1/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:17 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.cde_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'softimpute'}, 'static_minmax_scaler': {'clip': True}, 'ts_standard_scaler': {}, 'cde_classifier': {'n_iter': 2, 'n_temporal_units_hidden': 17, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:32 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_minmax_scaler->prediction.one_off.classification.cde_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'mean'}, 'static_minmax_scaler': {'clip': False}, 'ts_minmax_scaler': {'clip': False}, 'cde_classifier': {'n_iter': 2, 'n_temporal_units_hidden': 14, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:38 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_standard_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.cde_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'most_frequent'}, 'static_standard_scaler': {}, 'ts_standard_scaler': {}, 'cde_classifier': {'n_iter': 2, 'n_temporal_units_hidden': 6, 'lr': 0.0001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:45 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator '<Pipeline with ode_classifier>' 2/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:45 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:19:45 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.ode_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'softimpute'}, 'static_minmax_scaler': {'clip': True}, 'ts_standard_scaler': {}, 'ode_classifier': {'n_iter': 2, 'n_units_hidden': 17, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:00 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_minmax_scaler->prediction.one_off.classification.ode_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'mean'}, 'static_minmax_scaler': {'clip': False}, 'ts_minmax_scaler': {'clip': False}, 'ode_classifier': {'n_iter': 2, 'n_units_hidden': 14, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:06 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_standard_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.ode_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'most_frequent'}, 'static_standard_scaler': {}, 'ts_standard_scaler': {}, 'ode_classifier': {'n_iter': 2, 'n_units_hidden': 6, 'lr': 0.0001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:12 | INFO     | tempor.automl.seeker:search:400 | Running  search for estimator '<Pipeline with nn_classifier>' 3/3.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:12 | INFO     | tempor.automl.tuner:tune:228 | Baseline score computation skipped\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:12 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.nn_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'softimpute'}, 'static_minmax_scaler': {'clip': True}, 'ts_standard_scaler': {}, 'nn_classifier': {'n_iter': 2, 'n_units_hidden': 17, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:14 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6918608546257019, validation loss: 0.6910383105278015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:17 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6909429430961609, validation loss: 0.6872987151145935\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:20 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.692150354385376, validation loss: 0.6879752278327942\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:23 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6926363706588745, validation loss: 0.6881052851676941\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:26 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6915659308433533, validation loss: 0.6888220310211182\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:27 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_minmax_scaler->prediction.one_off.classification.nn_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'mean'}, 'static_minmax_scaler': {'clip': False}, 'ts_minmax_scaler': {'clip': False}, 'nn_classifier': {'n_iter': 2, 'n_units_hidden': 14, 'lr': 0.001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:27 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6920319199562073, validation loss: 0.6892786622047424\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:28 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6913070678710938, validation loss: 0.6893746852874756\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:30 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6924894452095032, validation loss: 0.6886722445487976\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:31 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6930248141288757, validation loss: 0.6894596815109253\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:32 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6921922564506531, validation loss: 0.6893032789230347\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:32 | INFO     | tempor.automl.tuner:objective:248 | \n",
      "Hyperparameters sampled from preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_standard_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.nn_classifier:\n",
      "{'plugin_params': {'static_tabular_imputer': {'imputer': 'most_frequent'}, 'static_standard_scaler': {}, 'ts_standard_scaler': {}, 'nn_classifier': {'n_iter': 2, 'n_units_hidden': 6, 'lr': 0.0001}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:33 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6914154291152954, validation loss: 0.6911771297454834\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:34 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.690178394317627, validation loss: 0.6934390664100647\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:35 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6933540105819702, validation loss: 0.6896734833717346\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:36 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.694333553314209, validation loss: 0.690070390701294\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.models.ts_model:_train:388 | Epoch:0| train loss: 0.6921719312667847, validation loss: 0.6909489631652832\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for <Pipeline with cde_classifier> scores:\n",
      "[0.4736742424242424, 0.4737373737373737, 0.4657828282828283].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for <Pipeline with ode_classifier> scores:\n",
      "[0.47575757575757577, 0.4779040404040404, 0.4779671717171716].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:435 | \n",
      "Evaluation for <Pipeline with nn_classifier> scores:\n",
      "[0.48478535353535346, 0.4361742424242424, 0.5071338383838383].\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:436 | \n",
      "All estimator definitions searched:\n",
      "['<Pipeline with cde_classifier>', '<Pipeline with ode_classifier>', '<Pipeline with nn_classifier>']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:437 | \n",
      "Best scores for each estimator searched:\n",
      "[0.4737373737373737, 0.4779671717171716, 0.5071338383838383]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:search:438 | \n",
      "Best hyperparameters for each estimator searched:\n",
      "[{'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'mean', '<candidates>(preprocessing.scaling.static)': 'static_minmax_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_minmax_scaler', '[ts_minmax_scaler](clip)': False, '[cde_classifier](n_iter)': 2, '[cde_classifier](n_temporal_units_hidden)': 14, '[cde_classifier](lr)': 0.001}, {'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'most_frequent', '<candidates>(preprocessing.scaling.static)': 'static_standard_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_standard_scaler', '[ts_minmax_scaler](clip)': False, '[ode_classifier](n_iter)': 2, '[ode_classifier](n_units_hidden)': 6, '[ode_classifier](lr)': 0.0001}, {'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'most_frequent', '<candidates>(preprocessing.scaling.static)': 'static_standard_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_standard_scaler', '[ts_minmax_scaler](clip)': False, '[nn_classifier](n_iter)': 2, '[nn_classifier](n_units_hidden)': 6, '[nn_classifier](lr)': 0.0001}]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:748 | \n",
      "Selected score 0.5071338383838383 for <Pipeline with nn_classifier> with hyperparameters:\n",
      "{'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'most_frequent', '<candidates>(preprocessing.scaling.static)': 'static_standard_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_standard_scaler', '[ts_minmax_scaler](clip)': False, '[nn_classifier](n_iter)': 2, '[nn_classifier](n_units_hidden)': 6, '[nn_classifier](lr)': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:748 | \n",
      "Selected score 0.4779671717171716 for <Pipeline with ode_classifier> with hyperparameters:\n",
      "{'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'most_frequent', '<candidates>(preprocessing.scaling.static)': 'static_standard_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_standard_scaler', '[ts_minmax_scaler](clip)': False, '[ode_classifier](n_iter)': 2, '[ode_classifier](n_units_hidden)': 6, '[ode_classifier](lr)': 0.0001}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_create_estimator_with_hps:748 | \n",
      "Selected score 0.4737373737373737 for <Pipeline with cde_classifier> with hyperparameters:\n",
      "{'<candidates>(preprocessing.imputation.static)': 'static_tabular_imputer', '[static_tabular_imputer](imputer)': 'mean', '<candidates>(preprocessing.scaling.static)': 'static_minmax_scaler', '[static_minmax_scaler](clip)': False, '<candidates>(preprocessing.scaling.temporal)': 'ts_minmax_scaler', '[ts_minmax_scaler](clip)': False, '[cde_classifier](n_iter)': 2, '[cde_classifier](n_temporal_units_hidden)': 14, '[cde_classifier](lr)': 0.001}\n"
     ]
    }
   ],
   "source": [
    "best_pipelines, best_scores = seeker.search()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">pipeline_seq</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.imputation.static.static_tabular_imputer-&gt;preprocessing.scaling.static.static_standard_scaler-&gt;preprocessing.scaling.temporal.ts_standard_scaler-&gt;prediction.one_off.classification.nn_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">predictor_category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_tabular_imputer'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'imputer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'most_frequent'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'imputer_params'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>\n",
       "        <span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_standard_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'with_mean'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'with_std'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'ts_standard_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'with_mean'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'with_std'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'nn_classifier'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_static_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_static_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_temporal_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_temporal_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'mode'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'RNN'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'window_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mPipeline\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mpipeline_seq\u001b[0m=\u001b[32m'preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_standard_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.nn_classifier'\u001b[0m,\n",
       "    \u001b[33mpredictor_category\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'static_tabular_imputer'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'imputer'\u001b[0m: \u001b[32m'most_frequent'\u001b[0m,\n",
       "            \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'imputer_params'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m\n",
       "        \u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'static_standard_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'with_mean'\u001b[0m: \u001b[3;92mTrue\u001b[0m, \u001b[32m'with_std'\u001b[0m: \u001b[3;92mTrue\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'ts_standard_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'with_mean'\u001b[0m: \u001b[3;92mTrue\u001b[0m, \u001b[32m'with_std'\u001b[0m: \u001b[3;92mTrue\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'nn_classifier'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'n_static_units_hidden'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "            \u001b[32m'n_static_layers_hidden'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "            \u001b[32m'n_temporal_units_hidden'\u001b[0m: \u001b[1;36m102\u001b[0m,\n",
       "            \u001b[32m'n_temporal_layers_hidden'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "            \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "            \u001b[32m'mode'\u001b[0m: \u001b[32m'RNN'\u001b[0m,\n",
       "            \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "            \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "            \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.0001\u001b[0m,\n",
       "            \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "            \u001b[32m'window_size'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "            \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "            \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "            \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'patience'\u001b[0m: \u001b[1;36m20\u001b[0m,\n",
       "            \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">pipeline_seq</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.imputation.static.static_tabular_imputer-&gt;preprocessing.scaling.static.static_standard_scaler-&gt;preprocessing.scaling.temporal.ts_standard_scaler-&gt;prediction.one_off.classification.ode_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">predictor_category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_tabular_imputer'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'imputer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'most_frequent'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'imputer_params'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>\n",
       "        <span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_standard_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'with_mean'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'with_std'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'ts_standard_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'with_mean'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'with_std'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'ode_classifier'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'atol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'rtol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'interpolation'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'cubic'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mPipeline\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mpipeline_seq\u001b[0m=\u001b[32m'preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_standard_scaler->preprocessing.scaling.temporal.ts_standard_scaler->prediction.one_off.classification.ode_classifier'\u001b[0m,\n",
       "    \u001b[33mpredictor_category\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'static_tabular_imputer'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'imputer'\u001b[0m: \u001b[32m'most_frequent'\u001b[0m,\n",
       "            \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'imputer_params'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m\n",
       "        \u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'static_standard_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'with_mean'\u001b[0m: \u001b[3;92mTrue\u001b[0m, \u001b[32m'with_std'\u001b[0m: \u001b[3;92mTrue\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'ts_standard_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'with_mean'\u001b[0m: \u001b[3;92mTrue\u001b[0m, \u001b[32m'with_std'\u001b[0m: \u001b[3;92mTrue\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'ode_classifier'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'n_units_hidden'\u001b[0m: \u001b[1;36m6\u001b[0m,\n",
       "            \u001b[32m'n_layers_hidden'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "            \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'atol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "            \u001b[32m'rtol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "            \u001b[32m'interpolation'\u001b[0m: \u001b[32m'cubic'\u001b[0m,\n",
       "            \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.0001\u001b[0m,\n",
       "            \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "            \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "            \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m500\u001b[0m,\n",
       "            \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "            \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'patience'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "            \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n",
       "            \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "            \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">pipeline_seq</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.imputation.static.static_tabular_imputer-&gt;preprocessing.scaling.static.static_minmax_scaler-&gt;preprocessing.scaling.temporal.ts_minmax_scaler-&gt;prediction.one_off.classification.cde_classifier'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">predictor_category</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #808000; text-decoration-color: #808000\">params</span>=<span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_tabular_imputer'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'imputer'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'mean'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'imputer_params'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'static_minmax_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'feature_range'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'clip'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'ts_minmax_scaler'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'feature_range'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'clip'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'cde_classifier'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_units_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_layers_hidden'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'nonlin'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'relu'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dropout'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'atol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'rtol'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.01</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'interpolation'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'cubic'</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'lr'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'weight_decay'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.001</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'batch_size'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'n_iter_print'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'random_state'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'patience'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'clipping_value'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'train_ratio'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.8</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'device'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'dataloader_sampler'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mPipeline\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[33mpipeline_seq\u001b[0m=\u001b[32m'preprocessing.imputation.static.static_tabular_imputer->preprocessing.scaling.static.static_minmax_scaler->preprocessing.scaling.temporal.ts_minmax_scaler->prediction.one_off.classification.cde_classifier'\u001b[0m,\n",
       "    \u001b[33mpredictor_category\u001b[0m=\u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[33mparams\u001b[0m=\u001b[1m{\u001b[0m\n",
       "        \u001b[32m'static_tabular_imputer'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'imputer'\u001b[0m: \u001b[32m'mean'\u001b[0m, \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m'imputer_params'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'static_minmax_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'feature_range'\u001b[0m: \u001b[1m[\u001b[0m\u001b[1;36m0\u001b[0m, \u001b[1;36m1\u001b[0m\u001b[1m]\u001b[0m, \u001b[32m'clip'\u001b[0m: \u001b[3;91mFalse\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'ts_minmax_scaler'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'feature_range'\u001b[0m: \u001b[1m[\u001b[0m\u001b[1;36m0\u001b[0m, \u001b[1;36m1\u001b[0m\u001b[1m]\u001b[0m, \u001b[32m'clip'\u001b[0m: \u001b[3;91mFalse\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[32m'cde_classifier'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'n_units_hidden'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "            \u001b[32m'n_layers_hidden'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'nonlin'\u001b[0m: \u001b[32m'relu'\u001b[0m,\n",
       "            \u001b[32m'dropout'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'atol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "            \u001b[32m'rtol'\u001b[0m: \u001b[1;36m0.01\u001b[0m,\n",
       "            \u001b[32m'interpolation'\u001b[0m: \u001b[32m'cubic'\u001b[0m,\n",
       "            \u001b[32m'lr'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "            \u001b[32m'weight_decay'\u001b[0m: \u001b[1;36m0.001\u001b[0m,\n",
       "            \u001b[32m'n_iter'\u001b[0m: \u001b[1;36m2\u001b[0m,\n",
       "            \u001b[32m'batch_size'\u001b[0m: \u001b[1;36m500\u001b[0m,\n",
       "            \u001b[32m'n_iter_print'\u001b[0m: \u001b[1;36m100\u001b[0m,\n",
       "            \u001b[32m'random_state'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'patience'\u001b[0m: \u001b[1;36m10\u001b[0m,\n",
       "            \u001b[32m'clipping_value'\u001b[0m: \u001b[1;36m1\u001b[0m,\n",
       "            \u001b[32m'train_ratio'\u001b[0m: \u001b[1;36m0.8\u001b[0m,\n",
       "            \u001b[32m'device'\u001b[0m: \u001b[3;35mNone\u001b[0m,\n",
       "            \u001b[32m'dataloader_sampler'\u001b[0m: \u001b[3;35mNone\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The best performing pipelines are returned, and can be used by calling .predict() and so on.\n",
    "\n",
    "for method in best_pipelines:\n",
    "    rich.pretty.pprint(method, indent_guides=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced customization\n",
    "\n",
    "You may further customize the AutoML tuning behavior by specifying the sampler an pruner, if desired.\n",
    "\n",
    "See the below example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_set_up_tuners:341 | Setting up estimators and tuners for study my_automl_study.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator cde_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator ode_classifier.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:20:37 | INFO     | tempor.automl.seeker:_init_estimator:557 | Creating estimator nn_classifier.\n"
     ]
    }
   ],
   "source": [
    "# 1. Import a Tuner:\n",
    "from tempor.automl.tuner import OptunaTuner\n",
    "\n",
    "# 2. Customize this as needed:\n",
    "import optuna\n",
    "\n",
    "custom_tuner = OptunaTuner(\n",
    "    study_name=\"my_automl_study\",\n",
    "    direction=\"maximize\",\n",
    "    # Customized sampler:\n",
    "    study_sampler=optuna.samplers.TPESampler(seed=12345, n_startup_trials=3),\n",
    "    # Customized pruner:\n",
    "    study_pruner=optuna.pruners.MedianPruner(interval_steps=2),\n",
    "    # Using default optuna storage object here, but may a provide custom one, e.g. redis.\n",
    "    study_storage=None,\n",
    ")\n",
    "\n",
    "# 3. Pass the Tuner to the {Method/Pipeline}Seeker:\n",
    "seeker = MethodSeeker(\n",
    "    study_name=\"my_automl_study\",\n",
    "    task_type=\"prediction.one_off.classification\",\n",
    "    estimator_names=[\n",
    "        \"cde_classifier\",\n",
    "        \"ode_classifier\",\n",
    "        \"nn_classifier\",\n",
    "    ],\n",
    "    metric=\"aucroc\",\n",
    "    dataset=dataset,\n",
    "    # Like so:\n",
    "    custom_tuner=custom_tuner,\n",
    ")\n",
    "\n",
    "# 4. Execute search:\n",
    "# results = seeker.search() ..."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supported tasks\n",
    "\n",
    "> ⚠️ The tasks for which benchmarking is supported are supported by AutoML. See the benchmarking tutorial.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🎉 Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards *Machine learning and AI for Medicine*, you can do so in the following ways!\n",
    "\n",
    "\n",
    "\n",
    "### ⭐ Star [TemporAI](https://github.com/vanderschaarlab/temporai) on GitHub\n",
    "\n",
    "- The easiest way to help our community is by just starring the repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "\n",
    "\n",
    "### Check out other projects from [vanderschaarlab](https://github.com/vanderschaarlab)\n",
    "- 📝 [HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n",
    "- 📊 [AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- 🤖 [SynthCity](https://github.com/vanderschaarlab/synthcity)\n",
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "RE_py38_temporai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}