1-1_Pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 1.1: ML Pipelines with ZenML\n",
    "\n",
    "[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/zenml-io/zenbytes/blob/main/1-1_Pipelines.ipynb)\n",
    "\n",
    "***Key Concepts:*** *ML Pipelines, Steps*\n",
    "\n",
    "In this notebook, we will learn how to easily convert existing ML code into ML pipelines using ZenML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we will build models with [sklearn](https://scikit-learn.org/stable/), you will need to have the ZenML sklearn integration installed. You can install ZenML and the sklearn integration with the following command, which will also restart the kernel of your notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install \"zenml[server]\"\n",
    "!zenml integration install sklearn -y\n",
    "%pip install pyparsing==2.4.2  # required for Colab\n",
    "\n",
    "import IPython\n",
    "\n",
    "# automatically restart kernel\n",
    "IPython.Application.instance().kernel.do_shutdown(restart=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab Note:** On Colab, you need an [ngrok account](https://dashboard.ngrok.com/signup) to view some of the visualizations later. Please set up an account, then set your user token below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NGROK_TOKEN = \"\"  # TODO: set your ngrok token if you are working on Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zenml.environment import Environment\n",
    "\n",
    "if Environment.in_google_colab():  # Colab only setup\n",
    "\n",
    "    # install and authenticate ngrok\n",
    "    !pip install pyngrok\n",
    "    !ngrok authtoken {NGROK_TOKEN}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an ML practitioner, you are probably familiar with building ML models using Scikit-learn, PyTorch, TensorFlow, or similar. An **[ML Pipeline](https://docs.zenml.io/user-guide/starter-guide)** is simply an extension, including other steps you would typically do before or after building a model, like data acquisition, preprocessing, model deployment, or monitoring. The ML pipeline essentially defines a step-by-step procedure of your work as an ML practitioner. Defining ML pipelines explicitly in code is great because:\n",
    "- We can easily rerun all of our work, not just the model, eliminating bugs and making our models easier to reproduce.\n",
    "- Data and models can be versioned and tracked, so we can see at a glance which dataset a model was trained on and how it compares to other models.\n",
    "- If the entire pipeline is coded up, we can automate many operational tasks, like retraining and redeploying models when the underlying problem or data changes or rolling out new and improved models with CI/CD workflows.\n",
    "\n",
    "Having a clearly defined ML pipeline is essential for ML teams that aim to serve models on a large scale."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ZenML Setup\n",
    "Throughout this series, we will define our ML pipelines using [ZenML](https://github.com/zenml-io/zenml/). ZenML is an excellent tool for this task, as it is straightforward and intuitive to use and has [integrations](https://zenml.io/integrations) with most of the advanced MLOps tools we will want to use later. Make sure you have ZenML installed (via `pip install zenml`). Next, let's run some commands to make sure you start with a fresh ML stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf .zen\n",
    "!zenml init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Experimentation ML Code\n",
    "Let us get started with some simple exemplary ML code. In the following, we train a Scikit-learn SVC classifier to classify images of handwritten digits. We load the data, train a model on the training set, then test it on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.base import ClassifierMixin\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "def train_test() -> None:\n",
    "    \"\"\"Train and test a Scikit-learn SVC classifier on digits\"\"\"\n",
    "    digits = load_digits()\n",
    "    data = digits.images.reshape((len(digits.images), -1))\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        data, digits.target, test_size=0.2, shuffle=False\n",
    "    )\n",
    "    model = SVC(gamma=0.001)\n",
    "    model.fit(X_train, y_train)\n",
    "    test_acc = model.score(X_test, y_test)\n",
    "    print(f\"Test accuracy: {test_acc}\")\n",
    "\n",
    "\n",
    "train_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Turning experiments into ML pipelines with ZenML"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, your ML workflows will, of course, be much more complicated than that. You might have complex preprocessing that you do not want to redo every time you train a model, you will need to compare the performance of different models, deploy them in a production setting, and much more. Here ML pipelines come into play, allowing us to define our workflows in modular steps that we can then mix and match.\n",
    "\n",
    "![Digits Pipeline](_assets/1-1/digits_pipeline.png)\n",
    "\n",
    "We can identify three distinct steps in our example: data loading, model training, and model evaluation. Let us now define each of them as a ZenML **[Pipeline Step](https://docs.zenml.io/user-guide/starter-guide)** simply by moving each step to its own function and decorating them with ZenML's `@step` [Python decorator](https://realpython.com/primer-on-python-decorators/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zenml.steps import step, Output\n",
    "\n",
    "\n",
    "@step\n",
    "def importer() -> Output(\n",
    "    X_train=np.ndarray,\n",
    "    X_test=np.ndarray,\n",
    "    y_train=np.ndarray,\n",
    "    y_test=np.ndarray,\n",
    "):\n",
    "    \"\"\"Load the digits dataset as numpy arrays.\"\"\"\n",
    "    digits = load_digits()\n",
    "    data = digits.images.reshape((len(digits.images), -1))\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        data, digits.target, test_size=0.2, shuffle=False\n",
    "    )\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "\n",
    "@step\n",
    "def svc_trainer(\n",
    "    X_train: np.ndarray,\n",
    "    y_train: np.ndarray,\n",
    ") -> ClassifierMixin:\n",
    "    \"\"\"Train an sklearn SVC classifier.\"\"\"\n",
    "    model = SVC(gamma=0.001)\n",
    "    model.fit(X_train, y_train)\n",
    "    return model\n",
    "\n",
    "\n",
    "@step\n",
    "def evaluator(\n",
    "    X_test: np.ndarray,\n",
    "    y_test: np.ndarray,\n",
    "    model: ClassifierMixin,\n",
    ") -> float:\n",
    "    \"\"\"Calculate the test set accuracy of an sklearn model.\"\"\"\n",
    "    test_acc = model.score(X_test, y_test)\n",
    "    print(f\"Test accuracy: {test_acc}\")\n",
    "    return test_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can use ZenML's `@pipeline` decorator to connect all of our steps into an ML pipeline.\n",
    "\n",
    "Note that the pipeline definition does not depend on the concrete step functions we defined above; it merely establishes a recipe for how data moves through the steps. This means we can replace steps as we wish, e.g., to run the same pipeline with different models to compare their performances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zenml.pipelines import pipeline\n",
    "\n",
    "\n",
    "@pipeline\n",
    "def digits_pipeline(importer, trainer, evaluator):\n",
    "    \"\"\"Links all the steps together in a pipeline\"\"\"\n",
    "    X_train, X_test, y_train, y_test = importer()\n",
    "    model = trainer(X_train=X_train, y_train=y_train)\n",
    "    evaluator(X_test=X_test, y_test=y_test, model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running ZenML Pipelines\n",
    "Finally, we initialize our pipeline with concrete step functions and call the `run()` method to run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits_svc_pipeline = digits_pipeline(\n",
    "    importer=importer(), trainer=svc_trainer(), evaluator=evaluator()\n",
    ")\n",
    "digits_svc_pipeline.run(unlisted=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it, we just built and ran our first ML pipeline! Great job!\n",
    "\n",
    "You can now visualize the pipeline run in ZenML's dashboard. To do so, run \n",
    "`zenml up` to spin up a ZenML dashboard locally, log in with username `default`\n",
    "and empty password, and navigate to the \"Runs\" tab in the \"Pipelines\" section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zenml.environment import Environment\n",
    "\n",
    "def start_zenml_dashboard(port=8237):\n",
    "    if Environment.in_google_colab():\n",
    "        from pyngrok import ngrok\n",
    "\n",
    "        public_url = ngrok.connect(port)\n",
    "        print(f\"\\x1b[31mIn Colab, use this URL instead: {public_url}!\\x1b[0m\")\n",
    "        !zenml up --blocking --port {port}\n",
    "\n",
    "    else:\n",
    "        !zenml up --port {port}\n",
    "\n",
    "start_zenml_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Viewing Pipeline Runs in the ZenML Dashboard](_assets/1-1/view_run_in_dashboard.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [next lesson](1-2_Artifact_Lineage.ipynb), you will see one of the best features of ML pipelines in action: automated artifact versioning and caching. See you there!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 64-bit ('zenbytes-dev')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "ec45946565c50b1d690aa5a9e3c974f5b62b9cc8d8934e441e52186140f79402"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}