keras_sklearn_compare_caip_e2e.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AirZHYMaHoa2"
   },
   "source": [
    "# Comparing Keras and Scikit models deployed on Cloud AI Platform with the What-if Tool\n",
    "\n",
    "In this notebook we'll use the UCI [wine quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality) to train both tf.keras and Scikit learn regression models that will predict the quality rating of a wine given 11 numerical data points about the wine. You'll learn how to:\n",
    "* Build, train, and then deploy tf.keras and Scikit Learn models to Cloud AI Platform\n",
    "* Use the What-if Tool to compare two different models deployed on CAIP\n",
    "\n",
    "You will need a Google Cloud Platform account and project to run this notebook. Instructions for creating a project can be found [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pOSv5xidIeDX"
   },
   "source": [
    "## Installing dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "python_version = sys.version_info[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-W6pVKiR8M1y"
   },
   "outputs": [],
   "source": [
    "# If you're running on Colab, you'll need to install the What-if Tool package and authenticate\n",
    "def pip_install(module):\n",
    "    if python_version == '2':\n",
    "        !pip install {module} --quiet\n",
    "    else:\n",
    "        !pip3 install {module} --quiet\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "\n",
    "if IN_COLAB:\n",
    "    pip_install('witwidget')\n",
    "\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tgZtpOds7RBP"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import witwidget\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from witwidget.notebook.visualization import WitWidget, WitConfigBuilder\n",
    "\n",
    "# This has been tested on TF 1.14\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LaWQ_7-DIg0C"
   },
   "source": [
    "## Download and process data\n",
    "\n",
    "In this section we'll:\n",
    "* Download the wine quality data directly from UCI Machine Learning\n",
    "* Read it into a Pandas dataframe and preview it\n",
    "* Split the data and labels into train and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3cynTjLY8LiC"
   },
   "outputs": [],
   "source": [
    "!wget 'http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tWVHSjqZ8e2S"
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('winequality-white.csv', index_col=False, delimiter=';')\n",
    "data = shuffle(data, random_state=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YfDFNvbHJk3k"
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uny6h1Tk9LDf"
   },
   "outputs": [],
   "source": [
    "labels = data['quality']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nmwemTTHO-7B"
   },
   "outputs": [],
   "source": [
    "print(labels.value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m22ZzLWkWdlk"
   },
   "outputs": [],
   "source": [
    "data = data.drop(columns=['quality'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GT3KiRibLiUM"
   },
   "outputs": [],
   "source": [
    "train_size = int(len(data) * 0.8)\n",
    "train_data = data[:train_size]\n",
    "train_labels = labels[:train_size]\n",
    "\n",
    "test_data = data[train_size:]\n",
    "test_labels = labels[train_size:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cZzev0UCXmz5"
   },
   "outputs": [],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XAOsNcBcIj9L"
   },
   "source": [
    "## Train tf.keras model\n",
    "\n",
    "In this section we'll:\n",
    "\n",
    "* Build a regression model using tf.keras to predict a wine's quality score\n",
    "* Train the model\n",
    "* Add a layer to the model to prepare it for serving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p64TNRFJ9TOP"
   },
   "outputs": [],
   "source": [
    "# This is the size of the array we'll be feeding into our model for each wine example\n",
    "input_size = len(train_data.iloc[0])\n",
    "print(input_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rsNQebpr-aB1"
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(200, input_shape=(input_size,), activation='relu'))\n",
    "model.add(Dense(50, activation='relu'))\n",
    "model.add(Dense(25, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zqHhJTwCgbwH"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D10kjJh1-dqN"
   },
   "outputs": [],
   "source": [
    "model.fit(train_data.values,train_labels.values, epochs=4, batch_size=32, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BxWHjEUTk9KX"
   },
   "source": [
    "## Deploy keras model to Cloud AI Platform\n",
    "\n",
    "In this section we'll:\n",
    "* Set up some global variables for our GCP project\n",
    "* Add a serving layer to our model so we can deploy it on Cloud AI Platform\n",
    "* Run the deploy command to deploy our model\n",
    "* Generate a test prediction on our deployed model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "89xSGhDgOoez"
   },
   "outputs": [],
   "source": [
    "# Update these to your own GCP project + model names\n",
    "GCP_PROJECT = 'your_gcp_project'\n",
    "KERAS_MODEL_BUCKET = 'gs://your_storage_bucket'\n",
    "KERAS_VERSION_NAME = 'v1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M5LY_q-lw1ur"
   },
   "outputs": [],
   "source": [
    "# Add the serving input layer below in order to serve our model on AI Platform\n",
    "class ServingInput(tf.keras.layers.Layer):\n",
    "  # the important detail in this boilerplate code is \"trainable=False\"\n",
    "  def __init__(self, name, dtype, batch_input_shape=None):\n",
    "    super(ServingInput, self).__init__(trainable=False, name=name, dtype=dtype, batch_input_shape=batch_input_shape)\n",
    "  def get_config(self):\n",
    "    return {'batch_input_shape': self._batch_input_shape, 'dtype': self.dtype, 'name': self.name }\n",
    "\n",
    "restored_model = model\n",
    "\n",
    "serving_model = tf.keras.Sequential()\n",
    "serving_model.add(ServingInput('serving', tf.float32, (None, input_size)))\n",
    "serving_model.add(restored_model)\n",
    "tf.contrib.saved_model.save_keras_model(serving_model, os.path.join(KERAS_MODEL_BUCKET, 'keras_export'))  # export the model to your GCS bucket\n",
    "export_path = KERAS_MODEL_BUCKET + '/keras_export'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8jVUIWZwW7K0"
   },
   "outputs": [],
   "source": [
    "# Configure gcloud to use your project\n",
    "!gcloud config set project $GCP_PROJECT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tPGC8JhhRAY2"
   },
   "outputs": [],
   "source": [
    "# Create a new model in our project, you only need to run this once\n",
    "!gcloud ai-platform models create keras_wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WaGbopHVIRdB"
   },
   "outputs": [],
   "source": [
    "# Deploy the model to Cloud AI Platform\n",
    "!gcloud beta ai-platform versions create $KERAS_VERSION_NAME --model keras_wine \\\n",
    "--origin=$export_path \\\n",
    "--python-version=3.5 \\\n",
    "--runtime-version=1.14 \\\n",
    "--framework='TENSORFLOW'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ypRd-P-ZPfaH"
   },
   "outputs": [],
   "source": [
    "%%writefile predictions.json\n",
    "[7.8, 0.21, 0.49, 1.2, 0.036, 20.0, 99.0, 0.99, 3.05, 0.28, 12.1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "othDFHKZcab-"
   },
   "outputs": [],
   "source": [
    "# Test the deployed model on an example from our test set\n",
    "# The correct score for this prediction is 7\n",
    "prediction = !gcloud ai-platform predict --model=keras_wine --json-instances=predictions.json --version=$KERAS_VERSION_NAME\n",
    "print(prediction[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WW5L9_JWmSuY"
   },
   "source": [
    "## Build and train Scikit learn model\n",
    "\n",
    "In this section we'll:\n",
    "* Train a regression model using Scikit Learn\n",
    "* Save the model to a local file using `pickle`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SGn5P-VoR2ld"
   },
   "outputs": [],
   "source": [
    "SKLEARN_VERSION_NAME = 'v1'\n",
    "SKLEARN_MODEL_BUCKET = 'gs://sklearn_model_bucket'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hD4SE8psB1UM"
   },
   "outputs": [],
   "source": [
    "scikit_model = LinearRegression().fit(train_data.values, train_labels.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hFGjWhJ9-JKZ"
   },
   "outputs": [],
   "source": [
    "# Export the model to a local file using pickle\n",
    "pickle.dump(scikit_model, open('model.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9-eS60E-lTKk"
   },
   "source": [
    "## Deploy Scikit model to CAIP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ri7u5kbyRnwL"
   },
   "source": [
    "In this section we'll:\n",
    "* Copy our saved model file to Cloud Storage\n",
    "* Run the gcloud command to deploy our model\n",
    "* Generate a prediction on our deployed model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VzfU9HaE-p2J"
   },
   "outputs": [],
   "source": [
    "# Copy the saved model to Cloud Storage\n",
    "!gsutil cp ./model.pkl gs://wine_sklearn/model.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RKInNmOBRaHO"
   },
   "outputs": [],
   "source": [
    "# Create a new model in our project, you only need to run this once\n",
    "!gcloud ai-platform models create sklearn_wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8pf5BnRo-1wJ"
   },
   "outputs": [],
   "source": [
    "!gcloud beta ai-platform versions create $SKLEARN_VERSION_NAME --model=sklearn_wine \\\n",
    "--origin=$SKLEARN_MODEL_BUCKET \\\n",
    "--runtime-version=1.14 \\\n",
    "--python-version=3.5 \\\n",
    "--framework='SCIKIT_LEARN'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zTxEHIDAlYRQ"
   },
   "outputs": [],
   "source": [
    "# Test the model usnig the same example instance from above\n",
    "!gcloud ai-platform predict --model=sklearn_wine --json-instances=predictions.json --version=$SKLEARN_VERSION_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qGfkfD6PiXia"
   },
   "source": [
    "## Compare tf.keras and Scikit models with the What-if Tool\n",
    "\n",
    "Now we're ready for the What-if Tool! In this section we'll:\n",
    "* Create an array of our test examples with their ground truth values. The What-if Tool works best when we send the actual values for each example input.\n",
    "* Instantiate the What-if Tool using the `set_compare_ai_platform_model` method. This lets us compare 2 models deployed on Cloud AI Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1qNIii9_SbPj"
   },
   "outputs": [],
   "source": [
    "# Create np array of test examples + their ground truth labels\n",
    "test_examples = np.hstack((test_data[:200].values,test_labels[:200].values.reshape(-1,1)))\n",
    "print(test_examples.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vqOFzh8pyG6Z"
   },
   "outputs": [],
   "source": [
    "# Create a What-if Tool visualization, it may take a minute to load\n",
    "# See the cell below this for exploration ideas\n",
    "\n",
    "# We use `set_predict_output_tensor` here becuase our tf.keras model returns a dict with a 'sequential' key\n",
    "\n",
    "config_builder = (WitConfigBuilder(test_examples.tolist(), data.columns.tolist() + ['quality'])\n",
    "  .set_ai_platform_model(GCP_PROJECT, 'keras_wine', KERAS_VERSION_NAME).set_predict_output_tensor('sequential').set_uses_predict_api(True)\n",
    "  .set_target_feature('quality')\n",
    "  .set_model_type('regression')\n",
    "  .set_compare_ai_platform_model(GCP_PROJECT, 'sklearn_wine', SKLEARN_VERSION_NAME))\n",
    "WitWidget(config_builder, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zScgqn1dScKS"
   },
   "source": [
    "## What-if Tool Exploration ideas\n",
    "\n",
    "* Look at the scatter plot of \"Inference value scikit_wine\" vs \"Inference value keras_wine\"\n",
    "  * Examples off of the diagonal represent wines for which the two models have large disagreement on the quality score. Click on some of these and explore the features. \n",
    "  * You can also click on individual examples, change some of the feature values for that example, and compare the impact of that change on the model's prediction\n",
    "  * Check out the partial dependence plots to see what features are causes the large skew between the two models.\n",
    "\n",
    "* Go to the Performance tab and see the overall performance of each model. Is one more accurate over the test data than the other?\n",
    "  * In this tab, use the \"Slice by\" dropdown to slice the data into subgroups and see how both models perform across those subgroups. Try slicing by alcohol. Which model has more consistent performance across the slices?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "What-If Tool with Keras and Scikit Learn Cloud AI Platform Models - end-to-end",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}