WIT_Age_Regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "WIT Age Regression",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UiNxsd4_q9wq",
        "colab_type": "text"
      },
      "source": [
        "### What-If Tool - age-predicting regression model\n",
        "\n",
        "Copyright 2019 Google LLC.\n",
        "SPDX-License-Identifier: Apache-2.0\n",
        "\n",
        "This notebook shows use of the [What-If Tool](https://pair-code.github.io/what-if-tool) on a regression model.\n",
        "\n",
        "This notebook trains a linear regressor to predict a person's age from their census data using the [UCI census data](https://archive.ics.uci.edu/ml/datasets/census+income).\n",
        "\n",
        "It then visualizes the results of the trained regressor on test data using the What-If Tool.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qqB2tjOMETmr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Install the What-If Tool widget if running in colab {display-mode: \"form\"}\n",
        "\n",
        "try:\n",
        "  import google.colab\n",
        "  !pip install --upgrade witwidget\n",
        "except:\n",
        "  pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jlwjF-Nnmoww",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Define helper functions {display-mode: \"form\"}\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import functools\n",
        "\n",
        "# Creates a tf feature spec from the dataframe and columns specified.\n",
        "def create_feature_spec(df, columns=None):\n",
        "    feature_spec = {}\n",
        "    if columns == None:\n",
        "        columns = df.columns.values.tolist()\n",
        "    for f in columns:\n",
        "        if df[f].dtype is np.dtype(np.int64) or df[f].dtype is np.dtype(np.int32):\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.int64)\n",
        "        elif df[f].dtype is np.dtype(np.float64):\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.float32)\n",
        "        else:\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.string)\n",
        "    return feature_spec\n",
        "\n",
        "# Creates simple numeric and categorical feature columns from a feature spec and a\n",
        "# list of columns from that spec to use.\n",
        "#\n",
        "# NOTE: Models might perform better with some feature engineering such as bucketed\n",
        "# numeric columns and hash-bucket/embedding columns for categorical features.\n",
        "def create_feature_columns(columns, feature_spec):\n",
        "    ret = []\n",
        "    for col in columns:\n",
        "        if feature_spec[col].dtype is tf.int64 or feature_spec[col].dtype is tf.float32:\n",
        "            ret.append(tf.feature_column.numeric_column(col))\n",
        "        else:\n",
        "            ret.append(tf.feature_column.indicator_column(\n",
        "                tf.feature_column.categorical_column_with_vocabulary_list(col, list(df[col].unique()))))\n",
        "    return ret\n",
        "\n",
        "# An input function for providing input to a model from tf.Examples\n",
        "def tfexamples_input_fn(examples, feature_spec, label, mode=tf.estimator.ModeKeys.EVAL,\n",
        "                       num_epochs=None, \n",
        "                       batch_size=64):\n",
        "    def ex_generator():\n",
        "        for i in range(len(examples)):\n",
        "            yield examples[i].SerializeToString()\n",
        "    dataset = tf.data.Dataset.from_generator(\n",
        "      ex_generator, tf.dtypes.string, tf.TensorShape([]))\n",
        "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)\n",
        "    dataset = dataset.batch(batch_size)\n",
        "    dataset = dataset.map(lambda tf_example: parse_tf_example(tf_example, label, feature_spec))\n",
        "    dataset = dataset.repeat(num_epochs)\n",
        "    return dataset\n",
        "\n",
        "# Parses Tf.Example protos into features for the input function.\n",
        "def parse_tf_example(example_proto, label, feature_spec):\n",
        "    parsed_features = tf.io.parse_example(serialized=example_proto, features=feature_spec)\n",
        "    target = parsed_features.pop(label)\n",
        "    return parsed_features, target\n",
        "\n",
        "# Converts a dataframe into a list of tf.Example protos.\n",
        "def df_to_examples(df, columns=None):\n",
        "    examples = []\n",
        "    if columns == None:\n",
        "        columns = df.columns.values.tolist()\n",
        "    for index, row in df.iterrows():\n",
        "        example = tf.train.Example()\n",
        "        for col in columns:\n",
        "            if df[col].dtype is np.dtype(np.int64) or df[col].dtype is np.dtype(np.int32):\n",
        "                example.features.feature[col].int64_list.value.append(int(row[col]))\n",
        "            elif df[col].dtype is np.dtype(np.float64):\n",
        "                example.features.feature[col].float_list.value.append(row[col])\n",
        "            elif row[col] == row[col]:\n",
        "                example.features.feature[col].bytes_list.value.append(row[col].encode('utf-8'))\n",
        "        examples.append(example)\n",
        "    return examples\n",
        "\n",
        "# Converts a dataframe column into a column of 0's and 1's based on the provided test.\n",
        "# Used to force label columns to be numeric for binary classification using a TF estimator.\n",
        "def make_label_column_numeric(df, label_column, test):\n",
        "  df[label_column] = np.where(test(df[label_column]), 1, 0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nu398ARdeuxe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Read training dataset from CSV {display-mode: \"form\"}\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "# Set the path to the CSV containing the dataset to train on.\n",
        "csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'\n",
        "\n",
        "# Set the column names for the columns in the CSV. If the CSV's first line is a header line containing\n",
        "# the column names, then set this to None.\n",
        "csv_columns = [\n",
        "  \"Age\", \"Workclass\", \"fnlwgt\", \"Education\", \"Education-Num\", \"Marital-Status\",\n",
        "  \"Occupation\", \"Relationship\", \"Race\", \"Sex\", \"Capital-Gain\", \"Capital-Loss\",\n",
        "  \"Hours-per-week\", \"Country\", \"Over-50K\"]\n",
        "\n",
        "# Read the dataset from the provided CSV and print out information about it.\n",
        "df = pd.read_csv(csv_path, names=csv_columns, skipinitialspace=True)\n",
        "\n",
        "df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "67DYIFxoevt2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Specify input columns and column to predict {display-mode: \"form\"}\n",
        "import numpy as np\n",
        "\n",
        "# Set the column in the dataset you wish for the model to predict\n",
        "label_column = 'Age'\n",
        "\n",
        "# Set list of all columns from the dataset we will use for model input.\n",
        "input_features = [\n",
        "  'Over-50K', 'Workclass', 'Education', 'Marital-Status', 'Occupation',\n",
        "  'Relationship', 'Race', 'Sex', 'Capital-Gain', 'Capital-Loss',\n",
        "  'Hours-per-week', 'Country']\n",
        "\n",
        "# Create a list containing all input features and the label column\n",
        "features_and_labels = input_features + [label_column]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BV4f_4_Lex22",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Convert dataset to tf.Example protos {display-mode: \"form\"}\n",
        "\n",
        "examples = df_to_examples(df)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YyLr-_0de1Ii",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Create and train the regressor {display-mode: \"form\"}\n",
        "\n",
        "num_steps = 200  #@param {type: \"number\"}\n",
        "\n",
        "# Create a feature spec for the classifier\n",
        "feature_spec = create_feature_spec(df, features_and_labels)\n",
        "\n",
        "# Define and train the classifier\n",
        "train_inpf = functools.partial(tfexamples_input_fn, examples, feature_spec, label_column)\n",
        "regressor = tf.estimator.LinearRegressor(\n",
        "    feature_columns=create_feature_columns(input_features, feature_spec))\n",
        "regressor.train(train_inpf, steps=num_steps)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NUQVro76e38Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Invoke What-If Tool for test data and the trained model {display-mode: \"form\"}\n",
        "\n",
        "num_datapoints = 2000  #@param {type: \"number\"}\n",
        "tool_height_in_px = 1000  #@param {type: \"number\"}\n",
        "\n",
        "from witwidget.notebook.visualization import WitConfigBuilder\n",
        "from witwidget.notebook.visualization import WitWidget\n",
        "\n",
        "# Load up the test dataset\n",
        "test_csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test'\n",
        "test_df = pd.read_csv(test_csv_path, names=csv_columns, skipinitialspace=True,\n",
        "  skiprows=1)\n",
        "test_examples = df_to_examples(test_df[0:num_datapoints])\n",
        "\n",
        "# Setup the tool with the test examples and the trained classifier\n",
        "config_builder = WitConfigBuilder(test_examples[0:num_datapoints]).set_estimator_and_feature_spec(\n",
        "    regressor, feature_spec).set_model_type('regression')\n",
        "WitWidget(config_builder, height=tool_height_in_px)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VlncgiQTDFw",
        "colab_type": "text"
      },
      "source": [
        "#### Exploration ideas\n",
        "\n",
        "- Create a scatter plot of age on the X-axis by inference value on the Y-axis to see how the predicted age compares to the actual age for all test examples.\n",
        "  - How does the model do overall? Is there a pattern to the errors it creates?\n",
        "\n",
        "- In the \"Performance\" tab, set the ground truth feature to \"age\". You can now see a simple breakdown of the mean error, mean absolute error, and mean squared error over the entire test set. Additionally, you can set a scatter axis or binning option to be any of those three calculations (i.e. you can make a scatter plot of age vs inference error).\n",
        "\n",
        "- Replace the LinearRegressor with a DNNRegressor (you'll need to decide on the size and number of the hidden layers). Can you create a better model? Does this model have the same pattern of performance issues as the previous model, or are there errors more evenly distributed?\n",
        "  - If you look at the [WIT Model Comparison](https://colab.research.google.com/github/pair-code/what-if-tool/blob/master/WIT_Model_Comparison.ipynb) notebook, you can see how to compare the two models on a single WIT.\n",
        "\n",
        "- In the \"Performance\" tab, slice the dataset by any feature or pair of features. Are there feature values for which the model seems to perform better or worse on?\n",
        "\n"
      ]
    }
  ]
}