2-2-12-AssignmentQuestion.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring DNN learning with TensorFlow\n",
    "\n",
    "In this assignment we'll dive a little deeper with a series of hands on exercises to better understand DNN learning with Tensorflow. Remember that if you are taking the class for a certificate we will be asking you questions about the assignment in the test!\n",
    "\n",
    "We start by setting up the problem for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load libraries\n",
    "import sys\n",
    "\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This script requires TensorFlow 2 and Python 3.\n",
    "if sys.version_info.major < 3:\n",
    "    raise Exception((f\"The script is developed and tested for Python 3. \"\n",
    "                     f\"Current version: {sys.version_info.major}\"))\n",
    "\n",
    "if tf.__version__.split('.')[0] != '2':\n",
    "    raise Exception((f\"The script is developed and tested for tensorflow 2. \"\n",
    "                     f\"Current version: {tf.__version__}\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "q3KzJyjv3rnA",
    "outputId": "7ad9fb4b-2468-4efb-b79a-eb649b417438"
   },
   "outputs": [],
   "source": [
    "# Load in fashion MNIST\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "\n",
    "# Define the base model\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)), \n",
    "                                    tf.keras.layers.Dense(512, activation=tf.nn.relu), \n",
    "                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural Networks learn the best when the data is scaled / normalized to fall in a constant range. One practitioners often use is the range [0,1]. How might you do this to the training and test images used here?\n",
    "\n",
    "*A hint: these images are saved in the standard [RGB](https://www.rapidtables.com/web/color/RGB_Color.html) format*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_images  = #YOUR CODE HERE#\n",
    "test_images = #YOUR CODE HERE#"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using these improved images lets compile our model using an adaptive optimizer to learn faster and a categorical loss function to differentiate between the the various classes we are trying to classify. Since this is a very simple dataset we will only train for 5 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compile the model\n",
    "model.compile(optimizer = tf.keras.optimizers.Adam(),\n",
    "              loss = 'sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# fit the model to the training data\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "\n",
    "# test the model on the test data\n",
    "model.evaluate(test_images, test_labels)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "-JJMsvSB-1UY"
   },
   "source": [
    "Once it's done training -- you should see an accuracy value at the end of the final epoch. It might look something like 0.8648. This tells you that your neural network is about 86% accurate in classifying the training data. I.E., it figured out a pattern match between the image and the labels that worked 86% of the time. But how would it work with unseen data? That's why we have the test images. We can call ```model.evaluate```, and pass in the two sets, and it will report back the loss for each. This should reach about .8747 or thereabouts, showing about 87% accuracy. Not Bad!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "rquQqIx4AaGR"
   },
   "source": [
    "But what did it actually learn? If we inference on the model using ```model.predict``` we get out the following list of values. **What does it represent?**\n",
    "\n",
    "*A hint: trying running ```print(test_labels[0])```*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RyEIki0z_hAD",
    "outputId": "cbb8727c-262b-4289-a201-188298c73bb8"
   },
   "outputs": [],
   "source": [
    "classifications = model.predict(test_images)\n",
    "print(classifications[0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "OgQSIfDSOWv6"
   },
   "source": [
    "Let's now look at the layers in your model. What happens if you double the number of neurons in the dense layer. What different results do you get for loss, training time etc? Why do you think that's the case? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GSZSwV5UObQP",
    "outputId": "4ddede25-bb2d-4567-8443-b10801218001"
   },
   "outputs": [],
   "source": [
    "NUMBER_OF_NEURONS = #YOUR_CODE_HERE#\n",
    "\n",
    "# define the new model\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                                    tf.keras.layers.Dense(NUMBER_OF_NEURONS, activation=tf.nn.relu),\n",
    "                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])\n",
    "\n",
    "# compile fit and evaluate the model again\n",
    "model.compile(optimizer = tf.keras.optimizers.Adam(),\n",
    "              loss = 'sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "model.evaluate(test_images, test_labels)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "-0lF5MuvSuZF"
   },
   "source": [
    "Consider the effects of additional layers in the network instead of simply more neurons to the same layer. First update the model to add an additional dense layer into the model between the two existing Dense layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "YOUR_NEW_LAYER = #YOUR_CODE_HERE#\n",
    "\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),\n",
    "                                    YOUR_NEW_LAYER,\n",
    "                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets then compile, fit, and evaluate our model. What happens to the error? How does this compare to the original model and the model with double the number of neurons?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b1YPa6UhS8Es"
   },
   "outputs": [],
   "source": [
    "# compile fit and evaluate the model again\n",
    "model.compile(optimizer = tf.keras.optimizers.Adam(),\n",
    "              loss = 'sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "model.evaluate(test_images, test_labels)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "HS3vVkOgCDGZ"
   },
   "source": [
    "Before you trained, you normalized the data. What would be the impact of removing that? To see it for yourself fill in the following lines of code to get a non-normalized set of data and then re-fit and evaluate the model using this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JDqNAqrpCNg0"
   },
   "outputs": [],
   "source": [
    "# get new non-normalized mnist data\n",
    "training_images_non = #YOUR_CODE_HERE#\n",
    "test_images_non = #YOUR_CODE_HERE#\n",
    "\n",
    "# re-compile, re-fit and re-evaluate\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                                    tf.keras.layers.Dense(512, activation=tf.nn.relu),\n",
    "                                    YOUR_NEW_LAYER,\n",
    "                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])\n",
    "model.compile(optimizer = tf.keras.optimizers.Adam(),\n",
    "              loss = 'sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(training_images_non, training_labels, epochs=5)\n",
    "model.evaluate(test_images_non, test_labels)\n",
    "classifications = model.predict(test_images_non)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "E7W2PT66ZBHQ"
   },
   "source": [
    "Sometimes if you set the training for too many epochs you may find that training stops improving and you wish you could quit early. Good news, you can! TensorFlow has a function called ```Callbacks``` which can check the results from each epoch. Modify this callback function to make sure it exits training early but not before reaching at least the second epoch!\n",
    "\n",
    "*A hint: logs.get(METRIC_NAME) will return the value of METRIC_NAME at the current step*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pkaEHHgqZbYv"
   },
   "outputs": [],
   "source": [
    "# define and instantiate your custom Callback\n",
    "class myCallback(tf.keras.callbacks.Callback):\n",
    "  def on_epoch_end(self, epoch, logs={}):\n",
    "    if( #YOUR_CODE_HERE# ):\n",
    "      self.model.stop_training = True\n",
    "callbacks = myCallback()\n",
    "\n",
    "# re-compile, re-fit and re-evaluate\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                            tf.keras.layers.Dense(512, activation=tf.nn.relu),\n",
    "                            YOUR_NEW_LAYER,\n",
    "                            tf.keras.layers.Dense(10, activation=tf.nn.softmax)])\n",
    "model.compile(optimizer = tf.keras.optimizers.Adam(),\n",
    "      loss = 'sparse_categorical_crossentropy',\n",
    "      metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5, callbacks=[callbacks])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "2-2-12-Exercise.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}