2-2-7-ExploringCategorical.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jL2nKxjS4S_r"
   },
   "source": [
    "# Start with a simple neural network for MNIST\n",
    "Note that there are 2 layers, one with 20 neurons, and one with 10.\n",
    "\n",
    "The 10-neuron layer is our final layer because we have 10 classes we want to classify.\n",
    "\n",
    "Train this, and you should see it get about 98% accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 84138,
     "status": "ok",
     "timestamp": 1600783606882,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gi5RQS-qsFWRcH0rj9XkKPRuO-VjT30YiqfpEk_=s64",
      "userId": "06401446828348966425"
     },
     "user_tz": 420
    },
    "id": "zrQ-DeQtybXZ",
    "outputId": "a24c70c8-2bd4-45e6-932e-749a1211805f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.4104 - accuracy: 0.8838 - val_loss: 0.2347 - val_accuracy: 0.9304\n",
      "Epoch 2/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.2216 - accuracy: 0.9364 - val_loss: 0.1985 - val_accuracy: 0.9410\n",
      "Epoch 3/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1879 - accuracy: 0.9463 - val_loss: 0.1734 - val_accuracy: 0.9484\n",
      "Epoch 4/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1675 - accuracy: 0.9519 - val_loss: 0.1609 - val_accuracy: 0.9519\n",
      "Epoch 5/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1514 - accuracy: 0.9565 - val_loss: 0.1519 - val_accuracy: 0.9561\n",
      "Epoch 6/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1408 - accuracy: 0.9589 - val_loss: 0.1457 - val_accuracy: 0.9575\n",
      "Epoch 7/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1308 - accuracy: 0.9621 - val_loss: 0.1422 - val_accuracy: 0.9600\n",
      "Epoch 8/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1234 - accuracy: 0.9639 - val_loss: 0.1345 - val_accuracy: 0.9602\n",
      "Epoch 9/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1171 - accuracy: 0.9657 - val_loss: 0.1385 - val_accuracy: 0.9593\n",
      "Epoch 10/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1122 - accuracy: 0.9671 - val_loss: 0.1368 - val_accuracy: 0.9606\n",
      "Epoch 11/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1071 - accuracy: 0.9688 - val_loss: 0.1324 - val_accuracy: 0.9616\n",
      "Epoch 12/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.1029 - accuracy: 0.9697 - val_loss: 0.1346 - val_accuracy: 0.9615\n",
      "Epoch 13/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0995 - accuracy: 0.9701 - val_loss: 0.1385 - val_accuracy: 0.9596\n",
      "Epoch 14/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0960 - accuracy: 0.9713 - val_loss: 0.1410 - val_accuracy: 0.9588\n",
      "Epoch 15/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0933 - accuracy: 0.9722 - val_loss: 0.1353 - val_accuracy: 0.9626\n",
      "Epoch 16/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0910 - accuracy: 0.9729 - val_loss: 0.1356 - val_accuracy: 0.9622\n",
      "Epoch 17/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0880 - accuracy: 0.9734 - val_loss: 0.1339 - val_accuracy: 0.9625\n",
      "Epoch 18/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0855 - accuracy: 0.9740 - val_loss: 0.1349 - val_accuracy: 0.9619\n",
      "Epoch 19/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0832 - accuracy: 0.9754 - val_loss: 0.1341 - val_accuracy: 0.9624\n",
      "Epoch 20/20\n",
      "1875/1875 [==============================] - 4s 2ms/step - loss: 0.0808 - accuracy: 0.9759 - val_loss: 0.1340 - val_accuracy: 0.9626\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7feecc40e748>"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "data = tf.keras.datasets.mnist\n",
    "\n",
    "(training_images, training_labels), (val_images, val_labels) = data.load_data()\n",
    "\n",
    "training_images  = training_images / 255.0\n",
    "val_images = val_images / 255.0\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                                    tf.keras.layers.Dense(20, activation=tf.nn.relu),\n",
    "                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(training_images, training_labels, epochs=20, validation_data=(val_images, val_labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JN92xbwK4i_S"
   },
   "source": [
    "## Examine the test data\n",
    "\n",
    "Using model.evaluate, you can get metrics for a test set. In this case we only have a training set and a validation set, so we can try it out with the validation set. The accuracy will be slightly lower, at maybe 96%.  This is because the model hasn't previously seen this data and may not be fully generalized for all data. Still it's a pretty good score.\n",
    "\n",
    "You can also predict images, and compare against their actual label. The [0] image in the set is a number 7, and here you can see that neuron 7 has a 9.9e-1 (99%+) probability, so it got it right!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 86
    },
    "colab_type": "code",
    "id": "Rzit5Te-4lT6",
    "outputId": "d9044da8-b552-412f-a59d-faa799f0a970"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 1s 2ms/step - loss: 0.1495 - accuracy: 0.9569\n",
      "[2.4921512e-09 1.3765138e-10 8.8281205e-08 1.0477231e-03 2.8455029e-12\n",
      " 4.0820678e-06 2.0070659e-16 9.9894780e-01 1.0296049e-07 2.9972372e-07]\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "\n",
    "model.evaluate(val_images, val_labels)\n",
    "\n",
    "classifications = model.predict(val_images)\n",
    "print(classifications[0])\n",
    "print(val_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6LkJGAiI5Cr3"
   },
   "source": [
    "## Modify to inspect learned values\n",
    "\n",
    "This code is identical, except that the layers are named prior to adding to the sequential. This allows us to inspect their learned parameters later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 777
    },
    "colab_type": "code",
    "id": "eyyJ3RMYpFXR",
    "outputId": "b9d80261-638e-4ee4-974c-b1141929535c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.3985 - accuracy: 0.8888\n",
      "Epoch 2/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2472 - accuracy: 0.9293\n",
      "Epoch 3/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2085 - accuracy: 0.9404\n",
      "Epoch 4/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1803 - accuracy: 0.9483\n",
      "Epoch 5/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1606 - accuracy: 0.9534\n",
      "Epoch 6/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1458 - accuracy: 0.9571\n",
      "Epoch 7/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1359 - accuracy: 0.9600\n",
      "Epoch 8/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1288 - accuracy: 0.9618\n",
      "Epoch 9/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1211 - accuracy: 0.9639\n",
      "Epoch 10/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1151 - accuracy: 0.9658\n",
      "Epoch 11/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1108 - accuracy: 0.9664\n",
      "Epoch 12/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1059 - accuracy: 0.9685\n",
      "Epoch 13/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1017 - accuracy: 0.9702\n",
      "Epoch 14/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0981 - accuracy: 0.9704\n",
      "Epoch 15/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0951 - accuracy: 0.9707\n",
      "Epoch 16/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0927 - accuracy: 0.9720\n",
      "Epoch 17/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0896 - accuracy: 0.9733\n",
      "Epoch 18/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0871 - accuracy: 0.9739\n",
      "Epoch 19/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0845 - accuracy: 0.9741\n",
      "Epoch 20/20\n",
      "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0829 - accuracy: 0.9748\n",
      "313/313 [==============================] - 1s 2ms/step - loss: 0.1495 - accuracy: 0.9569\n",
      "[2.4921512e-09 1.3765138e-10 8.8281205e-08 1.0477231e-03 2.8455029e-12\n",
      " 4.0820678e-06 2.0070659e-16 9.9894780e-01 1.0296049e-07 2.9972372e-07]\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "data = tf.keras.datasets.mnist\n",
    "\n",
    "(training_images, training_labels), (val_images, val_labels) = data.load_data()\n",
    "\n",
    "training_images  = training_images / 255.0\n",
    "val_images = val_images / 255.0\n",
    "layer_1 = tf.keras.layers.Dense(20, activation=tf.nn.relu)\n",
    "layer_2 = tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "model = tf.keras.models.Sequential([tf.keras.layers.Flatten(input_shape=(28,28)),\n",
    "                                    layer_1,\n",
    "                                    layer_2])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(training_images, training_labels, epochs=20)\n",
    "\n",
    "model.evaluate(val_images, val_labels)\n",
    "\n",
    "classifications = model.predict(val_images)\n",
    "print(classifications[0])\n",
    "print(val_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7pNYFF935PlE"
   },
   "source": [
    "# Inspect weights\n",
    "\n",
    "If you print layer_1.get_weights(), you'll see a lot of data. Let's unpack it. First, there are 2 arrays in the result, so let's look at the first one. In particular let's look at its size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "QACivjNKxFWW",
    "outputId": "928ee746-bf04-4941-9a94-152f3d00efc0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15680\n"
     ]
    }
   ],
   "source": [
    "print(layer_1.get_weights()[0].size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lqpHrDyp5acs"
   },
   "source": [
    "The above code should print 15680. Why?\n",
    "\n",
    "Recall that there are 20 neurons in the first layer.\n",
    "\n",
    "Recall also that the images are 28x28, which is 784.\n",
    "\n",
    "If you multiply 784 x 20 you get 15680.\n",
    "\n",
    "So...this layer has 20 neurons, and each neuron learns a W parameter for each pixel. So instead of y=Mx+c, we have \n",
    "y=M1X1+M2X2+M3X3+....+M784X784+C in every neuron! \n",
    "\n",
    "Every pixel has a weight in every neuron. Those weights are multiplied by the pixel value, summed up, and given a bias.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "TdXrHDEw6ACm",
    "outputId": "d17336de-c447-4bc7-943c-9fb0442c138c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    }
   ],
   "source": [
    "print(layer_1.get_weights()[1].size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EIOZ7rJy6Eg1"
   },
   "source": [
    "The above code will give you 20 -- the get_weights()[1] contains the biases for each of the 20 neurons in this layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yyFKpzxN6T-N"
   },
   "source": [
    "## Inspecting layer 2\n",
    "\n",
    "Now let's look at layer 2. Printing the get_weights will give us 2 lists, the first a list of weights for the 10 neurons, and the second a list of biases for the 10 neurons\n",
    "\n",
    "Let's look first at the weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "o9P_PVwXyKXJ",
    "outputId": "aab50faa-c291-4ed6-d4e6-76d3ec82da80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "print(layer_2.get_weights()[0].size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "daah2gq56fdb"
   },
   "source": [
    "This should return 200. Again, consider why?\n",
    "\n",
    "There are 10 neurons in this layer, but there are 20 neurons in the previous layer. So, each neuron in this layer will learn a weight for the incoming value from the previous layer. So, for example, the if the first neuron in this layer is N21, and the neurons output from the previous layers are N11-N120, then this neuron will have 20 weights (W1-W20) and it will calculate its output to be:\n",
    "\n",
    "W1N11+W2N12+W3N13+...+W20N120+Bias\n",
    "\n",
    "So each of these weights will be learned as will the bias, for every neuron.\n",
    "\n",
    "Note that N11 refers to Layer 1 Neuron 1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "Reyw9wC65o8z",
    "outputId": "37a5c21c-1d7b-4d3a-b8de-90f60186ad70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "print(layer_2.get_weights()[1].size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DS89WZag7GlB"
   },
   "source": [
    "...and as expected there are 10 elements in this array, representing the 10 biases for the 10 neurons.\n",
    "\n",
    "Hopefully this helps you see how the element of a simple neuron containing y=mx+c can be expanded greatly into a deep neural network, and that DNN can learn the parameters that match the 784 pixels of an image to their output!"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Exploring_Categorical.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}