From c78290c745033806e2b99c59254caf94c375c864 Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 18 Apr 2023 20:33:15 +0300 Subject: [PATCH] week02 --- week02_autodiff/README.md | 31 + week02_autodiff/homework.ipynb | 499 ++++++++++++++ week02_autodiff/mnist.py | 63 ++ week02_autodiff/notmnist.py | 52 ++ week02_autodiff/seminar_pytorch.ipynb | 896 ++++++++++++++++++++++++++ week02_autodiff/tensorflow.ipynb | 825 ++++++++++++++++++++++++ 6 files changed, 2366 insertions(+) create mode 100644 week02_autodiff/README.md create mode 100644 week02_autodiff/homework.ipynb create mode 100644 week02_autodiff/mnist.py create mode 100644 week02_autodiff/notmnist.py create mode 100644 week02_autodiff/seminar_pytorch.ipynb create mode 100644 week02_autodiff/tensorflow.ipynb diff --git a/week02_autodiff/README.md b/week02_autodiff/README.md new file mode 100644 index 00000000..daabb166 --- /dev/null +++ b/week02_autodiff/README.md @@ -0,0 +1,31 @@ +[__slides__](https://yadi.sk/i/eRVlESjqlIPBGw) + + +## Materials + +- __In english:__ + * Deep learning frameworks - [video](https://www.youtube.com/watch?v=Vf_-OkqbwPo) + * [PyTorch tutorial](https://www.youtube.com/watch?v=VMcRWYEKmhw) + * [Tensorflow tutorial](https://www.youtube.com/watch?v=FQ660T4uu7k) + +- __In russian:__ + * [Pytorch tutorial](https://yadi.sk/i/O3mQ76u43So3h9) __recommended__ + * [Tensorflow tutorial](https://www.youtube.com/watch?v=FQ660T4uu7k) (english only for now. Links are welcome) + +## More on DL frameworks + - A lecture on nonlinearities, intializations and other tricks in deep learning (karpathy) - [video](https://www.youtube.com/watch?v=GUtlrDbHhJM) + - A lecture on activations, recap of adaptive SGD and dropout (karpathy) - [video](https://www.youtube.com/watch?v=KaR4lIdI1MQ) + - [a deep learning neophite cheat sheet](http://www.kdnuggets.com/2016/03/must-know-tips-deep-learning-part-1.html) + - [bonus video] Deep learning philosophy: [our humble take](https://www.youtube.com/watch?v=9qyE1Ev1Xdw) (english) + - [reading] on weight initialization: [blog post](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization) + - [reading] pretty much all the module 1 of http://cs231n.github.io/ + + +## Practice + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yandexdataschool/Practical_DL/blob/spring33/week02_autodiff/seminar_pytorch.ipynb) + +As usual, go to `seminar_pytorch.ipynb` and follow instructions from there. You will also need to pass `homework_pytorch.ipynb` for full score. + +__Alternative (TensorFlow):__ a similar tutorial for tensorflow is provided in `tensorflow.ipynb`. From now on, you *can* submit assignments in any framework - but you will have to do some extra engineering in that case. However, unless you're already profficient with PyTorch, we recommend you stick to it. + diff --git a/week02_autodiff/homework.ipynb b/week02_autodiff/homework.ipynb new file mode 100644 index 00000000..9653a411 --- /dev/null +++ b/week02_autodiff/homework.ipynb @@ -0,0 +1,499 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "xOb-hGR2uh7t" + }, + "source": [ + "# Homework part I\n", + "\n", + "The first problem set contains basic tasks in PyTorch.\n", + "\n", + "__Note:__ Instead of doing this part of homework, you can prove your skills otherwise:\n", + "* A commit to PyTorch or PyTorch-based repos will do;\n", + "* Fully implemented seminar assignment in tensorflow or theano will do;\n", + "* Your own project in PyTorch that is developed to a state in which a normal human can understand and appreciate what it does." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "FCFZeFlGuh7v", + "outputId": "02d64913-e6e9-40d7-f7ec-04e444557a0e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1.10.2\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "import torch, torch.nn as nn\n", + "import torch.nn.functional as F\n", + "print(torch.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "HuMIhPfYuh71" + }, + "source": [ + "### Task I - tensormancy\n", + "\n", + "![img](https://media.giphy.com/media/3o751UMCYtSrRAFRFC/giphy.gif)\n", + "\n", + "When dealing with more complex stuff like neural network, it's best if you use tensors the way samurai uses his sword. \n", + "\n", + "\n", + "__1.1 The Cannabola__\n", + "[(_disclaimer_)](https://gist.githubusercontent.com/justheuristic/e2c1fa28ca02670cabc42cacf3902796/raw/fd3d935cef63a01b85ed2790b5c11c370245cbd7/stddisclaimer.h)\n", + "\n", + "Let's write another function, this time in polar coordinates:\n", + "$$\\rho(\\theta) = (1 + 0.9 \\cdot cos (8 \\cdot \\theta) ) \\cdot (1 + 0.1 \\cdot cos(24 \\cdot \\theta)) \\cdot (0.9 + 0.05 \\cdot cos(200 \\cdot \\theta)) \\cdot (1 + sin(\\theta))$$\n", + "\n", + "\n", + "Then convert it into cartesian coordinates ([howto](http://www.mathsisfun.com/polar-cartesian-coordinates.html)) and plot the results.\n", + "\n", + "Use torch tensors only: no lists, loops, numpy arrays, etc." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "URx7y3hyuh72" + }, + "outputs": [], + "source": [ + "theta = torch.linspace(-np.pi, np.pi, steps=1000)\n", + "\n", + "# compute rho(theta) as per formula above\n", + "rho = ### YOUR CODE\n", + "\n", + "# Now convert polar (rho, theta) pairs into cartesian (x,y) to plot them.\n", + "x = ### YOUR CODE\n", + "y = ### YOUR CODE\n", + "\n", + "\n", + "plt.figure(figsize=[6, 6])\n", + "plt.fill(x.numpy(), y.numpy(), color='green')\n", + "plt.grid()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "-eHnJPqyuh76" + }, + "source": [ + "### Task II: The Game of Life\n", + "\n", + "Now it's time for you to make something more challenging. We'll implement Conway's [Game of Life](http://web.stanford.edu/~cdebs/GameOfLife/) in _pure PyTorch_.\n", + "\n", + "While this is still a toy task, implementing game of life this way has one cool benefit: __you'll be able to run it on GPU!__ Indeed, what could be a better use of your GPU than simulating Game of Life on 1M/1M grids?\n", + "\n", + "![img](https://cdn.tutsplus.com/gamedev/authors/legacy/Stephane%20Beniak/2012/09/11/Preview_Image.png)\n", + "If you've skipped the URL above out of sloth, here's the Game of Life:\n", + "* You have a 2D grid of cells, where each cell is \"alive\"(1) or \"dead\"(0)\n", + "* Any living cell that has 2 or 3 neighbors survives, else it dies [0,1 or 4+ neighbors]\n", + "* Any cell with exactly 3 neighbors becomes alive (if it was dead)\n", + "\n", + "For this task, you are given a reference NumPy implementation that you must convert to PyTorch.\n", + "_[NumPy code inspired by: https://github.com/rougier/numpy-100]_\n", + "\n", + "\n", + "__Note:__ You can find convolution in `torch.nn.functional.conv2d(Z,filters)`. Note that it has a different input format.\n", + "\n", + "__Note 2:__ From the mathematical standpoint, PyTorch convolution is actually cross-correlation. Those two are very similar operations. More info: [video tutorial](https://www.youtube.com/watch?v=C3EEy8adxvc), [scipy functions review](http://programmerz.ru/questions/26903/2d-convolution-in-python-similar-to-matlabs-conv2-question), [stack overflow source](https://stackoverflow.com/questions/31139977/comparing-matlabs-conv2-with-scipys-convolve2d)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "d_8ydkevuh78" + }, + "outputs": [], + "source": [ + "from scipy.signal import correlate2d\n", + "\n", + "def np_update(Z):\n", + " # Count neighbours with convolution\n", + " filters = np.array([[1, 1, 1],\n", + " [1, 0, 1],\n", + " [1, 1, 1]])\n", + "\n", + " N = correlate2d(Z, filters, mode='same')\n", + "\n", + " # Apply rules\n", + " birth = (N == 3) & (Z == 0)\n", + " survive = ((N == 2) | (N == 3)) & (Z == 1)\n", + "\n", + " Z[:] = birth | survive\n", + " return Z" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "5EX2Vii8uh7_" + }, + "outputs": [], + "source": [ + "def torch_update(Z):\n", + " \"\"\"\n", + " Implement an update function that does to Z exactly the same as np_update.\n", + " :param Z: torch.FloatTensor of shape [height,width] containing 0s(dead) an 1s(alive)\n", + " :returns: torch.FloatTensor Z after updates.\n", + " \n", + " You can opt to create new tensor or change Z inplace.\n", + " \"\"\"\n", + " \n", + " #\n", + " \n", + " return Z\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "rX2wAml2uh8C" + }, + "outputs": [], + "source": [ + "# initial frame\n", + "Z_numpy = np.random.choice([0, 1], p=(0.5, 0.5), size=(100, 100))\n", + "Z = torch.from_numpy(Z_numpy).type(torch.FloatTensor)\n", + "\n", + "# your debug polygon :)\n", + "Z_new = torch_update(Z.clone())\n", + "\n", + "# tests\n", + "Z_reference = np_update(Z_numpy.copy())\n", + "assert np.all(Z_new.numpy() == Z_reference), \\\n", + " \"your PyTorch implementation doesn't match np_update. Look into Z and np_update(ZZ) to investigate.\"\n", + "print(\"Well done!\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "2c_KneQpuh8G" + }, + "outputs": [], + "source": [ + "%matplotlib notebook\n", + "plt.ion()\n", + "\n", + "# initialize game field\n", + "Z = np.random.choice([0, 1], size=(100, 100))\n", + "Z = torch.from_numpy(Z).type(torch.FloatTensor)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "fig.show()\n", + "\n", + "for _ in range(100):\n", + " # update\n", + " Z = torch_update(Z)\n", + "\n", + " # re-draw image\n", + " ax.clear()\n", + " ax.imshow(Z.numpy(), cmap='gray')\n", + " fig.canvas.draw()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "aXlR0iJjuh8L" + }, + "outputs": [], + "source": [ + "# Some fun setups for your amusement\n", + "\n", + "# parallel stripes\n", + "Z = np.arange(100) % 2 + np.zeros([100, 100])\n", + "# with a small imperfection\n", + "Z[48:52, 50] = 1\n", + "\n", + "Z = torch.from_numpy(Z).type(torch.FloatTensor)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "fig.show()\n", + "\n", + "for _ in range(100):\n", + " Z = torch_update(Z)\n", + " ax.clear()\n", + " ax.imshow(Z.numpy(), cmap='gray')\n", + " fig.canvas.draw()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "3HpYcyniuh8P" + }, + "source": [ + "More fun with Game of Life: [video](https://www.youtube.com/watch?v=C2vgICfQawE) and/or [Jupyter Notebook](https://nbviewer.jupyter.org/url/norvig.com/ipython/Life.ipynb)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hMvE8UoHuh8Q" + }, + "source": [ + "### Task III: Going deeper\n", + "\n", + "\n", + "Your third trial is to build your first neural network [almost] from scratch and pure PyTorch.\n", + "\n", + "This time you will solve yet another digit recognition problem, but at a greater scale\n", + "\n", + "* 10 different letters\n", + "* 20k samples\n", + "\n", + "We want you to build a network that reaches at least 80% accuracy and has at least 2 linear layers in it. Naturally, it should be nonlinear to beat logistic regression.\n", + "\n", + "\n", + "With 10 classes you will need to use __Softmax__ at the top instead of sigmoid and train using __categorical crossentropy__ (see [here](http://wiki.fast.ai/index.php/Log_Loss)). Write your own loss or use `torch.nn.functional.nll_loss`. Just make sure you understand what it accepts as input.\n", + "\n", + "Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) neural network should already give you an edge over logistic regression.\n", + "\n", + "\n", + "__[bonus kudos]__\n", + "If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! It should be possible to reach 90% without convnets.\n", + "\n", + "__SPOILERS!__\n", + "At the end of the notebook you will find a few tips and frequent errors.\n", + "If you feel confident enough, just start coding right away and get there ~~if~~ once you need to untangle yourself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_DL/fall21/week02_autodiff/notmnist.py" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "p1NcwbJLuh8R", + "outputId": "29382e11-0ebc-49e2-e14d-77b4315efef0", + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Parsing...\n", + "found broken img: ./notMNIST_small/F/Q3Jvc3NvdmVyIEJvbGRPYmxpcXVlLnR0Zg==.png [it's ok if <10 images are broken]\n", + "found broken img: ./notMNIST_small/A/RGVtb2NyYXRpY2FCb2xkT2xkc3R5bGUgQm9sZC50dGY=.png [it's ok if <10 images are broken]\n" + ] + } + ], + "source": [ + "from notmnist import load_notmnist\n", + "X_train, y_train, X_test, y_test = load_notmnist(letters='ABCDEFGHIJ')\n", + "X_train, X_test = X_train.reshape([-1, 784]), X_test.reshape([-1, 784])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "26anEwnwuh8V", + "outputId": "beffb493-6c1e-4098-e3e2-f5f3801b9b09" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADeCAYAAADYWw0uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYZEXV/z/V3RN3dmZ3Nue8pCUsOQkrINEAikgGBYmKCBh/vqKI4PuqqGRQQZIoKIISVQSJy7Kkhd1ll41szml2YnfX749zq6dvzfRM94Tt2enzeZ55evr2DXXPrapb9a1Tp4y1FkVRFEVRFEUpRCL5ToCiKIqiKIqi5AttDCuKoiiKoigFizaGFUVRFEVRlIJFG8OKoiiKoihKwaKNYUVRFEVRFKVg0cawoiiKoiiKUrBoY1hRFEVRFEUpWHplY9gY86Ixpt4YUxP8zct3mvKJMeZBY8wqY8xWY8x8Y8yF+U5TvjHGnG6MmWuM2W6MWWiM+US+05RPNI+0RPOIkFaPur+EMeaWfKcrnxhjdjPG/McYs8UYs8AYc0q+05RP1B5hjDHVxpi/BXXHUmPMmflOU77p6fVpr2wMB3zNWlsR/O2S78TkmRuBsdbaSuCzwPXGmP3ynKa8YYz5FPC/wJeBvsARwKK8Jir/aB5JQ/NIM2n1aAUwBKgDHs1zsvKGMSYGPAE8CVQDFwEPGmMm5zVheULt0Sq3AY1IeTkLuMMYs0d+k5Q/dob6tDc3hpUAa+1sa22D+xr8TchjkvLNj4HrrLXTrbVJa+0Ka+2KfCcqn2geaYHmkdY5FVgLvJzvhOSRXYHhwK+stQlr7X+AV4Fz8pusvKH2SMMY0wf4AvA/1toaa+0rwN8pUHsE9Pj6tDc3hm80xqw3xrxqjJmW78TkG2PM7caYWuBDYBXwdJ6TlBeMMVFgf2BQMJy33BhzqzGmLN9pyzeaRwTNI21yHnC/tdbmOyF5xGTYNmVHJ6SHoPYIMxlIWGvnp217DyhIZXhnqU97a2P4O8B4YARwN/APY0whq1xYay9Dhic+ATwGNLR9RK9lCFCEKFyfAPYBpgI/yGeiegKaR1JoHmkFY8xo4EjgvnynJc98iKjj3zLGFBljjkXsUp7fZOUNtUeYCmCLt20LUrcWIjtFfdorG8PW2jestdustQ3W2vuQIZsT852ufBMMYb0CjAQuzXd68kRd8HmLtXaVtXY9cBOaPwDNIwGaR1rnXOAVa+3ifCckn1hrm4CTgZOA1cDVwCPA8nymK1+oPVpQA1R62yqBbXlIS09gp6hPe2VjuBUsrQ/lFCoxCtQf1Fq7CamkC3mYNxs0j2ge8TkXVYUBsNbOstYeaa0dYK09DhmJnJHvdOULtUeI+UDMGDMpbdvewOw8pSev7Cz1aa9rDBtj+hljjjPGlBpjYsaYs5CZi8/lO235wBgzOAhpUmGMiRpjjgPOAP6T77TlkXuBrwe26Q9cicyELkg0j7SK5pE0jDGHIm5nBRtFIh1jzF7BO6bcGHMNMAz4Q56TlTfUHs1Ya7cjbmbXGWP6GGMOAz4HPJDflOWVHl+fxvKdgG6gCLgemeGaQPyZTrbWFmqsYYsMd9+JdH6WAldaa5/Ia6ryy0+AgUgPvh4Z0vtpXlOUXzSPtETzSJjzgMestYU61OtzDnAh8r55GfhUWjSWQkTtEeYy4B7El3oDcKm1tiCV4YAeX5+awp4UrCiKoiiKohQyvc5NQlEURVEURVGyRRvDiqIoiqIoSsHSqcawMeZ4Y8y8IJDyd7sqUTsrao8wao8wao+WqE3CqD3CqD3CqD3CqD3CqD06Tod9hoNVReYDn0LCZrwJnGGtndN1ydt5UHuEUXuEUXu0RG0SRu0RRu0RRu0RRu0RRu3ROTrTGD4E+FEQUxBjzPcArLU3ZjomWt7HFvWrZsqgdQAkg7BzkV4QArhme5JVa+JsrbHrrbWDsrFHcVWZLRtayaRSWawmH/awwTXdtZuC7FBvJdDI9mQJAJvqZTEhUyuDCcWb43J8fesThhPEaaCOBPHs7WFKbCl92k+0M4+fdfvI6o4N1ZLGfhXbAaiOyWeZZ9ZE2gkarfy4MS7X31wr91uyMdhhW204CdEoAPHxYqfYx3Iu29DYapK71B4muJGg7MYHyj67D+9cubJp9qgPzl1vi4C0fFAnNo5sD/LBliAfNOQ+cTxBnFq2NVlri6H9OiTr/JGJDPnGxOQZxvvJPcb7JgGoLK1P7dM/Ks+/xMj9Fhm5f2frOivfN8cl32yuFzsVbZHtkU1B/nH1bStpydke0TJbFq2kfrg8owF9agDoG5F0l5qEXDvILya4qHu2H9dXy+8LmsJp6wJMRO67fqTYoV8fKYN9o5K2MiPXdHZ0NFmxfY0tob42wepF9Tsuf+wATJE8q3hlsXyWuh/ko3irPAOztdY/dMeXly6kcZiko7RS6omKqHwOjYXrDVcHubzqytf6hJTNmuBze20JyYYG4mvWZ20P2AE28epmd99Ah9s9vk02JeXds36OlC2blDKTIE4dNSRt0khSdpw9TLHk66YgXyf6SJrLS+X5Vsak3FdEmp93UXD7zgoN1n3KuRqSUi9vbpL7bKqT7UVSzRHZ4tWprVDPdhptQ1aG7kxotRHAsrTvy4GD/J2MMRcBFwHEqvoz9qtXMePS2wFosFIhlpiiTiSjZ/CXJ2t47oVa7vnj1qXBpnbtUTqkL4fcfTpP7/I0kB97NNlE6NorE/J9TuMQAGbUjAfgsY/2BqBopqwoOeofUrATcz+SE0WiofOuSSxlA6tZyZLs7UE5B5mjMyfWvdSDhqiNx8O/77UXAB+dKYXns4e9BcA51a8BsJtn1prgngGWxeXHBzYeCsATM6cCMOFPYo/oC2+Hjo1W9Qdg480DABjwNUlLfNGSUFpdQV1jl3eZPUyRVDi2SRreGz5/CAAzfnwH0PF85PICwPzg3B+6fLA9yAdz9wGg/A1p9I14Zg0AifkL3Q0En0EjJ2jctFZhrbHLeZ/p6cuWtrBJTvkjE36+CfK4S1N04GAA1p0ka4xsOkYWTDp6YnM0xi9UzwRgUtEmAIbH5KW8LSl2mtMkL5QnNu0rnx9KeRn8D9mv8m/vyCWDToNrgKenJWd7RPty6NAzmPPDYQCce8DrAHyyYi4AuxRtBaBvRK5VZOT+5zXJNS/98Ez5/TOyUJjLT11BpELqiXnf3R2Azx4sZfGoShGp9ixeC8CQqOTlaPCMlsfFPtPrxzDj2Q3cesWC7s8f3YlXD8SGDAdg3TFjANi0e/BzTH4f+W95NiXPvCk/uHo1mdhx5aUr8O572YVSr0781CIADquW+uI7Az4KHZYI6otoUH/UBuXr91tk/YpXN0sZfeP9idS+NYv1dz/Ypj0kKTvOJn7d/PFFh6Z+m3FJx9o9rl525fevNbK43e+m7glAcrt0NNewgg/tW+mHdp09vPc7yUToa2z4KEnDp0YCsOEQuccDdpHFKo8ZIHXSYWULU8cMicqzLg3ua1HwOv+oUerjBQ3y7nlq5RQAVr4v34e9Jnmqz1PvAmlCTCtpfMM+3/69uXvIes+WtNbabvHGs9beDdwNUDp8VK+N45ahc9KmPap2GZJ3e7gC5j4nB22YyUH36+Q+swD4yWDJeNHDZYeFl8vvxzx/JQC7fUfaeIl10kgmEgFrfAu0aY9KU926PbyK1TWCG044QLZ/U6754pT75bNO0njRm2cD8N+XDwSgapEU0FidFOR4WXPhqRkuRcEV4u8d8ZSc4zMrAbhipVzr1d/uD8DWYG22j/aRBugnJ38VgOKgMZyxwe7dfosN2djDNTADNu0R3i3hq49Z4vIAwB7FZcGnNKi+UCHP/4bB0imIThMbv3ulVESnTb8IgLG3Bg3PV2X/5goqc6PYI7RDVvbIhFc5umcRmbIrAB9+XRpsdx39BwCOLf9nFietCH0ricoL7YjgUkcME/v8MvhMTJP7/p/vSCfixf+Vl2PfP00Pp9GGXy7pyQ59SbNHnwGj7IZpo1h84p1Ay5emn1bHXoGK8589/wzAycXT5NydaQx7ZTTSrwqAB06Q8nFYqeSX5sZO62mbECinfSMfU19c19ouXZc/upO0RixAzRelTXLtjfcAcGy51DN+46/pdNl/8j+lPO16uTQikrUtleKAnmkPr5yP+ql01Bqul+0v9pEGz8i3ZejtrL4bQvuvTUgD76RrrwGg+t6gvBhp+05OzmCNXc76Vq7cMik7wCZB/ndlKDpARl2uOeOxFrtGcpyq5SvIB5TKO+me/p8AmhvDGRZ465w9UvVTuP5uOlbeg2suljJ691RZT6RlOQ/fa8KWtLiE22cv6UewV/DOoUI+Ux2mPYNznBHUqf8jdeo//ng4ACNvfw9Is4dp0f5ok85MoFsOjEr7PhJY2Ynz7dSMHBZl2Yqm0CYK2B4llFFvQy8ztQdqj3RKKAMoTttU0DZRe4QZNiwKao8Umj/CqD3ClFBOkpBQUtD2yJXOKMNvApOMMeOAFcDpwJldkqqdkAP2KWXB4iaAYmNMMR2wR649xu7E9eySKZ/i4HvQMxwdE+Vw8fG/A+C2g6Vf9NQXDgagcs486tgGnbCHrzaZEulVzrtduoiLT/htaPfJ918KwISfiJo9bvusNk+frhu6/uqA38vnXxHl4tarTwbgratuAaDo2jdD53Bq3PYhUpRSNbPXI66kP3XUQGfsEWCT4e7u3vsuDH13Q86ZSHjKcjb4+cD1uPcMVMb5R94HwJZPSIP/gAeuAmDc90QRanafaH6mlfQHKO3yOsRT5SLl4tIx/3pxo3n9i78EYHBUXBucPRIpn7U21fwQztaxIDf5dnLbUyMrN0meHH/oxQBMuuKNVJor7UBIZm+PSNxStr5ZUXZ50VeSfHXG4YZpzZgRsmHO/LZvNheics1RsdogbWXh3zMoR463G6qxu1pgQ9fnjzzQ//KPgWZFuCYpPpTuGSSDZ+fyzeLjpCLa+49nADD05LndV152ACYm9+mU08aDZHTmuGA0JhHkD5cfDv6b1B+T7nk9OD5wK3IjbR0oL92Jf3+LvrELABdUNa9o31VukAMj8pZpHDsIgMjyFQBUmgEkbZKusIdv7+jEcQCsvkmu/fb+v/OOCCvCcYLR1yBfu9HK8kgxPn/YKu/a2xZMA2DDhmDUKJjHM2mUuOLdOfFPAIwrkt9/PFhcz274ptSpV3xJRm9nf3s/AGL/CbmMtEuHW1/W2jjwNeA5YC7wSCEvNxiLGW6+YRDAZNQeREyEXdgH1B6A2qM1IvLi+xitQwC1h080ZkDtkULzRxi1R5iIiVBKOag9OkRnlGGstU8DT3dRWrqc9AlB3vBBt3D0UcUAH1hr9+/2i3UzroferJ6G/S9dD3BLUpTAy/vJXMq375WJISuOLGEQY6H+1dzt4SnC0UHSAy5/TL4vniC90vWBX9lnvn01AOMeFhXBPWk3mcH3sXXKqom0oqAG923j0osf9kuZfHfc+6Lk/fa3vwZgYOATXBURRWPLRDm8fxu3NdAMA9uJ/OHsEiie0UqZSHH60Bmh3dobYcikxrWGm8DiK6DuHC4fuP2Kgt/nnye+olN3Px2AwSc3T0YL3YtlS1eVF1/NiOwjs5Om3vsBAM8Mcb61MoXfKTXOXq6+KDFynlzs1OT5/Prqj7tWQ1LStujUuwDYdbOMZoz54etu16ztEWmIU7aw2dfS+Qrnkm6ALVPEv7GiCwMw2ZikZXRMVJyW/sxts6RxYCp5O1V96pXRSKnktT2qVoV2c/nDt0fChuukf+0ndd2Z074uG174y85lj4DURNGAjy+U7wOj4UgGMxqknOz2q9UAuDEafzQsbQJXfu3h+QrHhg0F4GdnPNBi1xit531XN7jJuL5N/PLs1NWaUZK3Kl1SIoZYsgibtJNzvQ2HX4fWnSzzba6/6W4Ajgiin7jynKnOjPp1ajBad+Wq5kc180ZRcPs+Kb6+1fUyMlWdSky4HXDJgVJXHnfPqwBcVS2TMd275+bhMmp77k/lnbzm0B3nM6woiqIoiqIoOzWdUoZ7OuFed3aKxI7EAolkc3+kWb3unrSm+4o6v8iOqGDp+1eZILZv0Lv9/ehXAJhy5WWy488eyi2RxqTU2Ui5dEOrn5BzPzj2xdCuhz4kM42dIuxUmGQQaqW92fFtus66cFyBn3LRPyW01sm3fhuA978pYXKcTSv2Cc+GbvvkHSQVrizolU+VkBan9Hkh2EHyTab4lU7Fn9lQEewt3ebSIOZrVVoMyDGBQuD7eLmeftJT+spNEF/SxYkNfCLfOSDw87rnAgAmfznw4/LD4HQCX81oOFF8x355220A7FciaXMKgsvzvhbg7tXd4x+2DE399uByiQawdqvYrm+Z2OqcseLz60ZGmv2Pwz6xTglsIqyQPXLurwD4ziPny4b3s7plIZnEbNvO7EZ5ri4CSKaZ3JnYPEH2az2+Qwfx/NZzrdsW1g/uytTsOLwyGhkueegL/R4NdpB8kHMM8NZGsXYGPP99s7+EyXrp8FsBSFjx53d59UvPXA7A5MUy2tXCV7iH4fsKz/+G+Nae3OdZoPm9CM11gF8+X6iTkvf/5n4OgLf2eyR0bKru8Orc7cPleKcMS97rWD7x7bzlLJn788zPbgKgf1Sek1+HtuYD3Fpad33lHADGnjk3tU+fuNSdSVdXBHnFRWLysTOkcnz24iMAOOWP4iu8LCH2O+/FCwHY/cdrggO2Zr7hVlBlWFEURVEURSlYepUy7PdGTlvUHEz6zXnSY4uVBStm2e7qaf+/rPc0Sw3RS4q58k/iR/PrYaI+5upfly3pSpFT8rqLgUcHEV1+lsNBRnqFrnc6/y5xfVo49l6guaf8Sr0owBN/Ln6oiaBH6RThLllJy8U0doskBP7Hw38uPsSnfvYYAP4y4d8AfH3SiwD8ua/MIk5u2xbcU9jvqTM4H2cnOm/YXezg8omvJPjf939JfK7GnymRDSJ9Jc6uCWb+m/79UtdqGCOLiaybKtcYc7L4Zz068R+hc/p51eWxChP2y3Wz4w86V9LQ7/6Uj2yH8dWM+NHig3bH7b8BYLfisJrhK78uzU7F3uu18wEYfofsV/Rys0xb3CRxtEd6z/MfMQkEf9/Znwbguh9IHNnjyyXf+CqQ/6z2KhY7zbtY4vLytRwMkLTYhkb+ECwU8/Oh7+RwcDO145rCG7owz2aLb6eltdVt7d5j8cvoxoNFGT6wpPXy4uOikbhf790ssVTT8+LOhG+PeZdKfh8WC49DvBvUs7veJfWmG0do4SvcU/B9hceOBuDOL94d2q01P2EXacH51V722lmy74ogppFUYy3ixft5p85fpqCjoweRaIs69KEbfgFA/6g8J1dfZVKCHf47x7XBxpwm+demjQi2XHhInrqNtz6q6vYniGF/+afOk++bRAGevE7aT/H0yEXqM6woiqIoiqIo7dOrlGHXK3E9p4/+uEvqt8m3i6LnL5fY1SzOYV/b0Ejio0U8/+dgycYrpWfTUd9hX115tV6+/+jcrwAQ79P8uFcfLHZ48HyJjuD8KnP1N3T4PeCrxkn8yJdzOYkVla/uczKDdeFR0svelJBYpc5v6cIXvgzA5A1ir2aFsPtUBH829NbvSGzWVX+SlfjO6hv4mR4ufl9uOdUsV6LLDu+ZbDqgKcOOgq8slL9ZHvrd1osak3RlYXPzyqbRxaKEDv2vHNzwKznXpz4nPn1n/+xJAC6qkhGAbEczDrpCntn8PwZ5se1baB2nygQ2darMWbc9ATQrwr6a4afxkRpRY2+7Slb7Gv3kjPD505VRz5/N5QenXPX/gyjdN82T6BnDH5ZoEU75zRQD2HHmJ6R+urHdm0/DWmxDA88vDyaPB8qwry62x5CRm3K5anZEOqezrAyezc5O4xkbQ9/bq9v9PPrb50RZm9A0vXsS2F14ZTQ6WeY3PHn0LcEO4bjTnw9GrSa9Jys39nhfYa9e//BKWWb76DJ5fm3FFPbflePuC5Yi/2R4v/YiYCWGNngbEuQkhaYulEi1i/b/hczpmFAUVoSzjY3s39uqmyTMUjkypyY9glOuz9bfPzE/HF+/syNaqgwriqIoiqIoBUuvUoZb0FoY2cA/0ia6KbpEov1dfPp+3DWRB3xFaFsyWNXnDQkgGklTN0c/J/+fE78SgDmXS3QE358pW3wl+YCStTkdn8IY+l31cWhTxJuZPvilotS+8rkD+nRuNrTzW3pNYiMe/heJaLHwdIlhu/SzkqbJz9DlafPV6cN2XdDqfk7d9/27qudkGA1pZcW61Ize1Ox4OWfZE6KePvGe+LlveVJiun+rWnrpvrLlz6B2sSAPP1niNvPon1pPU1t4M/ZX3yLq67mV64H2fYRv2xyslniKRIgonRfMXHejRik7p5XLpFOCvQIe2M6tcpd8XfLFFx75JgAfnX1HcKZkkAaxh1++LqoWZTkXZdgmkyTrG9i0Mqyi5jqytO+g5QA4nSWlemXjr+mcQb1n0ll/403by9rfqScRCSuFLmrCE3vfGewgSlumWLN+mZ3bKKNhk+9eJ793fYq7FT/KwoeXS6x4P+LJqmDUb+Kd4Tvs8b7CnuL98Oec4i333dpz9uuh69fLKnyx50WNjZ90cJuX9lcWHTkkPKJjE4kOCcMAS78n9fmzQ6Qd4Neh7eHf29+3S33Y92WZa+Kerv8OA5rr0JKSlr+1hT/6lGzZjjL12ftRqzKsKIqiKIqiFCy9WxlupZdkE0HvwVd48ojp5sXxnBqejlO4hrwVdtpsb/WybBkY7YCyU1FGcure3DfexfAVxc+t8uZWnBvwRqCWuIgPrfU2uwl3LacQT/6BxDr89XFjAXjyBIlkcJURP/CUb3rzimu5461qFRsh/mkXDnk2tJtTI/wRghfr5JmWzfR66cEqe62peJn8uVzc5fgSUe+f/cY0AC69TxTiiog8M9/33B9xWHFckOld+NVsMGJ3l7YNFx4CwFv7ifrq+wi7NDi1YlajRI34x3mSZjtPZjh3ah5BYLtkfdh/b+zfJfbv7NPks8m6chXeL2Lk+M3JDkb5TSYoXuevYhb2FW+PaVUS+3MhEnGneXZ3LpnVq8SKOvdqSSZ3Lp3Gj5qw6Gp5JiNj2fleurjvLsrPif++AoDJ88PzIjrkY7+jSIsUkIqyMH4sALefdG9oV1cvHPemjBCNDEZUdjZf4blXSdQdFy2kLUXVH7F56K9HATAamS9gY22XN//dfMBAmdfxQSd8ZU00QrSikq+eFn6X5BrJyr+3+1fL+y+xbl1woVZGcQM12anr3376sZyumQ2XfnZ71vvuXDWOoiiKoiiKonQhvVsZVoBmNTylBKZhMsT06ywdiZHc0C/Cks+WtVjtxqklr9WL3xlr1oUP7I7V3jLh1OjAp83Wis/bc2eKSvnbk04EYCSvt3pch/D8MRsmS+zSaWXh+06psKlVj+QZ3Ll6GgCJDcHM9k4oCX7cZefvduhMifAx68CHJQ2eEuyrGp+ZKrEib8vp4oEiEyhQx17+auhn/xq+Qn7WrVcBMPzNILJMoHK7e+oU3kiT89P/9idPz+74uDv+1zlfOlbbuZjpny6X8vTIy7JyXywSzlexYOiqKJJosa0kIgpZn5jYsDwiZXZYcTiOTCZf2UxEozuwTHeUdCU0UAoT0/YF4LXDnQ9pHyB7X+HFTRKdZtdbRNHq8fF20whFCggS/tFFw4DmuNtOIXf2GHpbaegcPfY+PV/hyF7i7/v08a68yjureXVLIX3VVzcqML1eytH4ByQSj9PATbLtcuxHojmqSuqYORVSblOx7XOgqbqMNV/Yg6uqXwqlN9f3tz8atXybxKyvQuZxtDkPISa/+e+zrqBvDnKvKsOKoiiKoihKwaLKsNJjKC1vZNIBS1Pf/TiLL2+TuNGJLcGa461EQdhhOCUwSEPyPfG7HPle11+quVcdRE84MKym+P6Ifi/9jfcl1uNkgqgJ3kzvriD6z/7yj4SIbqHS+qrGkZWyemBOynDAhi/LRW4YIr7C/kxm//u3Vk8FYPhNb8gJnMqTiyKcY15zozDxxUvb2bPzxOo7d7xTJd1qit1BrnHLI5GdQBluhfrvbgZgYFQU4Vx9hY96XqL7TH7Pj6HeM31ogRaqKUBsmIxeXXdKOFqMs8Phsz4PQJ//yMhSSmXvQXN50mkZV1h8wTOtculwzxean/GZL0lc80mL3gpfJMcsv2+xqK6RalFhO6IMJyqSbDmiuQLpaDQpn621wXwftyFV/jM/30Q3jPDaHCbpqDKsKIqiKIqiFCy9ShkuM+Fe2X3fvin1/7KrpPdU5PVMkkF/oNRID/7jpmoA7vnmKUDaSmI7Qw99J6dfUS0nD3034++Lt8vMXaz0iHuEmuBUWLdCWaSlStLV1O1ZF/ruK8G+v1e/WV1fzP0IHoPeE9/plip16ysaHlC6ssPX7nt6+Fh/JrP//V9/EH/uIcnAV7gjZbmTcXPbpRPRRqL1nUtbTVKUof1eFcWqqT6wT12Qb1wo4USaOu75NxqXBZuC7cPlnAs+KVEEsl2h0BE1PdR3lNbzz5azJUbs9L0krrC730yKsO8r/HFcfIV3uUXKtrv7HutDm0Zrq2wuvnA8AKf3lSgFvnJq7grmf7BoB6Wyg2SIH/3fY5yvsCjEmXyF05ViZ4Px97X+TCON4TLVXmQnF7GpcfRA2X/psjb3b40BZds5a8qMrK+ZLU2Nub9zch09ygaTbUgdVBlWFEVRFEVRCphepQz7PYu9ikvT/m/dsc5XrtYWi4/fb8u0n7Cj6RtpYFr5R7gZ2H4vtaYpvEKNH9szr6RWKOvi8xrTwrf3hF3mhL67lYn8mcBOpR0yQ3ysU2pTV8Rl9m40tmYLANsC9aMk2vZa9lWR3KONmLJSIrvuzl2TfxtsCc/Ud/fv1LhX6+X7iL8uAZpnbXfk/l3kCbdKku0ipdg4n/MgKkmufoMAsbr292mLlYE9xn5plkuUfOZyj94xLnYon5SPXFfFi/XEaBJ+RIHS5vfLkVdPz+lUvq/wkf8OfIXf2Xl9hSN9+qR+uvLMx0O7OoX00/NPkO9/eyN8rp7qK+y9YxZcJc8e+XpKAAAgAElEQVRldBA/OpOvcGu+t6fMEz/p6Atvy7m9aDaRdmJIuzaKX8/XjJLzVGZ9V830jdTzyYq5qe/+3I6OYq13HhdppGc+ZkCVYUVRFEVRFKWA6VXKsE82sxP9WKT1gbJhEj3fV6u3UWwMI9tQFGsapQfcJ+MevRs3Q/uiQU51EWUq08pzf6mR/SPzZbW4VKe8K+RrL/axLZVnU5TB78sfgXmj3ukYK7K+ZGNllOXH9GNykeQA3w/VKeFOjbn6w9MAqFqxQE7Qno+5p266WMoA41+R364e/LSkOiHKUHGOUkciUF76RUQNeq5mDwCeP0k+WZLT6QCI1YXrKj8KS3tEPUdlF20kJyLhCB3JvqVt7d0usWg3SEh+RJAc1X3fN3bpNfumfntmiKya2V70CN+X9K0GURZ3+7mMrKRWh9yBq2p2FD8qzYqL9079dlGVxAD3ldN194wFoB+rgnP0UAXc8xW2h+0DwPRPuPg3Ugf5vsKO1uJKb7p/FAD9WQ6k5Sd3yaZw/oxmiGDjq87bh8lnR5ThskiSKcXbcPfTHX67OwuFe+eKoiiKoihKwdOrlWFl58LQ9mzzFjpOd/Zi24sr66tKue6fVRpEPXDqRM3+o4GwL3w6vlJ615IjACjbulh26MboG/UjRCmtipS1+ruvWr9TNzb4b25ru7dORQIO25x2zrAfqh9VY8v0wZImRBn24zW3wFO7zR4TUz/9dOjvAOgflfscG3NKd26+z83KodjptjpJY7wDM8EdsfrOKf0Jz08wtVJlR3yGAyLbOhf8uEuiSbQT4SWlSrqIDZnyhacSxsZKOfz1+b9tuWs7+pLvK3z6o98AYPzc18Np6mlKaSs4RdiNoJxy3n9b7OMU4YuXS0SXfveHV+bcGe4TYPk3JZ0ufnQmX2G/Dv7+mr1Sv1U/JJGpMs3daM9n2OHXc3VDOl5WDFDagdVi2z/xzjeyrsqwoiiKoiiKUrBoY1hRFEVRFEUpWHq1m0RWzuDeZKJSFy4mmselfguUJptkTaKOkUHYGp+KIm/54O6MqZarW0O3LMpgQq4g6/YKF1d/qC7qDXevmDsEgImIm0S7bgK5pMw714Y9/GVIwxOJkjZ8zVc3Tsj5mlUl9Zw4tjmsnD8k7U9mGTDHu8928ot/T2sPTC0mSv+oLLu6KVEbvpb32N3waKYQRf5iDM/MlGFUt1R2R4ht7/mTrXIl1pnlmFMTIeUcNi4PyYX+MkXy7BKbt4T3zxBSzs8Xc/6flKtjy5vHtdubOOeeuyur92+VhRIm/XKhpCXYb+eYOBd25Vjz1f0B+PGg21P7+HXTzN/JBLSB9HB3EM+VLPFJmSQ58+A7gx3kfjJNnPNdt/7+58NTv42IB4v+lJT4h8mls3ST8EkMzWFZeY96a5jTFOXAIEmZFknKlYjvJrETLB6jyrCiKIqiKIpSsPQqZdjv1cxqbJ7EsSye7XLMY+QcdT0w6Hsvp8YW81r9CE6r2NLq72UxeUZujYHuXKrUTQrxF7zI+TydUkBsSM1M7r0t9KsfesdXSgd7ce27Ukn3FazKY1fndPzcN8blfM1+0e18tuptXB/eV199NaNiqai4WS9t69ln02EtFRenCPkTZ7KlyAu5NGBm5yevxOp6vpq4w4hEU6peZJ/dAVj6A8kX350iSwMPjkk5umbWqQCMuWY7APFFS+QcKaVYjnN1QGKaqIRvHO+W4m0O8thaKK22uOl2Cfs3ZE0nlgjf0XiLbLg6cv/z3muxqysfN6zfBYBB974lxwa/7wwKOMDGb0recPfT3sQ5NzLw1xoJdDb6ruYJwin1vzGYeOiFMIx5c04zTcb06/2RQza1ex+Z2Bjvw583HsSBw2QhEH+ic0eJFe0czzcdVYYVRVEURVGUgqVXKcN1VnpcFUZCT533f1elfht8e9ADz1LxK+HN0Pce3WPvJWyOl/P4un05reIFoOXiASPLJazWR55PYIeWjyW8qILLDwtuOhiAmV+8CYDXGqoB2JaQUFjv1MrIweamcAixiqioiJPK1gAwqngDAPsWrwfgi3POlR2PyyGBNqygnL1rOE865cBfnnNVvAaAfrPFXs6KXaKkez519hAJtP/4Hs5fMLxEssOlzSkoo58Vey/K4dKlxrJLUR1+gHh/RCi12E48NyXcV6vG39v8/6FPXwJAJB4Oa5QM5haUbJFjl35a0rDo83cBzffrVGynHK1PiOI06A1RdTqj2SeLVNNIkUwQP2o/AH75e8mT+wQ+mi5fOPXrg4MfAuBbf5kKwOxpouYltsry5SYaXoq39EeyUMTgILyW8xOGzL7Cvi/xifNOBGDIzfI+8tXWnoy/6MimM8TOz466A4CaZLO0WRGRd/DD9x8NwPCmHq6Ae/Va4/EHAPDmfne5HYDsfYWveUmU/11q30/t40LyEQ/qmViwWNLqtbK59aiULZPqaZj7DJBFPOZFojkvd7y5vozH5+zNL1PKcG7LpmeipMh7vt05v6eL0FpUURRFURRFKVh6lTKs7Nxsry1l+ruTYZwow34P+IiqDwFYWCEz8JPbAh/a9ha8yEBrfmv9PpRznTrv9ND20RWi4N07+uU2z/mnbf0B+PXST4W2r5s5pENpxFqik8YDcErlQ8FGkRCc2uj7ed27WRSb5Acfeefq/Az91AIGwakafyz+3QM9tcwpYX4Q+hvW7ynfX/0g5yREMJS3osBlUojrh0sEiJJ3aDXtLfBGFqIvvp36v2+Wadz3/w0MfW9WWiRN7hk9uFWWX07O8Z5RB0iUdkPQ/M4SzYPOYkR53O1/JW85RXhLUmYZuIUuHE7J/PlQySAHfv5SAPr/QSIeOPVy+fcOBWD25OyWXIbmvOj2cVFImn4k9UCElUGawwu99GT8+nLEhQtC350aDPDrTWMBGHWv1Nk7U7QMgKYrZWTP1SXtPXN/+2vHil95Iq14ZyqlziJ9zH+C/6TeyrQAlT9X4piq2QAsqNgfU5NbuYttiTD46RISR4Xza674fsxVZWEH6O6c39NVqDKsKIqiKIqiFCztKsPGmFHA/cBQxLXtbmvtb4wx1cCfgbHAEuA0a23HpzXuJNTbWmbzJg3UYzCMYByjzSSabCPvMx1gijHmX/REe6TPtndKRKRz8ZSXrWji/CvWsnptnEjE8NWzK7niq/1YvqGO8y/dADnYo3izZfSTSRpOdr3wcPacViZqyh+GHSMbUspwB9WVVuLtDrw7WC707vD2QMfhO+9IvMzrBov/br23vOqdX/wEHyx6jIZtvwvljxF2YYfzx7YpgwDYo1gUYV9tjad8F+X7X5eKH+/A5Hygk356GZa0XfAr8a1eOOXOUJp8ZeHjFY1c+I31rF2bJBIxbBwxhQH7HcHA+hdztke9tcxvsuyTISZmPNBYokEff/2ekpYRTwUnyDV2ZqRZmTFeOTFl8izc6MSyH4h6OGeCqIe+Pdws9NUrk5x/xVpeW7oZjGFsYjijzSTiMcus+KvQgfwRL+ucprE60af9nXLEFmWvVi9b0cR5V6xh9doE0aAO4RBo2loHMMkY8xFZvGMS1X3Y+NkD+M3w2+R7IEZlWiLcV9hKN4XrA3uolKO/XfLzYEvr/vCt4S+7vO9TVwIw+b8ST7qtMtnWO4Yc7NFleP60zif70QmyHHVDK2X/9/eKb/TwDZ33Fe5We3j3Vv+ZAwF4Za/wCyBXxXRYhlj5XcGyFXG+8o21rFmbIBIxnHJGCedd0IfGAeVs37qNXOwR3bSdqkdncvU1ct+/HjYTaPmOyZXhfWTEcIPb4N61kR44ihWQTS0aB6621u4GHAxcbozZHfgu8Ly1dhLwfPC912MwTGIvDjXHcQCfZDkLqbFbWcKHVDMY4AMKyB6xmOHn1w5g9stjeO2pkdz+hy3MmdfITbdt5cjDS6DA7GFMhMmjj9P8ERCNwY0/7J/KHxvffZX69asL1h6uvIz9zdcZfeNXU/ljcWIO1ZEhUGD2ALHJ//1wAO+/NCpVh2xfup7lf54BsE3fMc11CGqPgrdHLAb/98Pmd+5D99eyYH6cxZveIBYpptDs0VW0qwxba1cBq4L/txlj5gIjgM8B04Ld7gNeBL7TLansQZSYMkoCn82YKaLc9qWBOtaxkv04kgV8ADnaw3a3s4pTtVoRTreO6piPkGPYkBjDhkg26lsRYddJxaxYHecfz9Xyp0eq+dGNQJb2MNvqKP33e3xrlahsvxomgXKdb5+bxb3+kMEA9P9IYhF05cpqKSU06j7l4SQbRX3dFhd7OaXA+TW7HnSpLaU0Voot2kYRxfSJ96MplmBd4yr2jx3Fgnju+WP93uHetD/jN2HDEQ5q33R+q/OzOX2zP3C0Za/dKTnOz3bhLwJF+EthRdhXEJyP3dihZYwdCt9fI37eVVtmUPXQTD7sQHnZEK/ggY2HsE+GmJi+j/kxXxQVbu7/Bqp24K+YtUoVci4OrhKoy04RbjhJZp0/9dX/C/arCNLix0CW7668RG/uS5S+lBPUH8nl7F90DAsS70GO+SNR0rnRnSWNA9vfKUdyUYaHDYkxeLDY1dUhi9fXsOG1BdAsLrVrk3g5bNjHtvDzjHr5wvf/vOBjWSWs7AnJL7ERwwHY/w5RySYXSb2TjVrmrzT3g7XiI7/b98R5NBvf2bbeMeRgj67C97VffH5Q7ozzg5fPb62emjpm5O3vAmmRbDrhK9yt9vAmEBRfuSr0PRv/cEiLYOOR9JeobANXZ7S3+pv/zt1tUhF16yxrGhYTLUsp0tnZw0o9+Modh8j362YG6c4tqoRf906rngfA45UTgbQILWkjbD0twEROzTBjzFhgKvAGMCRoKLsG8+AMx1xkjJlpjJmZqN3eudT2MOrsdraxmSqqaaSBEiMFNlt7NNHxZRR7IkuWNfHu+w0ctG8p69cnGTIkaKRmaw9b39ouOy11toZtdiNVkQE0Ul/w+WPLiu1sS26kKjKwQ+WlblPvskfj5o1p9YfmD2iuQ/ruOozGTbUATZDZJqH3S03ver9Ay3cMOdijN+aPXO0BvdsmS5Y1MeuDJvabWkxjYw2RSI7v3F5mj86QdTQJY0wF8FfgSmvtVpPlDH5r7d0EHpilw0f1/CmFWRK3cWbxOruwDzFTRLYdwHR7VJpqC7BtdDdLw24mZ1pXLLr7ZAD2u/jdLrnElpo4p16wipuuG0hl3whNNsJ/ascCba9M5tvDNjUy/RbppUZvFL/cpmS4C7n+aCnA/e9reV+dxsXPdZ+JsE9ZpL0iE5Fn2dRYy3u8xOTIVCJN2ccSbS1/lE7d2OYxvlJV/WHYHiaYVd/sWy2/p9SaQFkOqTfBtsiUXeXQW6Rnv2AXUYSdYuL7TzqFxG13+73w432Z/cIdTGZvoo2yj4nFgldZZtLtUTJ2pH3snX0zxsT0Yxk7/7fxv7kYgElXyEiDTam8niLu2yfteTXbKoixep7k0d/9+FcAjCuqCF3bfya+sjTo5c288+49ofojG/WstfwR95ThTCtXZWJ544Cc9s8G24FoEjXbk3zxgtXcdN1Abu5Tkt110vPHuJE2Wdn8zPwRk0zEImL3zefKM/3St58D4KpqGXnKJXqEe+4ugsWr35WRlOJNQYxwz0e1LTryjmktf3QYLwZybPxYAO489IHQbi7PT//Jgalt5bXB8pc53G97dOU71x8d2n7qQQC8sttdoWOz9RXOpOZ2h4esu5YrL7++bhCjqvpAZBPJovbbZS3sYQwDfidzZfb69BkAzDrwYblGMCqbHimkNfyRsLP7LgHgb2OPkA2ztmZxZ/klqxrLGFOENIQfstY+FmxeY4wZFvw+DFjbPUnseSRtklm8zlBGM9iMAKCYEhqsVICFZo+mJsuXLlzDGZ+v4PMnSaOgakARm9cGy04WmD2SNpGWP0YCmj/mvXY/A8dMTSsvpQVtjw9m/5EhQ/ZWewS4OuTMtDqkuH85QBEUnk0yvWNQe6g9kPJy6gWrQuUlVto3JeIUmj26gmyiSRjg98Bca+1NaT/9HTgP+Fnw+US3pLCHYa1lDjPpQ1/GmMmp7YMYziqWuq9Z2cOUFBMdM55jvzQ9tD1XZcfhq1GHlMoKZC/NCHxui5t7Z5/vew8Ao71Zr7nOHrXWcuFVa5k8KcbXLk6LxrrfXvzyntQs7uzyRxAntP990ks98LQvAjBj6qNAs0Lz6CdEnfzhiJMBiK+SVd+6UoXoKDaZZPbHf++S/IExmKJiThozO7TZ5Q9fhXwp8DKp+q8oWs4KydraIHGtSynRgaIM1hw2IbVt3dnSMPvPQbK6lJsd7SvCTqVwaXFqrVNU9vr15ax68mEq12xm0trSlCIzqGkYK+ML3eWyskfxRhj/sGXTcXI//aPlre7n7OHUukWnitqz79gvybWvD+LNzpwDpCnA3kqGkdJmNaTmBIksYC6R98uMPe8I7jscLcL5ijr8Z/Tw1n7c+e3FVNQOY4IZSQKZ7D2IYayyi91hOdWnTRWd8xle2dAv+K/rhkxtLPs6zFrLRVevY9dJRXzzkv6p7dWHTGDFozOdbN2+TZJAY/N1/dinDl/tu2tkEEHmZ/Lp8k1TkA+yUQf96BH7Pyirn45/Vs6ZSzSFtt4xS5mfvT1A6pBY6+lvbxVWd5zbb8mXxJf62PLwcM5uD18OwIS/pb3HurAu7lJ7uHN6ozDVX18a+p6tr7BjebDq58akPOdoDr7CiUBVHRqVNLm5MZlw79xdJhXxtYsrU9vL99iDrR+kYqN3qE028gJ5l172rIxo3D5CnqlTiMuC/O0r4b6fvqsHPz5JVnAdOUv2c1F4AKyLBtVDyMZN4jDgHOB9Y4wbU/8+0gh+xBhzAfAx8MXuSWLPYgsbWM3HVFDFdPsvACYyhTHskgoVBWyhQOzx6ox6HvzLNqbsVsRBx6wkYgzXf28Aw047iQU3PA4FZo/N25exatP7zfkjYZgY2bNg80fd8sVsmT2ThLNHk2FidK+Ctcf8t2p45YkNVEQtr235K9YmpP4wu/K+LTx7gNQhD/2lhim7FbPvMR8DED9jEaNOP4gVj86sDEJF6TuGXVjKfLUHhW2P9HfugceswBi4/nsDGHDo0Wx+63UKzR5dRTbRJF4hs9fV0V2bnJ5PPzOQYzi11d/240j+bf/ygbU2K7vYMZbEnY0pH8hce6Pt4eJr3jBkVovfElZUtc7GEzzkwBIaV45PzZp15/nxd0azy95XMPO9q7O2BzZYqSZQFQZ+RWIV/uA5mZF9/WBZ532fYlFu5vxIhssmf1WiADvf2FTY3S5QJVL+pBG32pt8umfl/BLdffevGM2x+/wPyffmBsc5f9REzvnDlJZgJo7nourfBVtaj1TgqDSi7H34w3EAWDM2vEOZ2KNPlfTyJw8UlfPqkeIjeXDJv1K7Nq/qFs4nft50dnBqtft94sOXADDpj9OZZE4NK2NJwOReXsy2WmLPv8XhM74KwOxDZEW+TH5tvmr99v5/lv0fk/1v3ihq71MrZTW4TTVyr5Xl8vtl419MnevcyvDojVOCnfrolBA/9rHv1/zSkBOZ9vyJxIJGnylqtst+5oic7OGoG9I519CPa50a27Z/f5t4ow7x8uwXNz38oDIaV8oqi85u0z4Y736eb63dP5vzRGsNA96Kwmfle7YjbO6ZuTjVfoSYTPvX2WZl1eW93V49B4Dx3w3UZjehKYf4um29Y7DZ2yN1SDsKcAtSvsJB2e4jSuW5Z/4rtNuE578MwMRrpGy4Mi7HdiCmeQa61B7B6KNL39YzRQF9fZKMNvorB/r45XtWo9QV3zn5ItnhI1GYTVlQF2UYjQMwRXKN+GpRYz+6WfyW3UhWptGmww8qI7FqYovfL99WQtHQgTQsXT4p40Vbwza/cxMbZH7K0pNlFODsR6cB8ODYF4Hm+j6e4X3gl7lvnfcXAB79k/jjxxeHFXgAG+3cyFZXoSvQKYqiKIqiKAVL9t13pcsxQDTSPPO/o77C7eGrd/K/9MY6qgg7tc2d2ykj4//9FQAmPfB6xxKbTDT3UteIcvnO58YC8LW/iHJ36wiZqbzgROlBT736awAM++VroVOlKxXptLZOur/CmNvHV1VigWqS6hEHhzXYsB9dVxAvj7Jx3/4pv25flfDZJ1DGF51yV6u/Z8apwM150akObhXAiKeI+0rw/CYJa3X6jdcAMOHO3H0ls2Xs1eJr9tBz4i54Vl8JNZrJr83lcZd29/v3B84LfbaFy+81ySCSSeCv7KIG7PoP8Zu845j7ADi+vCF0nLPT8jdkNGMsogznvCqeTyRKw+DwCEgmX9lMLNssPsODO6MMezRV7vhXS9GGWgY98A7fv0xiWrsRsWz9Hf14xA5fOXakj0RMmX4WAGPOkBEhm/I/z18wVROJEKmoYMlVMrI29PAVACx/W1S/CT94C2iu40xRWIF02z/8hYycPDPgVQDG/V1U0MmXSFzmlPrdiVjCOwwbrovGXR4u+/4qlj6+b/gpr1wKwMR33wGa67vk9vbD/Pn2jtXkVhf45bzP0O1EijqY37wV4uIrZLR1w1GSx8fdeiEAH54gcyVcfebqN3+E2X0/v1Le4Usflzjbz//w8NQlXVxvk+jcyJbDL6cxotic4jwriqIoiqIoSoGiynAB0BEfZNfLcr7AzvfRRRFwSqFTR3Z5+VwAJn9F1JhmZaQDCfZ7qUuXAbDwKIlWMeF28VVbePS9AMy6+nYAJu4r2ydfK1E0EgtSs/PbJZOAYw4QVWXDtaL03Tz8EaD5fvuUyXbnj9qVxPvAuoOaE9aeauFwSphPppGH1pREf1sm9ezUhccAsO3qYQAMmtF9ijAAkSjxJaKqPnDWCQDUP/BvAC6oEmXTqRJxT8X2YyL7qobL4wnn/562v1PInSL8z1opUz/5tsQw3v118YUbf4KLCS1+lv7Iy6B3vIzWGeXQGExRjNIBdaHNmZ6zX6ZTMXGXVwEZIvR3kKbycP5xvvUJwqMbfpqcteLJ3HUaay3J+npmXCUupDPuEeXzwBKpo1LP2XvePs5+vm++q0dd+ZrywBWp38Z/T/xmW1R3bfiMdjf1I8uY953dWfR5qR/d/bO7fBz9uqiaZY8HCq8XX3v+7RI3ePZnbgVgz5uuBGDyL2QELlXGvXjlOwMupvRz41xUmNZ9YB0unzr/3NQKm/dkyKfZjM54o5HRhtxGdPxyPmXIKlYXdXKE0r17g/S7lVcnXygx2088WuaCjP6prG567+iXg7S0vnqes9O1gyRyz/dvfz/12903jgXg5U3rgJYjn/6qfn5bxOFHNnLvqE2J2lRdng2qDCuKoiiKoigFiyrDPYhc1wPvTpp7Waktod/v3Cy+j3ffLFO3xwY+ojbShWn3FOJkEJdw4jninzXtJIkqEP+6+Iwu+KQoxVuOFKXsM7PPBGDTv0W1HPCB9FKLN4svnC1u7gvWV0uPf+1+sm3PI8XH6aYxohzcvVGUhAN+IGrK2HvkfiN9A7X6Oukx77J6UfgeOqH8lZY1svseH6e+Z+tT3t5qQR3B+QSf/cH5AJTfKn6mJc8EK2sZeQbdpgg70nzK7cwPAHjskxIV4vobPg3Af47+NdC8KlwmX2uXU331trX9FzdJHNGjnpL4sbv9eAkA5WvEfz0xVfwqx8XCtndK07sNMoJQ9bqMcjjrdMbP0kQMkZISDhjZcoZ2a7Qs08KoZzz1pAsUvoZ+YVs7G2fy1/XLycCyDi6tHIkSfUGi81z7GYnssPUXUt6f3ONBID0+ddvRIlza3mqQ4895S+ZDjPo/UcHGz2ieF9ETVVLTBKWrmu/RqZ9O3Vz2abnP3acPAWD9sRLBo+Jc8S3eJSKfX/j0+QAMfzdQhANfVxdtoifca65Muez99ndKwx+VO3HuFwCI/UdGH/zV+rLCm7sS8YJ+tOf7749cfKp6DjOirY8K5kzqmaatFgrEnpf7XfWS5IEjThL/8VWnSeJ/e9D9AEwrC48kO9Lr2sv7LQt9+tpspvo6U7l9sU72v3D6eQCM/Z1h0eLbWt23NVQZVhRFURRFUQoWVYYLHKcW1AYxMzcGysYb9aMAeGWrrPjz1PviOzv8WckyVc+KD9CgrZ6PaHcoI54fk5uBX/LUm8GnbD5+/7MBWHC6KILHHilrxJx/mfj57l4k56kP7nlNorkvuDkpkRge2yT+hn/7YB8Azr3xm3KNp+Va1YTvN6VWf1N8BlvofJ2ww9DiLXx79DOp75niC+eKi4DwQaPc87v1YwCYu314ap9nPhTHwuoXROkc9IrMCu7/0QLZwd2XNxLQbYpwOr5P+dr1AEz+SrB60hSZ+bzwbImfu8/h4t923lCZDX9QiajYfQOFbF1CVNs36uX+H14jvpLvvdocrnPig7JS3OQPxL8y4d23WSAK/lFfv0w2BOaxQRYr3hbMtl75VvheOlNOgpnxM1eIordppKzMVxmMDDjf1m1JeSYrE3K/79aPBuBfG+UZl/9b/Py7Mu5BtE7u6/m6QBEOdJfxRbIK5pCo5D1n+w8aJTLI05slEsSH/x1Ph0gbOUjOlkgBFcdLWr50iIzqLD9K/LkbdpNy0K9KVOjaeklT08fye7+5Ut6G/Gs5AKOWykiEq4dCcXV7kCLsKF61nVE/fY09GyVPXnS+VJRnVErd/e7xtwBw3dTDAHh8ruT/4ttlJC32mIx6WBPEcXd1fK5xi3sQ8YF92HDKIfxt5G8AWBUPxwvPhPN5HxiVfLz1jzIyWh1EhXHx6DtT/8Vqw99d2Sg2bdu7KHgv7Fm6jDJfXu4swX27+/Lf887ffPzjsvvPR8o8ju8fLu2HdftK2obtLfM5jh82J3XqMSVSb08qlt+mFIV9gpfF5XNNQt7nr2yXtsg/V+0mvy8eBMDQ/8ozqX5xCQATVr2Tln7PqG2gyrCiKIqiKIpSsBi7A3uyxph1wHZg/Q67aPcykJb3MsZaOyibg9UeYdQeYXpOSVwAACAASURBVNQeYdQeYdQeYdQeYQJ7LM1wnp2RTtkDel0eUXuE6Vx52ZGNYQBjzMxcl5PsqXTFvag9uv4cPQW1Rxi1Rxi1Rxi1R5iuupfeYhO1Rxi1R5jO3oe6SSiKoiiKoigFizaGFUVRFEVRlIIlH43hu/Nwze6iK+5F7dH15+gpqD3CqD3CqD3CqD3CdNW99BabqD3CqD3CdOo+drjPsKIoiqIoiqL0FNRNQlEURVEURSlYtDGsKIqiKIqiFCw7rDFsjDneGDPPGLPAGPPdHXXdrsAYM8oY84IxZq4xZrYx5hvB9h8ZY1YYY94N/k7M4Zxqj5bn3SltovYIo/YIo/YIo/YIo/Zoib5zw6g9wnRLmbHWdvsfEAUWAuOBYuA9YPcdce0uSv8wYN/g/77AfGB34EfANWqPztljZ7eJ2kPtofZQe6g9eq5N1B5qj/b+dpQyfCCwwFq7yFrbCPwJ+NwOunansdausta+Hfy/DZgLjOjEKdUeLdlpbaL2CKP2CKP2CKP2CKP2aIm+c8OoPcJ0R5nZUY3hEcCytO/L6XxhzwvGmLHAVOCNYNPXjDGzjDH3GGP6Z3katUdLeoVN1B5h1B5h1B5h1B5h1B4t0XduGLVHmK4qMzuqMWxa2bbTxXQzxlQAfwWutNZuBe4AJgD7AKuAX2Z7qla2FbI9oBfYRO0RRu0RRu0RRu0RRu3REn3nhlF7hOnKMrOjGsPLgVFp30cCK3fQtbsEY0wRYvSHrLWPAVhr11hrE9baJPBbZOghG9QeLdmpbaL2CKP2CKP2CKP2CKP2aIm+c8OoPcJ0dZnZUY3hN4FJxphxxphi4HTg7zvo2p3GGGOA3wNzrbU3pW0flrbbKcAHWZ5S7dGSndYmao8wao8wao8wao8wao+W6Ds3jNojTHeUmVjXJS8z1tq4MeZrwHPILMZ7rLWzd8S1u4jDgHOA940x7wbbvg+cYYzZBxleWAJcnM3J1B4t2cltovYIo/YIo/YIo/YIo/Zoib5zw6g9wnR5mdHlmBVFURRFUZSCRVegUxRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVh6bWPYGHO6MWauMWa7MWahMeYT+U5TvjDGjDXGPG2M2WSMWW2MudUYE8t3uvKJ5o9mjDE13l/CGHNLvtOVbzSPNGOMedEYU5+WR+blO035xBhTbYz5W5A3lhpjzsx3mvKJMeZBY8wqY8xWY8x8Y8yF+U5TvtH6QzDGlBhjfh+Uk23GmHeMMSfkO10+vbJBZIz5FPC/wJeAGcCw/KYo79wOrEXs0A/4F3AZcHM+E5UvNH+EsdZWuP+NMX2ANcCj+UtR/tE80ipfs9b+Lt+J6CHcBjQCQ4B9gKeMMe9Za2fnN1l540bgAmttgzFmV+BFY8w71tq38p2wfKD1R4gYsAw4EvgYOBF4xBizp7V2ST4Tlk6vbAwDPwaus9ZOD76vyGdiegDjgFuttfXAamPMs8AeeU5TPtH8kZlTkY7Ty/lOSJ7RPKK0StBh/AIwxVpbA7xijPk7cA7w3bwmLk94nQAb/E0ACrIxjNYfKay124EfpW160hizGNgPWJKPNLVGr3OTMMZEgf2BQcaYBcaY5YFbQFm+05ZHfgOcbowpN8aMAE4Ans1zmvKC5o92OQ+431pr852QfKF5JCM3GmPWG2NeNcZMy3di8shkIGGtnZ+27T0KW2DAGHO7MaYW+BBYBTyd5yTlBa0/2sYYMwQpQz1qFKXXNYaRYasiROH6BDKENRX4QT4TlWf+i1TUW4HlwEzg8bymKH9o/siAMWY0MpR1X77Tkmc0j7TkO8B4YARwN/APY8yE/CYpb1QAW7xtW4C+eUhLj8Faexlig08AjwEN+U1R3tD6IwPGmCLgIeA+a+2H+U5POr2xMVwXfN5irV1lrV0P3IT4qRQcxpgI8BxSOfUBBgL9EX+mQkTzR2bOBV6x1i7Od0LyjOYRD2vtG9babdbaBmvtfcCrFK49aoBKb1slsC0PaelRWGsT1tpXgJHApflOT57Q+qMVgrbIA4iv/dfynJwW9LrGsLV2E6J+Fuwwr0c1MArxGW6w1m4A7qVAC6bmjzY5F1WFNY9khwVMvhORJ+YDMWPMpLRte9PDhn3zTAzxGS44tP5oiTHGAL9HVPMvWGub8pykFvS6xnDAvcDXjTGDjTH9gSuBJ/OcprwQ9EoXA5caY2LGmH6IX+h7+U1ZXtH84WGMORQZAi/oKBJpaB4JMMb0M8YcZ4wpDeqQs4AjkBGngiOYEPQYcJ0xpo8x5jDgc4jqVXAEZeR0Y0yFMSZqjDkOOAP4T77Tlke0/ghzB7Ab8BlrbV17O+eD3toY/gnwJtKDnwu8A/w0rynKL58HjgfWAQuAOPDNvKYov2j+aMl5wGPW2oIf6g3QPNJMEXA9Un+sB74OnGytLeRYw5cBZUjklYeBSws4rJpFXCKWA5uAXwBXWmufyGuq8ovWHwHGmDHAxYjv9Oq0WOVn5TlpIUwBTxpXFEVRFEVRCpzeqgwriqIoiqIoSrtoY1hRFEVRFEUpWDrVGDbGHG+MmRcEli7IlXfSUXuEUXuEUXu0RG0SRu0RRu0RRu0RRu0RRu3RcTrsMxyssjIf+BTiOP8mcIa1dk7XJW/nQe0RRu0RRu3RErVJGLVHGLVHGLVHGLVHGLVH54h14tgDgQXW2kUAxpg/IeFlMhq+2JTYUvq0e+KGUbLP8MpNAPSNyEI2rtm+Oi7xzutWlwMQ2bS9I+nvUsrpSwN1JIi/Ya0d1JX2yJb4IDnXiCHrAYiaJAClRiwX6wKvGBs8BROEGF0ZL5Vr26hcKyLhA8v6DKSxYRuJeMMOs4eJyP0lKmXVy6YqSWtluURyqYhKPqow8hkzzWFS66z8vzFeAcCWWjlH8ebg3FtqvYsFn1n2JTuSP6J9+9jYoH7s2XcDAEnvYpEgEQsbZOGrxLxEkLYgcTticqx3regukg8mlEhQikxpfn/bAGLDBhJftb4p2zok2/xhYpKGREUJAE0Vcs1oaRyAiqJGACqjki+KjNitGCkvMRPko7S0NwX/xq38VpOUfL8tLtdoaCqSa2yXaxVtk3Pauvp20+sopy+1bOt6e7hnFIu5DcFnsENQbmzEhD9j8pkMDguKeGgbMTFMJCq2K44GtoyKrUsi8llqmoI0y/dYYNtokBZXn8zdPkD2WxqnPNqP2sTmLrdHd+Dqnoaxkh8ikaT7xdtT7rtoRWDjbswfsapyWzK4qsPRoE2WlVtzdmq5v0ntY73tNnRsJPjuylfjBrFjbH3md3uu9oBuyCM5vgfCxwZ5PxoNbbaJoB7Psf4upy911JC0yR1vj3bsYIqkfmyqKk5tS1QEZaEoqBOCMpNIBnkgLnaJbJfvxZul3raNXohi79qmrDT4bqlr2kJjvDarEtCZxvAIYFna9+XAQf5OxpiLgIsASinnIHN0+o/y6T30BdccDMC1x/8FgKPKlwDNL6Sfrz0GgHd/sQ8Aff88XX6IBJkqmejA7XSONXY5G1jNSpYsDTblbo9c8ey37ouHAHDD1fcA0C8iDbiJRVLhDox2PtMnrGTYaNBguHbdHgBsapKOyaSytQDcec4wNq2bx5qP3+x6e7jnHKTF3X+kTNJQc9SecsET5PcTpr4PwGGVHwFwaKkkaVC0Oft/0CiF9Y8bJe/94729ARj9uNxn6T9mhNMdNCyyrbg6kj9iA6sYecNlzJj2BwAavDjlJUbSfNoisdmWw6XRbIqkwrFNjW2mqSvwr1V1jzRmHhn/PJA5zZNePJ+aNz5g7a//lL6sbQubtJo/Um/foHPnlfdof0lDzWES83/lEbJf5STpXB86bAkAx/STSFijYhsBGB6VexgYlY5QTbJ5NdnVwSXWJKSz9Np2WW/hpfUTAZi/fIhc402piIf9V86ZnOWtOJoh74LkkfeZnrs9fLx6MFIqaYoMFLsQdBZsUZCHS+QZJiqlAZIok+0N/eSzbqDYr35A8zulYUDw4qqW59u3v9Q1I6sk+WMr5P7Hla0DYFLJGtkekzxaHZXjqoK0lhq51r7TzwdgzCVrWV2/kPe2/rPz9uhqWnlvRSqkQ7rwhvEAVPSpD3Yx3qFyzJD/EZsm35sbnKD9d1eu+aN4cCW73fxlIqbtuslvqKbuqZ3josHv0aARE0t1AJqFGHeO4kiQF01431jwvSzIDxsa5B217CGx48DfBfVusmVHPxt7yCHdl0dS74GkDaczm2NLgo5Tv6rgWDlHcutWOWdDhtWsM9R/a+xyPuTt9D13mD3aex/Gho6QNJ4wOrVtwyHyzIcPl7piQJnUIVsbpL5avq4/AH1mSH088nFpbsaXLnM3Ih9BZ8LGpVEdmbirbLeW1xf8Put76ExjuLXWdovSY629G1nLnkpTHe4OBkSDzDDkWSkYJQ1LALj9+lMBePh9ye8NQ6SgrPqqZJK5v7oTgHFHXgTA5Muk4LR4MJKQbO+rK8nOHh0+e/jwwfdIQbj50cNlwwDJTPEBYrfGtF5Z3SCx0QXfk1CQF1WtBKDJis2KTLi36ho1rjFz9JzPAlD8aWn8Juul8p9nquX45AKirGuR4pa3kL09Us81yPSOdZdKJ+Azl7wEwI8H3dXWaYCKFlsODjqTBw9/E4Cbg8/a46SBdOY1nwFg08/HAFD6ZLhx3MGOWNv2qBxpx9wdYfahomDuUewaaWJr9yx6Mi4/VUTEwLMaJe1j7o6wdq1hbctDQjZpLX+0qPz23g2ADy+X5/q/0x4B4LSK57NMZbH3KfSPlqf9L5+7BerxtDIJsfv9gUGo3V2DHaWfzqpv1QBw8WKpw1b8cRwAgx+cBUBye6B4ta/it2sPH2efNZfIe/DICySvnlj1z9B+fYIRt+qIPJOhUXev5XQfJd6n4OqX2Yc8BMD4ay9m+9vbZM2qMDnbo6tJ5b+098vGk6cAMP/IO7I6x6G7XQJA32DpI+PU+GSmIzKS0R77711qZ0zdudbQcWJL/Fqx7ZfO+jQAG34zFoDyx2fKjjZjPdu979yIp+J676LIXlIRrDxa3oPb9m5u0E4YJbXdblWrAdi7TyDKxKTxmwxGbrcmpK6cvk062v/8SM5Z/ZzU/wP+/I7sH7xzU2lq3STdYw+vQe7s4Br4Wz4/VX4/V9oAT0+R9Wg+ije/s8598ysAbP33UAAaN0pSGvoFZWGk5IU9TpMO45VXSv11/ltfBmDM9fJ78l0RvjefI+2AZ2/4JQAHPXg1Dbdk38TtzLj5cmSZX8dIYGUnzrdTU0IZ9YQWVlF7qD1SqD1aUlJSBeEWaEHbpIQyUHukiPXT/JGO5o8wao8wJZSRJNSbKmh75EpnlOE3gUnGmHHACuB04Mx2jzKmhbJT+ZR8f/G9yQBMvlhUuSqkB+Uer+tTjA4Ejl2vuxSAxRdKb3xc9Kuh452SmH6t7qKS/tRRA1BsjPn/7Z13eBzV1cZ/s6tuSW5ykatcZDAYMAYMoROaAwFCDTVAAJuEZtpHQhIg+RLyEUoIvQQInYCpSYhpodoYA46NDQZj3Issd3Vpy3x/nHtnNVdaaVe7WsvWfZ9Hz2pnp9w5c+fO3Pec854cErVHGqHdKhHtXtkgLsmAcmXnNnObN54h4QA/LvpOLZFZZ8Ag/PVMPWDMm+r/OgiArIYVQEvWttjpQ72bHnuY+9az774PSP94oywxNmZDRNi49REVC9nsXPsoN15plp81LgiI7V4pl8qzoQdlvXEfyex01M9XARDZuEk1tnWmr0P9o7qO4Htz+M0KYeFfGv2WtKED9NG2gtnWm1YIwx58bw695Le8pMcQNX4su3EfAKb/5FYARmTLtYt4xwyoNsg10w+KLEyvh/Qr7Q3Rn5Fmbdfr5Dr+ITNs0DF637ofvTz6dWnyjdKWC84Xr82aydKHvTCKQJBitwSiHbCH4ZX45j4J8Vl67H2+8ze9PTEupHUmWJ+/Gfcty1SYhOrnxkO4ZRMT5V307ZMbJXvMYOhI/+hkeC7xZvd4358Ky6dtrUNsCgLy1Gpwdey09J/qYWKPIr0Dp337FNNbdpGgPerdKF821TNGjf/tXaNMINF+oO+jv4+WCsa594gdxw/+OQAD7p6ZtD1SQaveZqDqTHmOll4sz9GHRogrI7WQRPEq/aRYcn8ig2YCEDxUbHfn/5QB8NqV4obKflPY8uJACdFolE61h8GM6zGn5jSxw/6/+ASAWwc+4FttxBuXATD2ysXesuFbJHwxrndMLd+oll/zo0sAeOCOhwDo94qEVZw5TxjmtybcBkBvZfv+n0WpMFJ92jy1xFf1w3XdMHApUp9+IfB8Ny5HScAJsBPjAcZg7SH2cPYEaw/A9o/WEJAXgBXYMQSw9jChSBNrDwXbP/yw9vAj4ATIk4mttUcHkAozjOu6rwOvJ7yBIwOcZviW3CIxHnmVEhPsMbpqJqsDqPRszMu6VJ/Db/gYgMP3F+Zs6XEPAzDxY2GMe//t49ih48SbphMlTim4LHBdd+9OO0giMAPLFSMcKPJ4CIovFEazZ0AY4bqorKOZUA3NiOnlx3xzjGz/osQn67mcOWPGdSlhINBxe5jXrP6EiQDceIfMvg/Pl2OaSX1v1gmLMOWDcwEo+VC+910gM+5ghSRSEYold0UHSnLR1p1EqaTiYDmzqw+T7n1JLwna1+zaNwc9Ib+/OQGAeVfLZ/D9/6rGt5ztdpn+0bWwNVl7LLpVEmeXnCIegRolbaDjTjX7pDmMGCNqMqOCAien1eXBZmxd3HXi8Ale7KNijhuj0ocfGfYRALc+I8l9754q/Say8Fu9aXL2CAQ9diZ6kMTpzZn0F91qWe6xuH6vT72rxgVtL9VnTfZOe4uCPvZS2bKDSgUa5r179rJDARh7m9yjKzrQPzoL5ngU/v5e3m8vl0tfDKhHarGKkdfnpRl0HedfM7LDz6GE7RHBoTqa027/N2F6UtIJvU/dx1p6K/yoiqj8iKDYrXZICy9Fp/YP85oHdxHvdfQe8TJ+vLMwoLF+LKykHov+d/0Eb1/PfCGerPyvpW/kbdDyB/JRJTmDnHrUDABuHiD5Bd59qpQ2pvZeBsCRD90NwFWnT5ENZ31BFtm4rjum42fcOlrYYUB/AL67S+J9vznIzwTr94n9/zQVgDF3CbsdacYsN/fct35QNd4o2+a/IrkPf1wmZPcvX3oGgP/u85zaQGy/sEno4OJ5lQTrDOWJNmAr0FlYWFhYWFhYWHRbpMQMJw1XGMSskWUAHHuEMMGLjhI2LmrG5RiZ+V7Gol6gmIysX/cCoGaazCKv/5VkLj70j4netpFNm33bbCN1icxAnZsX26aw9Zhdvf9njpWZnJ7Bmoywnunq5ZotWPtiGQD9QxKX3xkyXuYstOE4uY4v3fNnoGU81rKwzASPeepaAEbfL3HMY1Z95ltPW6NVTqZCpJ+K5srXor/L57+KRQXgvksk1vWuC0WpQrPSt5cKQz76WMncH/We2+o5WKSOaO8e1B6+L0tOkWug+26+Ym2DCcRdQssY2v3miuKD81SJ7K9S9tvYOzY8Vp4oMaDfKpk7k9E0oZd7zLEadrZGJYny2j4SY3jfFSJrNOZnelxK6BTUPiX/wlXj5Hc/FjtoNYh43h6NQicviYPF9gewKiI2WhMWb1NFWBSBVoYki35lg3wuqRGbrqmSMX7rVmmbWyVtyt4s9imdoWKyp2u1gC1JtS0TMMfTwK9iWijaxtpGZmx50FBQGlPuz2tq4VlLAyJugI3RHkDiWsbQviclk9Ax1xrhImWnthUUUoY5fteeLOP7bbdKHP5+eXJ8reyj1XJOX/p9ACr+IJ6fvDf/6+2zPOyTPWuBEvX5+XVyT4x47AIAlh4tnlA9bukxRKsLLb1C+taIWUmcYIJo8XyfKLKlRz32IQBTe0sCl2mHCY8IIzxcMcJaZcINxZ6HyT4btUSkVo+4+BGJH//qkvt8671cJR6y6Mo1LTWJ29p/Uq2xsLCwsLCwsLCw2IGQWWYYwHX55melACz+t3yWbZTYXi9WOGqwjAab6zHEevY2S2Jrvj/vHAC0tuL1F+/s7WLoH2b6jpGJggTbHFH/LHqnK1vG0puZ9d6mukKU+n79OgnL6n/PTN8+027HQLCFasRv7/wrAD0Dfibr5g07AfDuZfsDUPa+9CM93zRjkmJZ4K3EwikmT+t9akSUAPqQP8p53/aPUwGY/oToy077TOwy5tqPvfaDZYQ7A4F+TeT/PMaoxWJdE5vTm0zpDxf9AIC+P5VqeeEKlemsrmF2M89U0SuyzejHzwNgsWKITUakPRQ6fn3d+494HIC7hxwpC1YktBuB6+KGmggUCNt6zL5zfT9rNtJksdeGJXZ+/9evAiBvrdwn2WIGcjfLfZK/WVWT26oqRNXEWJZAlbBTTo14ZaJVquJgTY1qm153LQD9vM924OmXdoAp7ySYLGHdScISfjg2pmeuWbt4LLw5zl4+/G0A7imSONJotTJ+Gj2XITfI+nAx7THDZv94sUZY/Fk1o3zrRVXMeUiVIoyqGFatyhOKBlusq3/T8epNKr5/Ut8FAJxXXNlqG+Ihq5fcw17+UJqZYfP9QKtFvHqLaNf2V15J874f/azoRo+6WujZXERdyG3mEfDebzTM55DO8VFqUGOvkDyC5z6VegGnF4l3O894bbtmvKgMvZQ9BEIpBvHrtuo+r+zQeKz00z/dIyzsxFxh7DdH5P7X3qhDF/wIgOE3zvTvJ14BkSQQNfYx4tElALx5vrTlqAIZc15YKsxw/8avk7qPLDNsYWFhYWFhYWHRbZFRZthxHAJ5eQzZQ1iCHpNVxrWaPbnhkF5RfRrlVs3lsR0DkPeQzKBQsrOXn/Wqt8qrt0t9EDcix9yhYzoN3dEtZ0nM7etD722+EhB/Jm5m+b79qCh/9GdmGhsaB6r9w/4q2p2H5usZtCy/ZaOUwp1xjFToCaySuCwvLqlJ1TBP5toqZqcFaayVObJk9hldILqw8/eVWf6Y0KfGfra9lueOikG5W7hpxKuQYCa6iZBBI321QEqDlleINqZWW4nWCtsR6BGLTdcV43q+J3F6HCofkSSpy5i6gPSTSQXCdtyV2/Fqgu4ukoZ+z+CnfMu1ckHE6JPzm2ScHHOxUUWxveM0+79dQs7QIzU9Ls1+UDv3Kwd1pZwOz6OkxoI9r48f+6kZwzfqJNv+5MKqVtc7Kl/6050T1BjmqdBoe6ROeUYIsilcCK3VeWwGTzFIxd7f8NjZQMwbFoO+JqmPcbdfI3H65111n9qj3xOpYSqb9O4pdnNUKXHS5Zg0PJ2ho8Tj9+jNdwAxRljH62oVpjF/E+WqUdcb3m3dn5s9g9r1oup1VVu0V/KWr48G4HSlmlCt49KVwsbg7E3q2GU44dSY4RaKKYeLYsq9994FxOKUtZdNM8La25R7Q7Fvf2acfUrQY4KyT3it1Bm4doH0paMmPgtAzTcyvvVPcveWGbawsLCwsLCwsOi2yCgzHC3Op+7g3Vm9QWYVo5ZJjFtcllbNjoN9JTPZq/KlWTxjHlnwL5mx/36DxJr+uuRr77dnjj4WgPxXDTbEYFF3BJh17sdfJnZuzgLHq0plxm7dsUlYpwEPSpa3N8/rLHtFI6z/mbDQbwy539fWjxulre+fJjqzkVUSU+XFeaUhLqkFdJy6ntW3FyvdhRitHQ09HJiYmz77OmZ8ndadVn27tUzkvM1yf3gxgyoGOGQweWaFNhOajdPMCoGO8xLhIn8sYvyKc354MfWmd8hgkttkd0xPiD5fUwloO3SYmM+lDVPUuDRI61vHYnG1osmJ30hOwcoZQwA4WVVH1dc7qGJoNWtfOUGYtoHvq2PquNE0jK9hN8CGUGH7KxqI5ra/jg9OEmyk6h/BDjK6w4olbrY2mJxXKC5025W9g1ICnKPv+A8AY3Pk/jQZ4V1mCns+QjPCXl9RY0YqzwHjZhnay6+sUuD4vUjfNorWb7SuDrejN5qR6xIYJ+9QVzwgbKtmhLWCj6mOctBHlwIwalY773RpgHmPNM1VEQFKPKx4cWtbtQ/LDFtYWFhYWFhYWHRbZJQZbip0WHNgkPy5ig3Rsws9y1MMhI4HWvK4qAVctfs7AHy0RWJF118lsX4oFQlPf65BZurPPitaf7++LMYMb/yJivnLk+zQ/PUyw8n6z+eyQjyGeDvSJTYzYSt/LioLrw3WFaliM8pAnNJRZvzYkw9MAqB/SMWPdaI9nGCAYGExZ1wi2oWapdYM16X3iq5g6UJDu7AzGOF4MOPXt4N+saPAwUk6TrhNtHfpWmFZcrfK9Z/TJGPOwUpEItqCjW27QlvQqAaXFLtmIJzXMZvskPkS6YDOYVHPIx1LPmXqq77Vmsezak9awz2DACgJ+PuO7heeB0Fd7poJ9f5jp5FCD7sBNoZice+JKjaE8+LcGPHGvA6Mgdk1/m1i1e78fdl8To0q3ADA/KzeSR+zNehcEP3M/Pr3woi+3keoepMRvnvzcADKpoiqjX5biCkVpeF5YOzj649F637UyvMBiDaIjbIK5P4tZRSb9AAAIABJREFUnCnvUynl8+h+p65x34ckHvf4HpI/Yepn6z60ISLvVSPv9rc5rbHCZlPDfo/diGclJn7XJnk/KHtTqusmO7pZZtjCwsLCwsLCwqLbIrNqErlRgiNrKHlXBSWZsyjFun17i2jaFeZtBeC1Uw4AYNlJUqPl/578GwAP/ECyLKPLVqoDyKym7CkR65xxUWyWPXe/JwC4tXwXAEbnycznF2+eDkD5pZJR7jHEeqa0PTB/mslQs1vNlA/9sejw6fi05hWkTD1MsxKdjrse9A+xbSY4pEhRPtXfH8u1fT5QS2SupvWEB/1F4r1dHd/UtA21ortQvwg4/rZ4mdYZCNTUx/L6ntN17JIqWmNOc2csBOD3Z50LQP1AuddCPeQebCyWPhsSMpFo67KzHisdVc6akZULO9zOSF6KnEagdWbZU4Boi0mM08fiMUPxVCW6EkvtxSSqNi27UqpuTe4pVbda05bWY1TBy/IciR4oeQ1m/HbUUGI4aVeJsfxCfU9nJbqIG2BrKD/p7aKaGTaZ4DSOedm1HdvX2HxhZOcH+6bWACP3w9lHrvFbx92uVpBYa+0h1fjbX44BoMSojdCZdQtG/OLjTts3jsT36r6+9GYVF18mse6mNrsXM6yezcd8cR4AvT+eJ/vLRA6W0Q8j30iQ8JCb5bOjI4llhi0sLCwsLCwsLLotMsoMZwUj9C2upeBryY7Ub/Ca4dOZnEN3EdY27xiJ/YioWcvQrxYBcMXQswDo9xfZT+9jVUU6NUsLr5TtXtg80Tv2AaWihvDWtQcBMGOeMJ5nTpc4mzfPPVD29bh/FuZpj+oKQV0QJpOx8XSpwDK7XGcy+1nf1uBlvSsy4JkXJO566PLMVe4L9XJZdVzLGeWjbx0GwKjwLKMtO44CSCqIGvGnuv57Rtg2gzU027KjQesNO4oJKUjTflPpyZGcFG0eh8XpDMdC3H12hRh87WFT7GxWqWTp33/uA0As5tbUvgV44aHDgVjcZs538gz7okn2tVdusNVtz+sj6//PCPFQhpeKtno6GLZwNMCWxg4ww7k6fjR9mscmcmo6ts/dctWzPWvPdDaHtb+SsXJUtjDCJvs/ZZUwpv0eFV153UvN+NXOgFlJ1YTnhelIX3HlOaGrvb56pmbGZWTTMcIaZv+NvFai/lPKToaSVUagawHo96AOapVbZtjCwsLCwsLCwqLbIqPMsItD1HWIrFln/CBv8JGdJVOzYot8H64171QMrGa6xv56GQCXfCwxXHfvebL8/t8vAWiaJDHHi6pWe4eoGSAzvbUXiPLA8NOkDbOulHULfilV8XhcPoK7SgzYT14SZYObnjtDtjNrbpsxXpkkNrzKfUrDsliqv+w85Uvfaq0xGRoma3zrJqlHP+IJf6xwOmPZ4qF3fh2njP+8RduGvGPaOIPTzhSy/FtFGvpHwGjT+GJhS6b/6BAAGnrJDNnJAHHuKgIrb4sc7JDi932/m23tUjCbprR+vSpSrSHgZyFa7rNj5xutk6ztjvSPcH5yx8xRHSOrTFR53Fw530hPYRFDxfK9qaeMcQ29Aup77DhNvaShTb3VvdhT7tXCYsm+71Mgn3lZsjwUFXtVN0q+yJYqYZ76vSzHLHxevD7bkiE2PWwLfyXPI10Bs0bFT2pN4Qe2DPa2HfiYeAqiqv3hdesBeG7zvgDsNfC/vmPpWGKt31o1XljoAsUMp4Nhi7oONU3tiwabmrHkSdvSqXlsIrsmMa+VWbFxnPaCtMOWtolA0GNRo4cIw/zOXveoH0V9w1St+eRJWW9A2Hj+x/G+tTmGpIpWqtuliq+nCiOudZXNWGFT2enNOkl2GPCieOs9VY0MvCe0gK4FkKI9LDNsYWFhYWFhYWHRbZFRZhgXwpFgi2peXvWX74SdHXLfUFmuZqxRpSOrZ6qR9TLrvmv5EQB8e77MasrV5HvFUbJe6b1DvUOfcPFpALw48SEAri0WJQrelap1VdeIhnG+ihFe8mOpevebVySWK3c3UbbwquFt3qrOqetkz288YVcAXh/ur9zWljZrlqHr+MCbRwIwermKz82glm9BsJG9eizzvq8JyzELlkuN9haajplAF7q+GnlGHNf1Jd/I533fbIvmtAmzrV0K5qWNKsYlgbj4rtQrGvq0zQyburIH5QmDMvKDZwDIU+NsgRonmqskdBY8zdtDpG2jJ14MwKhrFEOsnw2ZIJoC/ljD4C5jAHhHKQuEXGFvtYdN2/Oex0/wdjG4VhhDU/P+nVXiYUQxw2alQj02r9tH9jniZfVDO1rAiSDqBqhtTJ6hzCkQNt8JShvcdITFGox/sDa5nUbVHaeVkdy8DjKvjrDumnFffIbsryQojLAZK/x8jeQxDXpSvK3tMqCGslNXh5OTQ9aQ4Tx16ENqiVxzM1bYrD9w7YJTABi4Qang7ACVfC0zbGFhYWFhYWFh0W2R4ZhhiETjsxia8Q2+t771FfRsWc2+ls0U5veHPxClCM2LuX1lVla4NMZm1t0jMVm73iez/A0nCYva+wnRrt2wXuJt++ysdEMLZSY6+jnJHl80RbZzhwyQHW7cBMRijpYfrdiUX06Le35ph5Hte+hVhhJGvPJXtKxG9E69zOx2ukW0ib0ZcAa1fAudJg7MX4nWeFwZkU8qjP6QiZhh1ceCfVSlozharEmjMj27sUgDTCpg55EALD63uOWqTVpz17/cE89Q+squ3qe+NQNu69818uVOG/sLpZVekWDbm6GxT3I8tb7nh2UVtrmeHiPCajSINPOSmHq5cfehGL2o4WGpVfseotpQMHqr73dPjzgDRJMXnxuWNn53o4zlI5SygFmF7LVaiasc9tdYhdOojhFV3ksdU7plqRo/9m67DX3G+8e4dMReRqMODU3Z7a9oYEAvUU5yctQ5KZY7rUjR4xYpUULe3yZ7XIktDQ7oD8D9Rzze5upeHYItoh/dXqxwIF/6yLqf7AFAuCA2YDiJPrbMx7YWi1Bva7p634AnVJy6zjdwnKRdVk19sll2+mAOUFrl8bzJZlx5+BN/BcDOjC/PFCwzbGFhYWFhYWFh0W2R4YA+B7ctHVI9+9CMpzHLMPXj+s+RqdaZZ0mc2Y3sJT83yiylqXcs9i3/FWGAn75FKtfs+XOp/LPsb7JPt07VjleZ1D1WSRsWXaD2EVLxy/MkRiZYLizSXneK+kH1bfsD8F38s0sbzNlp5aVy7N/1/4taI3k2YMorFwEwat22y+p2MCvTK2QiQ9WMaesp7OAPPpArekQPue4hNX/MbocZCxqV2CoiEpP2xojUm9qg4rd0/Nx166Ta1dwpUkWpsUT6rBPp/GvnBsVuuRuEPRr/4HwAbhkwt9W2dimY5vlO2NlHjp8BxFQEoKUnJVVopRRtl2P+T2LwOsIMN/X13x8x1rZtb4ZmgfSn3k6zQjqfwLt2aRAG0Xbsrew4vU5yEgb+yR8Dmom8AFMRKHqQ0mc/4D4AQq60Kduw4/V/PQ+AwRtnttinGSva73NltFPlQzNspsLPxSOl6uZzuTJAeDkaKYzDrgtNje0/4s22lPcUlnqtqYiQxmdCqmNTQ7/U4torThLVpEkFohal1RN0rPDmiLCto17wX88W/dKIla09ahwAc264P6X2NYc5Vly0UiryrnigGSMMHbougaIwRQfF3JXm2KHvV7OK7aAP63z72SYqEmmGZYYtLCwsLCwsLCy6LezLsIWFhYWFhYWFRbdFhsMkXAKBNtzLmuZPsPxj4XeSdDEoSxIcdDnn3Ao5rfV7xvx6g96Wz3tuEn/Vx7dLic1J+5wDQM5GFVqhhOVL7xAX2O++XQzAf+tEgP3ZPx4MwO9PFlmim++W0tADnm7pMks7jCIb2kWz9zkSSK9dGaZbRSPSLPFMu3u/aBL39k73iKxdBgr4xkW1m8OHDYM5rVCu68CgJC/SX5V83KKSbDqxTKgHdYz9CyRDQ4uRdxRLQu2L3ycKMxlpaa2E/jBbQhTyMlA6W8Ms0+21RcFsa5eC0TRdcn3Ks1MA+OanMVenvl90SIG+x9oqaNMWtFRRblDdoynYKbtXcrKHuu06DKKtMu0QGzfq3Vh/qo5K+xtUs7dG5TxWhCWxZnlTPwAW1pUCsGCTfK5cLf2j8Cs55tBXJS7E+VbGMM/lm8lEHG37mzYAsUQ5s/DAwiZxDRcvF3s0Hd0yK85VyXiBkKxjyt6ZUpYapxSuAOC5cSJtyeeqcFIqY13UIdqYfOLvbkVSxKcie6fkj5kooqklQdf3SS2hue6wGt93MyH0xnVSwCjwvr9YitkvzcSxVYfL9a6MyLNrbmMvb928QGJycnvnSL/TITVmqNlnT0pyni79bRaLSQYDcquYOuod73u8/qmxQIWLZn8pxWE8a2S0/nLnwDLDFhYWFhYWFhYW3RYZZYYdB7KD0XblSeLCmH0EKjcD0KCS8sJjywAY/k+Z9a34QVGLXRQ/Kwli5cedB8C9zz4NwFVfSFGOomdkVlZ1ipTR/LBK5gtvr5RZcr89pYzzPf/zYwAGvOIvz0g6BMrjQEvd6OSKyouljW8M1QkfqsxqnNldc2an0JFEgRNfmgrA6KXbvhxqXTSXz2tHcFqhJF8Nz5Lzrd1J2KS8b0X2LR2lStuF2vnbNSLBF+ErAEKq/nB2nFrHUaWtNVbN7gsdYYTfqdpFrfFla5ulhICRrOdkq76Ygdm6PpZmhs22dGkYCWGa5R79oCTS3X3icO+3oqB4n8pzhMk8QBVX6GhiXdBMJE6hbPXgki0JrWcmwywNyTh57tfiHaucJext8RJV4KBK1s/ZKuN0dnVs/AhUiT2cWvl069VnkwyAboOMUW5IPE/5LAVgjPr02uTtMHOi/ebzp/ZkGUc/GvsgEBtHTcZ8ZLbY7cPbZbztSDKluY0+lk7cWr+3JO6WqKr0KUlWuUBT+200JTjH5wlL/Vb+Hr7lntRavOTGBMYb75mf5NBktrGxnUIz8eAEAwQLi/nD+Fd9yyOGm+iNN4T1L0PkSuO9s5jet51/I3J75/32eLVCG+Oh2mekUhIWN5+7HwCzb/YXzdKeivfq5VqWvlGh2qwOkUKyaXEgxBEFq9BlqM3+qWUVg4o3fbjyUDn25s0dPmZXhWWGLSwsLCwsLCwsui0yywzjkhWIEuwnMaDhtUpHqINsZLRWYrgiimWpL5XZdcFLIpA9fF4sTlPvWc/wRp4p7OPFTwgrsuSIRwH428sixr0lIjGi026Uss0Dp31iHF1YSq+UZwfidRKGjhVWjHCwRJjSXc/+yrdaTBrJHytssg8AMxpk3VHT6v2HSiH+KFVsrivghXl7cctRcm00g7X6YGnTqH+oq5gmeSsfjL4XUfHJ7+0vxVred0pb307ZS7Miod1Ecu+xp+8GoGeWtHXavAlqgxfS1WIPUYNldENy7TJyDY1rYbalS8MYbnT52fBKiZn89yGjvd8iGzYCsOYakUCbf5Wwg1WqfGvvYGox5algQp+Vbf5uCunv+akUEij9hSzPXyhx8cOdZbJBnHG4+dJkeUrPc+bFwKrS15rVymCMsD6mbtOEX87x/W7KSzUa9Yg1W9YYTfz+0uVt4zHD+tps2lO+l8Qam/AxWsB1CDQkPlZqz8Heua0X2fAKMHXEa2h4Ppym1FyozUJxk0K4Zz6bjtmF43u8q5aI3TX7qjHg0471R/3cSAjaG6LsGTrRz7bWKU9uT0fadvl88UaXfisynx32sDdvAg5FbeQMeIV21OX7ZO0waQPbvgyz9uR5Y0mKzzvLDFtYWFhYWFhYWHRbZJQZjkQDbK3PI3+UxEUFPGa4YxmzmsnxvmuWQc9CmwtBmwU9FMqeku+7F58BQJ8HVXnQGVLcuUeVKsNozEI0Msq+KfusP34MAK+XSWyRmfVsorUSi2e/L0U2xsyU4DRTFWBbIHurw6B/ZsFR/uWXHvNvAN74jbC0Uc0qZCC+WSsMxIXRhpVHCvs+xCh1O+h1udVWpLd5Fp0BdU2j1TUtfgomJ9zQqXCCQYI9ezMyf7FvualwYTKd+c8IrRZZKHkCTq7yoJmxhyZ72xoSZC63hafJRItiRZO/B8D0QTKOmio8Zox1yx2m3iazzO2h44VxW6O+pxIP6kQh2Nh+I02lFO1BXHi1jLdZVYMAiOSrkuNZ/k/UZyAvdo2zc5UCQo58FuXJjdMjW54vZcWrfW1oT8XARFOvjjHm4UKXdQdFvWdhjfLs6HN+p16WF30q3iF9RgkXlUgg9t/0voa/L8XCZu/9kFpDqbw4/ud5j+d7JtaGJFDvwrwmmKiGgPZyIGqq/Ax6RvJ3TGhvfJrfVdp9GXYcZyjwBDAQCXt/yHXdvziO0wf4O1AGLANOc113x4uqNtDg1rIg8glNrtxEgxnBMKeckNvEfGYBjHMc5y26jT3q+JJPaaQBB6fb2yO0dTPrXnyWXR9fQSDgcNHZxVx+US/CjXV8O+sp6Gb2WL0mws+v2MzG9S6BgENg0heMOnX3bts/Vq4Ocd7llaytDBEIOEw+uyeXX9SLpkg9X6x+FbqZPaDtMQQodxznW7rRM6Z2XQ2fu++LPSIOg51RDGNUt7XHytUhzr18HRWVEYJqTKUXRCRMstvZQ98vTSGZYDS/X+qoprvZI11IhBkOA1e7rjvHcZwi4HM1WJ8HvOO67v85jvML4BfAdW3tKNoUpHZlERv2kDf7/jNkVtHRjFmnUNi3qJqiZ1f7yzW3xkaYs4khv10k/9wobGv2m58CLbOcve3cKOXsRrHTm7AbYjbv0McdwFqW0Yf+bKJyAfAOCdgjYSi7aPb2mCs+8P2s49FMmFnRsxpi9h39qMFwd7CcooNDObunxR7BqnqK31zInZvLALisl8RlT+29DIBHf34MENOAzgibHWemH+uzqnxzSR8AfnPSi2w+0OHyiUOpromy05GVrBk/mg3Tp9E/0JOtkP7+0YURDMLvbijm0PG9qK6JMviwBfTbewjL+Lrz7pdU0Ymehqwsh1tv7MuuuwWproly4KQKjji4gKUbZ9Gnx3A21i1L3B452bhDB7Bz7get/qxZHpN1C4SN81P3/rZib9saQ4Bq13XLE33GtH4AzV4pBrOHZM5ffIUoCmg7aUbdZMd+sly05T9aJDHkTkA9X5KIj3/uIFGqmJjrZ51NFv/C/u+zgTBrSg6jZ05/GtasZLb7Dn2c/snbw4VAAsywRnPPIcCSkx5MeNtU0ZYyR1aWw59u6Mueu+dSVwv7HL2Spskrqf1wDiTZP3LzQuw0JsZKm/ZfGZJ8nPDqNb7lCY8JHRg7qq4S72M8tvqRrcLQ937rOwDq9P0S6CP3i/u2d78EySbshhK2R0VTMbetnsTzI0VrOKoyA3RPMPWXqTa8JJ2RvxMPhgb56uv2B2LeuoF3plbrod0zcV13reu6c9T/1cBCYDBwAvC4Wu1x4EcptWQ7Qa6TT7EjwvJZTjYFFNFIPetZQymeFJO1Rze1R5/+2YwaJ8lURYUBBowsYGtlI5WhFQzOKderdRt7DBwQZI/dZNJSVBigqKw3DRtqu23/KB2QxYTd5SFXVBhg5/IcVleEqaz5lsE9d9OrdRt7QNtjCLBRrdZtbFLSP4ueOZLI7bfHauiG9igdkMWeu4sfX98zkU1V1M1ZCN3QHm3dL9l4oRXdxh7pQlIxw47jlAF7Ap8AA1zXXQvywuw4Tv92D1YH/T4NUHmIMHn9vYpzCQacGLGzkYHSIbIdFdu1SjI5NccZGLezt2mgWirCuDkys6m/T9aasVgYvdFvfiaHUPFzXuZsG2x1vVtLNVvoSR+aaCTXyQc3cXu0CyNTc91k0T78bb/7fKvFm1Wb2owXzTvH+3/Qh1JZJ53saqr2cCNRIlVVPH6fMMBTfyXnqWP5HrpUFBp+N13iuyNfLUr7ObRslCk5YMSeq3jtRXcOAeD4HnVqxQDLVoZY9GkYd6dTaYreSFaV0mBNV//YzrBsZYitizbSe5cBHesfuITcSAsGq8NojzRrLV4zTWIZAdU/5s5vZN8JeTSFN5IbFMYyUXtEswPUDy6iPFtnsBeqffsbaY4P2bVdt1qUOYaglNtTuWfMGM1lV4t+7uSeMwBoVM8TzRJqduxLVZ1z4yli1/LVftWJZHDxa2cDMGfvvwMxRQozPvmAPGlD3R5DZfma76hmC72C/WgKJ2cPJwrB1oUhEoLORekIzFjoeIgbjx0H+p7pd34JG+6thiT7R14wzJjiyri/N0RN5jN9eSneu4VShaq4QpjNeRP8dQJMtvr3M34IwJj16h2l2fPOvF/y1RiQqD3qq/OY9+4Y8Jhhf36BCTeQeR158/le/WPRY15whdhtxL8vBCSOt7X1E0XCHLfjOIXAi8BU13WrkthusuM4nzmO81m4oTapxnVlhN0wX/AxOzGerCRu6Ob2CNGFsnFShLWHHzW1UU69oILSQ35EMDev/Q0UdnR7jLv8ALJ7tF3+tzma22P9xszL93QWamqjnHFhJXf8roTiosRdjb7+0bTjjKfQsTFkR71fACLhxpTsEa7bsfqHHkPu+F0JwYLEy9s3t0n95hRmB10M6XjmRmp3rD6SChJihh3HyUZehJ92XfcltXid4zilagZSCrQ63XJd9yHgIYCe2f3dvtMXk3+OqnZSLKoSkaoqfSC9UevtMDIXt46RWdDKsOzHXb7Kt/6K4/p4///u/NeBWAWxX39+AgCjz1G1x3VscAJailE3yhd8zECG0d8ZDEAOuTS6qipTgvYodvrEOVFth6jv+/GT3/et1ppKRHNoRkjHIJX+qeUN44ZTL5mXVnsEgvS/XxQ8pv5UmPA7S2VGvJ96pxz7pGTPLzxGJr6RdbLrFqx+GmM/vUx0HV+pZp3L/iAZ6YsPu99bNxRy2eOcWsqO2om+dzfBW5/xHXk0Ok3gpqF/bGcIhVzOuGAtZ55UyIeHiA5zR/pH+W757jv1BUwqSM9LjxM22CtVYQylVtJcrUZLzYZ6+DcJdKByXCjkcsaFlZxxUiGH/yDI1mg9OcECGhuE4U3UHj36DXVrS7MYppRLIu142DZHxGsRbOx6k4p4Y0iYUDbEt0mb94sRK5xVKtzRg+cJo2TG7WpWTLOVJ750CQCjV4vqho41dhPQyHWUB1KvW7NAPYtkSGuh32pW+qoYH2DV359msLaHEyCHvKTskT9wqJuK+kk8daJtgVDI5ccXruPMkwo56dhC/vbtFr7rk09tbWOb9gC/TfqM7eduaCxsbTUA8gKGUlEHla70+4RXDZQYI+weMB6Ax6f+Wf3if7E37V42zdCRj0Ti3i+6Dyf8TpZf6o56bhOzzpLz2y+v7Zj27F5Gh+pMGQkjX0vfv+fd9JpvtdI3jdfYDrapXUrCcRwHeARY6LruHc1+eg04V/1/LvCque2OCNd1+YrP6EERw50x3vJ+DGIty/VXa49ubI8Lr6qkz4gi9j6nmT2cQax1l+mv3coeV1yzhbHlOVx5cW9veXfvHzuXZzN1SqxyQP+CUayp9kp1dxt7QNtjCNBXfe02NnFdl4p//p3ckv5+ewSGQDe1x+Sr17NzebZvDBlwwAjopvb4yp3d6v0SwgsN6Db2SBcSYYYPAM4B5juOM1ctux74P+B5x3EuQORTT21vR244TGT9elaskSomvc6Uql79HtD1v9VsOl6sh56lqVnbuv1kdv1wxSEARBs2qf3IaQ35Yyy78OHpxwEQaJB9j1o4Fx8SVLLYykYqWEEhPZnlvgXAaMYxnJ08qShgKwnYIx50DXg9k1xzrcQW6Vjh2KytdVbKZIz3/+x8AEo/ntdy5RTZ006xh7oWX1+4KwCzXhSmeC81gb69VGL3rp++OwBzz9wJgIiqpKXhVb1S8DQ7W5s5qr6lvQ/6u2bOdbyh3uc39+8JwNJjhRHW1+RvM2p4alo1hfkFLHrlI2iqF3sExjI/OhPS0D+2J3zyaRPPv1jPbmMjTDhiBUsbn2eXyft2qH+squ/N1fNOYdL3ngZa6sImi0gvv3qC2+BnPaINMZdqsF8/APp/36+Pmp2kPuqHn9Tz1LRqdhubwz5HrqIynMvRU8sZmbcHczdPhyTsEcmGmiGxMaDRlfPRzJIeB3QLvw2LnbKVq1jfBalo2aYDbY0hy1lUrKSiEnrGNIcZK7zw15KwebDyMJmxwlp14716+b7zn5YCENa603UqHyCRMdOoitV3vn8b05vXnCmeMbuBqvmfkVfWn1nuFwCMjuxBWWBnlkcXJmwP0RmOfTeVArYXzJjdwNPTahg3NocJR4hSe9mUeRw5eRgPPvffpPpHbWMOs5cPhxGt/35OkdQ+eORHJwJQ8LKqM2BqcZvPEP280KpM+vo3xi5A/QkTAbju9icAGK/2qWOzzT5x3TphkPPemy+HVsu3uuupcJe3er+sYgnJ2MNtaCTy5TecM+1SAL49W55n9ar6nTm2nrCT9Mf5aah+FxdxvG0r75eJ0OSeovShq+j2fl/dp2q9jipjtfsy7LruR8RPGzm8Q0fdjtHLKeEITmn1t704hLfdaQtc1+02drH28GP8PnnMXjaMX512AQDupzKQ4QTZK3gYb0f+3q3ssd/EXDasGuSVKz5p8ZEA1Drru2X/OGDfPJrWjPRCmG7dNAqAikAeE/v+iOlr7+1W9oC2xxBcFrmuu3dmW7RtceC++ezxj18B0O94Kf7kBLxHdbe0R9MaCa/S983Va0v1z93OHr2cEo5wTm11UlbgFlHlbipvZTOLdpDRCnQaIx+Xi1j8W4n9rH0gse1isy7Z/vSDhFF++eUDARiGYoJbUVdw/ysuSD1nSEdd77RDx8io2WSwRDxAE0/+wreayQC1h5J7CuIea1vUFG8T0Ujs2qhrdu3VPwfgn3ffCcRqtd88QOzy5j+katOVj0hVvWF3CQMeTSY5QLFE8cKNtp4lGayHXCPxg9MHPOz7XV+Ty6+7HIDCT1V1r65RyMSeAAAOY0lEQVTYz7ZTZFc4DLklQOWLcl37BqQfJMoQmxWdXj7iHgBO/+2VAJS9IjG7gah0gtqyIm/dLeeLFugXuz4LtNTwThQ1rtzbug8/+G8ptziq4uOk9gPg5kVpKq/3vreXwT+nvgyAQIWoUXldPaPlozof5j3n7D0OgBnH3Q5AyBXbayZYx+tmq/4z5dkpAJRVfNzq/hKByU71/nw9AJUR6bv9lXKI9iiZrOCtu04D4PZe8myLbNlK0nAh2LhDpB20UETZqaCiQ/vJ3hqg/2t5NB4iY4a2uzmGnPWHfwLw4ga5PwNKfSluXpOxPLiLhC98fV0sPnnB4TLe6DFjzPsSZXrBOHlvua6v37P50puSjzKyQfXDTlRNGn2DnN9PDhJN7SeGi3b51qiML7FnruTvTDp4MgBZ/0lfBdsWnlx1v317974ALJno172+YYkox2VVqLquKSp/ZFAx2cLCwsLCwsLCwqJrIbPMsCNv/8F3JeZz6VTRAW76pbD6OsY3kCdBXVHFkJoxtNEDJZbm5F5Sy3vew1I73YsZaU0hIeCfeXdFps5Uy1h7msTCvj5MYoV1bFE8Nsr8fY/Zosc78D8qProDme/bAmZ8ro7bOjYwFYCbb5UZoo79O6pArveXl4md7j1HNDpv++AHAAx8X+Z8PRcLKxOskAqVvhjREklmqh4rWd/r9pZtTpgkDO+tA1t3X8xulGNP/aUwwkXPW0a401BbD7Pn88NfXwPAS/97KwBDlJqCZnc0NPOnWSWTfdMxe19fpHQ+LxQ2z1QVaA2Jah1rBlm3radis+/YJG7fMX8VxrAj/pn8nBB7DI8p6JiZ36Zm6Be1cl+EK9bJ4u1kPEgKrZxTr79IjGFpi34i9tLX+fNGGT9HP7gSiMUKdzQGsTmiS2WfL1TLM++SXvJdx/Hq/qT7y+HSTbjhSFm/cNpsWZAE6eVEIat+x2CGQ4aaw6Qeizq0n8CmWoqem8WuxwmzufiwxwBoVA9drbx0cS/JDTjhadG2P+Nr0YmumCXhGQWqQJ0blD5SM1TsPHIfua4PlT8KxMYmgFVh6V/73SHPsfJXZCfj3lzpa6P2HoyaViPHUMvT0Q9bRSDo5UdsPK0EgOtfk3wc7X3VSjQ63I3rZdziP6pt6p1LM8TJeJtcowKmfv9b8qTkDC05RJ73G5RdSpRXZdUsUdEoQ5jhdnPO2oFlhi0sLCwsLCwsLLotMssMuypzWbG0g66U2ciZ/5aqPA9+dTIA+a/O9m/W6M/yPvBe+f3U1y8DoHytyvhsi43rarGxGs2YDN3uYF9hJ4+d/KFv1XhslI4704zw2rDMKIueEv1lfe7erI1OqtaWZpgMcQ+lKnHzkrMAePx+keZ6ZNhHvu0063LJ8eI54Hj50DPLTWrS2uDG7NkrIMcalhVfg7I5pq6VnI2FPxsLQNHszmOEswI7TkxnyucSCNL7cYmhO2+FsPGjbpGY8QeHyHJ9P5hxhia7FGNOBTEmONhifb1ulpGLr5m9ePvSywsDwnbcubkMgDd+eoCs+M1877yk8SSM0pyt/Grov0CVYDXVZRpUHLtuy7IarbsujFSqTEqXg+PgZGV757P0jxJz+cYIyZA340JjDLHY/vJvTgegcOUSWZzGvArdpjlVqgS5xwz72ftYFTK5lnVnS6xw4QvJM7yOC1nNYob1+Zp91fQodBTx1I2Sgb5nTc1s89n3Vu1o9V/HYofLJwuzPOpBUVr67vvCEJveIO1NeG/cK7JgXOv7M8ectWr432Xm2d46w38v65TOFQ94zQ9FXeLYAn8hkIuXSiysl3yt0VnvMNGI19fDK8XTNHeSaPru9oCc8Px9nwFi/fOdXUTrd9cX5Vk8fPJaACIbRdErmfhdrdSx4ZwJAPz6uicBKM9+F4Bxs6TC3OyJj/m2G/yef9xKlTm3zLCFhYWFhYWFhUW3RebVJJrPQpYsA+Dhq04C4Pq/PA7A5fv+FIBRz0hsZ9MAiRHpfZMwgQtrZIZVfplkNnoqDJ0VU9OZcN0WcW5rzpI4sdf7S0BOe5XmzOpF924SRiR3k0xP3e/tIb9vqfO2cTZL1b9otbDISWloZhgmQ6xVJlZ9T+w2/jJRmzjjItFcNLNyNXSsUUkHhDbv3iyMzsOPHgvAoLtU3wvN97ctjYywW1xA0wH7cNswXalIWIrsVtRSuirMtt42THTgz50kCg78e1pyO2w2fujcgxWHyXX93gkXA9B4uowb940TNmO/vKBqi3nh2+4I/vXjVHls5/enqiRO95aXRbe0/G7RxGStltxT934HWJ9N4R48uWl/9lIVGk0dWVM947uZ0ofL0AGPO47HAQDXxQ01sfECGf8WnSuMsB4/21MbqZw7AIBCR66Rp1OcBkZOe+UmFC/3LTc9DWY+yD/2/CsAF0z8mSz4JIn7xXUJhGLjuT7/ROPdtyXMWOpHtgpTeceT8q4w/D5doOajFtsmtP96YWNHny0qCkd9T5QdFp8tHpz99xRJu+NL5Pc8R1j1HEf6wuqQaN7OrhbB4g+WC1Md/EwUaIa9JjG1QxcuiB3TyFsqWCXP3pEvi3qJG5RzHiZCFuSh4nIzofyk963Go8h6UZwZdKIUsTtskig11VwqnooHdn0KgC+V5vuiOeJ1PX2evLs1fCpKWPnr/JUWG3vH3nXqdhaP/2E7i613zv4UgGtfOgeAMX8WD83gCrnWB74my+8cJ5EEOe/LGOr18BTHs+3nqWphYWFhYWFhYWGRZmwTnWEvhlWxabn/khnB7VVnAnDEHTIbu/qstwHYGJWYknNmSyGDET+RmUSLmUAXZDXbQtPIfJbevAcX7yaxwRtCwvxd1OdPag3NBLY9kzcZj9/3lxlT5EnR261UmaArI7Ea6NVRmQHfvuJoAIKnyG8difnJFDzW1ZgpD7hLYrDee0zYg38c/X0AVh8r65+wu9hhv8LvADgwX+L1mlt1ZoMoksyslhn+y1+KYsnAfwlT0+sNiUst3SLH8qyivRKdoBoRKnFZc36jF8esY/62B2ZHw9Tx1Oey5nyVB/DvDuxUsxjK9prlKX5G4rYRQpj/HS6xd5v3k6zjdUIY0me09PF9BkgW8lG9hL0Zmb1Bflfx4wOC+d4h61RFplXqMldE5Dw+qhHFl+lrJHZ83WLJxh6ssqyLPhAvxYiNEs8c1n03DWxPw7I8vvlpOaffJ7kBfx4qcXx5io2/f7NUSfz7Y1LDo+zPM33b72hKJ9FeBdQfOpHHf3MHAJUReT7o6m7ZhgeuTi0foC5FwdrOG/OCA6SC4fg86aMbIqLfqp9gJivVoNrQRzHFS69Ua5yexDEbo/RYXsNhX54AQCSq1DOy5LpnB6Tv5QUV6xmU7zmq/+er5bnqu/4sCOrqZPI9LyDrFQRiuT16mWZUW3x3/NsE1YjaT+17TqOM5f/zgrCBo//0FQBD1PgbSVUJRb07eNq2qjpruZL7VpwsjxWIooFTVOhbP7JBmFO3Ua7jMPzxvWY9Azmkv19F58o5lV/STlszme/k9X3VM9U4lTNd3tH6TJfFvxn9YwDWHyTXacPBct32GCkxxzlHy1i6sUG8dptqRYWidk2xd6jiufLOsfJheeY6M0T1aiRqrDSucenkLQD8caC8J7qhr+K0vWOwzLCFhYWFhYWFhUW3heNmkPlzHGc9UAtsyNhBOxcltDyX4a7r9ktkY2sPP6w9/LD28MPaww9rDz+sPfxQ9lgeZz/bI1KyB+xwfcTaw4/U7pdMvgwDOI7z2Y5SSzwd52Ltkf59dBVYe/hh7eGHtYcf1h5+pOtcdhSbWHv4Ye3hR6rnYcMkLCwsLCwsLCwsui3sy7CFhYWFhYWFhUW3xbZ4GX5oGxyzs5COc7H2SP8+ugqsPfyw9vDD2sMPaw8/0nUuO4pNrD38sPbwI6XzyHjMsIWFhYWFhYWFhUVXgQ2TsLCwsLCwsLCw6LawL8MWFhYWFhYWFhbdFhl7GXYcZ5LjON84jrPYcZxfZOq46YDjOEMdx3nXcZyFjuN86TjOFWr5TY7jrHYcZ676OyaJfVp7tNzvdmkTaw8/rD38sPbww9rDD2uPlrDPXD+sPfzolHvGdd1O/0Mq334HjARygHnALpk4dpraXwpMUP8XAYuAXYCbgGusPVKzx/ZuE2sPaw9rD2sPa4+uaxNrD2uP9v4yxQxPBBa7rrvEdd0m4DnghAwdO2W4rrvWdd056v9qYCEwOIVdWnu0xHZrE2sPP6w9/LD28MPaww9rj5awz1w/rD386Ix7JlMvw4OBlc2+ryL1m32bwHGcMmBP4BO16FLHcb5wHOdRx3F6J7gba4+W2CFsYu3hh7WHH9Yeflh7+GHt0RL2meuHtYcf6bpnMvUy7LSybLvTdHMcpxB4EZjqum4VcD8wChgPrAVuT3RXrSzrzvaAHcAm1h5+WHv4Ye3hh7WHH9YeLWGfuX5Ye/iRznsmUy/Dq4Chzb4PAdZk6NhpgeM42YjRn3Zd9yUA13XXua4bcV03CjyMuB4SgbVHS2zXNrH28MPaww9rDz+sPfyw9mgJ+8z1w9rDj3TfM5l6Gf4UKHccZ4TjODnA6cBrGTp2ynAcxwEeARa6rntHs+WlzVY7EViQ4C6tPVpiu7WJtYcf1h5+WHv4Ye3hh7VHS9hnrh/WHn50xj2Tlb7mxYfrumHHcS4F3kCyGB91XffLTBw7TTgAOAeY7zjOXLXseuAMx3HGI+6FZcCURHZm7dES27lNrD38sPbww9rDD2sPP6w9WsI+c/2w9vAj7feMLcdsYWFhYWFhYWHRbWEr0FlYWFhYWFhYWHRb2JdhCwsLCwsLCwuLbgv7MmxhYWFhYWFhYdFtYV+GLSwsLCwsLCwsui3sy7CFhYWFhYWFhUW3hX0ZtrCwsLCwsLCw6LawL8MWFhYWFhYWFhbdFv8PSCysRqLbFWMAAAAASUVORK5CYII=", + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "%matplotlib inline\n", + "plt.figure(figsize=[12, 4])\n", + "for i in range(20):\n", + " plt.subplot(2, 10, i+1)\n", + " plt.imshow(X_train[i].reshape([28, 28]))\n", + " plt.title(str(y_train[i]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "GLxKzWmouh8X" + }, + "outputs": [], + "source": [ + "#< a whole lot of your code > " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "IoT9Qr_-uh8g" + }, + "source": [ + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# SPOILERS!\n", + "\n", + "Recommended pipeline:\n", + "\n", + "* Adapt logistic regression from seminar assignment to classify one letter against others (e.g. A vs the rest)\n", + "* Generalize it to multiclass logistic regression.\n", + " - Either try to remember lecture 0 or google it.\n", + " - Instead of weight vector you'll have to use matrix (feature_id x class_id)\n", + " - Softmax (exp over sum of exps) can be implemented manually or as `nn.Softmax` (layer) or `F.softmax` (function)\n", + " - Probably better to use STOCHASTIC gradient descent (minibatch) for greater speed\n", + " - You can also try momentum/rmsprop/adawhatever\n", + " - in which case the dataset should probably be shuffled (or use random subsamples on each iteration)\n", + "* Add a hidden layer. Now your logistic regression uses hidden neurons instead of inputs.\n", + " - Hidden layer uses the same math as output layer (ex-logistic regression), but uses some nonlinearity (e.g. sigmoid) instead of softmax\n", + " - You need to train both layers, not just the output layer :)\n", + " - 50 hidden neurons and a sigmoid nonlinearity will do for a start. Many ways to improve.\n", + " - In ideal case this totals to 2 `torch.matmul`'s, 1 softmax and 1 ReLU/sigmoid\n", + " - __Make sure this neural network works better than logistic regression!__\n", + "\n", + "* Now's the time to try improving the network. Consider layers (size, neuron count), nonlinearities, optimization methods, initialization — whatever you want, but please avoid convolutions for now.\n", + "\n", + "* If anything seems wrong, try going through one step of training and printing everything you compute.\n", + "* If you see NaNs midway through optimization, you can estimate $\\log P(y \\mid x)$ as `F.log_softmax(layer_before_softmax)`." + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "homework.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.2" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/week02_autodiff/mnist.py b/week02_autodiff/mnist.py new file mode 100644 index 00000000..7b1edeb3 --- /dev/null +++ b/week02_autodiff/mnist.py @@ -0,0 +1,63 @@ +import sys +import os +import time + +import numpy as np + +__doc__="""taken from https://github.com/Lasagne/Lasagne/blob/master/examples/mnist.py""" + +def load_dataset(): + # We first define a download function, supporting both Python 2 and 3. + if sys.version_info[0] == 2: + from urllib import urlretrieve + else: + from urllib.request import urlretrieve + + def download(filename, source='http://yann.lecun.com/exdb/mnist/'): + print("Downloading %s" % filename) + urlretrieve(source + filename, filename) + + # We then define functions for loading MNIST images and labels. + # For convenience, they also download the requested files if needed. + import gzip + + def load_mnist_images(filename): + if not os.path.exists(filename): + download(filename) + # Read the inputs in Yann LeCun's binary format. + with gzip.open(filename, 'rb') as f: + data = np.frombuffer(f.read(), np.uint8, offset=16) + # The inputs are vectors now, we reshape them to monochrome 2D images, + # following the shape convention: (examples, channels, rows, columns) + data = data.reshape(-1, 1, 28, 28) + # The inputs come as bytes, we convert them to float32 in range [0,1]. + # (Actually to range [0, 255/256], for compatibility to the version + # provided at http://deeplearning.net/data/mnist/mnist.pkl.gz.) + return data / np.float32(256) + + def load_mnist_labels(filename): + if not os.path.exists(filename): + download(filename) + # Read the labels in Yann LeCun's binary format. + with gzip.open(filename, 'rb') as f: + data = np.frombuffer(f.read(), np.uint8, offset=8) + # The labels are vectors of integers now, that's exactly what we want. + return data + + # We can now download and read the training and test set images and labels. + X_train = load_mnist_images('train-images-idx3-ubyte.gz') + y_train = load_mnist_labels('train-labels-idx1-ubyte.gz') + X_test = load_mnist_images('t10k-images-idx3-ubyte.gz') + y_test = load_mnist_labels('t10k-labels-idx1-ubyte.gz') + + # We reserve the last 10000 training examples for validation. + X_train, X_val = X_train[:-10000], X_train[-10000:] + y_train, y_val = y_train[:-10000], y_train[-10000:] + + # We just return all the arrays in order, as expected in main(). + # (It doesn't matter how we do this as long as we can read them again.) + return X_train, y_train, X_val, y_val, X_test, y_test + + + + diff --git a/week02_autodiff/notmnist.py b/week02_autodiff/notmnist.py new file mode 100644 index 00000000..4c50f59b --- /dev/null +++ b/week02_autodiff/notmnist.py @@ -0,0 +1,52 @@ +import os +from glob import glob + +import numpy as np +from matplotlib.pyplot import imread +from skimage.transform import resize +from sklearn.model_selection import train_test_split + + +def load_notmnist(path='./notMNIST_small', letters='ABCDEFGHIJ', + img_shape=(28, 28), test_size=0.25, one_hot=False): + # download data if it's missing. If you have any problems, go to the urls + # and load it manually. + if not os.path.exists(path): + print("Downloading data...") + assert os.system( + 'wget http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz') == 0 + print("Extracting ...") + assert os.system( + 'tar -zxvf notMNIST_small.tar.gz > untar_notmnist.log') == 0 + + data, labels = [], [] + print("Parsing...") + for img_path in glob(os.path.join(path, '*/*')): + class_i = img_path.split(os.sep)[-2] + if class_i not in letters: + continue + try: + data.append(resize(imread(img_path), img_shape)) + labels.append(class_i,) + except BaseException: + print( + "found broken img: %s [it's ok if <10 images are broken]" % + img_path) + + data = np.stack(data)[:, None].astype('float32') + data = (data - np.mean(data)) / np.std(data) + + # convert classes to ints + letter_to_i = {l: i for i, l in enumerate(letters)} + labels = np.array(list(map(letter_to_i.get, labels))) + + if one_hot: + labels = (np.arange(np.max(labels) + 1) + [None, :] == labels[:, None]).astype('float32') + + # split into train/test + X_train, X_test, y_train, y_test = train_test_split( + data, labels, test_size=test_size, random_state=42) + + print("Done") + return X_train, y_train, X_test, y_test diff --git a/week02_autodiff/seminar_pytorch.ipynb b/week02_autodiff/seminar_pytorch.ipynb new file mode 100644 index 00000000..440fa471 --- /dev/null +++ b/week02_autodiff/seminar_pytorch.ipynb @@ -0,0 +1,896 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Hello, pytorch\n", + "\n", + "![img](https://pytorch.org/tutorials/_static/pytorch-logo-dark.svg)\n", + "\n", + "__This notebook__ will teach you to use pytorch low-level core. You can install it [here](http://pytorch.org/). For high-level interface see the next notebook.\n", + "\n", + "__Pytorch feels__ differently than tensorflow/theano on almost every level. TensorFlow makes your code live in two \"worlds\" simultaneously: symbolic graphs and actual tensors. First you declare a symbolic \"recipe\" of how to get from inputs to outputs, then feed it with actual minibatches of data. In pytorch, __there's only one world__: all tensors have a numeric value.\n", + "\n", + "You compute outputs on the fly without pre-declaring anything. The code looks exactly as in pure numpy with one exception: pytorch computes gradients for you. And can run stuff on GPU. And has a number of pre-implemented building blocks for your neural nets. [And a few more things.](https://medium.com/towards-data-science/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b)\n", + "\n", + "And now we finally shut up and let pytorch do the talking." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1.12.0+cu102\n" + ] + } + ], + "source": [ + "# if running in colab, execute this:\n", + "# !wget https://raw.githubusercontent.com/yandexdataschool/Practical_DL/fall19/week02_autodiff/notmnist.py -O notmnist.py\n", + "# !pip3 install torch==1.0.0 torchvision\n", + "\n", + "from __future__ import print_function\n", + "import numpy as np\n", + "import torch\n", + "print(torch.__version__) # it's okay if your version is different, as long as it's 1.0 or newer" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "X :\n", + "[[ 0 1 2 3]\n", + " [ 4 5 6 7]\n", + " [ 8 9 10 11]\n", + " [12 13 14 15]]\n", + "\n", + "X.shape : (4, 4)\n", + "\n", + "add 5 :\n", + "[[ 5 6 7 8]\n", + " [ 9 10 11 12]\n", + " [13 14 15 16]\n", + " [17 18 19 20]]\n", + "\n", + "X*X^T :\n", + "[[ 14 38 62 86]\n", + " [ 38 126 214 302]\n", + " [ 62 214 366 518]\n", + " [ 86 302 518 734]]\n", + "\n", + "mean over cols :\n", + "[ 1.5 5.5 9.5 13.5]\n", + "\n", + "cumsum of cols :\n", + "[[ 0 1 2 3]\n", + " [ 4 6 8 10]\n", + " [12 15 18 21]\n", + " [24 28 32 36]]\n", + "\n" + ] + } + ], + "source": [ + "# numpy world\n", + "\n", + "x = np.arange(16).reshape(4, 4)\n", + "\n", + "print(\"X :\\n%s\\n\" % x)\n", + "print(\"X.shape : %s\\n\" % (x.shape,))\n", + "print(\"add 5 :\\n%s\\n\" % (x + 5))\n", + "print(\"X*X^T :\\n%s\\n\" % np.dot(x, x.T))\n", + "print(\"mean over cols :\\n%s\\n\" % (x.mean(axis=-1)))\n", + "print(\"cumsum of cols :\\n%s\\n\" % (np.cumsum(x, axis=0)))" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "X :\n", + "tensor([[ 0., 1., 2., 3.],\n", + " [ 4., 5., 6., 7.],\n", + " [ 8., 9., 10., 11.],\n", + " [12., 13., 14., 15.]])\n", + "X.shape : torch.Size([4, 4])\n", + "\n", + "add 5 :\n", + "tensor([[ 5., 6., 7., 8.],\n", + " [ 9., 10., 11., 12.],\n", + " [13., 14., 15., 16.],\n", + " [17., 18., 19., 20.]])\n", + "X*X^T :\n", + "tensor([[ 14., 38., 62., 86.],\n", + " [ 38., 126., 214., 302.],\n", + " [ 62., 214., 366., 518.],\n", + " [ 86., 302., 518., 734.]])\n", + "mean over cols :\n", + "tensor([ 1.5000, 5.5000, 9.5000, 13.5000])\n", + "cumsum of cols :\n", + "tensor([[ 0., 1., 2., 3.],\n", + " [ 4., 6., 8., 10.],\n", + " [12., 15., 18., 21.],\n", + " [24., 28., 32., 36.]])\n" + ] + } + ], + "source": [ + "# pytorch world\n", + "\n", + "x = np.arange(16).reshape(4, 4)\n", + "\n", + "x = torch.tensor(x, dtype=torch.float32) # or torch.arange(0,16).view(4,4)\n", + "\n", + "print(\"X :\\n%s\" % x)\n", + "print(\"X.shape : %s\\n\" % (x.shape,))\n", + "print(\"add 5 :\\n%s\" % (x + 5))\n", + "print(\"X*X^T :\\n%s\" % torch.matmul(x, x.transpose(1, 0))) # short: x.mm(x.t())\n", + "print(\"mean over cols :\\n%s\" % torch.mean(x, dim=-1))\n", + "print(\"cumsum of cols :\\n%s\" % torch.cumsum(x, dim=0))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## NumPy and Pytorch\n", + "\n", + "As you can notice, pytorch allows you to hack stuff much the same way you did with numpy. No graph declaration, no placeholders, no sessions. This means that you can _see the numeric value of any tensor at any moment of time_. Debugging such code can be done with by printing tensors or using any debug tool you want (e.g. [gdb](https://wiki.python.org/moin/DebuggingWithGdb)).\n", + "\n", + "You could also notice the a few new method names and a different API. So no, there's no compatibility with numpy [yet](https://github.com/pytorch/pytorch/issues/2228) and yes, you'll have to memorize all the names again. Get excited!\n", + "\n", + "![img](http://i0.kym-cdn.com/entries/icons/original/000/017/886/download.jpg)\n", + "\n", + "For example, \n", + "* If something takes a list/tuple of axes in numpy, you can expect it to take *args in pytorch\n", + " * `x.reshape([1,2,8]) -> x.view(1,2,8)`\n", + "* You should swap _axis_ for _dim_ in operations like mean or cumsum\n", + " * `x.sum(axis=-1) -> x.sum(dim=-1)`\n", + "* most mathematical operations are the same, but types an shaping is different\n", + " * `x.astype('int64') -> x.type(torch.LongTensor)`\n", + "\n", + "To help you acclimatize, there's a [table](https://github.com/torch/torch7/wiki/Torch-for-Numpy-users) covering most new things. There's also a neat [documentation page](http://pytorch.org/docs/master/).\n", + "\n", + "Finally, if you're stuck with a technical problem, we recommend searching [pytorch forumns](https://discuss.pytorch.org/). Or just googling, which usually works just as efficiently. \n", + "\n", + "If you feel like you almost give up, remember two things: __GPU__ an __free gradients__. Besides you can always jump back to numpy with x.numpy()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Warmup: trigonometric knotwork\n", + "_inspired by [this post](https://www.quora.com/What-are-the-most-interesting-equation-plots)_\n", + "\n", + "There are some simple mathematical functions with cool plots. For one, consider this:\n", + "\n", + "$$ x(t) = t - 1.5 * cos( 15 t) $$\n", + "$$ y(t) = t - 1.5 * sin( 16 t) $$\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "t = torch.linspace(-10, 10, steps=10000)\n", + "\n", + "# compute x(t) and y(t) as defined above\n", + "x = # YOUR CODE\n", + "y = # YOUR CODE\n", + "\n", + "plt.plot(x.numpy(), y.numpy())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "if you're done early, try adjusting the formula and seing how it affects the function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Automatic gradients\n", + "\n", + "Any self-respecting DL framework must do your backprop for you. Torch handles this with the `autograd` module.\n", + "\n", + "The general pipeline looks like this:\n", + "* When creating a tensor, you mark it as `requires_grad`:\n", + " * __```torch.zeros(5, requires_grad=True)```__\n", + " * torch.tensor(np.arange(5), dtype=torch.float32, requires_grad=True)\n", + "* Define some differentiable `loss = arbitrary_function(a)`\n", + "* Call `loss.backward()`\n", + "* Gradients are now available as ```a.grads```\n", + "\n", + "__Here's an example:__ let's fit a linear regression on Boston house prices" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAs30lEQVR4nO2df4wc53nfv88tl+IeZWuPMaVSZ1FkDINqZVpkdLAZsAgiJhWd0LIuUiRGkAoVMKL+4QIWIVx9DoyQCmT4GtaR+lcBNTHKVLJKSpRPVFiANkymromKDuk7WmZFNnElU16xIl1yFUu3FJd3T/+4nePe7PvOr52ZnZn9fgDi7uZ2Z94Z3n7nmeenqCoIIYTkj4FeL4AQQkg0KOCEEJJTKOCEEJJTKOCEEJJTKOCEEJJTlqR5sI997GO6Zs2aNA9JCCG558SJE79U1ZXu7akK+Jo1a3D8+PE0D0kIIblHRH5u2k4XCiGE5BQKOCGE5BQKOCGE5BQKOCGE5BQKOCGE5JRAWSgi8haAXwGYBXBVVUdEZAWAvQDWAHgLwIOqeinuBU5O1bD70BnU6g2URDCriuFqBWNb12F04/DC6742+Tq+fews5gy9ucoDwKwCcwqURPDQZ2/BU6Prrfu+67aVOHL6At6pN3Cz4VhR1h90X7bXh90PIaT4SJBuhC0BH1HVX7Zt+3MAF1V1QkTGAQyp6le89jMyMqJh0ggnp2r46suvo9Gc7fhdpVzCN+5bj9GNw/ja5Ot47rWzgfcLAJs/sQI/Pvuecd9exwqDaf1e+7K9/v47h7H/RC3wfgghxUJETqjqiHt7Ny6UewHsaX2/B8BoF/sysvvQGavANpqz2H3oDADghWNvh9730Z9dDCTe7mOFwbR+r33ZXv/CsbdD7YcQ0h8EFXAF8F0ROSEij7W23aSq5wCg9fVG0xtF5DEROS4ixy9cuBBqce/UG4F+P5tCT3O/tYR5T9jttvOLsiZCSHEIKuCbVfU3APwegC+JyG8FPYCqPquqI6o6snJlRyWoJzdXK4F+XxIJtd8o+K0lzHvCbredX5Q1EUKKQyABV9V3Wl/PA/gOgM8AeFdEVgFA6+v5uBc3tnUdKuWS8XeVcgljW9cBAB767C2h9735Eyus+/Y6VhhM6/fal+31D332llD7IYT0B74CLiLLReQjzvcA7gbwUwAHADzaetmjAF6Je3GjG4fxjfvWY9hlaQ9XK4sCeE+Nrscjm1ZjwGKIlwew8LuSCB7ZtBrP//FvWvf9yKbVGK5WIK5jTU7VsHniMNaOH8TmicOYnKoFXr97X2Fe/9To+lD7IYT0B75ZKCLy65i3uoH5tMNvq+rXReTXAOwDsBrAWQAPqOpFr32FzUKJk27T8MJmlBBCSFzYslACpRHGRa8E3CS+AuDhTavx1Oj6QPvYPHEYNUPQcLhawdHxLXEtlRBCOkgijTA3mNLzFMDzr531dYM4hM0cIYSQpOkLAbeJrAKBc6nDZo4QQkjS9IWAe4lsUAs6bEYJIYQkTV8I+NjWdbBlige1oMNmlBBCSNKkOlKtV4xuHMbxn1/E86+dRXvINqwFPbpxmIJNCMkMfWGBA/O54k9v30ALmhBSGPpGwAkhpGj0hQsF6MwFr9Ub+OrLrwMArXBCSC7pGws8bGtXQgjJOoWywL3K5VmIQwgpGoWxwB0XSa3egOKai8SptGQhDiGkaBTGAvdzkcxcudrxHhbiEELyTGEE3OYKcSxxt7hXK2Xs+sLtDGASQnJLYVwoXtNsTLMvl1+3hOJNCMk1hRFwW68S2zzJWr0RaCgDIYRklcIIuK1XybBHkNId6CSEkDxRGB84YO9VYvKBOziBTrpTCCF5o1ACbsIR5t2Hzhgn6gDMBSeE5JPCCbi7mOeu21biyOkLeKfeQEnE6BNnLjghJI8USsBN/U6ee+3swu9N4s1ccEJIXimUgJuKeUyURDCnGmk6PSGEZIVCCbjNx+1mThVvTmxLeDWEEJIshRJwm4/bTVo+b6/mWoQQ0i2FEvAg4p2Wz5v9xwkhSVOYQh4AnkU7ACAA7r8znbmW7D9OCEmaQgn42NZ1KJds8+cBBXDk9IVU1sL+44SQpCmUgAMAfLwoaQko+48TQpKmUAK++9AZNOe8FTwtAbU112LOOSEkLnIbxDRlePhZ12kKaHsJP7NQCCFJIBogcyMuRkZG9Pjx413vx53hAcyL83VLBlBvNI3vGbYIKFP9CCFZR0ROqOqIe3suLXBbhsey8gAq5VKHsH/jvvVGUWaqHyEkz+TSB25zldRnmsae4DYxZqofISTP5NICv7laMZbN31ytWHuCm2CqHyEkz+TSAo8rw4OpfoSQPJNLAbeNTwvrt2aqHyEkzwR2oYhICcBxADVV/byIrACwF8AaAG8BeFBVLyWxSBNhXCVe+wCY6kcIySdhfOBfBvAGgI+2fh4H8H1VnRCR8dbPX4l5fYkTx43AD6YqEkKSIJALRUQ+DmAbgL9s23wvgD2t7/cAGI11ZQXBSVWs1RtQXEtVnJyq9XpphJCcE9QH/gyAfwtgrm3bTap6DgBaX280vVFEHhOR4yJy/MKFdBpJZQmmKhJCksJXwEXk8wDOq+qJKAdQ1WdVdURVR1auXBllF7mGqYqEkKQI4gPfDOALIvL7AJYB+KiIPAfgXRFZparnRGQVgPNJLjSveOWsE0JIN/ha4Kr6VVX9uKquAfBHAA6r6iMADgB4tPWyRwG8ktgqY2ZyqobNE4exdvwgNk8cTtQfzVRFQkhSdFOJOQFgn4h8EcBZAA/Es6RkSbv/CVMVCSFJkctuhN2weeKw0aUxXK3g6PiWHqyIEEK8KVQ3wm6wBQ9r9QY2TxymlUwIyQ25LKXvBlvwUADmahNCckXhBNwvQGkKKgo6R2kyV5sQknUK5QM3TeopDwiuX7YE9ZnmgmsEWBxUNPnEHZ7ZvoGuFEJIT7H5wAtlgZuqHptzikszzUWuEQA4Or4Fb05sw9HxLRj2yMkee/EkXSmEkExSKAEPUt1oco2Y3CoOzTnFrgOnYlkfIYTESaEEPGh1o1vonf7iNmyDkgkhpJcURsAnp2r44MOrgV5rEvq0/dxpVoMSQopJIfLATcFLABgsD6A5p2jOXgvUepWxDw2WcWmm09oWmT9GEJEP0vs7bDUo+4kTQkwUwgI3BS8BYGj5ddj9h3cEHr22857bUS5Jx3ZVBMoL9+v97Vjdj++dDtxilv3ECSE2CmGBe7VsDTNxx3ndE/tOYtaVXukIrNe+/Hp/m54S/M7Da5+0wgnpbwoh4N20bHXcE7V6AyWRDuFuxy/LxetGYntK8Fsv+4kTQmwUwoUStWVru3sCgKd4A/YbguMasb375mrFV3Bt67Udk/3ECSGFEHAnDTCor9shiFXsYBNY903A9j4vwfVaL/uJE0Js5MaF4peJEcTX7d6HVwm9m2Vl873O6yYw7Fqn2wdeKZd8bzTsJ04IsZELAY9jCINpH2G4NNM0HtPmGhFgUX/xboQ4TCCWENI/5ELA48jECOMusWE6ZtWSO14dLHdsoxATQuIkFz7wODIx4sracO/HFvdMsckjIaRPyYUFHsdk97A+b1OPcNMx37P0SbFtjwqrMQkhbnJhgceRiWHaR3lAjJWXQ4NlPLxpdaBjppHmx2pMQoiJXFjgQQKAQbJUTPvw2u/IrSt8rd67bluJ5187u8hajzvNj9WYhBAThZjIY2pmZUvRm5yqYdeBUwstYocGy9h5z+0AgmWItFdumtwsAuDhTavx1Ki9PW1Y1o4fNLpzBMCbE9tiOw4hJJsUeip9UAt1cqqGsRdPojl3TQ4vzTTx+N7pRe+1pSm6bxQmUVUAR05f8FxvWH92HDEAQkjxKISAB81S2X3ozCLx9qK9CZUjtgM+vVL81gNEy2kf27rO+ITBakxC+ptcBDH9CBpIDJtK6IirEzwMIt5e6wH8OxaaiNoqgBBSbAphgdss1LtuW4nNE4cXXBU3VMqhxqOVREIX//hZxrZURr+bC4uACCFuCiHgjrC1BycHBNj7o7cXXCa1egPlkmAAwFyAfVbKpdDiPSCLrWlTADVofjkhhPhRCBeKw4dXr0nzB1dmO/zdzVnFDYNlVCudZe7AfFYHcM1FMewjqiWZf0e1Uka5JHAOZ8vT3n3ojDWbhP5sQkhYCiPgQXudXJpp4r1GE8PVCh7ZtHqRX/np7Rvw1sS2hSZUXkOSh6sVfPPB+XFt9UZz0dxNwOzXtrlJFOkPVSaE5J9CuFCAcAFKp5px/4kavnHffL727kNnsGPvNHYfOoO7bluJ/Sdq1huC418POyLNlg7oZ+kTQoiJwljgUXzIjeYsvvryT7Bj7/SiMvXnXjvr2eP7G/etx5HTF0KPSONwBkJInBRGwI29TkqCzk4ni2k056yj0Nw4Pb5HNw5HGpHWi3RAZ9zb2vGD2DxxmP1TCCkQhXGh2Hqd7HBVWXZDu0Xt1d3QPYnHvc60/N1xDMLo5tjsnkhIshRGwAGzODp9S7rFbVHbcs+zVGDTqyZYvbxxENJPFMaFYsPkWglLSaRDmON2hyTh6ohjEEYUolSbEkLC42uBi8gyAD8AcF3r9S+p6k4RWQFgL4A1AN4C8KCqXkpuqdEwuVbCWOReVrXJ4je5DtzHN7XCTcJi7VUTrF7dOAjpN3zbyYqIAFiuqu+LSBnADwF8GcB9AC6q6oSIjAMYUtWveO0rqXayYdk8cTiQiHv5sk1MTtUw9tLJRTnhpYH56s/2oiL3TcG2nuFqZdFg5LCY2uyWBwTXL1uC+kwzMd90UudDSL9iayfr60LRed5v/Vhu/VMA9wLY09q+B8BoPEuND5tbYmzrOt/sFEdswojbk6+e6ijomZ3TjopQtzshKYvV7eapVsqAzBczBZ3sE8W1w3RJQtIhkA9cREoiMg3gPIDvqeoxADep6jkAaH290fLex0TkuIgcv3DBu092nHiNIRvdOOyZOijAQiOsMMJlmk5vo12ckxzLNrpxGEfHt+DNiW1Yft2SQBWjDlFHubF7IiHpECgLRVVnAWwQkSqA74jIp4IeQFWfBfAsMO9CibLIKPhlYAwNlq2Cq8CiSswksigGRLB2/CBurlaMlZ9JWKxhLf1usljYPZGQ5AmVhaKqdQB/C+BzAN4VkVUA0Pp6Pu7FdYOXWE1O1fD+ZXufEwCRsihsTbJMzKouKum//87hxC3WsJY+g5GEZJsgWSgrATRVtS4iFQC/C+DfATgA4FEAE62vryS50LB4ZWCEmczTTq3eWGQ1Hzl9YVFmya4v3N4xss1NyTDVp9GcxZHTF3wDfN0Wx4Sd7MNRboRkmyAulFUA9ohICfMW+z5V/RsR+Z8A9onIFwGcBfBAgusMjde0+G6qM9v7pTjU6g3s2DsNxbwVbhsaIQDmLFk/7VatLRWx21RDW7UqR7kRkk8KMZXejSl9rn1afNA0wqjYhjYAZgscuJb1Ylu7bX9Jp+axJJ6Q3lPoqfRuTMG39mnxJssyThR20TWJd7tVa1u7jW790X4CzWAkIdmlkALuF3xrdyUkZYkr5q1j2/5LIphT7RDNsILcjT+aPUsIyTeF7IUSJNvCyY9OaphCScRTjOdU8WZr+k+7WIYV5JkrVyP3TWHPErbbJfmmkAIephIwrMXrpPk549icbW6cNEEbNqEO23zr0kwzUHGNiX5PE4xaqERIViikCyVMtkWY5lYlEXzzwTs8G1gNWIKU7XhlckRx70RtEdtPaYImX3+v2u0SEheFzEIJg7HhU0kAhTGf26/n99rxg75l+k42TBDWjB8M9DoB8ObEtkCvdTCde9Z6mseB7TxtQewo15KQJOmrLJQwOEL15KunFkrrly9dgs/fsQovHHvbWHTzeGv4salVbNWjRB+YD26+cOxtjNy6IpBIegVC24liNYfNC88rNkvbltJZxCcQUkz6XsAdLjfnFr6vN5rYf6Lm6Qqp1RsYe+nkIku9Vm+gPCAol6SjaVQ7s6od2R62dL4gKY/dFNd0kyaYlxxxm09/VrXDEmehEskThQxihsXLQvOiOdvZKrY5p1gyIL7ZLe3ZHn6dE79x33rrWkzTgtIgTwFAm0Xt9Jxh10SSVyjg8LfQwtJozmFs6zo8s32D5/ud4/ql841uHLaW4M+p9kRw8pSC6JWV1N5uN2z/d0J6DQUcwSy0sDiZDF7Ws3PcIOl8SfYMj0KeUhDZn5wUFfrA4d20yfER27JVbL5ud9WnV1OoIOl8WWsslbcURLYEIEWEFjiCW2jXLbl2uYYGy9j9h3dgaNDcA9xd9em1/yCFR1mzIjk2jZDe0/d54DbaMyyqg2W8f/lqqB7iy5eWMHNldlF2hlfWhtfvvjb5+qLWuE6jrLBDl+MmL1kohOQdWx544QU8isiY3CXd4Aiuu0NhkKKZr02+vqj3uG3fvRZzQkhy9GUhT5Rue5NTNTyx76RvOXwY1PXVIUjZ9gvH3g6077Q6CdLqJiQ7FFrAw/a6cAQ/TvH2wytrY3LKu5jITfu52ab6dCO+QW+IFHlC0qHQAh421e3JV08lNuTBRtUSBHXEMizv1BsdPvNavYGxF08CgoWsmSgWe5AbInuME5IehRbwMKluk1M1zx4mSeEY2G6r9YMPr0a6mdxQKXfMAgXMjbkazVk8+eqpwNZykBsiO/wRkh6FTiMMk+rWqwrC9xpNY1m6bTCyF5VyCSLeI9jcXJppBi6HD1JMlKcCHxIfHIzRGwot4EFzpyenaokOOfaiUh7AE/tORrK2B8sDHedW7/IpwqscPsgNMWsVoyR58tQXp2gU2oUC+FfgRfU1x8VMWxfEKO8dwrzF/X/fu4zH905bW6SGwWQtT07VOmIEQ4Nl7Lzn9kVBU9ONsFySWAt8ggZJGUxNB7rNekehLfAgmP748oIAC4LpiLafeJdEFiz2asUcQB0QWWQ9TU7VMPbSyY4YwfuXry783rHAjMSY1BPU2qNVmB50m/WOvhfwXrlOusVdFBQUZ1bnBx9exefvWGXsluj0K3fEbvehM8aeL805xe5DZ3xvgs7r4iBoF8Q8dUvMO3Sb9Y6+FvDJqZpxIHEe6NaorTea2Pujt3H/ncPGbonO5KE14wc9b3Lv1BuBLC3ba8IGv2z7qdUbi/ZBqzA92Bend/S1gO8+dCbOp/vEGBosY/MnVsS+3+ac4sjpC9Ze40G4uVoJZGnZUjfDujm8jtW+D1t+Pa3C+Mlao7V+ovBBTC/yYo2pAkd/djGRfTsBviiupPLAteDk2Esnra11vVI3wwa/goyYazRncd2SAY5LSxG26+0NfS3gUYUrbaLkhAdlQAS1eiO0T71aKePzd6yyZp44eDXZiuLmcA9itq35vUYTT2/fwCwUUmj6WsCDWHNFx8laCSrew219Vfyu3XC1gqPjW4y/m5yqYSDkVHh3WuDT2zdg14FTxhvczdVKolYhUxRJFih8O1k/2vOXo2Z22Ih7f1mhPCC4ftmSQK0HBDD2PreJv63FrnEi0oBgDsCsq01AeUCw+4E7EhVv03Qk+n1JUtjayfZ1EBPAwlDbtya24entGyLNv7Tx8KbV1ok9eaY5p4H7xpiCk7a0w5KIVQRN72nOaYd4A8D1y5YkKqRMUSRZoe8FvB1HzL1EPGja4dBgGU+NrsfUn96NZ2K+MQTlme0bUCln47+4XeBsPu451dCNtEx0207AD6YokqyQjU93xjDltQLzgbuHN61elC61+RMrOkRdAGz79KqFn4PcGJLgyVdP4XIXpfpx4whclMKPMOl/SacKsnCFZAUKuAEnr9Xt/qg3mth/ooaxrevw5sQ2HB3fguf/+Dfx8KbVi0RcAew/UesoR5+5cjWdE2hxaaZpzYdOEpvVXx0sY/PE4YV4w+L3eKf4mW6q5QFBaWDxnuLuuxJ0LUxRJL2AAm5hdOMwBpd2JumYfJ1HTl+wjksDrgW9etVvPO1q0zlFp9iWBO9fvrqQcui+Xst8XD2mYpHtn7ml8w84hagxC1dIVvDNQhGRWwD8NYB/AmAOwLOq+h9EZAWAvQDWAHgLwIOqeslrX1nMQvFi7fhBox4IgDcntgV+nWN19oJeZcIsLQmutAp7RIBlSwbQ8HHnhB3QbLuuXumLvYbphyQK3WShXAXwhKr+UwCbAHxJRP4ZgHEA31fVTwL4fuvnQhHU1+n3ul4Gt6qD5Z70e7nSVpWpCl/xBjoHNEfti5LU9e52aAE7JJK48RVwVT2nqj9uff8rAG8AGAZwL4A9rZftATCa0Bp7RlBfp9/rehXcqpRLUM1nLnqjOYsn9p2M1Bfl5mol9gkxcYgv0w9J3ITygYvIGgAbARwDcJOqngPmRR7AjZb3PCYix0Xk+IULF7pcbroE9XX6vc6W1ZJkAMJZw3sJluEnjbutrRvbjfOu21bGbunGIb5+TwwcS0bCErgSU0SuB/DfAXxdVV8WkbqqVtt+f0lVh7z2kTcfeJy4J8UD84G9q7Mau4X8yKbVeGp0PQC7n7gXLC0JmhHO1/GJm3zH7ZW0zjQi21SibnzjtjgHMJ9vH8SP7eWzN7V1YHUnceiqElNEygD2A3heVV9ubX5XRFa1fr8KwPm4FltETJkqzVlFdbDcYUV267P+m5PnFr63Wf+94ErEm1Wt3sCOvdOLLOodrV7luw+dwV23rUSlXPKdStSNb9zLDRbUuvdytdG9QqLgK+AiIgD+CsAbqvoXbb86AODR1vePAngl/uUVB5t41GeaHe4Xd155WOqN5sIjeLt7Jy2S6F3uluT2gOfzr50N1JCsm1iE140wqNB6udpY3UmiEKQb4WYA/xLA6yIy3dr2JwAmAOwTkS8COAvggURWWBBuqJSNXfNuqJStXfOee+1s5OM5fl/gWq/mNNwpj2xajSOn0411BLHquy20cf5/Ht87bfx9UKG1/V/bWhuzupN4ESQL5YeqKqr6aVXd0Pr331T1/6nq76jqJ1tfk5k4UBAMU8sAAP94uWl8/H5qtLMSNCyN5iyefPXUQmDs4gcfdrW/IOw/UcuMz91BANx/Z/etZUc3DlufZLoVWlZ3kiiwEjMlbA2W5tTuQ42jKdOlmeaC7zhILnYYBgw3pUZz1jhjM2m8jqhAbE8FSQktqzuTpagZPn090CFNvKb/2MaIpTkxaLhawV23rQzkthEAT2/fgB0Wd8KsaqoVoOUBwfbP3IIjpy9Yr1dcvmT3RKA4qymzNpasKFWj7v7tbvdinqEFnhJ+2SAmgTG9x7E04wxKOul1I7euCGQ9P7xpNUY3DvsOGA5LZMtdgJFbV3hawXE29XK6SzoNzfIuAiaKVDVa5AwfWuAp4XzIn9h3MvAYMT9rzy8oWR4A/LwmzuO/84G1peAt7LMkGLl1PsskzpF0Ts4zAOzYOx3qBtCcVd8PY1yDp7JklfqtpZu1Rhk4nVWKnOFDAU8R5w/fVLBhsx69Hqv9BPT6ZWVcbs51jCG7ftkS1Geaiz7UG578biAhdsSyfV1+g439cDevsmV6eOH3YXRXpEYRtyw9ivutpdu1Fkn0ipzhQxdKysQZrHL2ZcOUY777gTsw9ad3L3r8n5yqGVMcbbR/iB13QhQq5RKe2b6hww0RxT10c7USeCBEVPdAlh7F/dbS7VqLNLSiyBk+tMB7gMmqjvq4O7px2GoBB53MHlaA3B/iyala6KCl1/xLvycL97HaP4xjL51Ec7ZzJbV6Axue/C52feH2yO6BLFmlfmvpdq220v4gopclNxOQbOC511DAM0C3j7vdfNiAcAJk2u/uQ2dCBy1t8y+dD3+jOYsBmU+zdB///juHceT0BeuH8clXTxmHZ9QbTYy9eBJNwyBkYP66rx0/aP2AZ+lR3G8t3a41quhlyc3UTtYyfOKCAp4Bug0YBfmwmawi5z1e4lsaEHzkuiV4r9G0foijWKAmIXF/+Od0Pmi6fOkS1BtNCOavy3OvncXQYBlPG5pI+VWdNufsza4ALHKpOPtz6PZGGSd+a4ljrVFEr0jBzzxAAc8AcTyae33YTFbR2IsnAYHR3dDOAIBdX7jd88MXJV/dJCSmD39zViEyH3xtt5wvzTQx9tJJAGbLzuvazaqiUi55Bm1NopOlR3G/tfRqrVlyM/UDFPAMkPSjuVEYLW4EN8059bWexraus/qeTQwNlo37s90EbLNE2zNi3HjdVNztaW2rNolOlh7F/dbiFnEn1uFsT8JXnSU3Uz/ALJQMkHSUvFvrx+/9oxuHsdwwANrGtk+v6tjmBELDYhPpsa3rUC517nGg9bv2Ypwo/U3yUJrtlW2TVKFOkTM+sggt8AyQ9ONutyX5QaynMJN/TH1JogRCHdaMH1zwa7fnlL94/CyO/mxxj7WSQdTD+ouzGqhz45dKmISvOktupn6AAp4RojyaB30ENgmUKcPDtL1dyLyOF+YmYbLou+354gQlHTE9/vOLHeINmN0ubtGpDpahOl8RuvvQmY7rmpdAXRR/dBy+6iy5mYoOXSg5JcwjsKl46KPLzL1BPrqsbCwy8jueV98WN7Y88rhoNGfx/DF7Uy6bb/vo+BY8vX0DLjfnUG80rdc1L4E6r2KcIhXq9DO0wHNKWCvQbRWtHT9o3O97jSamd94N4JrFvWPvNAYMqXftxzM9Ot9120rsP1HzdU104z6x4dX7xHQDcdbtd57O+/MQqPNzDcWZEpm14p1+gQKeU7q1Av1EyO3nDTJn0vToPHLrCt8PdtqWa7tIRTnPLOWDexHEHx2H6IaJCfRS6It4k6GA55RurUA/ETJZ+LZ1eBGkbcDg0hI+uNJ5rKHBMgaXLom1J/rypaVFaXS27pBu2s8z6UBdnELj5Y+Oy1cd9Gmwl8HfvASew0IBzyndWoF+IhTEKo7SG+OGShkfXLm6kDPuJc71mSa2fXpVhxsmKuWS4Ot/sH5hTUHa5wLm8+xG/LwEutciF+XGEfRpsJfBX9uxH7cEquMkScufAp5T4rACvUTIZuGXRDCnGrk3Rpiuh4r5GZv33zmMgz85Zy3oCYK7Za3fE4b7PIH5/utJuxt6JXLd3DiCPg32MvjrdYwkb5JJ35Ap4DkmyXQtm4UftvVtUFeMjUZzFi8cextzqhgaLONyczb0bE9n4lA7Xh9oARbllAPo+BA+vncaj++dRrVS9m010I6fQCc5Es7LEuzmxhH0abCXwV+/NNekbpJJ35CZRkiMmFIPo/Qtj0N4ZlWhmC+pDyve5ZIY3TxBxsE51tKuA6esNyGnw2HQCkabiNTqDayxZAb5rTcIfmmg3dw4gv6t9LJK02+kIZDMk0DSTx20wImVOCz8NAczG3G5uB0rtFZvdPQVN/U0bzRnfZ8ggvSL6QaBuflXGPyqMm393MO0n/U7/15WabYf2/b3mMSTQNJPHRRwkiimx+v2sW6mvOs4ac4pdh04ZRRtxTXhGu7yRmOyqNzB26gzmxXd+0u9LEFbHn4cNw43vazSdI7t9ksDyT0JJJ1ySgEnieJnddkKisLgNw2o3mguBE/dr3PE++j4FmsP8Xnf+5ynJW4qDooavHUTZcScGy9L0Cbucdw4skiaTwJJH4sCThInSrZLUNon9ETdj/M+m7W0857bAQC7DpwyCnF5oNPP3m3wdmHfJcEHH171nBQUBC9L0OZWiOPGkVXSfBJI8lgMYpKeEiS45MbxRDjBsqdG1+Po+BY8s31D6H0B8ymDwPwH7f47hxd+Long/juHFz6A0zvvxjPbN2Bo8FofmWqljN0P3BG5unRAOj+EzvkNDZYBxaK+LDv2TmNNhBa2XoFGtoDNL7TAU6aI5bx+eJ2z89WvIrJaKS9Yv9XBMnbe05m6Z3pcnbly1Td/3Dnu5FQN+0/UFn6eVcX+EzWM3Lpi0XqD/H8FebJwcs2d7of1RnNRW9wPPrzaMXjDnSHTft5+2NbezWN+P/49ZwnRBANIbkZGRvT48eOpHS9r2IInUdLz8kLQc56cqmHH3mmjL9vkgw563UzHdxPEBz64dElXw339KA9IoBF3trX3gn78e+4VInJCVUfc2+lCSRG/VK4iEvScRzcO4+FNqzvaylbKJajahw/40e46ADpb3La7Cmxuj0szzdCTa9wui2qljKHBMgTXXDbtNOc0tHh7rTkN+vHvOWvQhZIieekjHSdhzvmp0fXG7oU79k6H2rebdtdBHEMpglbS2VwWcWTeOPSyhW0//j1nDQp4iuSlj3Q3uAWyOlg2+qBt52wSPVuWRJTr5uXDNmVq2KjVG9j4Z99FfaYZ2vcbJvOmvSOjO12y14HGfvh7zjp0oaRI0aP9pnLt9y9f7RguHPac07pupkyNasU8uQiYd61EGQgcNPPGSWE8Or4Fb01sw9PbN3Td2iBOiv73nAcYxEyZIkftbUHAaqWM5deFCwK6+drk63jh2NuYVUVJBA999hY8Nbo+rqVbCROMDHOefr3ISyL45oOd6YlRiPo3F+R9Rf57zhK2ICYFnMTG2vGD1pLsNye2Lfwc9kPf62yHyakaHrf44b3wW6PXft3XLCpRr10S15xiH53IWSgi8i0ROS8iP23btkJEvicif9/6OhT3gkn+CDIoN8wwZodeZzuMbhyOVJXot8bRjcOLioLaicuPHPXaxX3No/y/E3+C+MD/M4DPubaNA/i+qn4SwPdbP5M+J4hPNIowZCHbIUrFKOC/xp333N61H3lyqobNE4ex1lChGfXaxX3Ne30TLiq+Aq6qPwBw0bX5XgB7Wt/vATAa77JIHgnSFzqKMPhZ9l4CFhe2vG4/gswMbd/v0GAZ1y0ZwI6904HOxc+yDfJUFOb3UZ8MsnATLiJR0whvUtVzAKCq50TkRtsLReQxAI8BwOrVqyMejuQFv1LzKKlnXo2Y0pwhaTo3W+C2fY1B9xvlXPwmvkRtZxp3G1SmHCZD4mmEqvqsqo6o6sjKlSuTPhzJOFFSz7ws+14/mttcK0OD5VjGz3XrXoo6WSmuiUwOTDlMhqgW+Lsisqplfa8CcD7ORZHiErVxks2yT/LRPEjWRJz9nqO6l/ws26jtTONsgzq6cRjHf35xUSqo0+mRRCeqgB8A8CiAidbXV2JbESk8cQpDUo/mYdwZcZ1P3O4l5zyykLoXpNMjCY+vgIvICwB+G8DHROQXAHZiXrj3icgXAZwF8ECSiyTERre+2smpGp589dRCub8zZT7OaeJBRdRWyv/Bh1cxOVUL3Qo2yE0oLYFPejq7iazcvJLEV8BV9SHLr34n5rUQEppue1mPvXRyURdAZ8q8uw+3Q1jXTFhLHsCiG4qzJq9gpu0JwE804w4Aewlm2lkoaQa3ewl7oZDcM7pxGEfHt+DNiW04Or4l8Ad096EzxhauzTk1tnwFwrtmwgYmRzcOY3Bpp10VJTDrJ5pxBoCTSmeMSq+D22lBASd9i5f1N6saS9ZEFMszLmvVTzTjtIr9BDPtLJR+yTungJO+xcv6c9Lmuk2ji2J5RnmPqZjJTzTjtIrjTmfstjgrbYu/V1DASd8ytnVdR6tbh1q9gd2HzmBs67rQrhn3McJanmHfY3NfAPAUzW6sYrfAVgP0dHEKi26uVvBO6/qahDmOvin9knfOgQ6kb7EFDR3iCHyFCbK2BwGrrZL69xr+AyO83BdeN56oAWBTgLA8ICiXZFFMwS2YQQOLcWSsxJmf70WvM13YTpaQFray+DQGB3fTvjVoG9+4iNr3Pej1Tft8opJmm2NbO1la4IS06GXgqxurM+0+I7br8V6jiemdd4d+n3t7Xvqm9CK33Q194IS06GXgq5ubR9r+3qjdIYNe37z4r7OQ6UIBJ6RFL4Wjm5tH3I2n/PC6Tl4ByKDXN+3ziUoWMl3oQiGkRVqBLxPdtgSIs79MkGMB5uu0eeKwZ0DV9j7TMbIm2G7ibrkbBQYxCckISWY0pJUtkZcAZFykdV0ZxCQk4yRldabZFyQvAci46PWTAn3ghBScNPuC5CUAWRRogROSEr0q+kgzW8KvvW3R27umDQWckBToZXvTtN0aJrdCv7R3TRu6UAhJgV62N82CW6Nf2rumDS1wQlKgl0UfvUyPdMhC0UsRoYATkgK9zs7odbZEr8+/qNCFQkgKZMGN0Uv6/fyTghY4ISmQBTdGL+n3808KVmISQkjGsVVi0oVCCCE5hQJOCCE5hQJOCCE5hQJOCCE5hQJOCCE5JdUsFBG5AOADAL9M7aDR+Ri4zjjhOuOF64yXrK/zVlVd6d6YqoADgIgcN6XDZA2uM164znjhOuMlL+t0QxcKIYTkFAo4IYTklF4I+LM9OGYUuM544TrjheuMl7yscxGp+8AJIYTEA10ohBCSUyjghBCSU1ITcBH5nIicEZF/EJHxtI4bFhF5S0ReF5FpEclM60QR+ZaInBeRn7ZtWyEi3xORv299HerlGltrMq1zl4jUWtd0WkR+v5drbK3pFhE5IiJviMgpEflya3umrqnHOjN1TUVkmYj8SEROttb5ZGt71q6nbZ2Zup5BScUHLiIlAP8bwL8A8AsAfwfgIVX9X4kfPCQi8haAEVXNVFK/iPwWgPcB/LWqfqq17c8BXFTVidZNcUhVv5LBde4C8L6q/vterq0dEVkFYJWq/lhEPgLgBIBRAP8KGbqmHut8EBm6piIiAJar6vsiUgbwQwBfBnAfsnU9bev8HDJ0PYOSlgX+GQD/oKr/R1WvAPivAO5N6diFQFV/AOCia/O9APa0vt+D+Q92T7GsM3Oo6jlV/XHr+18BeAPAMDJ2TT3WmSl0nvdbP5Zb/xTZu562deaStAR8GMDbbT//Ahn8I2yhAL4rIidE5LFeL8aHm1T1HDD/QQdwY4/X48W/EZGftFwsPXf1tCMiawBsBHAMGb6mrnUCGbumIlISkWkA5wF8T1UzeT0t6wQydj2DkJaAi2FbVu96m1X1NwD8HoAvtVwCpDv+I4BPANgA4ByAb/Z0NW2IyPUA9gN4XFX/sdfrsWFYZ+auqarOquoGAB8H8BkR+VSPl2TEss7MXc8gpCXgvwBwS9vPHwfwTkrHDoWqvtP6eh7AdzDv/skq77Z8pI6v9HyP12NEVd9tfWjmAPwnZOSatnyg+wE8r6ovtzZn7pqa1pnVawoAqloH8LeY9ytn7no6tK8zy9fTi7QE/O8AfFJE1orIUgB/BOBASscOjIgsbwWKICLLAdwN4Kfe7+opBwA82vr+UQCv9HAtVpwPcIs/QAauaSuY9VcA3lDVv2j7VaauqW2dWbumIrJSRKqt7ysAfhfAaWTvehrXmbXrGZTUKjFbaTnPACgB+Jaqfj2VA4dARH4d81Y3ACwB8O2srFNEXgDw25hve/kugJ0AJgHsA7AawFkAD6hqTwOIlnX+NuYfTRXAWwD+teMX7RUi8s8B/A8ArwOYa23+E8z7lzNzTT3W+RAydE1F5NOYD1KWMG8Y7lPVPxORX0O2rqdtnf8FGbqeQWEpPSGE5BRWYhJCSE6hgBNCSE6hgBNCSE6hgBNCSE6hgBNCSE6hgBNCSE6hgBNCSE75/1rZsxPK4QnpAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from sklearn.datasets import load_boston\n", + "boston = load_boston()\n", + "plt.scatter(boston.data[:, -1], boston.target)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "w = torch.zeros(1, requires_grad=True)\n", + "b = torch.zeros(1, requires_grad=True)\n", + "\n", + "x = torch.tensor(boston.data[:, -1] / 10, dtype=torch.float32)\n", + "y = torch.tensor(boston.target, dtype=torch.float32)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "y_pred = w * x + b\n", + "loss = torch.mean((y_pred - y)**2)\n", + "\n", + "# propagete gradients\n", + "loss.backward()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The gradients are now stored in `.grad` of those tensors that require them." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dL/dw = \n", + " tensor([-47.3514])\n", + "dL/db = \n", + " tensor([-45.0656])\n" + ] + } + ], + "source": [ + "print(\"dL/dw = \\n\", w.grad)\n", + "print(\"dL/db = \\n\", b.grad)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you compute gradient from multiple losses, the gradients will add up at tensors, therefore it's useful to __zero the gradients__ between iteratons." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1Z0lEQVR4nO2de3hU9Zn4P98ME5iAEm5yGYmA9UGqUVBUKv1ZxbZUwRqv1Nu6fdq6W7e2ujVtbG1BVx/Zpa1u27Vb2u2vtF6Kt0YE+8NWUFu6omBApIBWQGBAQCUIJJhJ8v39MXMmk5lznTkzc2byfp4nJJk5l/d8ybznPe9Vaa0RBEEQyo+qUgsgCIIg5IYocEEQhDJFFLggCEKZIgpcEAShTBEFLgiCUKb0K+bJhg8frseNG1fMUwqCIJQ9a9eufU9rPSLz9aIq8HHjxrFmzZpinlIQBKHsUUq9Y/a6uFAEQRDKFFHggiAIZYoocEEQhDJFFLggCEKZIgpcEAShTHGVhaKU2g4cArqATq31VKXUUGAxMA7YDlyttT7gt4DNLTEWLN9CrLWdkFJ0aU20NkLjzIk0TImmtruzeQOPrN5Bt0lvrnAVdGno1hBSimvOGcs9DfWWx77g5BGs3Lyf3a3tjDE5Vy7yuz2W1fZejyMIQuWj3HQjTCrwqVrr99Je+w/gA631fKVUEzBEa/1tu+NMnTpVe0kjbG6JccdTG2iPd2W9FwmHuO/yehqmRLmzeQMPvbzD9XEBpp84lNd2HDQ9tt25vGAmv92xrLa/4swoT66NuT6OIAiVhVJqrdZ6aubr+bhQLgUWJX9eBDTkcSxTFizfYqlg2+NdLFi+BYBHV+/0fOxVb3/gSnlnnssLZvLbHctq+0dX7/R0HEEQ+gZuFbgGnlNKrVVK3ZR8baTWeg9A8vtxZjsqpW5SSq1RSq3Zv3+/J+F2t7a7er+rCD3NnWTxso/X162uLxeZBEGoHNwq8Ola6zOAi4B/UUqd5/YEWuuFWuupWuupI0ZkVYLaMqY24ur9kFKejpsLTrJ42cfr61bXl4tMgiBUDq4UuNZ6d/L7PuD3wNnAXqXUaIDk931+C9c4cyKRcMj0vUg4ROPMiQBcc85Yz8eefuJQy2PbncsLZvLbHctq+2vOGevpOIIg9A0cFbhSaqBS6hjjZ+CzwBvAEuDG5GY3Ak/7LVzDlCj3XV5PNMPSjtZGegXw7mmo5/ppdVRZGOLhKlLvhZTi+ml1PPyVT1ge+/ppdURrI6iMczW3xJg+fwXjm5Yxff4KmltiruXPPJaX7e9pqPd0HEEQ+gaOWShKqQkkrG5IpB0+orW+Vyk1DHgMqAN2AFdprT+wO5bXLBQ/yTcNz2tGiSAIgl9YZaG4SiP0i1IpcDPlq4DrptVxT0O9q2NMn7+CmEnQMFobYVXTDL9EFQRByKIQaYRlg1l6ngYefnmHoxvEwGvmiCAIQqHpEwrcSslqcJ1L7TVzRBAEodD0CQVup2TdWtBeM0oEQRAKTZ9Q4I0zJ2KVKe7WgvaaUSIIglBoijpSrVQ0TImy5p0PePjlHaSHbL1a0A1ToqKwBUEIDH3CAodErvj9cyaLBS0IQsXQZxS4IAhCpdEnXCiQnQsea23njqc2AIgVLghCWdJnLHCvrV0FQRCCTkVZ4Hbl8lKIIwhCpVExFrjhIom1tqPpcZEYlZZSiCMIQqVRMRa4k4ukraMzax8pxBEEoZypGAVu5QoxLPFM5V4bCTPv86dIAFMQhLKlYlwodtNszGZfDuzfT5S3IAhlTcUocKteJVbzJGOt7a6GMgiCIASVilHgVr1KojZBysxApyAIQjlRMT5wsO5VYuYDNzACneJOEQSh3KgoBW6GoZgXLN9iOlEHJBdcEITypOIUeGYxzwUnj2Dl5v3sbm0npJSpT1xywQVBKEcqSoGb9Tt56OUdqffNlLfkgguCUK5UlAI3K+YxI6QU3VrnNJ1eEAQhKFSUArfycWfSrTXb5s8qsDSCIAiFpaIUuJWPO5Ni+bztmmsJgiDkS0UpcDfKu1g+b+k/LghCoamYQh7AtmgHQAFXnFmcuZbSf1wQhEJTUQq8ceZEwiGr+fOggZWb9xdFFuk/LghCoakoBQ6AgxelWApU+o8LglBoKkqBL1i+hXi3vQYvlgK1aq4lOeeCIPhF2QYxzTI8nKzrYirQ9BJ+yUIRBKEQKO0ic8Mvpk6dqtesWZP3cTIzPCChnPv3q6K1PW66T9RCgUqqnyAIQUcptVZrPTXz9bK0wK0yPAaEq4iEQ1mK/b7L602VsqT6CYJQzpSlD9zKVdLaFjftCW6ljCXVTxCEcqYsLfAxtRHTsvkxtRHLnuBmSKqfIAjlTFla4H5leEiqnyAI5UxZKnCr8Wle/daS6icIQjnj2oWilAoBa4CY1nq2UmoosBgYB2wHrtZaHyiEkGZ4cZXYHQMk1U8QhPLEiw/8G8Am4Njk703A81rr+UqppuTv3/ZZvoLjx43ACUlVFAShELhyoSiljgdmAb9Me/lSYFHy50VAg6+SVQhGqmKstR1NT6pic0us1KIJglDmuPWBPwB8C+hOe22k1noPQPL7cWY7KqVuUkqtUUqt2b+/OI2kgoSkKgqCUCgcFbhSajawT2u9NpcTaK0Xaq2naq2njhgxIpdDlDWSqigIQqFw4wOfDnxeKXUxMAA4Vin1ELBXKTVaa71HKTUa2FdIQcsVu5x1QRCEfHC0wLXWd2itj9dajwO+AKzQWl8PLAFuTG52I/B0waT0meaWGNPnr2B80zKmz19RUH+0pCoKglAo8qnEnA88ppT6ErADuMofkQpLsfufSKqiIAiFoiy7EebD9PkrTF0a0doIq5pmlEAiQRAEeyqqG2E+WAUPY63tTJ+/QqxkQRDKhrIspc8Hq+ChAsnVFgShrKg4Be4UoDQLKiqyR2lKrrYgCEGnonzgZpN6wlWKQQP60doWT7lGoHdQ0cwnbvDAnMniShEEoaRY+cArygI3q3qMd2sOtMV7uUYAVjXNYNv8WaxqmkHUJie78fH14koRBCGQVJQCd1PdaOYaMXOrGMS7NfOWbPRFPkEQBD+pKAXutroxU9Eb/cWtsBqULAiCUEoqRoE3t8Q48lGnq23NFH2x/dzFrAYVBKEyqYg8cLPgJUBNuIp4tybe1ROotStjH1IT5kBbtrWtVOIcbpS8m97fXqtBpZ+4IAhmVIQFbha8BBgysD8Lrjzd9ei1uZecQjiksl7XGld54U69vw2r+9bF61y3mJV+4oIgWFERFrhdy1YvE3eM7b752Hq6MtIrDQVrdyyn3t9mTwlO12F3TLHCBaFvUxEKPJ+WrYZ7ItbaTkipLMWdjlOWi92NxOopwUle6ScuCIIVFeFCybVla7p7ArBV3mB9QzBcI1Z7j6mNOCpcK3mtzin9xAVBqAgFbqQBuvV1G7ixig2sFGzmTcBqPzuFayev9BMXBMGKsnGhOGViuPF1Zx7DroQ+kwFh83ud3U0gmiFnpg88Eg453mikn7ggCFaUhQL3YwiD2TG8cKAtbnpOK9eIgl79xfNRxF4CsYIg9B3KQoH7kYnhxV1ihdk5ay1yx2trwlmviSIWBMFPysIH7kcmhl9ZG5nHsYp7FrHJoyAIfZSysMD9mOzu1edt1iPc7JwHLfqkWL2eK1KNKQhCJmVhgfuRiWF2jHCVMq28HFIT5rppda7OWYw0P6nGFATBjLKwwN0EAN1kqZgdw+64U08Y6mj1XnDyCB5+eUcva93vND+pxhQEwYyKmMhj1szKKkWvuSXGvCUbUy1ih9SEmXvJKYC7DJH0yk0zN4sCrptWxz0N1u1pvTK+aZmpO0cB2+bP8u08giAEk4qeSu/WQm1uidH4+Hri3T3q8EBbnFsXr+u1r1WaYuaNwkypamDl5v228nr1Z/sRAxAEofKoCAXuNktlwfItvZS3HelNqAxlW+XQK8VJHsgtp71x5kTTJwypxhSEvk1ZBDGdcBtI9JpKaChXI3joRnnbyQPOHQvNyLVVgCAIlU1FWOBWFuoFJ49g+vwVKVfF4EjY03i0kFKei3+cLGOrVEanm4sUAQmCkElFKHBDsaUHJ6sULH5lZ8plEmttJxxSVAHdLo4ZCYc8K+8q1duaNgugus0vFwRBcKIiXCgGH3X2qOYjHV1Z/u54l2ZwTZjaSHaZOySyOqDHRRF1UKohldijNhImHFIYp7PK016wfItlNon4swVB8ErFKHC3vU4OtMU52B4nWhvh+ml1vfzK98+ZzPb5s1JNqOyGJEdrI/zw6sS4ttb2eK+5m2Du17Zyk2iKP1RZEITypyJcKOAtQGlUMz65NsZ9lyfytRcs38Jti9exYPkWLjh5BE+ujVneEAz/utcRaVbpgE6WviAIghkVY4Hn4kNuj3dxx1Ovc9vidb3K1B96eYdtj+/7Lq9n5eb9nkekyXAGQRD8pGIUuGmvk5Aiu9NJb9rj3Zaj0DIxenw3TInmNCKtFOmAxri38U3LmD5/hfRPEYQKomJcKFa9Tm7LqLLMh3SL2q67YeYknkw5i+Xv9mMQRj7nlu6JglBYKkaBg7lyNPqW5EumRW2Vex6kAptSNcEq5Y1DEPoSFeNCscLMteKVkFJZitlvd0ghXB1+DMLIhVyqTQVB8I6jBa6UGgC8BPRPbv+E1nquUmoosBgYB2wHrtZaHyicqLlh5lrxYpHbWdVmFr+Z6yDz/GatcAthsZaqCVapbhyC0NdwbCerlFLAQK31YaVUGPgL8A3gcuADrfV8pVQTMERr/W27YxWqnaxXps9f4UqJ2/myzWhuidH4xPpeOeGhqkT1Z3pRUeZNwUqeaG2k12Bkr5i12Q1XKQYN6EdrW7xgvulCXY8g9FWs2sk6ulB0gsPJX8PJLw1cCixKvr4IaPBHVP+wcks0zpzomJ1iKBsvyu2uZzZmFfR0deusitBMd0KhLNZMN09tJAwqUczkdrJPLq4dSZcUhOLgygeulAoppdYB+4A/aq1XAyO11nsAkt+Ps9j3JqXUGqXUmv377ftk+4ndGLKGKVHb1EEFqUZYXhSX2XR6K9KVcyHHsjVMibKqaQbb5s9iYP9+ripGDXId5SbdEwWhOLjKQtFadwGTlVK1wO+VUqe6PYHWeiGwEBIulFyEzAWnDIwhNWFLhauhVyVmIbIoqpRifNMyxtRGTCs/C2GxerX088like6JglB4PGWhaK1bgReAzwF7lVKjAZLf9/ktHIe3wqs3wxPD4RGV9hWCZafC3hcsd7VTVs0tMQ4fte5zAuSURWHVJMuMLq17lfRfcWa04BarV0tfgpGCEGzcZKGMAOJa61alVAT4NPDvwBLgRmB+8vvTvkoWexb+ciV0mSmLbji4EZ6/AE6dB6fNzdrCLgPDy2SeXiK1tveymldu3t8rs2Te50/JGtmWSchkqk97vIuVm/c7BvjyLY7xOtlHRrkJQrBxk4VyGokgZYiExf6Y1vpupdQw4DGgDtgBXKW1/sDuWK6zUA5vTVjYpsrbhP7Doe4qmHQ7DJoAwJ3NG0ynxd93eT23LV7nunzeLUaf71qboRFG4NRpQLFVKqIfhUNebgJehkULglA4rLJQgjmV/tWb4a2f5XwenfoHdncM47Zdt/PqkfrUtHi3aYS5YjW0AcwtcOjJejFTmnbHK3RqnpTEC0LpKa+p9Lvy88ao1D8Q7f8+iyfcwYHOQXxv611AvakrwU801krXTHmnuzHMAod2t9h8/dFOClqCkYIQXIJZSt++x9fDKQVDw4f56chvwiOKhk3H8+cp3+OsYYUrHNXY9/kOKWUasPSqkPPxR+eaJigIQjAIpgUeGQ3tu30/rEqr3hl+dA2PR29Aj4GdHcdx+67beOVIvW/nCillq4y7tU75vNPxWurf1tGZym33SqmaXQUJcREJ5UwwLfDjLy3aqZSCuv77WDzhDrbVz+aNU65g9uAXrbeHXuPYjNcyMdIErbCynL023zrQFs/Zau7raYLyBCKUO8G0wCfdDlt/7T4LxQcM63xQ6CN+UreAH7OAH++dwwP7bkhtE1KKH159um0DqyqLIGU6dql76c233FriuVrNfSlN0MzSlicQodwJZhYKOOSBF4f0pYnrKu7d8yUe+/By2zS68U3LHMv0jWwYN4xrWuZqu/Q0RLf0lTRBq+u0CmLnspaCUEjKKwsFIHoxzHoDNv0A3nkMOt4vugjpPvNq1c28Mb9g3phfsPm1Oq76071c9+kLgN6tYmttSvQhEdx8dPVOpp4w1JWSjLr0iediNVtNMaok5Q3Wvn6rlM5KfAIRKpPgWuBWBMAyhx7rvK27P9/e9XWWHvwUkGjXiiKraVQmmZauVTDNzHp0OlaxKJcAoN1TUaYlXolPIEL5k3M72cBhWOYnfRUiY0omhlKJr4FJn/m2+tn87ZTLmXnMC/SrUrYphNC7t4pT58T7Lq8npMxCpebTgopBOQUArSxqI4VTuiYK5Ur5WeBmbH4AXvtX7EteioOxnFrBm8O+zmV/ucjR1+pmAIKVFVkqf205DW3oK75+oXKpHAvcjJNvhWu74dyHIXxsSUUxLPMq4OT3f8zfJl3E/558I2cP3JC1rWEZuknnK2TP8FwopxRE6U8uVCrBDWLmwrhrE18A2x+B1TdB15GSiqSA0dWJcn5IPCP8+r1ZLHjva6lUQjfpfF47CRaacktBlJYAQiVSGRa4GeOuhTmH4Vqd+DrjflDu+3X7TcoyV/DF4cv426SLaNh0PDxzKnMvGOA4gixoVqSMTROE0lMZPnCv7H0BVlwE+mipJUmhgVfbzuD2HTfTVTO+V0ZHZmtco1GW16HLflMuWSiCUO6UVztZH7FVMtsfgZe/CN0dRZXJFSMvhHMWcuefjvDQyzssNwuKMhcEoXD0SQXuKftg7wvwly+gP9oLuncRTynR2n2zrWJkVojVLQjFp08qcK+pbobCr69ex/zoTxjfP9ERMQjK3Phv2hMfxq07b7dU5umDIcym+uSjfN3eEEXJC4K/9EkF7jV3esrdz2WVwZ89cAMPjP0Bo8OJUv4gKfMPO2v4yo7v9VLmRq+VzHFyZhWiXi12NzdEybkWBP+p7DxwC7zkTje3xEx7mLxypJ5zNy9i/IalnLfll7z04el064QSLeK9rxdGRsvgcBuLJ9zB30+dzY3DElOMBkfCWcobIN6ts8r72+Nd3PXMRqbPX8H4pmVMn7/CtpLSTe63XYc/QRD8paIVuJdUNzcKZmfHKP5h+71M2LCU8RuWMm/3V+jsViVX5v2qYN6YX7CtfjYtH5vJq5OuMS0cMuNAW9x1ObybG2I5FfgI/tHcEnNtCAj+UdEK3G3udHNLLKchx4vev5SPvfEM4zcs5ZYdjRzu6l8yZW5Y5UrB8PAhFk+4g631s/n5Cf/G2Op3XR/Hzlp2c0MMWsWoUHjKqS9OpVHRPnA3uOn255Vbj/stt4x8jKqkI6PUfnOtE6mG2z8aQ1PsFsdsFrMYQXNLjLue2djLzTSkJszcS05JdU20GkIRDikWXJk9CCNX3AZJJZhaHMqpL0650id94G4w89nmywP7buDEDQnL/P++NysQPvMqBRMG7E6Njlvz8Ru4cMjfTLevUqqX9dTcEqPxifVZMYLDRztT7xsWmCk+Xrdba0+swuIhbrPS0ecVeC6uEy/cveerKZ/5nK33sbejtuTKXCkY3u8Avxz7Ld4+9ZJUANSgS+teym7B8i2m/c3j3ZoFy7c43gSN7fzAbZBUgqnFQ9xmpaNPu1CaW2LctnhdSZrQnj1wQ2ByzdP/BN6PH8PNO7/j6GYxMMR2WkOr1E2vbg674QwKUsew+n+VcWn+I6mjhaf8RqoVgQXLt5Ssg/grR+qZ8eZCAGYPfpH7jv8pg6p6ngaKqdDTzzW8+lCqc+KfD53Od3ffws6OUZb7GlaW05OMVepm+gffcHMAlh98qy6IQC9XidVoO7EK/aevjOYLIn3aAncaQFwqbj3ut3x95GIUwQiAAnTqKu7Z8yUWvX9p6r1wlWLBVacD0PjEessxclbWWC7BL7dB59pImI86u8UqFCoCCWKaEFRr7IF9NzAhWTi0qa0uED7zcFV3Ktf8rVMv4asjn2HO2WNZsHwLty5eZ6m87dre5hL8ykwNteJgezxQ7XcFoRD0aReK2ZCEILGzYxQX/f1BAG4c9jTfHf0rwqpH1mJb58b5wkrzrZE/hyM/pz1azbf0N1JDndNxsqSrPE6Fz/SX3z9nMvOWbKS13dxVUsghDpKiKASBPu1CAXrlLxutWf3C7+OlsyD6Q64cujJxnoC4WT7squG7sX/ppczTA4uGgrNzg1i5Ocz2CVcpuoGu7t6rbLh2Cqm8JWgnFJM+2czKK3bFKLlw/bQ6lr2+xzSY5hezB7/Ivx//n9RU9fQ0D0JGy+HuCHfs+lpKmacrOCvfd0gpfni1ueK12seMITVhWr7/2dwvwgEpXBGKjfjAXdAwJcqqphlEbXzjbnXjkJow9zTU0/L9z/LAnMm2x8yHpQc/xSkbn2J8Wq751qNjUtWXxcbwmR8TaucndQvYVj+bbfWzeWrcP/HwnxJPDFY+7m6tLS1YL0UhrQW8YYIUrgjBQRS4CWY9PyCR2XDdtLpegbHpJw7NUuoKmHXa6NTvbm4MfmGkJ47fsJSmPd/maFe/kgdAlYJJkR08NuYGeETRfNK3Tfuz2AWVvQScCx2clsIVISiIAjfByHQYUtN7CHJre5wn18ZonDmRbfNnsappBg9/5RNcN62ulxLXwJNrY1nl6G0dncW5gCSL9/8fPrFtGedt+SXLD55DZwBK+gFOj2zkpYlf5i8Tv5jqmug0ENnsphquUoSqet8+wyFV8MHKMtBZCAqiwC1omBKlpjo7ScesHHvl5v1Z7or07YygVyF94VZoDbs6RvFP73yPj6W5WfZ/dEzJlfnx/fenerNsPPkizn3nC3B4q+n2Zp0l55w9NvsPuAjX47bLpSAUGscgplJqLPAbYBTQDSzUWv+nUmoosBgYB2wHrtZaH7A7VtCDmJm4nejjtJ2XAJzfOGXC3Djsae4c/Qv6qdJns0DPDSUeOobqaf8N46613LYcg4mSfijkQj5BzE7gm1rrScA04F+UUh8HmoDntdYnAc8nf68o3Po6nbYrZXCrtiZsG3hd9P6lnPRG8KYNVXcfQv/1OnhEwfLpppZ5sYOJ+Q4tkA6Jgt84KnCt9R6t9WvJnw8Bm4AocCmwKLnZIqChQDKWDLe+TqftShXcioRDrrNRMqcN3bKjkSNd1aVV5sYP7/8VlpwIj1TBslNh7wuA/Y3T7wkxfihf6ZAo+I0nH7hSahwwBVgNjNRa74GEkgeOs9jnJqXUGqXUmv379+cpbnFx6+t02s4qq6WQAQhDhoMmVYpuSE9PvGVHI21pyrxUCh00HNwIz18AjyheqvsMXx7RuxVuJBzigpNH+G7p+qF8nZ4YZCyZ4BXXhTxKqUHAi8C9WuunlFKtWuvatPcPaK2H2B2j3HzgfnJn84bsSfEhRWeX9j3udv20Ou5pSLSDLYT/fWz1uyw/6Z+JVHWW3G+uk//8+dDpfH/3LWzvGEXIokQ/H9+4XeOzB+ZMduXHtvPZm7V1kOpOwSCvQh6lVBh4EnhYa/1U8uW9SqnRyfdHA/v8ErYSMctUiXdpamvCWdZ5vjpx6fo9qZ+trP982Nkxio9vbA7GtCESPvPzjl3Pyolf5u36S3iw7m7TPPN8fON2bjC31r2dq03cK0IuOCpwpZQC/gfYpLX+UdpbS4Abkz/fCDydua/Qg5XyaG3L7pqXmVfuldb2eOoRPN294zeZ04Z2fHRcyQOgIaWZOXg1L038MtvqZ9Py8S8we/CLQH6xCLsboVtFa+dqk+pOIRfcpBF+EvgzsIFEGiHAd0j4wR8D6oAdwFVa6w/sjtWXXSiT73rOtGtebSTMurnZfTvubN7AQy/vyOucmY/gxUhnvH5aHcN2/oybhyykOtk5seRuluSfeLcKEzrjP+DkW3M6TnNLjFsXrzN9L99JP+WYEikUD2lmVWKm3P2caSFPlYIfXW3uQ7XaxwtDasLUVPdjd2s7A8JVtMe7nXfKg0g41MsVcPbADfzbmJ9x0oAdgRhQAUD1MPjcKzBoguddC6VopcOhYIeMVCsxVg2WujWWY8T8aMp0oC2eugn4rbyrVEL+dNrjXb2CiK8cqWfmW4me5mOr3+XOUb/gwmNfIaQS75dEoXe8n0hLRMHgj8PUn8LI813tahVszLeMXsaSFZZKLaASC7xIOLkvzCy4YlZwRmsjXHDyCFduGwXcP2ey7UBoN73Qzx64gf+uu5ch/Q4HwzIHOOYkOHuhrUKvVGWQSaVcZyU83Ug72RLjlA1iFqwy28fQc34GJY2bx9QThhJyoUmvm1ZHw5SobVDQjVnwypF6ztj0u0AEQFMceiuRZ/7MybZ9WVY1zUg1NCsXJeCFSqoareQMH3GhFAnjQ/7Nx9a7HiPm9FjtZKGHq8DJa2I8/hsfWDPZeh0zpJh6wlDAv5F0rxypZ+bWRdx3eT01He9w5NXvclHtKvqreOks80Nbkm4WYOSFcM5CGDQhUFapkyz5yGqn9MrthlXJGT6iwIuI8YfvxYdqN9fRSYEOGhDmaLw7awzZoAH9aG2L9/pQT77rOVeKON6lUx/i9BtMPq6eaC/lEmXc72/ntl23M7b6XX4Y/QFnDdqc2rYkCn3v87DkRDTwyXgth2rOZWHb5exsHWUZvyg0mW4Bw0I2ZHF634lKUnpjaiOmf5+V0L9dFHiR8TNYZexjldrW2hbn/jmTHc/V3BIzTXG0Iv1DbCjycU3LPMtv5YeMJj9wOztGcfW2H6ReH1v9LveO+QmfPGZ9STJaFDA83MoNw5/l+mHPpl7f/fpIGPM714FQP3CykPO1oCtJ6RUq8BwERIGXADOrOtfHXePDavVhczOZ3asvMPND3NwS8zzAOaSUZRDJ6snCaLilgLMGbuA3475H/xKV86efMxrem/CbR6+ATz1RlPM7Wcj5WtD5KL0guZmgsjN8RIEHgHwfd/O1MLw8Fpsdd8HyLZ77uVjNvzQ+/O3xLtM0xUg4xBVnRlm5OcKkjc18PbqMrw/7OaFkjZkm/1YEORN7Eh7tDx/7Eky6Pac8c7c4Wcj5WtC5Kr18/5YLhRtDphwRBR4A8n3cdfNhM7OKjH3slG+oSnFM/34cbI9bfohz8YuaKZLMD3+3TgRNB1b3o7U9jiKxLg+9vIMhNWHunzOZhimzgAdTx1B7X4BXbkJ/+Fbi92Jrc90Bb/0s8TXmYpj6k4Iocqebth9ug1yUXiUFP8sBUeABwI+Akd2Hzcwqanx8PahEUNKOKmDe50+x/fBZWXt2mCkSsw9/vEujVCL4Gk8zxw+0xWl8Yj2QYdmNPB8ueZMJTcs4a+AG/nPsAkaFPyhN8HP3s7DE8JVXAd0wYCSMvTxvC93ppl0qt0ElBT/LAVHgAaDQASNTxZjpm7Ag3q0drafGmRNpfGK9483AYEhN2PR4VjcBq3YC6RkxmYypjfBKaz2f2Pwbxla/y03Dn+LiwX9maL9DQCmyWZL5nEf39ljoqhpOuApOuzsnZe5kIWcqcSPWYbxeCF91JQU/ywEp5AkAhZ5ynq/147R/w5QoA00GQFsx67TRWa8ZgVCvWCn9xpkTCYcSR9zZMYrv7b6ZMzc9yokblvLHj62CcdeBKrH9ojtg+8OJfPNnJ1sWDuWKXTFOoQp1Cv23LPRGLPAAUOjH3VxcHJn7O+Fl8s/KzdmTmXIJhBqMa1qW6r+SnlP++JodrHq7d4PMUEjRVn0CTHkIzn0I9r7Ahy99kWM6tgMlbLbVur6ncMhFOb8bnCoQC+GrruSMjyAiCjwg5BIwcvsIbBbQMsvwMHs93XqyO5+Xm4SZRZ9vzxejgtSwJNe880GW8gYTt8vI8zn2qm00t8R47aUH+eaQBzg21AaUUJkb5fwJKSB6CZx5v2c3Sy7+aD981ZWa8RFExIVSpnh5BDYbJHDsgLDpcY8dEDYdOOB0Pru+LZlY5ZH7RXu8i4dXWzflMlNSDVOi3P2Ne1lZ/yYf3/wHztvyS3773sUc7BxY4v4sGmJLEtb540Pg1Ztdu1rshj7bvSeUD9KNsEzJty+11YzH9MEE6RZ3lYs5k5kW+gUnj+DJtTHHLnDF7LqYKXOm3GbXObb6Xe4Z+ys+NfBloButA9DXvHoYnHC1bTaLXRc+MG/pkGuHvqAV71Qa0g+8wsg3XcspWyDzw2/V5MqsrD6dqScMdfxgFzvFLD2g5uY6d3aM4h/f/k7qxrbmhZ9y0o47GBw6DJSwp/lbP4Otv4ZPPgHRi7M2ceOP9kPpeineKaWir8SbjCjwMiXfdC2nQg+zAJiVHHa4aRtQUx3iSEf2uYxpQn5a5wOrQ73S6Ky6Q2aSfp1nnf81mlsuY8HyLYTatnFf3S84t2Z1aSpAu9rhxVmkOrBn5Jnb+aP98lW7Ld4pZZVmUCtE80UUeJmSb6Wdk3XmxirOpTfG4EiYIx2dqZxxO+Xc2hZn1mmjs9wwuRIOKe69rD4lk5v2uWB+nb2V35cSfulNP4BdT0P7u/SMjy0Wyesw8sz/vhDO+nmirN8luVqobp8GS1mlaXXuWxevY8HyLQW1xgtp+YsCL1P8SNeys8CsLPyQUnRrnXNvDC9dDzXw5NoYV5wZZdnre/KaDxrNkNfpCSPzOiHhq7dc60ET4KwHE1/Qo9C3PQKdB3OWO2d0F7zyZVhzM3R3JF4L1cDYy0wLh/KxUN0+DZayStPuHIW0xgtt+UsQUzDFrzFUfgQoDWVaWxPmaLzL82xPs8CuVRAXesbBRdOUt1Xf9dpI2LHVAIe3wuvfh3d+h+7uKnkAVGt47ejpvH/qg3z2nHOB/ILibv9WCjUQ2g1u/g4LIYdf1ywj1QRPmKUe5pKh4Id11aU1mkRJvVflHQ4pUzePm3FwhrU0b8lGS2u9tT1O4+Pr7SsYB01IFA1d08ktOxo52FlT0vFxSsGZkfVc+PdP8tqKRL91K+Xm5v/P7d9KKas0nUYaQmGeBAr91CEuFMESP4Jc+VaB5k2GkjT8kbHW9qwe5mY9zdvjXY7+dzf9YgyWHvwUSw9+CkikJ945+hd8+tjVVFH8bJaQ0kzZ04h+pJFt9dDe3Z/lH36CH+29np0dowBv7Wedrr+UVZpupkcVIge+0L1hxIUiFBSzx+v0sW5W+eV+UhsJM7B/P1ulHc3zRpOeP2+QGbxVyrox1+zBL3Lf8T9lUFVChlK6WbSGj7r78YcPP8nAqfem3CyVQjGn1Pt1LskDF0qCk9U1PodRbJk4TQNqbY+ngqeZ2xnKe1XTDEt/5ZCa7NmimZhVl3oJ3mZa5jcNf4rPDf4rw/q1ooFQERW6UjAg1MllQ16AbTPg+KdM88zLlWI+CRT6XGKBCyUl3yBnz4Se/XkdZ/v8WY6Vi/OWbDRVxOEqxYKrTi9Idakx0OLYrp3cGl3CxbV/ZUDXe3kf1ztVMHgSTP1pUWd/CgkkiCkEEjfBpUwMY9QIlt3TUM+qphk8MGey52NBIssFEtbSFWdGU7+HlOKKM6Mp/+66uZ/lgTmTGVLT00emNhLOUt7gPkhVpbI/hMb1DakJg05Y7zs6RvGv225iUsuv+cdtczna3d/zdeZHNxzcmGiy9fpdRT63YIW4UIpMJZbzOmF3zcZ3p4rI2kg4Zf3W1oSZe0l26p7Z42pbR6dj/rhx3uaWGE+ujaV+79KaJ9fGmHrC0F7yuvn/chO8TU+P1ElFnd4W98hHnVmDNzTwwqGz+Myb/8XNI3/PFcNeorr7kKM8vvLGPDjwGry/Gn10L91aUYXmg65aDg7/PBPOm1vQeaBCD+JCKSLFDJ4EBbfX3NwS47bF60x92WY+aLfrZnb+TNz4wGuq++U13NeJcJVyNeLOVPavjeupAj36LvQ7FuKtno7jJxpQKgRjZuXUBlfIRlwoAcCpwX4l4vaaG6ZEuW5aXVY/kUg4hNbWwwecSM9RhuwWt+l5yFZujwNtcc+TazJzo2sjYYbUhFH0uGzSiXdrz8o7JbNRBXpZDK7pgqsOwDm/pFQfbwWJSlCjDW7zCbD3hZLIUumIC6WI9MWBr16u+Z6GetPuhbctXufp2Jmkuz38GErhtn+HlbvFj8wbA8t84hO/BCMvSFSA7ni8p5y+FLTtSPjOT50Hp80tnRwViCjwItIXBr5mKsjamrCpD9rqms2UnlXxRS7rZufDNmsQZkWstZ0pdz9Ha1vccyzDS3FTekfGzHRJxypGowL03IcC0GyLhO/8jXk9v1cPSWS1jLu2+LJUCOJCKSKVPvDVbGrP4aOdqeHCBl6vuVjrZlYSXhsxn1wECddKLgOB3WbeRMIh5l5yCquaZrB9/izunzM599YG6W6Wa7vg828nBjtXFTubJY2OA/DX6+D5T5dOhjJHgphFppKzUKyCgEYlZD7XfGfzBh5dvZMurQkpxTXnjOWehnq/RLfESzDSy3U69SIPKcUPr85OT8wFx785o9nWzt9DV1ve58uJ6OXwqSdLc+4yQCoxA0IlD3y18kkfbI+zbu5nU783t8TsW7Nm4Ca9r1AYx7/Vwg+fTnrFp1PbUKfjdmvtm/J2bGdquFoMXr8L/ca84g6oiD2VCHRKkZAnHF0oSqlfKaX2KaXeSHttqFLqj0qpt5LfhxRWTKEccDMo18swZoNSZ+80TImmsli84CRjw5Ror6KgdPyKi+S0dqfN5Wvv/pAt7XV0aYXW0KUVW4+OoUNbu5TyZs3XEt8Pb00Mb/59FB6pSnz3MMy5L+HGB/5r4HMZrzUBz2utTwKeT/4u9HHc+KpzUShByN7JpWIUnGWce8kpefv3jSea8U3LmD5/Ra+bYa5r9+y+icx860FO3PAM4zcs5cQNzzDjzYX80/bvQKhAQfeDmyD2LCw7NTFVqH03oBPf3/pZ4vXYs4U5d5niqMC11i8BH2S8fCmwKPnzIqDBX7GEcsRNX+hcFIqTZW+nwPzCKq/bCTczQ9OPO6QmTP9+Vdy2eJ2ra3F6onHzVOTl/TdD58GsN+Ckrybmb/pKN/zlysScTzO62hPviyWewlUQUyk1DliqtT41+Xur1ro27f0DWmtTN4pS6ibgJoC6uroz33nnHR/EFsqVXCaUODWZKmV1q13TKq9y5FKp67SeuVb/etpv7wvwyk1w6C1X15k3J321Z3RdH6FklZha64Va66la66kjRowo9OmEgJNLSqCdZV9q/7iVa2VITdjzTaQQ7qVcJyt52m/k+XDJm3CthgtXwuBT6FEtHkKhymVOxa6n3R+zwsk1C2WvUmq01nqPUmo0sM9PoYTKJdf+yFbZO4X0j7tJ+fSz33Ou7iWnIqdcM59y2m/k+QkXSzrPfxr2Pu+8r3bXN4aj7zpv06twaQ9ERsPxl8Kk2yuqN0uuCnwJcCMwP/ldbomCa/xMpSxUdauXaeJ+XU8u12JWPZr+RBOIuoML/8SuJRcRPfT/rCcNnToP3l6YDFw6MGCU/fuxZ7N96UYg9K2fJX4P1cDYy+C0u8taobtJI3wU+F9golJql1LqSyQU92eUUm8Bn0n+LghFJ98qzeaWGFPufo5xTcsY17SMyXc9l1J6frlm3AZZrdwxRz7qtNzHztXhJmWzGAFggDl/a2TO1vuyUxM7xsGFLyZ6pBx/qbuD2W13eKt9INSgqw22P5xotvXE8LJNU5RKTKHsydXKbG6J0fjE+qwugOEqldWH28Bs9qXTObwEEZtbYtz1zMas/jG5BGYLFeC0wu7/YXzTMtNWwb3W8/DWRKqgnfINRWDWRhg03vz9V2/usbK9EorAJ58I5Pg4aScrVCwNU6KsaprBtvmzWNU0w7XyWbB8i2kL13i3Nm35Ct5dM14t+YYpUWqqsz2buVj/Tj51v58y8k5nHDQhoUCt8swNBWulvCG/AGcZpimKAhf6LHbBwS6tfWmglUtg0q/ArJPS9DMA7HQzcO3qil7ck2ceGQOqKvH9pK8mLG8n67h9j2fZe9HVngh+lgmiwIU+i501bfiSc+7+53AOu3Pnso+ZL9tJaeZa5GOGr+mMgybQ3O+7TN/yW8avX8L0Lb+lud937S1vg8hoz7JnUUZpiqLAhT5L48yJWa1uDWKt7SxYvoXGmRM9u2Yyz+HVkve6j5X7ArBVmvkEgDNvGLUuero0TInSOHMiY2oj7E6ur1nQNJd+OSncBkLtcJOmGBCkG6HQZzEUmVnQEJw7Cno5h5sga3oQsDZZUn+w3XlghJ37wu7Gk2sOu1mKZbhKEQ6pXjGFzJuB29RMu+tx/H+YdDts/bVzFoodTmmKBoe3svWluzjmvWUMC33A+11DOTR8VlGHOksWiiAkyaXM3y/yyQhxleHhI7n2fXe7vnlfj1keuBfclOrHnqXzpSvop49mvdWpBtDvvCd9zWaRfuCC4EApux7mY3UWe1Sf277vbvfLfD3v6zECoZt+AO88Bh3vu9sPEpkukxrtt0nmmpspbyDx+l+uTMhQYEtcfOCCkMTPoJ5X8rl5FHtUX67dId2ury/XY4yQu/K9nvFxoYH2+7hJU4TEjcGxUCiZzVLg3uaiwAUhSSlnluZz88i1YVWu2K2TXQDS7fr6fj3GxKE5hxMNtz7/du5piuA+S+Wdxwre21x84IKQRql6h/hdFVlorNbJTfVnyXuz5MsjVWDqpfdIKOLazWLlAxcFLggBoZDKrViKs9gB1ZLw+6i7pltucNnbXIKYghBwCjXw2ktnxXwpdkC1JBx/ae79VjLZ9XRewynEBy4IFU4xh16UMo5QNCbd7t9c0DyLhsQCF4QiUSr/bzHTI+2KgyrC/w09Tbescs1DkcRXR+YoYRPcFg1ZIApcEIpAMd0YmRTbrWHmCirl9ReE9FzzXU8nLOkBo5JTfxph0wJ3bpY8S//FhSIIRaCUszuD4NYo9ezSgmDkml8Wg2u6Et/PejCRR+7GzeKmaMgBUeCCUARKWeVZ7DxxM0p5/SXBj97mLhAXiiAUgVJnZxQqw8Utpb7+kuDkZslTeYNY4IJQFILgxiglffb67dwsPiAWuCAUgVxbt1YKff36C4VUYgqCIAQcGWosCIJQYYgCFwRBKFNEgQuCIJQposAFQRDKFFHggiAIZUpRs1CUUvuBI8B7RTtp7gxH5PQTkdNfRE5/CbqcJ2itR2S+WFQFDqCUWmOWDhM0RE5/ETn9ReT0l3KRMxNxoQiCIJQposAFQRDKlFIo8IUlOGcuiJz+InL6i8jpL+UiZy+K7gMXBEEQ/EFcKIIgCGWKKHBBEIQypSAKXCn1OaXUFqXU35VSTSbvK6XUj5Pvv66UOqMQcvgg5/lKqYNKqXXJr++XSM5fKaX2KaXesHg/KOvpJGfJ11MpNVYptVIptUkptVEp9Q2TbUq+ni7lDMJ6DlBKvaKUWp+U8y6TbYKwnm7kLPl6ekZr7esXEALeBiYA1cB64OMZ21wM/AFQwDRgtd9y+CTn+cDSYstmIut5wBnAGxbvl3w9XcpZ8vUERgNnJH8+BngzoH+fbuQMwnoqYFDy5zCwGpgWwPV0I2fJ19PrVyEs8LOBv2utt2qtO4DfAZmjly8FfqMTvAzUKqVGF0CWfOUMBFrrl4APbDYJwnq6kbPkaK33aK1fS/58CNgEZE4VKPl6upSz5CTX6HDy13DyKzMzIgjr6UbOsqMQCjwK7Ez7fRfZf3hutik0bmX4RPKx6w9KqVOKI5pngrCebgnMeiqlxgFTSFhj6QRqPW3khACsp1IqpJRaB+wD/qi1DuR6upATArCeXiiEAlcmr2Xe6dxsU2jcyPAaiR4EpwM/AZoLLVSOBGE93RCY9VRKDQKeBG7VWn+Y+bbJLiVZTwc5A7GeWusurfVk4HjgbKXUqRmbBGI9XcgZiPX0QiEU+C5gbNrvxwO7c9im0DjKoLX+0Hjs0lo/C4SVUsOLJ6JrgrCejgRlPZVSYRJK8WGt9VMmmwRiPZ3kDMp6psnTCrwAfC7jrUCsp4GVnEFbTzcUQoG/CpyklBqvlKoGvgAsydhmCfAPyej0NOCg1npPAWTJS06l1CillEr+fDaJ9Xq/yHK6IQjr6UgQ1jN5/v8BNmmtf2SxWcnX042cAVnPEUqp2uTPEeDTwOaMzYKwno5yBmE9veL7VHqtdadS6mvAchKZHr/SWm9USv1z8v3/Bp4lEZn+O9AGfNFvOXyS80rgq0qpTqAd+ILWuuiPfkqpR0lEyIcrpXYBc0kEYQKzni7lDMJ6TgduADYk/aEA3wHq0uQMwnq6kTMI6zkaWKSUCpFQeI9prZcG7fPuUs4grKcnpJReEAShTJFKTEEQhDJFFLggCEKZIgpcEAShTBEFLgiCUKaIAhcEQShTRIELgiCUKaLABUEQypT/DwlWLEwvsUOnAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "loss = 44.59417\n" + ] + } + ], + "source": [ + "from IPython.display import clear_output\n", + "\n", + "for i in range(100):\n", + "\n", + " y_pred = w * x + b\n", + " loss = torch.mean((y_pred - y)**2)\n", + " loss.backward()\n", + " \n", + " with torch.no_grad():\n", + " w.data = w - 0.05 * w.grad.data\n", + " b.data = b - 0.05 * b.grad.data\n", + "\n", + " # zero gradients\n", + " w.grad.zero_()\n", + " b.grad.zero_()\n", + "\n", + " # the rest of code is just bells and whistles\n", + " if (i + 1) % 5 == 0:\n", + " clear_output(True)\n", + " plt.scatter(x.data.numpy(), y.data.numpy())\n", + " plt.scatter(x.data.numpy(), y_pred.data.numpy(),\n", + " color='orange', linewidth=5)\n", + " plt.show()\n", + "\n", + " print(\"loss = \", loss.data.numpy())\n", + " if loss.item() < 0.5:\n", + " print(\"Done!\")\n", + " break" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Bonus quest__: try implementing and writing some nonlinear regression. You can try quadratic features or some trigonometry, or a simple neural network. The only difference is that now you have more weights and a more complicated `y_pred`. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# High-level pytorch\n", + "\n", + "So far we've been dealing with low-level torch API. While it's absolutely vital for any custom losses or layers, building large neura nets in it is a bit clumsy.\n", + "\n", + "Luckily, there's also a high-level torch interface with a pre-defined layers, activations and training algorithms. \n", + "\n", + "We'll cover them as we go through a simple image recognition problem: classifying letters into __\"A\"__ vs __\"B\"__.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data...\n", + "Extracting ...\n", + "Parsing...\n", + "found broken img: ./notMNIST_small/A/RGVtb2NyYXRpY2FCb2xkT2xkc3R5bGUgQm9sZC50dGY=.png [it's ok if <10 images are broken]\n", + "Done\n", + "Train size = 2808, test_size = 937\n" + ] + } + ], + "source": [ + "from notmnist import load_notmnist\n", + "X_train, y_train, X_test, y_test = load_notmnist(letters='AB')\n", + "X_train, X_test = X_train.reshape([-1, 784]), X_test.reshape([-1, 784])\n", + "\n", + "print(\"Train size = %i, test_size = %i\" % (len(X_train), len(X_test)))" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAADHCAYAAAAAoQhGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAeFUlEQVR4nO3deZTU1ZUH8O/t6qah2RdZZV/iLmhHMMocEmOimAmM0USNohkMGjWJSxbGmCE6mJgo7hkdjATE3QGjYzCKjIlxQ1oluKCAiIogoLI0DTRN150/upxDc+8PftW1vvL7OcfT9OVW1auqV4+fdd8iqgoiIgpPWaEbQERELcMBnIgoUBzAiYgCxQGciChQHMCJiALFAZyIKFAcwImIAsUBvAiJyEUiUiMi9SIys9DtIcoWEekiIg+LSJ2IvCciZxS6TSErL3QDyLUGwFQAXwfQpsBtIcqm3wPYCaAHgOEA/iwi/1DVNwraqkAJV2IWLxGZCmB/VT2n0G0hypSItAWwEcAhqrosFZsN4ENVnVzQxgWKX6EQUb4MA9D42eCd8g8ABxeoPcHjAE5E+dIOwOY9YpsBtC9AW0oCB3AiypetADrsEesAoLYAbSkJHMCJKF+WASgXkaG7xQ4HwAJmC3EAL0IiUi4irQEkACREpLWIcMYQBU1V6wDMBXCViLQVkWMAjAMwu7AtCxcH8OJ0BYDtACYDODP15ysK2iKi7LgATVNj1wO4D8APOIWw5TiNkIgoULwCJyIKFAdwIqJAcQAnIgoUB3AiokBlNICLyAki8raIrBAR7mVAJYN9m0LQ4lkoIpJA08T84wGsBrAIwOmq+mbUbVpJpbZG2xY93t4b48SyMLlGO1SZ2M7Ofm67NvUm1iax081NIGli3lOIbJcTa1T/3+KtjZUmtr3OxgCgcqNtF+q2p9GywtqBOuzU+nReSldR9e0SIOV2CUNjO78Plm2qy3VzghTVtzNZHHIUgBWquhIAROR+NE3Kj+zkrdEWI+W4DB7S53UQTUaM4MlG5w78z/zOY6pNbNXJ/t0ec8hyEzu8wwdubvuyHSZWIU67IjRowsQ2N9p/bADghU8Hmdjriwa6uYMfdD48L70Wu10os+2K5L0PGVqoC7J1V0XTt0tBolt3E6v9kt8Hqx5eaIMRn098jqZAR/XtTL5C6QNg9xFqdSrWjIhMSh1OUNMAe5VKVITYtykImQzgsb64UNXpqlqtqtUV8P+3iajIsG9TEDL5CmU1gL67/b4/mk6SyQ7nf5sk4f8vuu7aZXMr/Q/UmotGmtiY0xe5uW0SNr5q0Rfd3KUzDzSxDUv6ubnl6/bcURPQTVucxIi3p5PdfXPn/p3c1PUjWptYYuRWN/fEP/7dxJ7fONjNff/2oSbW8e4X3Vz3vYx4bt57WQC57dulIOrrMuersS2j7dcl4371lHvzBY91NTFt8GtJ7lcrn6OvVYDMrsAXARgqIgNFpBWA0wA8mp1mERUU+zYFocVX4Kq6S0QuAvAEmnbNm8FNaagUsG9TKDLaolRV5wGYl6W2EBUN9m0KAVdiEhEFigM4EVGgCn/KS8QkfSmvMLGoavT2cUeZ2D9f7U98f2vrEhN75Zoj3Ny2c2tMbFjSn7GSjoznWWzYYEIJu44IANDrr/Er9U+072tiayZ+wc390S/mmtjzFw1xcz/8oZ2FoIv8BUJS0crmRs1CoIKRMv9zq85i3g2n2tW8P+3yjnv7P43/qom1e8hZ3AN/VlqRzGLKG16BExEFigM4EVGgOIATEQWKAzgRUaDyX8Tco2gZuTzeKVytuGGUmztn/E0mdu7Ui93crne+YGJt5SU31+MV2SJFFHrg7ZToVX8k4t9X734b09jdL+J+k1vtEvueNz7v5j54Y08TW3nNoW7u03OuNbGTpv3Mze15k328qNecxc08cSYaRBULEwcNM7GHRk53Mu0WDwCw5YxaE2v3kN+syB1HP0d4BU5EFCgO4EREgeIATkQUKA7gRESB4gBORBSo/M5CETvrJKqavf6RA0zs/MHz3dxf/NO3TKzrB3a2CRAxo8GbARLRtqKd+RB1bqBHI5Ybp3HwgrSyr+Ogyf5r/r3Hf2hi986a5uaO63uJiQ3+iX9QhHkvGzI+z5gc6SxZXz7BHshwWCt/xonnnhEzTGzygWe5uY1Lnf0jPmfnZ/IKnIgoUBzAiYgCxQGciChQHMCJiAKVURFTRFYBqAXQCGCXqlbv9QZqix8rZo9wU7/T72UTe+rQDm5uolu9iZUdZougAJBc8pYNRu5Jbl+eqOLN+1O+ZGLf+hd7yjsALNva3cRaldml8Jt2tnFvv3KB3V+7/7TFbm5y2zYbjCr0OEvso56vVh9kYu+d2M7N7T/FLo//6YkT3Nx7591iYuds/LGb2/fqPe43i4WqtPt2KYjoF14fKGvf3s396bhHYj3U5qTdIxwADmtl+/y7p+7n5va7yhYxI7fmKNF9wrMxC+XLqvpxFu6HqNiwb1NR41coRESBynQAVwBPisjLIjIpGw0iKhLs21T0Mv0K5RhVXSMi3QHMF5G3VPWZ3RNSnX8SALRGVYYPR5Q37NtU9DK6AlfVNamf6wE8DMCcLqyq01W1WlWrK1CZycMR5Q37NoWgxVfgItIWQJmq1qb+/DUAV+3tNg092mLN2c1na/xu5N1u7p2jncMbdL2bu2W2rYhvXtDFze1tD6WPrFxHHqjg2DnEVtWndvdPXt/WzS7HryqzS9Mb1D+koWKYbe/4r37dzd3x1XS2A/C3FHC9aF/Ii2dscFNvKP+miQ34hb/s/vJ/Pc/Ebr/zP93cKTXnNvtdn/fvM10t6dulIJ0ZHOvOPMTNPaO93e7iuR22bx/T2p9h5fnGeP99XeK8I6U62yRKJl+h9ADwsDRNPSoHcK+q/iUrrSIqLPZtCkKLB3BVXQng8Cy2hagosG9TKDiNkIgoUBzAiYgCldf9wLt0rcXpZy1oFrv2qjPc3I7r7B7Qq64+2s19+7DbTKx6zg9a0MKW02T8vajrnf24Exr/9puTtgj5p6FPuLmHXnKBifX+nX/SvJRX2GAae6Vfv+Q4N3fZ9+z7M/It//3pNNsWrL5/r5/b72erm/0u5ze4eRSPNvpFc8/Y855140c+Z6fMJ99ta2LLJ9g+AQDbnL59bc9X3dwxJ37fxCofX+TmprMtRkh4BU5EFCgO4EREgeIATkQUKA7gRESB4gBORBSovM5C+WRje8ye03ymQr+7/RkRydH2oIe/T7gu4p5tlTsZsTo+V6Qs/mEClWJf9kpxZoBEiFpi76k7ZEfsXFca2wk0bIp/+vikyx9243Pm2YMiBt/wtpu7akfzQzt21tol2xShzPmAJP1+tesrR5rY1O53urkvXGa2jEF5XZ1N9M/zQCLqsBHHB2fYWSRDHo9985LAK3AiokBxACciChQHcCKiQHEAJyIKVF6LmK0+qkP/q19qFosq/W2evNXEuidssTJS/FpIVmhj/H8Lt6mz5DuNrbgbI181q/3L8QuLroil9J423bbFzp3Y8SM3/usrxpvYkEvstgoA0O/al5v9vqY+/uN/3kmZ/YBEvdXrf2j3ur9x4wA3t3zByzboFEwvW3uEe/tpvV7xG+H4/dH3mtgtfY93c3d9sNoGowqmGv/zVWi8AiciChQHcCKiQHEAJyIKFAdwIqJA7XMAF5EZIrJeRF7fLdZFROaLyPLUz865bSZR9rFvU+jizEKZCeBWAHftFpsMYIGqXiMik1O//3yf96R2E/WyQw5wUx87zC7VbVT/JOuEt9w7z4VkScSfrVHlLJtP51T6KrG5xy452c3tecvC2O3SXZkdiHDqUH/j/XRc+vU/m9j/VPZ2c7W+fo9A2m/6TGSrbxcrb8k8/MMMygcNcHOfqf6DiY3+/U/c3D5wtsZwlug/9hf/cJZp37OzUDYn7SwYADihysZ+fko/N7fnDXYWiiTivzbFap9X4Kr6DIBP9wiPAzAr9edZAMZnt1lEuce+TaFr6XfgPVR1LQCkfnbPXpOICop9m4KR84U8IjIJwCQAaA3n/3mIAsW+TYXW0ivwdSLSCwBSP9dHJarqdFWtVtXqClS28OGI8oZ9m4LR0ivwRwGcDeCa1M9HWtqANcd3cePesvmoYkZH8YubGUtjGXnVP2wbvtH3RDd3bW17E6twiqB19f7+1omnO5lYz9tq3FyN2OPZ4xV1ogo69WO/aGJX7ndH7MdqjHhtL+z0gYnNGf01N7fiKWfZduay1reLgbdkHvC79rLze7m5axrtffS7c7mbG7e3DfzvLW584wS7HUKHsvjbQRx8ylI3/skNNqaN8T8bxSrONML7ALwA4AsislpEJqKpcx8vIssBHJ/6nSgo7NsUun1egavq6RF/dVxEnCgI7NsUOq7EJCIKFAdwIqJAcQAnIgpUXg908Gw5MLPl27mUzpLa3tfZJcQN1/m53bA21n12jP3ogEYsmXY3rY9Ycp7O8+308/dj59Y7B1g0RrTB2yZg4wH+FL3uT8VuwueD815HvadSaV/TG0/+o5v7jacvMrFhG/wZQFJh3z9viwZ99Q339mevtFtCPDr0L26ut9XErAF+pxh77EQTK3t2sZvrbj+QxmyufOIVOBFRoDiAExEFigM4EVGgOIATEQWq4EXMTj1qC92E7HAKH1H7DWfMWQed1h7GUQVPp1CzbLpdMg8A7w61y+a9YiUAVDr7n2/TnXtpYHPbu4dzSnghpbMVwscT7KnwJ1X5e8ff8p/2vYp6R7zl6em0693HBtngJf5jeX2oY5m/rcaKM2xxddiz/v267WURk4iIsokDOBFRoDiAExEFigM4EVGgCl7ETKq/XzERpSed/a1HTbKHB397pb8Joy56zQbTKIRrGp/xfnPWmNjiC+qdTGB4ZfxzAK78ylwTu7/zoW5u48aNNuitaAZacpB2VvEKnIgoUBzAiYgCxQGciChQHMCJiAIV50zMGSKyXkRe3y32KxH5UEQWp/4bm9tmEmUf+zaFLs4slJkAbgVw1x7xG1Q1Ysfr+Lasaxc7N1ngiu9eedX3fC6/jZoV4B0/nka7hk1a5MZPfuZ4E5s7ZL6bG7XEPq4263M2U2kmcti3cyaNGSCNY+ySeQC4tc8MExv1s/Pd3I74xMTSOe3em6kh5f7Qs2vlKhOb+PpZbu7LRz5oYluTO9zcCR0+NrGbTznAze16xwsmFrUtRlpbWOTAPq/AVfUZAJ/moS1EecW+TaHL5Dvwi0RkSep/QztnrUVEhce+TUFo6QB+G4DBAIYDWAtgWlSiiEwSkRoRqWmAPyGfqIiwb1MwWjSAq+o6VW1U1SSAOwActZfc6apararVFfDPNiQqFuzbFJIWLaUXkV6q+tnJvP8C4PW95e9Nh6V2r2gAwD/bUFnUctYc8QotUUWLNT/5kon1/8a7bu7a2vYmVpGw1Z+tO/xBofxpe9xxz9tr3FxtiF+wTOf5fvrbATZotwgH4O8H3giv2uXrvCz+3uGZymbfzpV0CoirL/ALyE9us+9Jpwfs8noAgHNQsSQirv1i7oEfVcT0+pv8qat/J0faUGPkTuVW1Skf+X/h9GNNFucEin0O4CJyH4AxALqJyGoAUwCMEZHhaNrXfRWA83LXRKLcYN+m0O1zAFfV053wnTloC1FesW9T6LgSk4goUBzAiYgCxQGciChQBT/Qoc+TdokrAKy9dKuJdU9U5bo5zUn8f9+2Hb7dxB4b9rifm7SzKqrKbKW/Qf0ZJBWjbKX/2JNOdnPbjn3PBiOW0rsHAkTM/Kn8s11iP2XDwW7ulfu9YWKJiNf29k19TKzV35wDBRB9MnpJcV7/qJlB5fvb1+7Zo293c798809NrHfD834TnBkjyR2ZzQzSnfFv3/0h238A4MZLB5jYxZ1Xxb7fuQfd7cbPHDHJxPRVvw3utgZ53EKDV+BERIHiAE5EFCgO4EREgeIATkQUqPwWMcUWRBrfXOamnrR4oom9Uv1AWo+VT9qYRsHT2x87/spybFNbAHr2MHvqNgAc/sMLTKznTVHFKmdbA3eDZ7+Q9tDyEW6uV8SMcu2Tdg+FIfUvurlSucdWA/V5ftPzwHtPtMEvAK48t7+JdUu0dXO3D7dF9xU3jnJztcIpF+eogqxl9o4T2/3P1mmJN2Pfr7dPeNRr8853OpjYoFf9+/X2Cc/nOQC8AiciChQHcCKiQHEAJyIKFAdwIqJAcQAnIgpUXmeh7OzZFu9Nan7ASb+r/BkR3X7bxsTW3meX1wNAr3LnZPs8r7MW50CGKFXOAQfeUvoo3lL8KLVH2up7z9i3RlrbCWz/OP5WB3dt6ebGvzB1uQ127eLmrrqg+aniO+/4W+zHLzoRWxa4M04ics87dZ6JPbfD75enHWwPAOlXaU+fB4AGjXdIQ64k1e+D63bZg03q//8sjubaSPzP17knPWViT19hHwtI7/2BZn9Q4hU4EVGgOIATEQWKAzgRUaA4gBMRBSrOocZ9AdyFptpXEsB0Vb1JRLoAeADAADQd/vptVd24t/vq2rkW3z3lf5vFHntnjJvb8R67fHrMLLuHMQC8PfE2EyvL32pWAIAm4y/jrle7DD2hmd2+Cn6RpurN1rHv1xWxlN5T0bE+du6tvznVjXf++AUTW/UfR7u5+x+9utnvH97vn8AeJZt9O1PekmzA37Jg83dHurkXd7Z7f4++8AdubtXDC02sprx37DbkjFcAjCj+ma0UAFS9OsjNvbDTByZW721pAeDnXW0h/ZFvne/mtn/AjlPpvJeZinMFvgvAZap6IIBRAC4UkYMATAawQFWHAliQ+p0oJOzbFLR9DuCqulZVX0n9uRbAUgB9AIwDMCuVNgvA+By1kSgn2LcpdGl9By4iAwCMALAQQA/VpkmXqZ/dI24zSURqRKSmbmNmxzAR5UqmfbsB8b8+IsqW2AO4iLQDMAfAxaq6Je7tVHW6qlaranXbzvEn0xPlSzb6dgXs97FEuRZrABeRCjR18HtU9bONp9eJSK/U3/cCsD43TSTKHfZtClmcWSgC4E4AS1X1+t3+6lEAZwO4JvXzkX3d16eftMcDs7/SLPbLKfe4uTOfPMrEBvzSzlAAgGO/aE9kr++c3839xdmIPkql2Je90lleH3UqfeeEXbJ+xrtfdnP73rzYxKLmlegupyqfxvLfSw+3S5ABYNhMOxNi4Cz/vWwcc4SJ3fFdO8sIAKb84NzmgY/S2xkim307U5qM/zr3nvSOG39wq13uXfWIXTIPwD9NPWLbBO9U+nyKevzkDrtNxHXPnOjmXvjN6Sa2LenPQqlM2M/ilu/UurntnTNm0nkvMxXnnTkGwFkAXhORxanY5Wjq3A+KyEQA7wPw54URFS/2bQraPgdwVX0W0QeUHZfd5hDlD/s2hY4rMYmIAsUBnIgoUHmtTlSsq0Pva5vv//1vh9sCJACcPt8WXxYd4U9D7HCWLTC07+XnegU8bfSLhZLGNsitVtj9y6849FA3d9lWO624lbP2f9NOe58AsOJvA01s0LTX3dzktm02GLVfsVfEiiikYtRhJnTjQ86+7AAGTrF7vie+MMTN/fcZfzCx78+4yM3t+0Tz+xV1nmsx8gqIESeZ65cON7G5Q2Y5mcDgB+xy7yFJu9QbAKTCfj6iTrsPSd/HI/7imzbUriz+1M+7R8xw45cPO8PEGpf5ReZ0tgmIi1fgRESB4gBORBQoDuBERIHiAE5EFCgO4EREgcrvGlmxy2KHnPmqmzrv0YNMbMJrdhN6AJg/1m5E37jkLb8JaSwLTmcD9n5X2pkWi66Mmsbin/5tbXKj/WFP3m6MnFmSTuXbztGJer3kNVtp7/9inX+vxw43sV/PtkubAeA7//1jExs81b62gDOToiG/2ye0lJTZdkadm7F8ol3WHWXIA/Fn4UTNvCpG6bS1at5iNz59sx0jJnVc4+ZuTm43seGV/oywVd/uYWJ9p/qzUKTcvpeZzvzhFTgRUaA4gBMRBYoDOBFRoDiAExEFKr9FTLUFiagi2X7ffNvEbr75627u/Gemmdipv/FPsN/vdmcf6jQKgF4hAgDgFKYiefsFe1WsiP2Z3cdKpygVcb/efuBRhVwvvupq//T4ud+93sTOvO4yN3fwrbZg6S37BpwCUIbLkjPiFOgRdTp5vT1+rXxgfzd34dduMrHL1o722/DiEtusiM9XPveszpWy1q1NzNsjHAB+89xYE5s01m7bkK7x33rWxF7+tT9GeMXYqL5txoOI+RS8AiciChQHcCKiQHEAJyIKFAdwIqJA7XMAF5G+IvK0iCwVkTdE5Mep+K9E5EMRWZz6z1YJiIoY+zaFLs4slF0ALlPVV0SkPYCXRWR+6u9uUNXr0nrEPWYKRB6m4FRnh/7IX0p/zl/tjIZJUx91c9/6fi8Te+l31W5uhzn2UIngNr13l9L7Je2yKnva/Uf/OtzNPev8v5jY8xuXu7k/GT/RxLovjrk8Hjl9zbPXt9WZmZPGzKC3r+7sxrsn2prYk+8f4Ob2xpuxHy/qAImiFDG7SBsj9h9wtFobf0uCpPN4W5P+7JZf97Azfw785QVurrfdRuRcoD3fn4jEOIcarwWaNt9Q1VoRWQqgz75uR1Ts2LcpdGl9By4iAwCMAPDZpfBFIrJERGaIiHsJISKTRKRGRGoaYOe/EhUD9m0KUewBXETaAZgD4GJV3QLgNgCDAQxH01WMXU0DQFWnq2q1qlZXIP4ZdET5wr5NoYo1gItIBZo6+D2qOhcAVHWdqjaqahLAHQCOyl0ziXKDfZtCJrqP5cciIgBmAfhUVS/eLd4r9R0iROQSACNV9bS93VcH6aIj5biYLXOWsUctTXaWdXvLbAFg9Y+OMLGxp/sFtdZldmn5XTX+cvEuL9kiSdcl/v7MFR9tMjHdtMUmJiL+fe3UwYR27u8XwTYMt69D3VF+u845xJ5g/vwng9zcTf/Vz8Ta3++fgJ7pexnXQl2ALfpp7D0Nstm32/Tqq4POvrRZrK6fXyg895/+amKXd7NbRwBAo7PFQjKiojX8xQk2WNPRze03b6O9338sdXNR5rxXmRZBo7avcMaj8l493dRV59i+WX+I3csbAP46+hYT27+8nZvrveaJqG0t0vC99+0WCH9/7mA3t2pN88d7Z/b12P7RB+ZFizML5RgAZwF4TUQWp2KXAzhdRIajqT66CsB5Me6LqJiwb1PQ4sxCeRaA98/lvOw3hyh/2LcpdFyJSUQUKA7gRESB4gBORBSofc5Cyaa0ZqGkwdu0PnLDeq967lXZAdSfYGesvHeyf79jDrGzCA5t96GbW1VmF31USPyq/g61M14277LL4AHgxY0DTWzpQhsDgMEP1pqY1rweu11pHR6Qg6Xc6c5CyabuB3XVU2af2Cy2ZZc/N3zdNjuLaPUmf7aIN1mjssLOjgKAXu3t+ze66wo3d860r5pY55nOYSeI+HxlMFsIQORnzu0XRx3qpo664xUTe6duPzf3/Vo7S2vzdn+mmvcJTyb9a10RZ9ZMmb/Ef2DnT0yscyt/1kzb8uZjxINnPoH1b35iegOvwImIAsUBnIgoUBzAiYgCxQGciChQeS1iisgGAO+lfu0G4OO8PXj+8HkVTn9V9atYObZb3w7hdWqpUn1uITwvt2/ndQBv9sAiNarqn6QQMD6vz7dSfp1K9bmF/Lz4FQoRUaA4gBMRBaqQA/j0Aj52LvF5fb6V8utUqs8t2OdVsO/AiYgoM/wKhYgoUHkfwEXkBBF5W0RWiMjkfD9+NqUOvF0vIq/vFusiIvNFZHnqp39UThETkb4i8rSILBWRN0Tkx6l48M8tl0qlb7Nfh/Pc8jqAi0gCwO8BnAjgIDSdfHJQPtuQZTMBnLBHbDKABao6FMCC1O+h2QXgMlU9EMAoABem3qdSeG45UWJ9eybYr4OQ7yvwowCsUNWVqroTwP0AxuW5DVmjqs8A+HSP8Dg0nbOI1M/x+WxTNqjqWlV9JfXnWgBLAfRBCTy3HCqZvs1+Hc5zy/cA3gfAB7v9vjoVKyU9PjsQN/Wze4HbkxERGQBgBICFKLHnlmWl3rdL6r0vlX6d7wHc26uZ02CKlIi0AzAHwMWquqXQ7Sly7NuBKKV+ne8BfDWAvrv9vj+ANXluQ66tE5FeAJD6ub7A7WkREalAUye/R1XnpsIl8dxypNT7dkm896XWr/M9gC8CMFREBopIKwCnAXg0z23ItUcBnJ3689kAHilgW1pERATAnQCWqur1u/1V8M8th0q9bwf/3pdiv877Qh4RGQvgRgAJADNU9eq8NiCLROQ+AGPQtJvZOgBTAPwJwIMA+gF4H8CpqrpnQaioicixAP4O4DUAn50PdTmavi8M+rnlUqn0bfbrcJ4bV2ISEQWKKzGJiALFAZyIKFAcwImIAsUBnIgoUBzAiYgCxQGciChQHMCJiALFAZyIKFD/B0AGctKTTGscAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "for i in [0, 1]:\n", + " plt.subplot(1, 2, i + 1)\n", + " plt.imshow(X_train[i].reshape([28, 28]))\n", + " plt.title(str(y_train[i]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's start with layers. The main abstraction here is __`torch.nn.Module`__" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Base class for all neural network modules.\n", + "\n", + " Your models should also subclass this class.\n", + "\n", + " Modules can also contain other Modules, allowing to nest them in\n", + " a tree structure. You can assign the submodules as regular attributes::\n", + "\n", + " import torch.nn as nn\n", + " import torch.nn.functional as F\n", + "\n", + " class Model(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " self.conv1 = nn.Conv2d(1, 20, 5)\n", + " self.conv2 = nn.Conv2d(20, 20, 5)\n", + "\n", + " def forward(self, x):\n", + " x = F.relu(self.conv1(x))\n", + " return F.relu(self.conv2(x))\n", + "\n", + " Submodules assigned in this way will be registered, and will have their\n", + " parameters converted too when you call :meth:`to`, etc.\n", + "\n", + " .. note::\n", + " As per the example above, an ``__init__()`` call to the parent class\n", + " must be made before assignment on the child.\n", + "\n", + " :ivar training: Boolean represents whether this module is in training or\n", + " evaluation mode.\n", + " :vartype training: bool\n", + " \n" + ] + } + ], + "source": [ + "from torch import nn\n", + "import torch.nn.functional as F\n", + "\n", + "print(nn.Module.__doc__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There's a vast library of popular layers and architectures already built for ya'.\n", + "\n", + "This is a binary classification problem, so we'll train a __Logistic Regression with sigmoid__.\n", + "$$P(y_i | X_i) = \\sigma(W \\cdot X_i + b) ={ 1 \\over {1+e^{- [W \\cdot X_i + b]}} }$$\n" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# create a network that stacks layers on top of each other\n", + "model = nn.Sequential(\n", + " nn.Linear(784, 1), # add first \"dense\" layer with 784 input units and 1 output unit.\n", + " nn.Sigmoid() # add softmax activation for probabilities. Normalize over axis 1\n", + " \n", + ")\n", + "\n", + "\n", + "# note: you can also add layers with model.add_module('l1', ), all layer names must be unique\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Weight shapes: [torch.Size([1, 784]), torch.Size([1])]\n" + ] + } + ], + "source": [ + "print(\"Weight shapes:\", [w.shape for w in model.parameters()])" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "tensor([ 0.4526, 0.4411, 0.5917])" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# create dummy data with 3 samples and 784 features\n", + "x = torch.tensor(X_train[:3], dtype=torch.float32)\n", + "y = torch.tensor(y_train[:3], dtype=torch.float32)\n", + "\n", + "# compute outputs given inputs, both are tensors\n", + "y_predicted = model(x)[:, 0]\n", + "\n", + "y_predicted # display what we've got" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's now define a loss function for our model.\n", + "\n", + "The natural choice is to use binary crossentropy (aka logloss, negative llh):\n", + "$$ L = {1 \\over N} \\underset{X_i,y_i} \\sum - [ y_i \\cdot log P(y_i | X_i) + (1-y_i) \\cdot log (1-P(y_i | X_i)) ]$$\n", + "Your task is to implement crossentropy loss __manually__ without using `torch.nn.functional`. \n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "crossentropy = # YOUR CODE\n", + "\n", + "loss = # YOUR CODE\n", + "\n", + "assert tuple(crossentropy.size()) == (\n", + " 3,), \"Crossentropy must be a vector with element per sample\"\n", + "assert tuple(loss.size()) == tuple(\n", + "), \"Loss must be scalar. Did you forget the mean/sum?\"\n", + "assert loss.data.numpy() > 0, \"Crossentropy must non-negative, zero only for perfect prediction\"\n", + "assert loss.data.numpy() <= np.log(\n", + " 3), \"Loss is too large even for untrained model. Please double-check it.\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Note:__ you can also find crossentropy loss in `torch.nn.functional`, just type __`F.`__. However, it operates on raw logits instead of probabilities." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Torch optimizers__\n", + "\n", + "When we trained Linear Regression above, we had to manually .zero_() gradients on both our tensors. Imagine that code for a 50-layer network.\n", + "\n", + "Again, to keep it from getting dirty, there's `torch.optim` module with pre-implemented algorithms:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "opt = torch.optim.RMSprop(model.parameters(), lr=0.01)\n", + "\n", + "# here's how it's used:\n", + "loss.backward() # add new gradients\n", + "opt.step() # change weights\n", + "opt.zero_grad() # clear gradients" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# dispose of old tensors to avoid bugs later\n", + "del x, y, y_predicted, loss, y_pred" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Putting it all together" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# create network again just in case\n", + "model = nn.Sequential()\n", + "model.add_module('first', nn.Linear(784, 1))\n", + "model.add_module('second', nn.Sigmoid())\n", + "\n", + "opt = torch.optim.Adam(model.parameters(), lr=1e-3)" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "step #0 | mean loss = 0.573\n", + "step #10 | mean loss = 0.371\n", + "step #20 | mean loss = 0.218\n", + "step #30 | mean loss = 0.159\n", + "step #40 | mean loss = 0.141\n", + "step #50 | mean loss = 0.127\n", + "step #60 | mean loss = 0.131\n", + "step #70 | mean loss = 0.107\n", + "step #80 | mean loss = 0.116\n", + "step #90 | mean loss = 0.101\n" + ] + } + ], + "source": [ + "history = []\n", + "\n", + "for i in range(100):\n", + "\n", + " # sample 256 random images\n", + " ix = np.random.randint(0, len(X_train), 256)\n", + " x_batch = torch.tensor(X_train[ix], dtype=torch.float32)\n", + " y_batch = torch.tensor(y_train[ix], dtype=torch.float32)\n", + "\n", + " # predict probabilities\n", + " y_predicted = # YOUR CODE\n", + "\n", + " assert y_predicted.dim(\n", + " ) == 1, \"did you forget to select first column with [:, 0]\"\n", + "\n", + " # compute loss, just like before\n", + " loss = # YOUR CODE\n", + "\n", + " # compute gradients\n", + " \n", + "\n", + " # Adam step\n", + " \n", + "\n", + " # clear gradients\n", + " \n", + "\n", + " history.append(loss.data.numpy())\n", + "\n", + " if i % 10 == 0:\n", + " print(\"step #%i | mean loss = %.3f\" % (i, np.mean(history[-10:])))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Debugging tips:__\n", + "* make sure your model predicts probabilities correctly. Just print them and see what's inside.\n", + "* don't forget _minus_ sign in the loss function! It's a mistake 99% ppl do at some point.\n", + "* make sure you zero-out gradients after each step. Srsly:)\n", + "* In general, pytorch's error messages are quite helpful, read 'em before you google 'em.\n", + "* if you see nan/inf, print what happens at each iteration to find our where exactly it occurs.\n", + " * If loss goes down and then turns nan midway through, try smaller learning rate. (Our current loss formula is unstable).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Evaluation\n", + "\n", + "Let's see how our model performs on test data" + ] + }, + { + "cell_type": "code", + "execution_count": 254, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Test accuracy: 0.96585\n" + ] + } + ], + "source": [ + "# use your model to predict classes (0 or 1) for all test samples\n", + "predicted_y_test = # YOUR CODE\n", + "\n", + "assert isinstance(predicted_y_test, np.ndarray), \"please return np array, not %s\" % type(\n", + " predicted_y_test)\n", + "assert predicted_y_test.shape == y_test.shape, \"please predict one class for each test sample\"\n", + "assert np.in1d(predicted_y_test, y_test).all(), \"please predict class indexes\"\n", + "\n", + "accuracy = np.mean(predicted_y_test == y_test)\n", + "\n", + "print(\"Test accuracy: %.5f\" % accuracy)\n", + "assert accuracy > 0.95, \"try training longer\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## More about pytorch:\n", + "* Using torch on GPU and multi-GPU - [link](http://pytorch.org/docs/master/notes/cuda.html)\n", + "* More tutorials on pytorch - [link](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)\n", + "* Pytorch examples - a repo that implements many cool DL models in pytorch - [link](https://github.com/pytorch/examples)\n", + "* Practical pytorch - a repo that implements some... other cool DL models... yes, in pytorch - [link](https://github.com/spro/practical-pytorch)\n", + "* And some more - [link](https://www.reddit.com/r/pytorch/comments/6z0yeo/pytorch_and_pytorch_tricks_for_kaggle/)\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/week02_autodiff/tensorflow.ipynb b/week02_autodiff/tensorflow.ipynb new file mode 100644 index 00000000..f815ce2c --- /dev/null +++ b/week02_autodiff/tensorflow.ipynb @@ -0,0 +1,825 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Down the rabbit hole with Tensorflow\n", + "\n", + "![img](https://images.exxactcorp.com/CMS/landing-page/resource-center/supported-software/deep-learning/tensorflow/TensorFlow.png)\n", + "\n", + "In this seminar, we're going to play with [Tensorflow](https://www.tensorflow.org/) and see how it helps you build deep learning models.\n", + "\n", + "If you're running this notebook outside the course environment, you'll need to install tensorflow:\n", + "* `pip install tensorflow` should install cpu-only TF on Linux & Mac OS\n", + "* If you want GPU support from offset, see [TF install page](https://www.tensorflow.org/install/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "\n", + "# session is main tensorflow object. You ask session to compute stuff for you.\n", + "sess = tf.InteractiveSession()\n", + "\n", + "# # print current version of tf.\n", + "print(tf.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Warming up\n", + "For starters, let's implement a python function that computes the sum of squares of numbers from 0 to N-1.\n", + "* Use numpy or python\n", + "* An array of numbers 0 to N - numpy.arange(N)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "-" + } + }, + "outputs": [], + "source": [ + "def sum_squares(N):\n", + " return " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "sum_squares(10**8)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Same with tensorflow__" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# \"i will insert N here later\"\n", + "N = tf.placeholder('int64', name=\"input_to_your_function\")\n", + "\n", + "# a recipe on how to produce {sum of squares of arange of N} given N\n", + "result = tf.reduce_sum((tf.range(N)**2))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "\n", + "# dear session, compute the result please. Here's your N.\n", + "print(sess.run(result, {N: 10**8}))\n", + "\n", + "# hint: run it several times to let tensorflow \"warm up\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# How it works: computation graphs\n", + "\n", + "\n", + "1. create placeholders for future inputs;\n", + "2. define symbolic graph: a recipe for mathematical transformation of those placeholders;\n", + "3. compute outputs of your graph with particular values for each placeholder\n", + " * ```sess.run(outputs, {placeholder1:value1, placeholder2:value2})```\n", + " * OR output.eval({placeholder:value}) \n", + "\n", + "Still confused? We gonna fix that." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Placeholders and constants__" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# placeholder that can be arbitrary float32 scalar, vertor, matrix, etc.\n", + "arbitrary_input = tf.placeholder('float32')\n", + "\n", + "# input vector of arbitrary length\n", + "input_vector = tf.placeholder('float32',shape=(None,))\n", + "\n", + "# input vector that _must_ have 10 elements and integer type\n", + "fixed_vector = tf.placeholder('int32',shape=(10,))\n", + "\n", + "# you can generally use None whenever you don't need a specific shape\n", + "input1 = tf.placeholder('float64',shape=(None, 100, None))\n", + "input2 = tf.placeholder('int32',shape=(None, None, 3, 224, 224))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can create new __tensors__ with arbitrary operations on placeholders, constants and other tensors.\n", + "\n", + "* tf.reduce_sum(tf.arange(N)\\**2) are 3 sequential transformations of placeholder N\n", + "* there's a tensorflow symbolic version for every numpy function\n", + " * `a + b, a / b, a ** b, ...` behave just like in numpy\n", + " * np.zeros -> tf.zeros\n", + " * np.sin -> tf.sin\n", + " * np.mean -> tf.reduce_mean\n", + " * np.arange -> tf.range\n", + " \n", + "There are tons of other stuff in tensorflow, see the [docs](https://www.tensorflow.org/api_docs/python) or learn as you go with __shift+tab__." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# elementwise multiplication\n", + "double_the_vector = input_vector * 2\n", + "\n", + "# elementwise cosine\n", + "elementwise_cosine = tf.cos(input_vector)\n", + "\n", + "# elementwise difference between squared vector and it's means - with some random salt\n", + "vector_squares = input_vector ** 2 - tf.reduce_mean(input_vector) + tf.random_normal(tf.shape(input_vector))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Practice 1: polar pretzels\n", + "_inspired by [this post](https://www.quora.com/What-are-the-most-interesting-equation-plots)_\n", + "\n", + "There are some simple mathematical functions with cool plots. For one, consider this:\n", + "\n", + "$$ x(t) = t - 1.5 * cos( 15 t) $$\n", + "$$ y(t) = t - 1.5 * sin( 16 t) $$\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "t = tf.placeholder('float32')\n", + "\n", + "\n", + "# compute x(t) and y(t) as defined above.\n", + "x = ###YOUR CODE\n", + "y = ###YOUR CODE\n", + "\n", + "\n", + "x_points, y_points = sess.run([x, y], {t: np.linspace(-10, 10, num=10000)})\n", + "plt.plot(x_points, y_points);" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Visualizing graphs with Tensorboard\n", + "\n", + "It's often useful to visualize the computation graph when debugging or optimizing. \n", + "Interactive visualization is where tensorflow really shines as compared to other frameworks. \n", + "\n", + "There's a special instrument for that, called Tensorboard. You can launch it from console:\n", + "\n", + "__```tensorboard --logdir=/tmp/tboard --port=7007```__\n", + "\n", + "If you're pathologically afraid of consoles, try this:\n", + "\n", + "__```import os; os.system(\"tensorboard --logdir=/tmp/tboard --port=7007 &\")```__\n", + "\n", + "_(but don't tell anyone we taught you that)_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "One basic functionality of tensorboard is drawing graphs. One you've run the cell above, go to `localhost:7007` in your browser and switch to _graphs_ tab in the topbar. \n", + "\n", + "Here's what you should see:\n", + "\n", + "\n", + "\n", + "Tensorboard also allows you to draw graphs (e.g. learning curves), record images & audio ~~and play flash games~~. This is useful when monitoring learning progress and catching some training issues.\n", + "\n", + "One researcher said:\n", + "```\n", + "If you spent last four hours of your worktime watching as your algorithm prints numbers and draws figures, you're probably doing deep learning wrong.\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can read more on tensorboard usage [here](https://www.tensorflow.org/get_started/graph_viz)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Practice 2: mean squared error\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Quest #1 - implement a function that computes a mean squared error of two input vectors\n", + "# Your function has to take 2 vectors and return a single number\n", + "\n", + "\n", + "\n", + "mse =\n", + "\n", + "compute_mse = lambda vector1, vector2: sess.run(, {})" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Tests\n", + "from sklearn.metrics import mean_squared_error\n", + "\n", + "for n in [1, 5, 10, 10 ** 3]:\n", + " \n", + " elems = [np.arange(n),np.arange(n,0,-1), np.zeros(n),\n", + " np.ones(n),np.random.random(n),np.random.randint(100,size=n)]\n", + " \n", + " for el in elems:\n", + " for el_2 in elems:\n", + " true_mse = np.array(mean_squared_error(el,el_2))\n", + " my_mse = compute_mse(el,el_2)\n", + " if not np.allclose(true_mse,my_mse):\n", + " print('Wrong result:')\n", + " print('mse(%s,%s)' % (el,el_2))\n", + " print(\"should be: %f, but your function returned %f\" % (true_mse,my_mse))\n", + " raise ValueError,\"Что-то не так\"\n", + "\n", + "print(\"All tests passed\") " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Tensorflow variables\n", + "\n", + "The inputs and transformations have no value outside function call. That's a bit unnatural if you want your model to have parameters (e.g. network weights) that are always present, but can change their value over time.\n", + "\n", + "Tensorflow solves this with `tf.Variable` objects.\n", + "* You can assign variable a value at any time in your graph\n", + "* Unlike placeholders, there's no need to explicitly pass values to variables when `s.run(...)`-ing\n", + "* You can use variables the same way you use transformations \n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# creating shared variable\n", + "shared_vector_1 = tf.Variable(initial_value=np.ones(5))\n", + "\n", + "# initialize all variables with initial values\n", + "sess.run(tf.global_variables_initializer())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# evaluating shared variable (outside symbolicd graph)\n", + "print(\"initial value\", sess.run(shared_vector_1))\n", + "\n", + "# within symbolic graph you use them just as any other inout or transformation, not \"get value\" needed" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# setting new value manually\n", + "sess.run(shared_vector_1.assign(np.arange(5)))\n", + "\n", + "#getting that new value\n", + "print(\"new value\", sess.run(shared_vector_1))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# tf.gradients - why graphs matter\n", + "* Tensorflow can compute derivatives and gradients automatically using the computation graph\n", + "* Gradients are computed as a product of elementary derivatives via chain rule:\n", + "\n", + "$$ {\\partial f(g(x)) \\over \\partial x} = {\\partial f(g(x)) \\over \\partial g(x)}\\cdot {\\partial g(x) \\over \\partial x} $$\n", + "\n", + "It can get you the derivative of any graph as long as it knows how to differentiate elementary operations" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "my_scalar = tf.placeholder('float32')\n", + "\n", + "scalar_squared = my_scalar ** 2\n", + "\n", + "#a derivative of scalar_squared by my_scalar\n", + "derivative = tf.gradients(scalar_squared, [my_scalar])[0]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "x = np.linspace(-3,3)\n", + "x_squared, x_squared_der = sess.run([scalar_squared, derivative], {my_scalar:x})\n", + "\n", + "plt.plot(x, x_squared,label=\"x^2\")\n", + "plt.plot(x, x_squared_der, label=\"derivative\")\n", + "plt.legend();" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Why autograd is cool" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "my_vector = tf.placeholder('float32',[None])\n", + "\n", + "#Compute the gradient of the next weird function over my_scalar and my_vector\n", + "#warning! Trying to understand the meaning of that function may result in permanent brain damage\n", + "\n", + "weird_psychotic_function = tf.reduce_mean((my_vector+my_scalar)**(1+tf.nn.moments(my_vector,[0])[1]) + 1./ tf.atan(my_scalar))/(my_scalar**2 + 1) + 0.01*tf.sin(2*my_scalar**1.5)*(tf.reduce_sum(my_vector)* my_scalar**2)*tf.exp((my_scalar-4)**2)/(1+tf.exp((my_scalar-4)**2))*(1.-(tf.exp(-(my_scalar-4)**2))/(1+tf.exp(-(my_scalar-4)**2)))**2\n", + "\n", + "der_by_scalar = \n", + "der_by_vector = " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "#Plotting your derivative\n", + "scalar_space = np.linspace(1, 7, 100)\n", + "\n", + "y = [sess.run(weird_psychotic_function, {my_scalar:x, my_vector:[1, 2, 3]})\n", + " for x in scalar_space]\n", + "\n", + "plt.plot(scalar_space, y, label='function')\n", + "\n", + "y_der_by_scalar = [sess.run(der_by_scalar, {my_scalar:x, my_vector:[1, 2, 3]})\n", + " for x in scalar_space]\n", + "\n", + "plt.plot(scalar_space, y_der_by_scalar, label='derivative')\n", + "plt.grid()\n", + "plt.legend();" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Almost done - optimizers\n", + "\n", + "While you can perform gradient descent by hand with automatic grads from above, tensorflow also has some optimization methods implemented for you. Recall momentum & rmsprop?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "y_guess = tf.Variable(np.zeros(2,dtype='float32'))\n", + "y_true = tf.range(1,3,dtype='float32')\n", + "\n", + "loss = tf.reduce_mean((y_guess - y_true + tf.random_normal([2]))**2) \n", + "\n", + "optimizer = tf.train.MomentumOptimizer(0.01,0.9).minimize(loss,var_list=y_guess)\n", + "\n", + "# same, but more detailed:\n", + "# updates = [[tf.gradients(loss,y_guess)[0], y_guess]]\n", + "# optimizer = tf.train.MomentumOptimizer(0.01,0.9).apply_gradients(updates)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "from IPython.display import clear_output\n", + "\n", + "sess.run(tf.global_variables_initializer())\n", + "\n", + "guesses = [sess.run(y_guess)]\n", + "\n", + "for _ in range(100):\n", + " sess.run(optimizer)\n", + " guesses.append(sess.run(y_guess))\n", + " \n", + " clear_output(True)\n", + " plt.plot(*zip(*guesses), marker='.')\n", + " plt.scatter(*sess.run(y_true), c='red')\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Logistic regression example\n", + "Implement the regular logistic regression training algorithm\n", + " \n", + "We shall train on a two-class MNIST dataset. \n", + "\n", + "This is a binary classification problem, so we'll train a __Logistic Regression with sigmoid__.\n", + "$$P(y_i | X_i) = \\sigma(W \\cdot X_i + b) ={ 1 \\over {1+e^{- [W \\cdot X_i + b]}} }$$\n", + "\n", + "\n", + "The natural choice of loss function is to use binary crossentropy (aka logloss, negative llh):\n", + "$$ L = {1 \\over N} \\underset{X_i,y_i} \\sum - [ y_i \\cdot log P(y_i | X_i) + (1-y_i) \\cdot log (1-P(y_i | X_i)) ]$$\n", + "\n", + "Mind the minus :)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "X, y = load_digits(2, return_X_y=True)\n", + "\n", + "print(\"y [shape - %s]:\" % (str(y.shape)), y[:10])\n", + "print(\"X [shape - %s]:\" % (str(X.shape)))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print('X:\\n', X[:3,:10])\n", + "print('y:\\n', y[:10])\n", + "plt.imshow(X[0].reshape([8,8]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# inputs and shareds\n", + "weights = \n", + "input_X = \n", + "input_y = " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "predicted_y_proba = \n", + "\n", + "loss = \n", + "\n", + "train_step = " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from sklearn.metrics import roc_auc_score\n", + "\n", + "for i in range(5):\n", + " \n", + " loss_i, _ = sess.run([loss, train_step], ###)\n", + " \n", + " print(\"loss at iter %i: %.4f\" % (i, loss_i))\n", + " \n", + " print(\"train auc:\", roc_auc_score(y_train, sess.run(predicted_y_proba, {input_X: X_train})))\n", + " print(\"test auc:\", roc_auc_score(y_test, sess.run(predicted_y_proba, {input_X: X_test})))\n", + "\n", + " \n", + "print (\"resulting weights:\")\n", + "plt.imshow(sess.run(weights).reshape(8, -1))\n", + "plt.colorbar();" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Practice 3: my first tensorflow network\n", + "Your ultimate task for this week is to build your first neural network [almost] from scratch and pure tensorflow.\n", + "\n", + "This time you will same digit recognition problem, but at a larger scale\n", + "* images are now 28x28\n", + "* 10 different digits\n", + "* 50k samples\n", + "\n", + "Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) NN should already have ive you an edge over logistic regression.\n", + "\n", + "__[bonus score]__\n", + "If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! The milestones would be 95%/97.5%/98.5% accuraсy on test set.\n", + "\n", + "__SPOILER!__\n", + "At the end of the notebook you will find a few tips and frequently made mistakes. If you feel enough might to shoot yourself in the foot without external assistance, we encourage you to do so, but if you encounter any unsurpassable issues, please do look there before mailing us." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from mnist import load_dataset\n", + "\n", + "# [down]loading the original MNIST dataset.\n", + "# Please note that you should only train your NN on _train sample,\n", + "# _val can be used to evaluate out-of-sample error, compare models or perform early-stopping\n", + "# _test should be hidden under a rock untill final evaluation... But we both know it is near impossible to catch you evaluating on it.\n", + "X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()\n", + "\n", + "print (X_train.shape,y_train.shape)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "plt.imshow(X_train[0,0])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "```\n", + "\n", + "\n", + "# SPOILERS!\n", + "\n", + "Recommended pipeline\n", + "\n", + "* Adapt logistic regression from previous assignment to classify some number against others (e.g. zero vs nonzero)\n", + "* Generalize it to multiclass logistic regression.\n", + " - Either try to remember lecture 0 or google it.\n", + " - Instead of weight vector you'll have to use matrix (feature_id x class_id)\n", + " - softmax (exp over sum of exps) can implemented manually or as T.nnet.softmax (stable)\n", + " - probably better to use STOCHASTIC gradient descent (minibatch)\n", + " - in which case sample should probably be shuffled (or use random subsamples on each iteration)\n", + "* Add a hidden layer. Now your logistic regression uses hidden neurons instead of inputs.\n", + " - Hidden layer uses the same math as output layer (ex-logistic regression), but uses some nonlinearity (sigmoid) instead of softmax\n", + " - You need to train both layers, not just output layer :)\n", + " - Do not initialize layers with zeros (due to symmetry effects). A gaussian noize with small sigma will do.\n", + " - 50 hidden neurons and a sigmoid nonlinearity will do for a start. Many ways to improve. \n", + " - In ideal casae this totals to 2 .dot's, 1 softmax and 1 sigmoid\n", + " - __make sure this neural network works better than logistic regression__\n", + " \n", + "* Now's the time to try improving the network. Consider layers (size, neuron count), nonlinearities, optimization methods, initialization - whatever you want, but please avoid convolutions for now." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.2" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}