drafts/Measures of Dispersion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dispersion\n",
    "By Evgenia \"Jenny\" Nitishinskaya and Delaney Granizo-Mackenzie\n",
    "\n",
    "Notebook released under the Creative Commons Attribution 4.0 License.\n",
    "\n",
    "---\n",
    "Dispersion measures how spread out a set of data is. This corresponds to risk when our data set is returns over time. Data with low dispersion is heavily clustered around the mean, while high dispersion a indicates many very large and very small values.\n",
    "\n",
    "Let's generate an array of random integers to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X: [ 3  8 34 39 46 52 52 52 54 57 60 65 66 75 83 85 88 94 95 96]\n",
      "Mean of X: 60.2\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "\n",
    "np.random.seed(121)\n",
    "X = np.sort(np.random.randint(100, size=20))\n",
    "print 'X:', X\n",
    "mu = np.mean(X)\n",
    "print 'Mean of X:', mu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Range\n",
    "\n",
    "Range is simply the difference between the maximum and minimum values in a dataset. Not surprisingly, it is very sensitive to outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Range of X: 93\n"
     ]
    }
   ],
   "source": [
    "print 'Range of X:', np.ptp(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mean absolute deviation\n",
    "\n",
    "The mean absolute deviation is the average of the distances of observations from the arithmetic mean. We use the absolute value of the deviation, so that 5 above the mean and 5 below the mean both contribute 5, because otherwise the deviations always sum to 0.\n",
    "\n",
    "$$ MAD = \\frac{\\sum_{i=1}^n |X_i - \\mu|}{n} $$\n",
    "\n",
    "where $n$ is the number of observations and $\\mu$ is their mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean absolute deviation of X: 20.52\n"
     ]
    }
   ],
   "source": [
    "abs_dispersion = [abs(mu - x) for x in X]\n",
    "MAD = sum(abs_dispersion)/len(abs_dispersion)\n",
    "print 'Mean absolute deviation of X:', MAD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variance and standard deviation\n",
    "\n",
    "The variance $\\sigma^2$ is defined as the average of the squared deviations around the mean:\n",
    "$$ \\sigma^2 = \\frac{\\sum_{i=1}^n (X_i - \\mu)^2}{n} $$\n",
    "\n",
    "This is sometimes more convenient than the mean absolute deviation because absolute value is not differentiable, while squaring is smooth, and some optimization algorithms rely on differentiability.\n",
    "\n",
    "Standard deviation is defined as the square root of the variance, $\\sigma$, and it is the easier of the two to interpret because it is in the same units as the observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance of X: 670.16\n",
      "Standard deviation of X: 25.8874486962\n"
     ]
    }
   ],
   "source": [
    "print 'Variance of X:', np.var(X)\n",
    "print 'Standard deviation of X:', np.std(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to interpret standard deviation is by referring to Chebyshev's inequality. This tells us that the proportion of samples within $k$ standard deviations (that is, within a distance of $k \\cdot$ standard deviation) of the mean is at least $1 - 1/k^2$ for all $k>1$.\n",
    "\n",
    "Let's check that this is true for our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observations within 1.25 stds of mean: [34, 39, 46, 52, 52, 52, 54, 57, 60, 65, 66, 75, 83, 85, 88]\n",
      "Confirming that 0.75 > 0.36\n"
     ]
    }
   ],
   "source": [
    "k = 1.25\n",
    "dist = k*np.std(X)\n",
    "l = [x for x in X if abs(x - mu) <= dist]\n",
    "print 'Observations within', k, 'stds of mean:', l\n",
    "print 'Confirming that', float(len(l))/len(X), '>', 1 - 1/k**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bound given by Chebyshev's inequality seems fairly loose in this case. This bound is rarely strict, but it is useful because it holds for all data sets and distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semivariance and semideviation\n",
    "\n",
    "Although variance and standard deviation tell us how volatile a quantity is, they do not differentiate between deviations upward and deviations downward. Often, such as in the case of returns on an asset, we are more worried about deviations downward. This is addressed by semivariance and semideviation, which only count the observations that fall below the mean. Semivariance is defined as\n",
    "$$ \\frac{\\sum_{X_i < \\mu} (X_i - \\mu)^2}{n_<} $$\n",
    "where $n_<$ is the number of observations which are smaller than the mean. Semideviation is the square root of the semivariance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Semivariance of X: 689.512727273\n",
      "Semideviation of X: 26.2585743572\n"
     ]
    }
   ],
   "source": [
    "# Because there is no built-in semideviation, we'll compute it ourselves\n",
    "lows = [e for e in X if e <= mu]\n",
    "semivar = sum(map(lambda x: (x - mu)**2,lows))/len(lows)\n",
    "\n",
    "print 'Semivariance of X:', semivar\n",
    "print 'Semideviation of X:', math.sqrt(semivar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A related notion is target semivariance (and target semideviation), where we average the distance from a target of values which fall below that target:\n",
    "$$ \\frac{\\sum_{X_i < B} (X_i - B)^2}{n_{<B}} $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target semivariance of X: 188\n",
      "Target semideviation of X: 13.7113092008\n"
     ]
    }
   ],
   "source": [
    "B = 19\n",
    "lows_B = [e for e in X if e <= B]\n",
    "semivar_B = sum(map(lambda x: (x - B)**2,lows_B))/len(lows_B)\n",
    "\n",
    "print 'Target semivariance of X:', semivar_B\n",
    "print 'Target semideviation of X:', math.sqrt(semivar_B)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}