1/Applying Statistical Tests - 02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Applied Statistical Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "In the previous chapter we introduced descriptive statistics and hypothesis testing.  In this chapter we will make extensive use of the concepts covered to introduce you to:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A/B Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Inquiry\n",
    "\n",
    "Suppose you work for a company that makes towels, and they want to send an email, like they've done many times in the past, to see how much this email affects sale of their towels.  Now let's say that, in the past 10 percent of the people sent the email bought a towel.  But, we aren't sure if this email maximizes profits.  In other words, would a different email mean we make more profit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Set Up\n",
    "\n",
    "In order to test this question, we can set up an experiment.  Here we will set up a randomized test group and a randomized control group.  \n",
    "\n",
    "The test group will be sent an email, with slightly different copy, or possibly with a picture.  Some specific change will be made, in any event.\n",
    "\n",
    "The control group will get the same email as last time.  This way, we can directly compare, as much as possible between the old email and the new one.  There are many things you typically need to control for, or account for in experimental design.  Some things to account for in this scenario are:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "1) Age\n",
    "\n",
    "2) Gender\n",
    "\n",
    "3) Race\n",
    "\n",
    "4) Location\n",
    "\n",
    "5) Time of Day\n",
    "\n",
    "6) Time of Year\n",
    "\n",
    "7) Approximate Disposable Income\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "By setting up the experiment with a test and control group, we account for time of day and time of year directly.  Some of the other information like location, age, gender, and approximate disposable income; we may not be able to account for.  Some of this may be accounted by selecting randomly from our population.  But there is no guarantee of that.  It is best to account for as many variables of interest as you can, while remaining ethical.  We will cover the ethics of variable accounting in the next section in detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Running Our Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "In order to evaluate our set up, we will assume that our conversion rate follows a Bernoulli random variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bernoulli Random Variables\n",
    "\n",
    "a Bernoulli random variable takes on value '1' with probably `p`, and value '0' with probability `1-p`.  \n",
    "\n",
    "For our case, a value of '1' represents a person who chose to buy a towel by going to the website after reading our email.  The value of '0' represents no towel being bought after reading our email."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Informally, we can think of a Bernoulli random variable as a model for the set of possible outcomes of any single experiment.\n",
    "\n",
    "Specifically, a Bernoulli random variable takes on value '1' with probability `p`, and value '0' with probability `1-p`.  \n",
    "\n",
    "For our case, a value of '1' represents a person who chose to buy a towel by going to the website after reading our email.  The value of '0' represents no towel being bought after reading our email."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### An Aside Regarding Assumptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "By setting up our experiment in this way, we have actually assumed quiet a bit.  Let's formally state our assumptions here.  It is good practice, generally speaking to always list out your assumptions and confirm with a domain experiment, someone who knows the problem, but maybe not the statistics, before caring out any experiments.  If you work in a place where tasks are created for you, then before you get started, it's a good idea to confirm with the person who created the task, to make sure you are both aligned on assumptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Making Our Assumptions Explicit\n",
    "\n",
    "1) We assume here that the only reason a person would buy a towel is because of this email.\n",
    "\n",
    "This is a pretty strong assumption.  Specifically we are only looking at the marginal effect of changing the language in an email.  It could be the case that our entire control group or our entire test group happened to need a new towel, and this email reminded them.  It could be all of the people in group one or group two just moved, and didn't have _any_ towels.  Or it could be that everyone in a group always buys things due to a sale.  \n",
    "\n",
    "We could control for some of this stuff, which will be touched upon in a later section.  But for now, we aren't controlling for any of it.  \n",
    "\n",
    "2) We assume non-uniformity of preferences.\n",
    "\n",
    "A preference is a propensity or likelihood to consume a product, given the choice to consume it and the ability.  So if you are more likely to consume chocolate ice cream over vanilla ice cream given the choice between the two, and the money to buy either, then we say you have a preference for chocolate ice cream.\n",
    "\n",
    "By assuming non-uniformity of preferences, we are saying, implicitly that there is some utility function for towels for each individual.  And that at price `towel_price`, you will not buy, because it would cost more than the utility you would derive.  But at price `towel_sale_price`, you will buy, because it would cost less than the utility you would derive.\n",
    "\n",
    "3) We assume evenly distributed life circumstances.\n",
    "\n",
    "We assume everyone on this email is capable of buying a towel from us.  This may seem like a riddiculous assumption, but it's really not.  You have no idea what circumstance someone is in.\n",
    "\n",
    "4) We assume evenly distributed access to the email\n",
    "\n",
    "What if everyone who didn't buy a towel deactivated their email, or died, or doesn't speak english, or possibly, that their email client filtered out the email.  If we don't know for sure, we don't know the effectiveness of the email campaign.\n",
    "\n",
    "The reason it's important to explicitly state your assumptions, is not necessarily for the code itself.  Although it can inform the tests you run or the models you create.  The most important reason to write down your assumptions is for future people working with your code, which is more often than not yourself, as well as for your managers.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simulating Some Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "\n",
    "test_size = 1000\n",
    "control_size = 1000\n",
    "test_probability = 0.15\n",
    "control_probability = 0.1\n",
    "test_dist = stats.bernoulli(test_probability)\n",
    "control_dist = stats.bernoulli(control_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Note: we don't actually know the test probability, so \"pretend\" you can't see it.  We wouldn't know this for the population in the real world.  Also, we probably can't be \"sure\" our process follows a Bernoulli random variable.  It could be some other generative process, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 1, 1, 1, 1, 0, 0, 1, 0]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def generative_process():\n",
    "    random_number = random.randint(0, 100)\n",
    "    if random_number < 15:\n",
    "        return 1\n",
    "    elif random_number % 2 == 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1\n",
    "    \n",
    "[generative_process() for _ in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The important difference here is there are multiple cases by which our generative process outputs a '1', which we may be important!  The more you know about your underlying data generation process, the better.  But usually it is impossible to discern this.  The important thing to note is, this is an assumption."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "test_df = pd.DataFrame()\n",
    "test_df[\"converted\"] = test_dist.rvs(size=test_size)\n",
    "test_df[\"group\"] = \"test\"\n",
    "control_df = pd.DataFrame()\n",
    "control_df[\"converted\"] = control_dist.rvs(size=control_size)\n",
    "control_df[\"group\"] = \"control\"\n",
    "df = test_df.append(control_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>total</th>\n",
       "      <th>rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>92</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>153</td>\n",
       "      <td>1000</td>\n",
       "      <td>0.153</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         converted  total   rate\n",
       "group                           \n",
       "control         92   1000  0.092\n",
       "test           153   1000  0.153"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = df.pivot_table(values='converted', index='group', aggfunc=np.sum)\n",
    "# add additional columns to the pivot table\n",
    "summary['total'] = df.pivot_table(values='converted', index='group', aggfunc=lambda x: len(x))\n",
    "summary['rate'] = df.pivot_table(values='converted', index='group')\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "We may believe our test was successful at this point, and in fact, we are done.  However, how do we know that our test was successful?  How do we know that the two samples differences weren't just because of demographic differences?  Time to verify against all of the assumptions we made in the prior section!\n",
    "\n",
    "In order to do this we will look at the chi-squared test.  Recall, the chi-squared test's null hypothesis states the two variables are independent.  One could also think of the null hypothesis as, \"is there a statistically significant difference between the expected frequencies and the observed frequencies in one or more categories of a contingency table\".  \n",
    "\n",
    "This sounds perfect!  So let's bring in our other variables:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Age\n",
    "* Gender\n",
    "* Location\n",
    "* Race\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Note: Here we assume that location is a proxy for income.  So we'll specify location as high, medium and low; where high means an expensive area to live, medium means a middle class area to live, and low means a poor area to live.\n",
    "\n",
    "Note: We will specify Gender as a binary variable.  While in the real world this assumption is not valid, we will make this simplifying assumption here.\n",
    "\n",
    "Note: For Race we will specify one of four categories:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* white\n",
    "* black\n",
    "* asian\n",
    "* hispanic\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Note: For Race we will specify one of four categories:\n",
    "\n",
    "While this does not encompass all of the races, it works as a simplifying set of assumptions, for this example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* 18 - 24\n",
    "* 24 - 35\n",
    "* 35 - 50\n",
    "* 50 - 70\n",
    "* 70+\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Note: For Age we will specify one of five categories:\n",
    "Now let's generate some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def generate_data(df, column, choices):\n",
    "    \"\"\"\n",
    "    Generates categorical data given choices.\n",
    "    \n",
    "    Parameters:\n",
    "    * df - pd.DataFrame: the data to add a column to\n",
    "    * column - str: the column to generate\n",
    "    * choices - list: the list of possible choices\n",
    "    \n",
    "    Returns:\n",
    "    A dataframe with the newly generated column.\n",
    "    \"\"\"\n",
    "    df[column] = [random.choice(choices)\n",
    "                  for _ in range(df.shape[0])]\n",
    "    df = pd.concat([df, pd.get_dummies(df[column])], axis=1)\n",
    "    df = df.drop(column, axis=1)\n",
    "    return df\n",
    "\n",
    "def generate_contingency_table(df, demographics):\n",
    "    \"\"\"\n",
    "    Generates a contingency table from a list of choices.\n",
    "    We hard code converted, because that is 'always' one of the variables of interest.\n",
    "    \n",
    "    Parameters:\n",
    "    * df - pd.DataFrame: the dataframe to convert to a contingency table.\n",
    "    * demographics - list: a list of demographics, aka the classes from\n",
    "    all categorical classes of interest.\n",
    "    \n",
    "    Note: one way to get the demographics is:\n",
    "    categories = []\n",
    "    for column in categorical_columns:\n",
    "        categories += list(df[column].unique())\n",
    "    \"\"\"\n",
    "    summary = df.pivot_table(values='converted', index='group', aggfunc=np.sum)\n",
    "    for demographic in demographics:\n",
    "        summary[demographic] = df.pivot_table(\n",
    "            values=demographic,\n",
    "            index=\"group\",\n",
    "            aggfunc=np.sum\n",
    "        )\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "gender_choices = [\"male\", \"female\"]\n",
    "age_choices = [\"18-24\", \"24-35\", \"35-50\", \"50-70\", \"70+\"]\n",
    "race_choices = [\"white\", \"black\", \"asian\", \"hispanic\"]\n",
    "location_choices = [\"high\", \"medium\", \"low\"]\n",
    "\n",
    "test_df = generate_data(test_df, \"gender\", gender_choices)\n",
    "test_df = generate_data(test_df, \"age\", age_choices)\n",
    "test_df = generate_data(test_df, \"race\", race_choices)\n",
    "test_df = generate_data(test_df, \"location\", location_choices)\n",
    "control_df = generate_data(control_df, \"gender\", gender_choices)\n",
    "control_df = generate_data(control_df, \"age\", age_choices)\n",
    "control_df = generate_data(control_df, \"race\", race_choices)\n",
    "control_df = generate_data(control_df, \"location\", location_choices)\n",
    "\n",
    "df = pd.DataFrame()\n",
    "df = df.append(test_df)\n",
    "df = df.append(control_df)\n",
    "\n",
    "# does list concatenation\n",
    "demographics = gender_choices + age_choices + race_choices + location_choices\n",
    "table = generate_contingency_table(df, demographics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Before we run our hypothesis test, let's look at our contingency table!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>male</th>\n",
       "      <th>female</th>\n",
       "      <th>18-24</th>\n",
       "      <th>24-35</th>\n",
       "      <th>35-50</th>\n",
       "      <th>50-70</th>\n",
       "      <th>70+</th>\n",
       "      <th>white</th>\n",
       "      <th>black</th>\n",
       "      <th>asian</th>\n",
       "      <th>hispanic</th>\n",
       "      <th>high</th>\n",
       "      <th>medium</th>\n",
       "      <th>low</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>106</td>\n",
       "      <td>500.0</td>\n",
       "      <td>500.0</td>\n",
       "      <td>199</td>\n",
       "      <td>223</td>\n",
       "      <td>194</td>\n",
       "      <td>205</td>\n",
       "      <td>179</td>\n",
       "      <td>249</td>\n",
       "      <td>234.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>262.0</td>\n",
       "      <td>301.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>349.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>137</td>\n",
       "      <td>531.0</td>\n",
       "      <td>469.0</td>\n",
       "      <td>205</td>\n",
       "      <td>207</td>\n",
       "      <td>204</td>\n",
       "      <td>187</td>\n",
       "      <td>197</td>\n",
       "      <td>231</td>\n",
       "      <td>258.0</td>\n",
       "      <td>256.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>332.0</td>\n",
       "      <td>329.0</td>\n",
       "      <td>339.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         converted   male  female  18-24  24-35  35-50  50-70  70+  white  \\\n",
       "group                                                                       \n",
       "control        106  500.0   500.0    199    223    194    205  179    249   \n",
       "test           137  531.0   469.0    205    207    204    187  197    231   \n",
       "\n",
       "         black  asian  hispanic   high  medium    low  \n",
       "group                                                  \n",
       "control  234.0  255.0     262.0  301.0   350.0  349.0  \n",
       "test     258.0  256.0     255.0  332.0   329.0  339.0  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19.04123960342558, 0.16337944030998258)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "from scipy.stats import chi2_contingency\n",
    "\n",
    "table_transpose = table.T\n",
    "chi2, pvalue, degrees_of_freedom, expected = chi2_contingency(table_transpose)\n",
    "\n",
    "chi2, pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's go over the definitions of each of these:\n",
    "\n",
    "* chi2 - this is the test statistic.  It is calculated as a normalized sum of squared deviations between observed and theoretical frequencies.  In the two group setting we can take one as the observation and the other as the theoretical.\n",
    "\n",
    "* p value - this is the probability value of obtaining test results at least as extreme as the results actually observed during the test, assuming the null hypothesis is correct.  So the lower the p value, the more likely we are to reject the null hypothesis.\n",
    "\n",
    "* degress of freedom - since we are doing the test for homogenity, here the degrees of freedom are:\n",
    "\n",
    "$$ df = (Rows - 1) * (Columns -1)  $$ \n",
    "\n",
    "Where Rows := number of categories, aka the rows of the contingency table.\n",
    "\n",
    "Where Columns := number of independent groups\n",
    "\n",
    "Notice, we satisfy this requirement by taking the transpose of the contingency table\n",
    "\n",
    "* expected value - this is the theoretical values for the contingency table, given the null hypothesis is true.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Analyzing Our Test\n",
    "\n",
    "Now let's see what our test tells us about our data.  First let's explicitly state the null hypothesis and alternative hypothesis:\n",
    "\n",
    "* null hypothesis: there exists no relationship between the two groups.  If the null hypothesis is true, then it might be the case that the difference in our conversion rates is significant.\n",
    "\n",
    "* alternative hypothesis: there exists a relationship between the two groups.  In other words, the two groups are statistically similar.  If this is the case, then it might be that the difference in our conversion rates is due to chance.\n",
    "\n",
    "Based on the p-value, it looks like we fail to reject our null hypothesis.  With a test statistic of approximately 12.64, the normalized sum of squared deviations appears large enough that there is no relationship.  This appears to confirm what our contingency table seemed to be telling us.  Let's go a little deeper.\n",
    "\n",
    "Let's next verify, for academic purposes that the degrees of freedom match up with what we expect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "degrees_of_freedom == (table.shape[0]-1) * (table.shape[1]-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now, just for comparison, let's look at the expected values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>male</th>\n",
       "      <th>female</th>\n",
       "      <th>18-24</th>\n",
       "      <th>24-35</th>\n",
       "      <th>35-50</th>\n",
       "      <th>50-70</th>\n",
       "      <th>70+</th>\n",
       "      <th>white</th>\n",
       "      <th>black</th>\n",
       "      <th>asian</th>\n",
       "      <th>hispanic</th>\n",
       "      <th>high</th>\n",
       "      <th>medium</th>\n",
       "      <th>low</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>121.593693</td>\n",
       "      <td>489.848878</td>\n",
       "      <td>502.752699</td>\n",
       "      <td>202.490722</td>\n",
       "      <td>201.49812</td>\n",
       "      <td>202.987022</td>\n",
       "      <td>195.542511</td>\n",
       "      <td>190.083202</td>\n",
       "      <td>244.676289</td>\n",
       "      <td>244.179988</td>\n",
       "      <td>272.469133</td>\n",
       "      <td>231.276167</td>\n",
       "      <td>316.143602</td>\n",
       "      <td>315.151001</td>\n",
       "      <td>361.306974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>123.406307</td>\n",
       "      <td>497.151122</td>\n",
       "      <td>510.247301</td>\n",
       "      <td>205.509278</td>\n",
       "      <td>204.50188</td>\n",
       "      <td>206.012978</td>\n",
       "      <td>198.457489</td>\n",
       "      <td>192.916798</td>\n",
       "      <td>248.323711</td>\n",
       "      <td>247.820012</td>\n",
       "      <td>276.530867</td>\n",
       "      <td>234.723833</td>\n",
       "      <td>320.856398</td>\n",
       "      <td>319.848999</td>\n",
       "      <td>366.693026</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          converted        male      female       18-24      24-35  \\\n",
       "group                                                                \n",
       "control  121.593693  489.848878  502.752699  202.490722  201.49812   \n",
       "test     123.406307  497.151122  510.247301  205.509278  204.50188   \n",
       "\n",
       "              35-50       50-70         70+       white       black  \\\n",
       "group                                                                 \n",
       "control  202.987022  195.542511  190.083202  244.676289  244.179988   \n",
       "test     206.012978  198.457489  192.916798  248.323711  247.820012   \n",
       "\n",
       "              asian    hispanic        high      medium         low  \n",
       "group                                                                \n",
       "control  272.469133  231.276167  316.143602  315.151001  361.306974  \n",
       "test     276.530867  234.723833  320.856398  319.848999  366.693026  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(expected, columns=table_transpose.columns, index=table_transpose.index).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>male</th>\n",
       "      <th>female</th>\n",
       "      <th>18-24</th>\n",
       "      <th>24-35</th>\n",
       "      <th>35-50</th>\n",
       "      <th>50-70</th>\n",
       "      <th>70+</th>\n",
       "      <th>white</th>\n",
       "      <th>black</th>\n",
       "      <th>asian</th>\n",
       "      <th>hispanic</th>\n",
       "      <th>high</th>\n",
       "      <th>medium</th>\n",
       "      <th>low</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>92</td>\n",
       "      <td>499.0</td>\n",
       "      <td>501.0</td>\n",
       "      <td>216</td>\n",
       "      <td>202</td>\n",
       "      <td>206</td>\n",
       "      <td>195</td>\n",
       "      <td>181</td>\n",
       "      <td>251</td>\n",
       "      <td>250</td>\n",
       "      <td>265.0</td>\n",
       "      <td>234</td>\n",
       "      <td>319.0</td>\n",
       "      <td>311.0</td>\n",
       "      <td>370.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>153</td>\n",
       "      <td>488.0</td>\n",
       "      <td>512.0</td>\n",
       "      <td>192</td>\n",
       "      <td>204</td>\n",
       "      <td>203</td>\n",
       "      <td>199</td>\n",
       "      <td>202</td>\n",
       "      <td>242</td>\n",
       "      <td>242</td>\n",
       "      <td>284.0</td>\n",
       "      <td>232</td>\n",
       "      <td>318.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>358.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         converted   male  female  18-24  24-35  35-50  50-70  70+  white  \\\n",
       "group                                                                       \n",
       "control         92  499.0   501.0    216    202    206    195  181    251   \n",
       "test           153  488.0   512.0    192    204    203    199  202    242   \n",
       "\n",
       "         black  asian  hispanic   high  medium    low  \n",
       "group                                                  \n",
       "control    250  265.0       234  319.0   311.0  370.0  \n",
       "test       242  284.0       232  318.0   324.0  358.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now let's verify our calculation of test statistic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.041239603425577"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summation = 0\n",
    "expected_df = pd.DataFrame(\n",
    "    expected, columns=table_transpose.columns, \n",
    "    index=table_transpose.index).T\n",
    "for index, row in table_transpose.iterrows():\n",
    "    for column in table_transpose.columns:\n",
    "        deviation = row[column] - expected_df[index][column]\n",
    "        deviation_squared = deviation ** 2\n",
    "        normed_deviation_squared = deviation_squared / expected_df[index][column]\n",
    "        summation += normed_deviation_squared\n",
    "summation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(summation, 5) == round(chi2, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Success!  Now we are sure that we have some belief that our test and control are different, accounting for at least some demographic variables, we are ready to do our final exploration.  \n",
    "\n",
    "This is an important point - if something passes a hypothesis test (or fails) that does not mean that the test is conclusive.  That's why I used the word may.  When you do statistics, machine learning, data science or whatever we are calling it these days, you should build up evidence that you believe your hypothesis is true.  You should imagine perhaps a court room, or some other sort of collection of people, that you have to convince of your results.  \n",
    "\n",
    "Each piece of evidence you bring to the table helps you make your case, but it does not and likely will not lead to a conclusive pattern.  It can only give you more of a chance that your hypothesis is right or wrong.  This is in direct conflict with the business world, where everything usually moves pretty fast.  But, if you build up the right tooling and can replicate the scientific process quickly, you can usually at least develop some confidence in what you are seeing.\n",
    "\n",
    "With this in mind we move onto the next leg in our scientific journey, data visualization.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualizing Categorical Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAESBJREFUeJzt3XuwnVV9xvHvYxCp4hguR8QEGtR0FK0iRorVOgjVCl5gKuKtJUWmmbG0yuCl2IpWa6daqljajhrFGqmilBZJLcUyAfFWhKDIRcoYGREil3AVZECBX//YKzObuJKzE7LPPuR8PzN79nrXu973/A6zTx7e29qpKiRJ2tCjJl2AJGl2MiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVKXASFJ6tpu0gU8HLvuumstWrRo0mVI0iPKJZdccktVTU037hEdEIsWLWL16tWTLkOSHlGSXDvKOE8xSZK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSuh7RT1JvDc975+cmXYJmoUtOPHLSJUgTN9YjiCQ/TnJ5kkuTrG59Oyc5N8kP2/tOrT9JTk6yJsllSfYdZ22SpE2biVNML6mqfapqSVs+HlhVVYuBVW0Z4GBgcXstAz4+A7VJkjZiEtcgDgVWtPYK4LCh/s/VwIXA/CS7T6A+SRLjvwZRwP8kKeCTVbUc2K2qbmjrbwR2a+0FwHVD217f+m5AmoN+8oHfnHQJmoX2fO/lM/azxh0QL6qqtUmeCJyb5P+GV1ZVtfAYWZJlDE5Bseeee269SiVJDzHWU0xVtba93wycCewH3LT+1FF7v7kNXwvsMbT5wta34T6XV9WSqloyNTXt911IkrbQ2AIiyeOSPH59G3gZcAWwEljahi0FzmrtlcCR7W6m/YE7h05FSZJm2DhPMe0GnJlk/c/5QlWdk+Ri4PQkRwPXAke08WcDhwBrgHuAo8ZYmyRpGmMLiKq6BnhOp/9W4KBOfwHHjKseSdLmcaoNSVKXASFJ6jIgJEldBoQkqcuAkCR1GRCSpC4DQpLUZUBIkroMCElSlwEhSeoyICRJXQaEJKnLgJAkdRkQkqQuA0KS1GVASJK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVKXASFJ6jIgJEldBoQkqcuAkCR1GRCSpC4DQpLUNfaASDIvyfeSfKUt75XkO0nWJPlSku1b/2Pa8pq2ftG4a5MkbdxMHEG8DbhqaPnDwElV9TTgduDo1n80cHvrP6mNkyRNyFgDIslC4BXAp9tygAOBM9qQFcBhrX1oW6atP6iNlyRNwLiPID4GvAt4sC3vAtxRVfe35euBBa29ALgOoK2/s42XJE3A2AIiySuBm6vqkq2832VJVidZvW7duq25a0nSkHEeQbwQeHWSHwNfZHBq6R+A+Um2a2MWAmtbey2wB0Bb/wTg1g13WlXLq2pJVS2ZmpoaY/mSNLeNLSCq6t1VtbCqFgGvB86rqjcB5wOHt2FLgbNae2Vbpq0/r6pqXPVJkjZtEs9B/DlwXJI1DK4xnNL6TwF2af3HAcdPoDZJUrPd9EMevqr6GvC11r4G2K8z5l7gtTNRjyRpej5JLUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVKXASFJ6jIgJEldBoQkqcuAkCR1GRCSpC4DQpLUZUBIkroMCElSlwEhSeoyICRJXQaEJKnLgJAkdRkQkqQuA0KS1GVASJK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVLX2AIiyQ5JLkry/SRXJnl/698ryXeSrEnypSTbt/7HtOU1bf2icdUmSZreOI8g7gMOrKrnAPsAL0+yP/Bh4KSqehpwO3B0G380cHvrP6mNkyRNyLQBkeQ3kqxKckVbfnaS90y3XQ3c3RYf3V4FHAic0fpXAIe19qFtmbb+oCQZ+TeRJG1VoxxBfAp4N/BLgKq6DHj9KDtPMi/JpcDNwLnAj4A7qur+NuR6YEFrLwCuaz/jfuBOYJfOPpclWZ1k9bp160YpQ5K0BUYJiMdW1UUb9N3fHbmBqnqgqvYBFgL7AU/fzPp6+1xeVUuqasnU1NTD3Z0kaSNGCYhbkjyVwekhkhwO3LA5P6Sq7gDOB14AzE+yXVu1EFjb2muBPdrP2A54AnDr5vwcSdLWM0pAHAN8Enh6krXAscBbptsoyVSS+a39a8BLgasYBMXhbdhS4KzWXtmWaevPq6oa8feQJG1l2003oKquAX43yeOAR1XVXSPue3dgRZJ5DILo9Kr6SpIfAF9M8kHge8ApbfwpwKlJ1gC3MeJ1DknSeGw0IJIct5F+AKrqo5vacbuY/dxO/zUMrkds2H8v8NpNlytJmimbOoJ4/IxVIUmadTYaEFX1/pksRJI0u0x7DSLJDgyecn4msMP6/qp68xjrkiRN2Ch3MZ0KPAn4PeACBremjnqhWpL0CDVKQDytqk4Afl5VK4BXAL813rIkSZM2SkD8sr3fkeRZDB5ge+L4SpIkzQbTXoMAlifZCTiBwcNsOwLvHWtVkqSJG+VBuU+35gXAU8ZbjiRpthjlLqb5wJHAouHxVfXW8ZUlSZq0UU4xnQ1cCFwOPDjeciRJs8UoAbFDVXWn3ZAkbbtGeg4iyR8n2T3JzutfY69MkjRRoxxB/AI4EfhL2ndCtHcvWEvSNmyUgHg7g4flbhl3MZKk2WOUU0xrgHvGXYgkaXYZ5Qji58ClSc4H7lvf6W2ukrRtGyUgvtxekqQ5ZJQnqVe075Tes6qunoGaJEmzwLTXIJK8CrgUOKct75Nk5bgLkyRN1igXqf+KwXdI3wFQVZfiLa6StM0babrvqrpzgz6n3JCkbdwoF6mvTPJGYF6SxcBbgW+PtyxJ0qRt9Agiyamt+SMG30d9H3Aa8DPg2PGXJkmapE0dQTwvyZOB1wEvAT4ytO6xwL3jLEySNFmbCohPAKsYXJBePdQfnItJkrZ5Gz3FVFUnV9UzgM9U1VOGXntVleEgSdu4ae9iqqq3zEQhkqTZZZTbXCVJc5ABIUnqMiAkSV0GhCSpa2wBkWSPJOcn+UGSK5O8rfXvnOTcJD9s7zu1/iQ5OcmaJJcl2XdctUmSpjfOI4j7gbdX1d7A/sAxSfYGjgdWVdViBs9ZHN/GHwwsbq9lwMfHWJskaRpjC4iquqGqvtvadwFXAQuAQ4EVbdgK4LDWPhT4XA1cCMxPsvu46pMkbdqMXINIsgh4LvAdYLequqGtuhHYrbUXANcNbXZ965MkTcDYAyLJjsC/A8dW1c+G11VVMZi2Y3P2tyzJ6iSr161btxUrlSQNG2tAJHk0g3D4fFX9R+u+af2po/Z+c+tfC+wxtPnC1vcQVbW8qpZU1ZKpqanxFS9Jc9w472IKcApwVVV9dGjVSmBpay8FzhrqP7LdzbQ/cOfQqShJ0gwb5QuDttQLgT8ELk9yaev7C+BDwOlJjgauBY5o684GDgHWAPcAR42xNknSNMYWEFX1TQZTg/cc1BlfwDHjqkeStHl8klqS1GVASJK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVKXASFJ6jIgJEldBoQkqcuAkCR1GRCSpC4DQpLUZUBIkroMCElSlwEhSeoyICRJXQaEJKnLgJAkdRkQkqQuA0KS1GVASJK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkrrEFRJLPJLk5yRVDfTsnOTfJD9v7Tq0/SU5OsibJZUn2HVddkqTRjPMI4rPAyzfoOx5YVVWLgVVtGeBgYHF7LQM+Psa6JEkjGFtAVNXXgds26D4UWNHaK4DDhvo/VwMXAvOT7D6u2iRJ05vpaxC7VdUNrX0jsFtrLwCuGxp3fev7FUmWJVmdZPW6devGV6kkzXETu0hdVQXUFmy3vKqWVNWSqampMVQmSYKZD4ib1p86au83t/61wB5D4xa2PknShMx0QKwElrb2UuCsof4j291M+wN3Dp2KkiRNwHbj2nGS04ADgF2TXA+8D/gQcHqSo4FrgSPa8LOBQ4A1wD3AUeOqS5I0mrEFRFW9YSOrDuqMLeCYcdUiSdp8PkktSeoyICRJXQaEJKnLgJAkdRkQkqQuA0KS1GVASJK6DAhJUpcBIUnqMiAkSV0GhCSpy4CQJHUZEJKkLgNCktRlQEiSugwISVKXASFJ6jIgJEldBoQkqcuAkCR1GRCSpC4DQpLUZUBIkroMCElSlwEhSeoyICRJXQaEJKnLgJAkdRkQkqQuA0KS1GVASJK6ZlVAJHl5kquTrEly/KTrkaS5bNYERJJ5wD8DBwN7A29Isvdkq5KkuWvWBASwH7Cmqq6pql8AXwQOnXBNkjRnzaaAWABcN7R8feuTJE3AdpMuYHMlWQYsa4t3J7l6kvVsY3YFbpl0EbNB/n7ppEvQQ/nZXO992Rp7+fVRBs2mgFgL7DG0vLD1PURVLQeWz1RRc0mS1VW1ZNJ1SBvyszkZs+kU08XA4iR7JdkeeD2wcsI1SdKcNWuOIKrq/iR/CnwVmAd8pqqunHBZkjRnzZqAAKiqs4GzJ13HHOapO81WfjYnIFU16RokSbPQbLoGIUmaRQyIOSDJ/CR/soXbHpvksVu7JmlzJFmU5I1buN0V46hpLjAg5ob5wBYFBHAsYEBo0hYB3YBIMquupW5L/A87N3wIeGqSS4FzgZuBI4DHAGdW1fuSPA44ncHzJ/OAvwZ2A54MnJ/klqp6yUSq1yNekiOBdwAFXAacAHyGwQNw64CjquonST4L/AxYAjwJeFdVncHgM/yM9hleAdwO/D6wIzAvyQHA3zGYy62AD1bVl2bsF9xGGRBzw/HAs6pqnyQvAw5nMPdVgJVJXgxMAT+tqlcAJHlCVd2Z5DjgJVXlU6zaIkmeCbwH+O2quiXJzgz+kV9RVSuSvBk4GTisbbI78CLg6QyehTqDwWf4HVX1yrbPPwL2BZ5dVbcleQ2wD/AcBqFzcZKvz9TvuK3yFNPc87L2+h7wXQZ/hIuBy4GXJvlwkt+pqjsnWKO2LQcC/7b+fzKq6jbgBcAX2vpTGQTCel+uqger6gcMjmI35ty2L9r2p1XVA1V1E3AB8Pyt+UvMRR5BzD0B/raqPvkrK5J9gUOADyZZVVUfmPHqJLhvqL2piYd+Pu5C5jqPIOaGu4DHt/ZXgTcn2REgyYIkT0zyZOCeqvpX4EQGh+8bbittifOA1ybZBaCdYvo2g+l0AN4EfGOafUz3OfwG8Lok85JMAS8GLnpYVcsjiLmgqm5N8q12u99/Mzi0/98kAHcDfwA8DTgxyYPAL4G3tM2XA+ck+akXqbUlqurKJH8DXJDkAQanN/8M+Jck76RdpJ5mN5cBDyT5PvBZBheph53J4LTV9xlcpH5XVd2YZNHW+j3mIp+kliR1eYpJktRlQEiSugwISVKXASFJ6jIgJEldBoQkqcuAkB4mZxPVtsqAkKaR5IQkVyf5ZpLTkrwjydeSfCzJauBt7XsHzktyWZJVSfZs2342yeFD+7q7vR+Q5OtJ/qvt+xNJ/HvUrOIHUtqEJM8HXsNgltCDGUxDvd72VbWkqj4C/COD2UmfDXyeweyk09mPwRPFewNPZTB9tTRrGBDSpr0QOKuq7q2qu4D/HFo3/H0Dm5qddGMuqqprquoB4LQRt5FmjAEhbblRZhO9n/Z31k4hbT+0bsN5bpz3RrOKASFt2reAVyXZoc2A+8qNjNvY7KQ/Bp7X2q8GHj20zX5J9mrB8Trgm1uzcOnh8u4LaROq6uIkKxnMJnoTgy9W6n2Z0sZmJ/0UcFabhfQcHnrUcTHwTwxm0j2fwYyk0qzhbK7SNJLsWFV3J3ks8HVgWVV992Hu8wCGvkJTmo08gpCmtzzJ3sAODO5UeljhID1SeAQhSeryIrUkqcuAkCR1GRCSpC4DQpLUZUBIkroMCElS1/8Dz7ktvvoY7QcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "gender_data = pd.DataFrame()\n",
    "gender_data = gender_data.append({\n",
    "    \"female\": df[df[\"group\"] == \"test\"][\"female\"].sum(),\n",
    "    \"group\": \"test\"\n",
    "}, ignore_index=True)\n",
    "gender_data = gender_data.append({\n",
    "    \"female\": df[df[\"group\"] == \"control\"][\"female\"].sum(),\n",
    "    \"group\": \"control\"\n",
    "}, ignore_index=True)\n",
    "\n",
    "sns.barplot(x=\"group\", y=\"female\", data=gender_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "As you can see it looks like there is a pretty close to even gender split between test and control, so we can probably rule out that lone interaction as the source of our variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEIhJREFUeJzt3X+wHWV9x/H3p6AogkXMNfKzYWj6I1qNeGGwqIU6VaG2YFWEVsEf09gWOlB/FZlarC2trYot/UGNAxKroqlKSVuqxUgFLAqBIhAoNYMgRAhBHASdognf/nE29YBPck8ge88h9/2aOXP2PLvP3u9lzs2HfXb32VQVkiQ93I+NuwBJ0mQyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlq2nHcBTwa8+bNqwULFoy7DEl6TLnqqqvurqqpmbZ7TAfEggULWLVq1bjLkKTHlCS3jrKdQ0ySpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTb0FRJJ9klyc5IYkq5Oc1LW/K8naJNd0ryOG+rwjyZokNyV5SV+1SZJm1ud9EBuAt1TV1Ul2Ba5KclG37gNV9b7hjZMsAo4BngHsCXw+yU9V1cYea5QkbUZvRxBVdUdVXd0t3wfcCOy1hS5HAp+oqgeq6uvAGuCgvuqTJG3ZrNxJnWQB8BzgK8AhwIlJjgNWMTjK+DaD8PjyULfb2XKgSNu1b7z758ZdgibQvn943az9rN5PUifZBfg0cHJVfQc4C9gfWAzcAbx/K/e3JMmqJKvWr1+/zeuVJA30GhBJHscgHD5WVZ8BqKp1VbWxqh4EPsQPh5HWAvsMdd+7a3uIqlpaVdNVNT01NeNcU5KkR6jPq5gCnA3cWFVnDLXvMbTZy4Hru+UVwDFJdkqyH7AQuKKv+iRJW9bnOYhDgNcC1yW5pms7FTg2yWKggFuANwFU1eoky4EbGFwBdYJXMEnS+PQWEFV1GZDGqgu30Od04PS+apIkjc47qSVJTY/pBwZtC89920fGXYIm0FXvPW7cJUhj5xGEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUlNvAZFknyQXJ7khyeokJ3Xtuye5KMnXuvendO1JcmaSNUmuTXJAX7VJkmbW5xHEBuAtVbUIOBg4Icki4BRgZVUtBFZ2nwEOBxZ2ryXAWT3WJkmaQW8BUVV3VNXV3fJ9wI3AXsCRwLJus2XAUd3ykcBHauDLwG5J9uirPknSls3KOYgkC4DnAF8B5lfVHd2qO4H53fJewG1D3W7v2iRJY9B7QCTZBfg0cHJVfWd4XVUVUFu5vyVJViVZtX79+m1YqSRpWK8BkeRxDMLhY1X1ma553aaho+79rq59LbDPUPe9u7aHqKqlVTVdVdNTU1P9FS9Jc1yfVzEFOBu4sarOGFq1Aji+Wz4euGCo/bjuaqaDgXuHhqIkSbNsxx73fQjwWuC6JNd0bacC7wGWJ3kjcCtwdLfuQuAIYA3wPeD1PdYmSZpBbwFRVZcB2czqFzW2L+CEvuqRJG0d76SWJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqam3gEhyTpK7klw/1PauJGuTXNO9jhha944ka5LclOQlfdUlSRpNn0cQ5wIvbbR/oKoWd68LAZIsAo4BntH1+bskO/RYmyRpBr0FRFVdAtwz4uZHAp+oqgeq6uvAGuCgvmqTJM1sHOcgTkxybTcE9ZSubS/gtqFtbu/aJEljMtsBcRawP7AYuAN4/9buIMmSJKuSrFq/fv22rk+S1JnVgKiqdVW1saoeBD7ED4eR1gL7DG26d9fW2sfSqpququmpqal+C5akOWxWAyLJHkMfXw5susJpBXBMkp2S7AcsBK6YzdokSQ+1Y187TnIecCgwL8ntwGnAoUkWAwXcArwJoKpWJ1kO3ABsAE6oqo191SZJmtlIAZFkPvCnwJ5VdXh3WerzqurszfWpqmMbzVva/nTg9FHqkST1b9QhpnOBzwF7dp//Bzi5j4IkSZNh1ICYV1XLgQcBqmoD4BCQJG3HRg2I7yZ5KoNzByQ5GLi3t6okSWM36knqNzO40mj/JF8CpoBX9VaVJGnsRg2I1cAvAD8NBLgJZ4KVpO3aqP/IX15VG6pqdVVdX1U/AC7vszBJ0nht8QgiydMZzIn0xCTPYXD0APBkYOeea5MkjdFMQ0wvAV7HYOqLM4ba7wNO7akmSdIE2GJAVNUyYFmSV1TVp2epJknSBJhpiOk1VfVRYEGSNz98fVWd0egmSdoOzDTE9KTufZe+C5EkTZaZhpg+2L3/0eyUI0maFKNO1jcF/CawYLhPVb2hn7IkSeM26o1yFwCXAp/HOZgkaU4YNSB2rqrf77USSdJEGfVO6n9JckSvlUiSJspMl7neRzeDK3BqkgcYPPENoKrqyX0WJ0kan5muYtoVIMlHgUuAS6vqxtkoTJI0XqMOMZ0NPB04M8nNST6V5KQe65IkjdlIJ6mr6uIklwAHAocBvwU8E/irHmuTJI3RqPdBrGRwV/XlDC53PbCq7uqzMEnSeI06xHQt8H0GRw3PAp6Z5Im9VSVJGrtRh5h+DyDJrgym//4wg3MSO/VWmSRprEYdYjoReAHwXOAW4BwGQ02SpO3UqHdSP4HBA4OuqqoNM20sSXrsG3WI6X19FyJJmiyjnqSWJM0xBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSU28BkeScJHcluX6obfckFyX5Wvf+lK49Sc5MsibJtUkO6KsuSdJo+jyCOBd46cPaTgFWVtVCYGX3GeBwYGH3WgKc1WNdkqQR9BYQVXUJcM/Dmo8ElnXLy4Cjhto/UgNfBnZLskdftUmSZjbb5yDmV9Ud3fKdwPxueS/gtqHtbu/afkSSJUlWJVm1fv36/iqVpDlubCepq6qAegT9llbVdFVNT01N9VCZJAlmPyDWbRo66t43PZVuLbDP0HZ7d22SpDGZ7YBYARzfLR8PXDDUflx3NdPBwL1DQ1GSpDEY9XkQWy3JecChwLwktwOnAe8Blid5I3ArcHS3+YXAEcAa4HvA6/uqS5I0mt4CoqqO3cyqFzW2LeCEvmqRJG0976SWJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqWnHcfzQJLcA9wEbgQ1VNZ1kd+CTwALgFuDoqvr2OOqTJI33COKwqlpcVdPd51OAlVW1EFjZfZYkjckkDTEdCSzrlpcBR42xFkma88YVEAX8e5Krkizp2uZX1R3d8p3A/PGUJkmCMZ2DAJ5fVWuTPA24KMl/D6+sqkpSrY5doCwB2HffffuvVJLmqLEcQVTV2u79LuB84CBgXZI9ALr3uzbTd2lVTVfV9NTU1GyVLElzzqwHRJInJdl10zLwYuB6YAVwfLfZ8cAFs12bJOmHxjHENB84P8mmn//xqvpskiuB5UneCNwKHD2G2iRJnVkPiKq6GXh2o/1bwItmux5JUtskXeYqSZogBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWqauIBI8tIkNyVZk+SUcdcjSXPVRAVEkh2AvwUOBxYBxyZZNN6qJGlumqiAAA4C1lTVzVX1feATwJFjrkmS5qRJC4i9gNuGPt/etUmSZtmO4y5gayVZAizpPt6f5KZx1rOdmQfcPe4iJkHed/y4S9BD+d3c5LRsi738xCgbTVpArAX2Gfq8d9f2/6pqKbB0NouaK5KsqqrpcdchPZzfzfGYtCGmK4GFSfZL8njgGGDFmGuSpDlpoo4gqmpDkhOBzwE7AOdU1eoxlyVJc9JEBQRAVV0IXDjuOuYoh+40qfxujkGqatw1SJIm0KSdg5AkTQgDYo5IsluS33mEfU9OsvO2rkkaVZIFSX79Efa7vo+a5gIDYu7YDXhEAQGcDBgQGqcFQDMgkkzcudTthf9h5473APsnuQa4CLgLOBrYCTi/qk5L8iRgOYP7T3YA/hiYD+wJXJzk7qo6bCzV6zEtyXHAW4ECrgXeCZzD4Aa49cDrq+obSc4FvgNMA08H3l5Vn2Lw/f3Z7vu7DPg28GvALsAOSQ4F/oLBPG4F/ElVfXLWfsHtlAExd5wCPLOqFid5MfBKBnNfBViR5IXAFPDNqvplgCQ/XlX3JnkzcFhVeSertlqSZwB/APx8Vd2dZHcG/8gvq6plSd4AnAkc1XXZA3g+8DMM7oP6FIPv71ur6mXdPl8HHAA8q6ruSfIKYDHwbAahc2WSS2brd9xeOcQ0N724e/0XcDWDP8SFwHXALyX58yQvqKp7x1ijth+/CPzjpv/BqKp7gOcBH+/W/wODQNjkn6rqwaq6gcER7OZc1O2Lrv95VbWxqtYBXwQO3Ja/xFzkEcTcFODPquqDP7IiOQA4AviTJCur6t2zXp3mugeGlrc08dB3+y5krvMIYu64D9i1W/4c8IYkuwAk2SvJ05LsCXyvqj4KvJfBIfzD+0pb6wvAq5I8FaAbYvpPBlPpAPwGcOkM+5jpO3gp8OokOySZAl4IXPGoqpZHEHNFVX0ryZe6S/7+jcHh/eVJAO4HXgP8JPDeJA8CPwB+u+u+FPhskm96klpbq6pWJzkd+GKSjQyGNn8X+HCSt9GdpJ5hN9cCG5N8FTiXwUnqYeczGLb6KoOT1G+vqjuTLNhWv8dc5J3UkqQmh5gkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwIaRtwRlFtjwwIaQRJ3pnkpiSXJTkvyVuT/EeSv0yyCjipe/bAF5Jcm2Rlkn27vucmeeXQvu7v3g9NckmSf+32/fdJ/JvUxPDLKM0gyYHAKxjMFHo4g6moN3l8VU1X1fuBv2YwQ+mzgI8xmKF0JgcxuKt4EbA/gymspYlgQEgzOwS4oKr+t6ruA/55aN3wMwe2NEPp5lxRVTdX1UbgvBH7SLPCgJAenVFmFN1A97fWDSE9fmjdw+e6ce4bTQwDQprZl4BfSfKEbgbcl21mu83NUHoL8Nxu+VeBxw31OSjJfl1wvBq4bFsWLj0aXnkhzaCqrkyygsGMousYPFip9TClzc1Q+iHggm4m0s/y0KOOK4G/YTCT7sUMZiWVJoKzuUojSLJLVd2fZGfgEmBJVV39KPd5KEOP0ZQmjUcQ0miWJlkEPIHBlUqPKhykxwKPICRJTZ6kliQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWr6Pzj1DDBRLJqbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "ethnicity_data = pd.DataFrame()\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"white\": df[df[\"group\"] == \"test\"][\"white\"].sum(),\n",
    "    \"group\": \"test\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"white\": df[df[\"group\"] == \"control\"][\"white\"].sum(),\n",
    "    \"group\": \"control\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"black\": df[df[\"group\"] == \"control\"][\"black\"].sum(),\n",
    "    \"group\": \"control\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"black\": df[df[\"group\"] == \"test\"][\"black\"].sum(),\n",
    "    \"group\": \"test\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"asian\": df[df[\"group\"] == \"control\"][\"asian\"].sum(),\n",
    "    \"group\": \"control\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"asian\": df[df[\"group\"] == \"test\"][\"asian\"].sum(),\n",
    "    \"group\": \"test\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"hispanic\": df[df[\"group\"] == \"control\"][\"hispanic\"].sum(),\n",
    "    \"group\": \"control\"\n",
    "}, ignore_index=True)\n",
    "ethnicity_data = ethnicity_data.append({\n",
    "    \"hispanic\": df[df[\"group\"] == \"test\"][\"hispanic\"].sum(),\n",
    "    \"group\": \"test\"\n",
    "}, ignore_index=True)\n",
    "\n",
    "sns.barplot(x=\"group\", y=\"white\", data=ethnicity_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAD/xJREFUeJzt3X+wXGV9x/H3p0G0ii3QxIhADIOZsbEqxSul/gScUkEU6g+UVqFoJ47FjoxVm/6wOFanVGvbsVptHJFgFUUrklbEMsGCaC0Ei/xSxpSCgIEEsfyQihK+/WPPrQt9krsXsvds7n2/Znb2nGfPOftdZm8+nOc559lUFZIkPdjP9F2AJGkyGRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNe3SdwEPx+LFi2v58uV9lyFJO5XLLrvstqpaMtN2O3VALF++nA0bNvRdhiTtVJLcMMp2djFJkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaduo7qaX57LvvfGrfJWgCLfvTK+fsvRZ8QDzjrWf0XYIm0GXvPb7vEqTe2cUkSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElS09gCIsm+Sb6c5JokVyd5U9e+Z5Lzk3yne96ja0+S9yfZmOSKJAeOqzZJ0szGeQZxH/D7VbUSOBg4KclKYDWwvqpWAOu7dYAjgBXdYxXwoTHWJkmawdgCoqo2VdU3uuW7gG8BewNHA2u7zdYCx3TLRwNn1MDXgd2T7DWu+iRJ2zcnYxBJlgO/DPw7sLSqNnUv3QIs7Zb3Bm4c2u2mrk2S1IOxB0SS3YB/BE6uqjuHX6uqAmqWx1uVZEOSDVu2bNmBlUqSho01IJI8gkE4fKKqPtc13zrdddQ9b+7abwb2Hdp9n67tAapqTVVNVdXUkiVLxle8JC1w47yKKcBHgW9V1V8NvbQOOKFbPgE4Z6j9+O5qpoOBO4a6oiRJc2yXMR772cBrgCuTXN61/RFwKnBWktcBNwDHdq+dCxwJbATuAU4cY22SpBmMLSCq6mIg23j5BY3tCzhpXPVIkmbHO6klSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1jS0gkpyWZHOSq4ba3pHk5iSXd48jh177wyQbk1yb5NfHVZckaTTjPIM4HXhho/2vq+qA7nEuQJKVwKuAp3T7/F2SRWOsTZI0g7EFRFVdBNw+4uZHA5+qqnur6r+AjcBB46pNkjSzPsYg3pjkiq4Lao+ubW/gxqFtburaJEk9meuA+BCwP3AAsAl432wPkGRVkg1JNmzZsmVH1ydJ6sxpQFTVrVW1taruBz7CT7uRbgb2Hdp0n66tdYw1VTVVVVNLliwZb8GStIDNaUAk2Wto9TeA6Suc1gGvSvLIJPsBK4BL5rI2SdID7TKuAyc5EzgEWJzkJuAU4JAkBwAFXA+8HqCqrk5yFnANcB9wUlVtHVdtkqSZjS0gquq4RvNHt7P9u4F3j6seSdLseCe1JKlppIBIcn6S3YfW90jypfGVJUnq26hnEIur6r+nV6rqB8DjxlOSJGkSjBoQ9ydZNr2S5IkMBpolSfPUqIPUfwxcnORCIMBzgVVjq0qS1LuRAqKqzktyIHBw13RyVd02vrIkSX3bbhdTkid3zwcCy4DvdY9lXZskaZ6a6QzizQy6klpzJhVw2A6vSJI0EbYbEFW1qns+dG7KkSRNipHvpE7yLGD58D5VdcYYapIkTYCRAiLJxxlM0305MD1HUgEGhCTNU6OeQUwBK6vKex8kaYEY9Ua5q4DHj7MQSdJkGfUMYjFwTZJLgHunG6vqJWOpSpLUu1ED4h3jLEKSNHlGvZP6wnEXIkmaLKNO931wkkuT3J3kx0m2Jrlz3MVJkvoz6iD1B4DjgO8APwv8DvDBcRUlSerfyL8oV1UbgUVVtbWqPga8cHxlSZL6Nuog9T1JdgUuT/IeYBP+XKkkzWuj/iP/mm7bNwI/BPYFXjauoiRJ/Rv1KqYbujOI5cDngGur6sfjLEyS1K9R52J6EfBh4D8Z/KLcfkleX1VfHGdxkqT+jDoG8T7g0G6gmiT7A18ADAhJmqdGHYO4azocOtcBd42hHknShBj1DGJDknOBsxhM8/0K4NIkLwWoqs+NqT5JUk9GDYhHAbcCz+/WtzC4Ye7FDALDgJCkeWbUq5hOHHchkqTJMupcTO9J8nNJHpFkfZItSV497uIkSf0ZdZD68Kq6EzgKuB54EvDWcRUlSerfqAEx3RX1IuAzVXXHmOqRJE2IUQep/znJt4H/Ad6QZAnwo/GVJUnq20hnEFW1GngWMFVVP2EwH9PR4yxMktSv7Z5BJDmsqi6Yvt+haxvexMtbJWmemqmL6fnABfz0fodpwfsfJGle225AVNUp3eIbGEzvvXxon2rtI0maH0a9iunzDM4ifgLcPfTYpiSnJdmc5Kqhtj2TnJ/kO93zHl17krw/ycYkVyQ58KF9HEnSjjLqVUz7VNVsf2L0dAa/ZX3GUNtqYH1VnZpkdbf+B8ARwIru8SvAh7pnSVJPRj2D+FqSp87mwFV1EXD7g5qPBtZ2y2uBY4baz6iBrwO7J9lrNu8nSdqxZrqK6UoGYw27ACcmuQ64l26QuqqeNsv3W1pVm7rlW4Cl3fLewI1D293UtW3iQZKsAlYBLFu2bJZvL0ka1UxdTEeN642rqpLMeqC7qtYAawCmpqYcKJekMZnpKqYbdvD73Zpkr6ra1HUhbe7abwb2Hdpun65NktSTUccgdpR1wAnd8gnAOUPtx3dXMx0M3DHUFSVJ6sGoVzHNWpIzgUOAxUluAk4BTgXOSvI64Abg2G7zc4EjgY3APYC/PyFJPRtbQFTVcdt46QWNbQs4aVy1SJJmb667mCRJOwkDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKadunjTZNcD9wFbAXuq6qpJHsCnwaWA9cDx1bVD/qoT5LU7xnEoVV1QFVNdeurgfVVtQJY361LknoySV1MRwNru+W1wDE91iJJC15fAVHAvyS5LMmqrm1pVW3qlm8BlvZTmiQJehqDAJ5TVTcneRxwfpJvD79YVZWkWjt2gbIKYNmyZeOvVJIWqF7OIKrq5u55M3A2cBBwa5K9ALrnzdvYd01VTVXV1JIlS+aqZElacOY8IJI8Jsljp5eBw4GrgHXACd1mJwDnzHVtkqSf6qOLaSlwdpLp9/9kVZ2X5FLgrCSvA24Aju2hNklSZ84DoqquA57eaP8+8IK5rkeS1DZJl7lKkiaIASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTRMXEElemOTaJBuTrO67HklaqCYqIJIsAj4IHAGsBI5LsrLfqiRpYZqogAAOAjZW1XVV9WPgU8DRPdckSQvSpAXE3sCNQ+s3dW2SpDm2S98FzFaSVcCqbvXuJNf2Wc88sxi4re8iJkH+8oS+S9AD+d2cdkp2xFGeOMpGkxYQNwP7Dq3v07X9n6paA6yZy6IWiiQbqmqq7zqkB/O72Y9J62K6FFiRZL8kuwKvAtb1XJMkLUgTdQZRVfcleSPwJWARcFpVXd1zWZK0IE1UQABU1bnAuX3XsUDZdadJ5XezB6mqvmuQJE2gSRuDkCRNCANigUiye5LffYj7npzk0Tu6JmlUSZYn+c2HuN9V46hpITAgFo7dgYcUEMDJgAGhPi0HmgGRZOLGUucL/8MuHKcC+ye5HDgf2AwcCzwSOLuqTknyGOAsBvefLAL+DFgKPAH4cpLbqurQXqrXTi3J8cBbgAKuAN4OnMbgBrgtwIlV9d0kpwN3AlPA44G3VdVnGXx/f7H7/q4FfgC8FNgNWJTkEOA9DOZxK+BdVfXpOfuA85QBsXCsBn6pqg5IcjjwcgZzXwVYl+R5wBLge1X1IoAkP19VdyR5M3BoVXknq2YtyVOAPwGeVVW3JdmTwT/ya6tqbZLXAu8Hjul22Qt4DvBkBvdBfZbB9/ctVXVUd8zfBg4EnlZVtyd5GXAA8HQGoXNpkovm6jPOV3YxLUyHd4//AL7B4A9xBXAl8GtJ/iLJc6vqjh5r1PxxGPCZ6f/BqKrbgV8FPtm9/nEGgTDt81V1f1Vdw+AMdlvO745Ft/+ZVbW1qm4FLgSeuSM/xELkGcTCFODPq+rv/98LyYHAkcC7kqyvqnfOeXVa6O4dWt7exEM/HHchC51nEAvHXcBju+UvAa9NshtAkr2TPC7JE4B7quofgPcyOIV/8L7SbF0AvCLJLwB0XUxfYzCVDsBvAV+Z4RgzfQe/ArwyyaIkS4DnAZc8rKrlGcRCUVXfT/LV7pK/LzI4vf+3JAB3A68GngS8N8n9wE+AN3S7rwHOS/I9B6k1W1V1dZJ3Axcm2cqga/P3gI8leSvdIPUMh7kC2Jrkm8DpDAaph53NoNvqmwwGqd9WVbckWb6jPsdC5J3UkqQmu5gkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwIaQdwRlHNRwaENIIkb09ybZKLk5yZ5C1J/jXJ3yTZALyp++2BC5JckWR9kmXdvqcnefnQse7ung9JclGSL3TH/nAS/yY1MfwySjNI8kzgZQxmCj2CwVTU03atqqmqeh/wtwxmKH0a8AkGM5TO5CAGdxWvBPZnMIW1NBEMCGlmzwbOqaofVdVdwD8NvTb8mwPbm6F0Wy6pquuqaitw5oj7SHPCgJAenlFmFL2P7m+t60Ladei1B89149w3mhgGhDSzrwIvTvKobgbco7ax3bZmKL0eeEa3/BLgEUP7HJRkvy44XglcvCMLlx4Or7yQZlBVlyZZx2BG0VsZ/LBS68eUtjVD6UeAc7qZSM/jgWcdlwIfYDCT7pcZzEoqTQRnc5VGkGS3qro7yaOBi4BVVfWNh3nMQxj6GU1p0ngGIY1mTZKVwKMYXKn0sMJB2hl4BiFJanKQWpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnpfwEuIrw3ilXf1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(x=\"group\", y=\"hispanic\", data=ethnicity_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEIlJREFUeJzt3XuwXWV9xvHvU0BQoQIlRgxgGJtqo1XEI6XVOiBTFWoLVotgFYrOxKnYytTLoFOLo9LaKtrBttg4UkKrKF4ocaQoRiriDU4ohpvUVEGIQII4XLxVwq9/7HXKJr7J2YGsszc538/MnrXWuy7nd5h98rDetda7UlVIkrSpXxp3AZKkyWRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktS047gLeCj22muvWrx48bjLkKSHldWrV99eVQtm2+5hHRCLFy9menp63GVI0sNKkhtH2c4uJklSkwEhSWoyICRJTQaEJKnJgJAkNfUWEEn2TXJxkmuTXJPk9V3725OsS3Jl9zliaJ+3JFmb5PokL+irNknS7Pq8zfVe4A1VdUWS3YDVSS7q1r2/qt47vHGSpcAxwFOAxwNfSPJrVbWxxxolSZvR2xlEVd1SVVd083cD1wGLtrDLkcDHqupnVfVdYC1wUF/1SZK2bE6uQSRZDDwD+EbX9Loka5KcmWSPrm0RcNPQbjez5UCRJPWo9yepk+wKfAo4qaruSnIG8E6guulpwKu24njLgGUA++2337YvWJoQ33vHb4y7BE2g/f7qqjn7Wb2eQSTZiUE4fKSqPg1QVbdV1caqug/4EPd3I60D9h3afZ+u7QGqanlVTVXV1IIFsw4lIkl6kPq8iynAh4Hrqup9Q+17D232YuDqbn4lcEySnZPsDywBLuurPknSlvXZxfRs4JXAVUmu7NreChyb5AAGXUw3AK8BqKprkpwLXMvgDqgTvYNJksant4CoqkuBNFZdsIV9TgVO7asmSdLofJJaktT0sH4fxLbwzDedPe4SNIFWv+e4cZcgjZ1nEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNvQVEkn2TXJzk2iTXJHl9175nkouSfLub7tG1J8npSdYmWZPkwL5qkyTNrs8ziHuBN1TVUuBg4MQkS4GTgVVVtQRY1S0DHA4s6T7LgDN6rE2SNIveAqKqbqmqK7r5u4HrgEXAkcCKbrMVwFHd/JHA2TXwdWD3JHv3VZ8kacvm5BpEksXAM4BvAAur6pZu1a3Awm5+EXDT0G43d22SpDHoPSCS7Ap8Cjipqu4aXldVBdRWHm9Zkukk0xs2bNiGlUqShvUaEEl2YhAOH6mqT3fNt810HXXT9V37OmDfod336doeoKqWV9VUVU0tWLCgv+IlaZ7r8y6mAB8Grquq9w2tWgkc380fD5w/1H5cdzfTwcCdQ11RkqQ5tmOPx3428ErgqiRXdm1vBd4NnJvk1cCNwNHduguAI4C1wI+BE3qsTZI0i94CoqouBbKZ1Yc1ti/gxL7qkSRtHZ+kliQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNfUWEEnOTLI+ydVDbW9Psi7Jld3niKF1b0myNsn1SV7QV12SpNH0eQZxFvDCRvv7q+qA7nMBQJKlwDHAU7p9/inJDj3WJkmaRW8BUVWXAHeMuPmRwMeq6mdV9V1gLXBQX7VJkmY3jmsQr0uypuuC2qNrWwTcNLTNzV2bJGlM5jogzgCeCBwA3AKctrUHSLIsyXSS6Q0bNmzr+iRJnTkNiKq6rao2VtV9wIe4vxtpHbDv0Kb7dG2tYyyvqqmqmlqwYEG/BUvSPDanAZFk76HFFwMzdzitBI5JsnOS/YElwGVzWZsk6YF27OvASc4BDgH2SnIzcApwSJIDgAJuAF4DUFXXJDkXuBa4Fzixqjb2VZskaXYjBUSSPavqjk3a9u/uOGqqqmMbzR/ewvanAqeOUo8kqX+jdjF9Jskvzyx0zy18pp+SJEmTYNSA+GsGIbFrkmcCnwBe0V9ZkqRxG6mLqao+m2Qn4PPAbsCLq+q/e61MkjRWWwyIJB9gcEF5xmOA/2HwsBtV9ed9FidJGp/ZziCmN1le3VchkqTJssWAqKoVAEkeDfx05tbTbiC9nfsvT5I0LqNepF4FPHJo+ZHAF7Z9OZKkSTFqQOxSVffMLHTzj+qnJEnSJBg1IH6U5MCZhe5W15/0U5IkaRKMOtTGScAnknwfCPA44GW9VSVJGrtRn4O4PMmTgSd1TddX1c/7K0uSNG5bM1jfk4ClwC7Agd1zEGf3U5YkadxGHazvFAYjsy4FLgAOBy4FDAhJ2k6NepH6pcBhwK1VdQLwdAZPVUuStlOjBsRPurfA3duN6rqeB74BTpK0nRn1GsR0kt0ZvCZ0NXAP8LXeqpIkjd2odzG9tpv9YJILgV+uqjX9lSVJGrfZRnM9cEvrquqKbV+SJGkSzHYGcdrQ/PCw3+mWn7fNK5IkTYTZRnM9FCDJI4HXAs9hEAxfBs7ovTpJ0tiMepF6BXAXcHq3/HIGz0Ac3UdRkqTxGzUgnlpVS4eWL05ybR8FSZImw6jPQVyR5OCZhSS/yS++bU6StB2Z7S6mqxhcc9gJ+GqS73XLTwC+1X95kqRxma2L6UVzUoUkaeLMdhfTjXNViCRpsox6DUKSNM8YEJKkJgNCktRkQEiSmgwISVKTASFJauotIJKcmWR9kquH2vZMclGSb3fTPbr2JDk9ydoka7Y0zLgkaW70eQZxFvDCTdpOBlZV1RJgVbcMcDiwpPssw5FiJWnseguIqroEuGOT5iMZjAxLNz1qqP3sGvg6sHuSvfuqTZI0u7m+BrGwqm7p5m8FFnbzi4Cbhra7uWuTJI3J2C5SV1XxwLfUjSTJsiTTSaY3bNjQQ2WSJJj7gLhtpuuom67v2tcB+w5tt0/X9guqanlVTVXV1IIFC3otVpLms7kOiJXA8d388cD5Q+3HdXczHQzcOdQVJUkag1HfKLfVkpwDHALsleRm4BTg3cC5SV4N3Mj9ryy9ADgCWAv8GDihr7okSaPpLSCq6tjNrDqssW0BJ/ZViyRp6/kktSSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqWnHcfzQJDcAdwMbgXurairJnsDHgcXADcDRVfXDcdQnSRrvGcShVXVAVU11yycDq6pqCbCqW5YkjckkdTEdCazo5lcAR42xFkma98YVEAV8PsnqJMu6toVVdUs3fyuwsLVjkmVJppNMb9iwYS5qlaR5aSzXIIDnVNW6JI8FLkryreGVVVVJqrVjVS0HlgNMTU01t5EkPXRjOYOoqnXddD1wHnAQcFuSvQG66fpx1CZJGpjzgEjy6CS7zcwDzweuBlYCx3ebHQ+cP9e1SZLuN44upoXAeUlmfv5Hq+rCJJcD5yZ5NXAjcPQYapMkdeY8IKrqO8DTG+0/AA6b63okSW2TdJurJGmCGBCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKlp4gIiyQuTXJ9kbZKTx12PJM1XExUQSXYA/hE4HFgKHJtk6XirkqT5aaICAjgIWFtV36mq/wU+Bhw55pokaV6atIBYBNw0tHxz1yZJmmM7jruArZVkGbCsW7wnyfXjrGc7sxdw+7iLmAR57/HjLkEP5HdzxinZFkd5wigbTVpArAP2HVrep2v7f1W1HFg+l0XNF0mmq2pq3HVIm/K7OR6T1sV0ObAkyf5JHgEcA6wcc02SNC9N1BlEVd2b5HXA54AdgDOr6poxlyVJ89JEBQRAVV0AXDDuOuYpu+40qfxujkGqatw1SJIm0KRdg5AkTQgDYp5IsnuS1z7IfU9K8qhtXZM0qiSLk7z8Qe53dR81zQcGxPyxO/CgAgI4CTAgNE6LgWZAJJm4a6nbC//Dzh/vBp6Y5ErgImA9cDSwM3BeVZ2S5NHAuQyeP9kBeCewEHg8cHGS26vq0LFUr4e1JMcBbwQKWAO8DTiTwQNwG4ATqup7Sc4C7gKmgMcBb66qTzL4/v569/1dAfwQ+ENgV2CHJIcAf8dgHLcC3lVVH5+zX3A7ZUDMHycDT62qA5I8H3gpg7GvAqxM8lxgAfD9qvo9gCSPqao7k/wFcGhV+SSrtlqSpwB/Cfx2Vd2eZE8G/8ivqKoVSV4FnA4c1e2yN/Ac4MkMnoP6JIPv7xur6kXdMf8EOBB4WlXdkeQlwAHA0xmEzuVJLpmr33F7ZRfT/PT87vNfwBUM/hCXAFcBv5vkb5P8TlXdOcYatf14HvCJmf/BqKo7gN8CPtqt/1cGgTDj36vqvqq6lsEZ7OZc1B2Lbv9zqmpjVd0GfAl41rb8JeYjzyDmpwB/U1X//AsrkgOBI4B3JVlVVe+Y8+o03/1saH5LAw/9qO9C5jvPIOaPu4HduvnPAa9KsitAkkVJHpvk8cCPq+rfgPcwOIXfdF9pa30R+KMkvwLQdTF9lcFQOgB/DHx5lmPM9h38MvCyJDskWQA8F7jsIVUtzyDmi6r6QZKvdLf8/QeD0/uvJQG4B3gF8KvAe5LcB/wc+NNu9+XAhUm+70Vqba2quibJqcCXkmxk0LX5Z8C/JHkT3UXqWQ6zBtiY5JvAWQwuUg87j0G31TcZXKR+c1XdmmTxtvo95iOfpJYkNdnFJElqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEjbgCOKantkQEgjSPK2JNcnuTTJOUnemOQ/k/x9kmng9d27B76YZE2SVUn26/Y9K8lLh451Tzc9JMklST7bHfuDSfyb1MTwyyjNIsmzgJcwGCn0cAZDUc94RFVNVdVpwAcYjFD6NOAjDEYonc1BDJ4qXgo8kcEQ1tJEMCCk2T0bOL+qflpVdwOfGVo3/M6BLY1QujmXVdV3qmojcM6I+0hzwoCQHppRRhS9l+5vretCesTQuk3HunHsG00MA0Ka3VeA30+ySzcC7os2s93mRii9AXhmN/8HwE5D+xyUZP8uOF4GXLotC5ceCu+8kGZRVZcnWclgRNHbGLxYqfUypc2NUPoh4PxuJNILeeBZx+XAPzAYSfdiBqOSShPB0VylESTZtaruSfIo4BJgWVVd8RCPeQhDr9GUJo1nENJolidZCuzC4E6lhxQO0sOBZxCSpCYvUkuSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1/R/1BPrdYy5oygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(x=\"group\", y=\"black\", data=ethnicity_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The three groups:\n",
    "* white\n",
    "* black\n",
    "* hispanic \n",
    "\n",
    "all appear to have very little variation.  However, as we can see below, the difference in size between black folks from test and control appears to be different.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEGxJREFUeJzt3Xu05WVdx/H3J0BZCgnIcYIBG6OxGk1HPBKKGeiKADXwhlAJAatxFRqUl4WuDFJZUXhpaYWNQQylKCnIZAjSQCKmwpkJucpyloLMOMBBiEskyfDtj/2b3OIzM/vA7LM3c96vtfbav/38nue3v5u1z3z43Z6dqkKSpEf7qVEXIEkaTwaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU3bjrqAx2PXXXetBQsWjLoMSXpCWbly5V1VNbG5fk/ogFiwYAFTU1OjLkOSnlCS3DpIPw8xSZKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmp7Qd1JvCS98xzmjLkFjaOXpR426BGnk3IOQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElS05yfzVUaV9997y+PugSNoWf+6XWz9l7uQUiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1DS0gkuyZ5PIkNya5IckJXfspSdYmuaZ7HNI35l1JVie5OclvDKs2SdLmDfM+iIeBt1XVqiQ7AiuTXNqt+3BVfaC/c5JFwBHAc4DdgX9L8uyqWj/EGiVJGzG0PYiqWldVq7rl+4GbgPmbGHIo8KmqeqiqvgOsBvYZVn2SpE2blXMQSRYALwC+3jW9Jcm1Sc5KsnPXNh+4rW/YGjYdKJKkIRp6QCTZAfgscGJV3QecAewFLAbWAR+c4faWJJlKMjU9Pb3F65Uk9Qw1IJJsRy8cPlFV5wNU1R1Vtb6qHgE+zo8OI60F9uwbvkfX9mOqamlVTVbV5MTExDDLl6Q5bZhXMQU4E7ipqj7U175bX7fXANd3y8uBI5I8OcmzgIXAVcOqT5K0acO8imk/4E3AdUmu6dreDRyZZDFQwC3AmwGq6oYk5wE30rsC6nivYJKk0RlaQFTVlUAaqy7axJhTgVOHVZMkaXDeSS1JajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqWloAZFkzySXJ7kxyQ1JTujad0lyaZJvdc87d+1J8pEkq5Ncm2TvYdUmSdq8Ye5BPAy8raoWAfsCxydZBJwErKiqhcCK7jXAwcDC7rEEOGOItUmSNmNoAVFV66pqVbd8P3ATMB84FFjWdVsGHNYtHwqcUz1fA3ZKstuw6pMkbdqsnINIsgB4AfB1YF5VretW3Q7M65bnA7f1DVvTtUmSRmDoAZFkB+CzwIlVdV//uqoqoGa4vSVJppJMTU9Pb8FKJUn9hhoQSbajFw6fqKrzu+Y7Nhw66p7v7NrXAnv2Dd+ja/sxVbW0qiaranJiYmJ4xUvSHDfMq5gCnAncVFUf6lu1HDi6Wz4auLCv/ajuaqZ9gXv7DkVJkmbZtkPc9n7Am4DrklzTtb0bOA04L8lxwK3A4d26i4BDgNXAg8AxQ6xNkrQZQwuIqroSyEZWv6LRv4Djh1WPJGlmvJNaktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpadtBOiXZDzgF+NluTICqqp8bXmmSpFEaKCCAM4E/AlYC64dXjiRpXAwaEPdW1ReGWokkaawMGhCXJzkdOB94aENjVa0aSlWSpJEbNCB+pXue7Gsr4OVbthxJ0rgYKCCq6oBhFyJJGi+D7kGQ5JXAc4DtN7RV1Xs30f8s4FXAnVX13K7tFOD3gOmu27ur6qJu3buA4+idBP/DqrpkRp9EkrRFDXQfRJKPAW8E3krvEtc30LvkdVPOBg5qtH+4qhZ3jw3hsAg4gl4AHQT8bZJtBvoEkqShGPRGuZdU1VHAPVX1Z8CLgWdvakBVXQHcPeD2DwU+VVUPVdV3gNXAPgOOlSQNwaAB8T/d84NJdgd+COz2GN/zLUmuTXJWkp27tvnAbX191nRtPyHJkiRTSaamp6dbXSRJW8CgAfH5JDsBpwOrgFuAcx/D+50B7AUsBtYBH5zpBqpqaVVNVtXkxMTEYyhBkjSIQa9iel+3+Nkknwe2r6p7Z/pmVXXHhuUkHwc+371cC+zZ13WPrk2SNCKbDIgkL6+qy5K8trGOqjp/Jm+WZLeqWte9fA1wfbe8HPhkkg8BuwMLgatmsm1J0pa1uT2IXwMuA17dWFf07qxuSnIusD+wa5I1wMnA/kkWd2NvAd4MUFU3JDkPuBF4GDi+qpzzSZJGaJMBUVUnd8/HzHTDVXVko/nMTfQ/FTh1pu8jSRqOQe+DOCHJT6fn75OsSnLgsIuTJI3OoFcxHVtV9wEHAk8H3gScNrSqJEkjN2hApHs+BDinqm7oa5MkbYUGDYiVSb5ILyAuSbIj8MjwypIkjdqgk/UdR+/mtu3oTfm9K725liRJW6lBA+JY4AR6N7BdA+wLfBX46JDqkiSN2KCHmE4AXgTc2v02xAuA/xpaVZKkkRs0IH5QVT8ASPLkqvom8AvDK0uSNGqDHmJa003W9zng0iT3ALcOryxJ0qgNOlnfa7rFU5JcDjwNuHhoVUmSRm7gnxzdoKq+NIxCJEnjZdBzEJKkOcaAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU1DC4gkZyW5M8n1fW27JLk0ybe655279iT5SJLVSa5Nsvew6pIkDWaYexBnAwc9qu0kYEVVLQRWdK8BDgYWdo8lwBlDrEuSNIChBURVXQHc/ajmQ4Fl3fIy4LC+9nOq52vATkl2G1ZtkqTNm+1zEPOqal23fDswr1ueD9zW129N1yZJGpGRnaSuqgJqpuOSLEkylWRqenp6CJVJkmD2A+KODYeOuuc7u/a1wJ59/fbo2n5CVS2tqsmqmpyYmBhqsZI0l812QCwHju6WjwYu7Gs/qruaaV/g3r5DUZKkEdh2WBtOci6wP7BrkjXAycBpwHlJjgNuBQ7vul8EHAKsBh4EjhlWXZKkwQwtIKrqyI2sekWjbwHHD6sWSdLMeSe1JKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1LTtKN40yS3A/cB64OGqmkyyC/BpYAFwC3B4Vd0zivokSaPdgzigqhZX1WT3+iRgRVUtBFZ0ryVJIzJOh5gOBZZ1y8uAw0ZYiyTNeaMKiAK+mGRlkiVd27yqWtct3w7Maw1MsiTJVJKp6enp2ahVkuakkZyDAF5aVWuTPAO4NMk3+1dWVSWp1sCqWgosBZicnGz2kSQ9fiPZg6iqtd3zncAFwD7AHUl2A+ie7xxFbZKknlkPiCRPTbLjhmXgQOB6YDlwdNftaODC2a5NkvQjozjENA+4IMmG9/9kVV2c5GrgvCTHAbcCh4+gNklSZ9YDoqq+DTy/0f594BWzXY8kqW2cLnOVJI0RA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaxi4gkhyU5OYkq5OcNOp6JGmuGquASLIN8DfAwcAi4Mgki0ZblSTNTWMVEMA+wOqq+nZV/S/wKeDQEdckSXPSuAXEfOC2vtdrujZJ0izbdtQFzFSSJcCS7uUDSW4eZT1bmV2Bu0ZdxDjIB44edQn6cX43Nzg5W2IrPztIp3ELiLXAnn2v9+ja/l9VLQWWzmZRc0WSqaqaHHUd0qP53RyNcTvEdDWwMMmzkjwJOAJYPuKaJGlOGqs9iKp6OMlbgEuAbYCzquqGEZclSXPSWAUEQFVdBFw06jrmKA/daVz53RyBVNWoa5AkjaFxOwchSRoTBsQckWSnJH/wGMeemOQpW7omaVBJFiT5rcc47vph1DQXGBBzx07AYwoI4ETAgNAoLQCaAZFk7M6lbi38Dzt3nAbsleQa4FLgTuBw4MnABVV1cpKnAufRu/9kG+B9wDxgd+DyJHdV1QEjqV5PaEmOAt4OFHAt8B7gLHo3wE0Dx1TVd5OcDdwHTAI/A7yzqj5D7/v7S933dxlwD/BaYAdgmyT7A39Jbx63At5fVZ+etQ+4lTIg5o6TgOdW1eIkBwKvpzf3VYDlSV4GTADfq6pXAiR5WlXdm+SPgQOqyjtZNWNJngP8CfCSqroryS70/pFfVlXLkhwLfAQ4rBuyG/BS4Bfp3Qf1GXrf37dX1au6bf4usDfwvKq6O8nrgMXA8+mFztVJrpitz7i18hDT3HRg9/hPYBW9P8SFwHXAryf5iyS/WlX3jrBGbT1eDvzzhv/BqKq7gRcDn+zW/yO9QNjgc1X1SFXdSG8PdmMu7bZFN/7cqlpfVXcAXwJetCU/xFzkHsTcFODPq+rvfmJFsjdwCPD+JCuq6r2zXp3muof6ljc18dB/D7uQuc49iLnjfmDHbvkS4NgkOwAkmZ/kGUl2Bx6sqn8CTqe3C//osdJMXQa8IcnTAbpDTP9BbyodgN8GvryZbWzuO/hl4I1JtkkyAbwMuOpxVS33IOaKqvp+kq90l/x9gd7u/VeTADwA/A7w88DpSR4Bfgj8fjd8KXBxku95klozVVU3JDkV+FKS9fQObb4V+Ick76A7Sb2ZzVwLrE/yDeBseiep+11A77DVN+idpH5nVd2eZMGW+hxzkXdSS5KaPMQkSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASFuAM4pqa2RASANI8p4kNye5Msm5Sd6e5N+T/FWSKeCE7rcHLktybZIVSZ7ZjT07yev7tvVA97x/kiuS/Gu37Y8l8W9SY8Mvo7QZSV4EvI7eTKEH05uKeoMnVdVkVX0Q+Ci9GUqfB3yC3gylm7MPvbuKFwF70ZvCWhoLBoS0efsBF1bVD6rqfuBf+tb1/+bApmYo3ZirqurbVbUeOHfAMdKsMCCkx2eQGUUfpvtb6w4hPalv3aPnunHuG40NA0LavK8Ar06yfTcD7qs20m9jM5TeArywW/5NYLu+MfskeVYXHG8ErtyShUuPh1deSJtRVVcnWU5vRtE76P2wUuvHlDY2Q+nHgQu7mUgv5sf3Oq4G/preTLqX05uVVBoLzuYqDSDJDlX1QJKnAFcAS6pq1ePc5v70/YymNG7cg5AGszTJImB7elcqPa5wkJ4I3IOQJDV5klqS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSp6f8A4Kv94xyzhE0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(x=\"group\", y=\"asian\", data=ethnicity_data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "It's worth it to note that because this data was entirely randomly generated, there is no implicit data generation process to analyze, but if this were the real world, I would say this information warrants further inquiry.  In fact, if this were the real world, it may be enough evidence to explain the variation in the difference between test and control.  We will look at how we would measure that next.  But do keep in mind, we may not find anything in this case because the data was randomly generated.  And in fact, any conclusions we do draw will be entirely by chance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd73b2f86a0>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGPZJREFUeJzt3XuYVnW99/H3V1FR05BDVgwG5iE8ICloaRGpSbERPKNlgqh0UB4PjzutK0nNvbOt+ykPTyo7j5Wo2fa4DVPU3Lg9NCAawuMlEeGgJoKoFKOA3+ePe6GjgcwaZu57hnm/rmuuWeu3Dvf3Hu7hM+u31vqtyEwkSWqujWpdgCSpYzE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSunSVjuOiGuAEcDLmblb0dYduBnoC8wHjsrMVyMigEuA4cDfgbGZOaPYZgzw/WK3F2Tm9et67Z49e2bfvn1b9f1I0oZu+vTpr2Rmr3WtF2015EhEDAGWATc0CY5/A5Zk5oURcTawTWaeFRHDgQlUgmMf4JLM3KcImnpgEJDAdGCvzHz1g1570KBBWV9f3ybvS5I2VBExPTMHrWu9NuuqysyHgSXvax4FrD5iuB44pEn7DVnxGNAtIj4GDAPuy8wlRVjcB3y5rWqWJK1btc9xbJuZLxbTLwHbFtO9geebrNdQtK2tXZJUIzU7OZ6VPrJW6yeLiPERUR8R9YsWLWqt3UqS3qfNTo6vxV8j4mOZ+WLRFfVy0b4Q6NNkvbqibSEw9H3tD61px5k5CZgElXMcrVu2pGpZsWIFDQ0NNDY21rqUDVbXrl2pq6tjk002adH21Q6OO4ExwIXF9zuatJ8SETdROTn+WhEu9wL/GhHbFOsdBHy3yjVLqqKGhga22mor+vbtS+WCS7WmzGTx4sU0NDTQr1+/Fu2jLS/HnUzlaKFnRDQAP6ASGLdExAnAX4CjitXvoXJF1Vwql+MeD5CZSyLih8AfivXOz8z3n3CXtAFpbGw0NNpQRNCjRw/Wp0u/zYIjM49Zy6ID1rBuAievZT/XANe0YmmS2jlDo22t78/XO8clSaUYHJKkUqp9clzSetrrn2+odQlt6qJD+5PPv9KsdXfp0/Mf2iZOnMiQIUM48MADW7u0mpo5cyYvvPACw4cPL7Xd0KFDufjiixk0aJ03hDebwSFpg3L++efXuoRWt3LlSmbOnEl9fX3p4GgLBsdabOh/1ZUx/aLjal2CxIQTj+OlFxby5ptvcuy48Rx+9NcYO3Ys9fX1RATjxo3j9NNPZ+zYsYwYMYIjjjiC888/n7vuuovly5ez7777ctVVVxERDB06lH322YcHH3yQpUuXcvXVV/P5z39+ja+7atUqzjrrLKZMmcJGG23ESSedxIQJE5g6dSpnnnkmK1euZPDgwVxxxRVsttlm9O3blzFjxnDXXXexYsUKfv3rX7PTTjux/fbbM3PmTLp16wbAjjvuyLRp09hoo4345je/yYIFCwD46U9/yn777ce5557Ln/70J+bNm8d2223HI488wvLly5k2bRrf/e53GTFiBBMmTGDWrFmsWLGCc889l1GjRrF8+XKOP/54nnrqKT71qU+xfPnyVv+3MDgkdQg/vPgSunXbhsbG5YwecRC77r4HCxcuZNasWQAsXbr0H7Y55ZRTmDhxIgBf//rXufvuuzn44IOByl/xTzzxBPfccw/nnXce999//xpfd9KkScyfP5+ZM2fSpUsXlixZQmNjI2PHjmXq1KnstNNOHHfccVxxxRWcdtppAPTs2ZMZM2bws5/9jIsvvpif//znjBo1ittuu43jjz+exx9/nE984hNsu+22fPWrX+X000/nc5/7HAsWLGDYsGHMmTMHgNmzZzNt2jQ233xzrrvuOurr67n88ssB+N73vsf+++/PNddcw9KlS9l777058MADueqqq9hiiy2YM2cOTz/9NHvuuWfr/kPgyXFJHcSvrvkPDh02lGNGfYWXXlzIihVvMW/ePCZMmMCUKVPYeuut/2GbBx98kH322Yfdd9+dBx54gGeeeeadZYcddhgAe+21F/Pnz1/r695///184xvfoEuXyt/Z3bt359lnn6Vfv37stNNOAIwZM4aHH374A/c9evRobr75ZgBuuukmRo8e/c7+TznlFAYOHMjIkSN5/fXXWbZsGQAjR45k8803X2Ndv/vd77jwwgsZOHAgQ4cOpbGxkQULFvDwww9z7LHHAjBgwAAGDBjwwT/YFvCIQ1K798Sjj/DotN9z4+33sPnmWzD2qFG89dZbPPXUU9x7771ceeWV3HLLLVxzzbu3fDU2NvLtb3+b+vp6+vTpw7nnnvueYUw222wzADbeeGNWrlzZqvWuad+f/exnmTt3LosWLeL222/n+9+vPGbo7bff5rHHHqNr167/sJ8tt9xyra+RmfzmN79h5513btXam8MjDknt3rI3XmfrD3dj8823YN7c53jqyem8umQxb7/9NocffjgXXHABM2bMeM82q0OiZ8+eLFu2jFtvvbVFr/2lL32Jq6666p0AWLJkCTvvvDPz589n7ty5APziF7/gC1/4wgfuJyI49NBDOeOMM+jfvz89evQA4KCDDuKyyy57Z72ZM2eucfutttqKN9544535YcOGcdlll7H6mUpPPvkkAEOGDOHGG28EYNasWTz99NMtedsfyOCQ1O597gv7s2rVSg7ef19+cuEP2ePTe/HySy8ydOhQBg4cyLHHHsuPfvSj92zTrVs3TjrpJHbbbTeGDRvG4MGDW/TaJ554Ittttx0DBgxgjz324MYbb6Rr165ce+21HHnkkey+++7vnOBel9GjR/PLX/7ynW4qgEsvvZT6+noGDBjALrvswpVXXrnGbb/4xS8ye/ZsBg4cyM0338w555zDihUrGDBgALvuuivnnHMOAN/61rdYtmwZ/fv3Z+LEiey1114tet8fpM2eAFhLrfEEQK+qepdXVbUvG/pn86JD+/PRPs0bfG9N93GoeebMmUP//v3f01bzJwBKkjZMnhyXJODee+/lrLPOek9bv379uO2222pUUftlcEgSlZPNw4YNq3UZHYJdVZKkUgwOSVIpBockqRTPcUjq1Fr78ubmXr4+ZcoUTj31VFatWsWJJ57I2Wef3ap1tCWPOCSpylatWsXJJ5/Mb3/7W2bPns3kyZOZPXt2rctqNoNDkqrsiSeeYIcddmD77bdn00035eijj+aOO+6odVnNZnBIUpUtXLiQPn36vDNfV1fHwoULa1hROQaHJKkUg0OSqqx37948//zz78w3NDTQu3fvGlZUjsEhSVU2ePBgnnvuOf785z/z1ltvcdNNNzFy5Mhal9VsXo4rqVOrxejPXbp04fLLL2fYsGGsWrWKcePGseuuu1a9jpYyOCSpBoYPH87w4cNrXUaL2FUlSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpXo4rqVNbcP7urbq/7Sb+cZ3rjBs3jrvvvpuPfOQjzJo1q1Vfvxo84pCkKhs7dixTpkypdRktVpPgiIjTI+KZiJgVEZMjomtE9IuIxyNibkTcHBGbFutuVszPLZb3rUXNktRahgwZQvfu3WtdRotVPTgiojfwv4BBmbkbsDFwNPBj4CeZuQPwKnBCsckJwKtF+0+K9SRJNVKrrqouwOYR0QXYAngR2B+4tVh+PXBIMT2qmKdYfkBERBVrlSQ1UfXgyMyFwMXAAiqB8RowHViamSuL1RqA1WMM9waeL7ZdWazf4/37jYjxEVEfEfWLFi1q2zchSZ1YLbqqtqFyFNEP+DiwJfDl9d1vZk7KzEGZOahXr17ruztJ0lrU4nLcA4E/Z+YigIj4T2A/oFtEdCmOKuqA1c9RXAj0ARqKrq0PA4urX7akDVFzLp9tbccccwwPPfQQr7zyCnV1dZx33nmccMIJ696wnahFcCwAPhMRWwDLgQOAeuBB4AjgJmAMsPrJ7XcW848Wyx/IzKx20ZLUWiZPnlzrEtZLLc5xPE7lJPcM4I9FDZOAs4AzImIulXMYVxebXA30KNrPAM6uds2SpHfV5M7xzPwB8IP3Nc8D9l7Duo3AkdWoS5K0bt45LqldyQR7o9vW+v58DQ5J7UrD0kbe+vsbhkcbyUwWL15M165dW7wPBzmU1K5c++gCjgfqunVlXbf6xjLv2WqJrl27UldX1+LtDQ5J7cobb67i0of+3Kx1p190XBtXozWxq0qSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFJqEhwR0S0ibo2I/xcRcyLisxHRPSLui4jniu/bFOtGRFwaEXMj4umI2LMWNUuSKmp1xHEJMCUzPwXsAcwBzgamZuaOwNRiHuArwI7F13jgiuqXK0larerBEREfBoYAVwNk5luZuRQYBVxfrHY9cEgxPQq4ISseA7pFxMeqXLYkqVCLI45+wCLg2oh4MiJ+HhFbAttm5ovFOi8B2xbTvYHnm2zfULS9R0SMj4j6iKhftGhRG5YvSZ1bLYKjC7AncEVmfhr4G+92SwGQmQlkmZ1m5qTMHJSZg3r16tVqxUqS3qsWwdEANGTm48X8rVSC5K+ru6CK7y8XyxcCfZpsX1e0SZJqoOrBkZkvAc9HxM5F0wHAbOBOYEzRNga4o5i+EziuuLrqM8BrTbq0JElV1qVGrzsB+FVEbArMA46nEmK3RMQJwF+Ao4p17wGGA3OBvxfrSpJqpCbBkZkzgUFrWHTAGtZN4OQ2L0qS1CzeOS5JKsXgkCSV0qzgiIipzWmTJG34PvAcR0R0BbYAehZjR0WxaGvWcBOeJGnDt66T498ATgM+Dkzn3eB4Hbi8DeuSJLVTHxgcmXkJcElETMjMy6pUkySpHWvW5biZeVlE7Av0bbpNZt7QRnVJktqpZgVHRPwC+CQwE1hVNCdgcEhSJ9PcGwAHAbsUN+NJkjqx5t7HMQv4aFsWIknqGJp7xNETmB0RTwBvrm7MzJFtUpUkqd1qbnCc25ZFSJI6juZeVfX7ti5EktQxNPeqqjd494l8mwKbAH/LzK3bqjBJUvvU3COOrVZPR0QAo4DPtFVRkqT2q/TouFlxOzCsDeqRJLVzze2qOqzJ7EZU7utobJOKJEntWnOvqjq4yfRKYD6V7ipJUifT3HMcPudbkgQ0/0FOdRFxW0S8XHz9JiLq2ro4SVL709yT49cCd1J5LsfHgbuKNklSJ9Pc4OiVmddm5sri6zqgVxvWJUlqp5obHIsj4tiI2Lj4OhZY3JaFSZLap+YGxzjgKOAl4EXgCGBsG9UkSWrHmns57vnAmMx8FSAiugMXUwkUSVIn0twjjgGrQwMgM5cAn26bkiRJ7Vlzg2OjiNhm9UxxxNHcoxVJ0gakuf/5/zvwaET8upg/EviXtilJktSeNffO8Rsioh7Yv2g6LDNnt11ZkqT2qtndTUVQGBaS1MmVHlZdktS5GRySpFIMDklSKTULjmLokicj4u5ivl9EPB4RcyPi5ojYtGjfrJifWyzvW6uaJUm1PeI4FZjTZP7HwE8ycwfgVeCEov0E4NWi/SfFepKkGqlJcBTP8vgn4OfFfFC51PfWYpXrgUOK6VHFPMXyA4r1JUk1UKsjjp8C3wHeLuZ7AEszc2Ux3wD0LqZ7A88DFMtfK9aXJNVA1YMjIkYAL2fm9Fbe7/iIqI+I+kWLFrXmriVJTdTiiGM/YGREzAduotJFdQnQLSJW35BYBywsphcCfQCK5R9mDc8CycxJmTkoMwf16uUzpiSprVQ9ODLzu5lZl5l9gaOBBzLza8CDVJ7zATAGuKOYvrOYp1j+QGZmFUuWJDXRnu7jOAs4IyLmUjmHcXXRfjXQo2g/Azi7RvVJkqjx0OiZ+RDwUDE9D9h7Des0UhmNV5LUDrSnIw5JUgdgcEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSql6cEREn4h4MCJmR8QzEXFq0d49Iu6LiOeK79sU7RERl0bE3Ih4OiL2rHbNkqR31eKIYyXwvzNzF+AzwMkRsQtwNjA1M3cEphbzAF8Bdiy+xgNXVL9kSdJqVQ+OzHwxM2cU028Ac4DewCjg+mK164FDiulRwA1Z8RjQLSI+VuWyJUmFmp7jiIi+wKeBx4FtM/PFYtFLwLbFdG/g+SabNRRtkqQaqFlwRMSHgN8Ap2Xm602XZWYCWXJ/4yOiPiLqFy1a1IqVSpKaqklwRMQmVELjV5n5n0XzX1d3QRXfXy7aFwJ9mmxeV7S9R2ZOysxBmTmoV69ebVe8JHVytbiqKoCrgTmZ+X+aLLoTGFNMjwHuaNJ+XHF11WeA15p0aUmSqqxLDV5zP+DrwB8jYmbR9j3gQuCWiDgB+AtwVLHsHmA4MBf4O3B8dcuVJDVV9eDIzGlArGXxAWtYP4GT27QoSVKzeee4JKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJXSpdYFqP1bcP7utS6h3dhu4h9rXYJUcx5xSJJKMTgkSaXYVSWpw7Ib9V3V7Eb1iEOSVIrBIUkqpcMER0R8OSKejYi5EXF2reuRpM6qQwRHRGwM/F/gK8AuwDERsUttq5KkzqlDBAewNzA3M+dl5lvATcCoGtckSZ1SRwmO3sDzTeYbijZJUpVtMJfjRsR4YHwxuywinq1lPRuST0BP4JVa19Eu/CBqXYGa8LPZROt8Nj/RnJU6SnAsBPo0ma8r2t6RmZOASdUsqrOIiPrMHFTrOqT387NZGx2lq+oPwI4R0S8iNgWOBu6scU2S1Cl1iCOOzFwZEacA9wIbA9dk5jM1LkuSOqUOERwAmXkPcE+t6+ik7AJUe+VnswYiM2tdgySpA+ko5zgkSe2EwdHJRUS3iPh2C7c9LSK2aO2apDIiom9EfLWF281qi5o2dAaHugEtCg7gNMDgUK31BdYYHBHRYc7jdiT+UHUh8MmImAncB7wMHAVsBtyWmT+IiC2BW6jcP7Mx8ENgW+DjwIMR8UpmfrEm1avDi4jjgDOBBJ4GzgGuoXJz3yLg+MxcEBHXAa8Dg4CPAt/JzFupfIb7F5/h64FXgcOADwEbR8RQ4N+ojHWXwAWZeXPV3uAGyODQ2cBumTkwIg4CjqAyNlgAd0bEEKAX8EJm/hNARHw4M1+LiDOAL2amd+6qRSJiV+D7wL6Z+UpEdKfyn//1mXl9RIwDLgUOKTb5GPA54FNU7uW6lcpn+MzMHFHscyywJzAgM5dExOHAQGAPKmH0h4h4uFrvcUNkV5WaOqj4ehKYQeWXc0fgj8CXIuLHEfH5zHythjVqw7I/8OvVf3xk5hLgs8CNxfJfUAmK1W7PzLczczaVo961ua/YF8X2kzNzVWb+Ffg9MLg130Rn4xGHmgrgR5l51T8siNgTGA5cEBFTM/P8qlcnwZtNpj9ocKa/tXUhnZlHHHoD2KqYvhcYFxEfAoiI3hHxkYj4OPD3zPwlcBGVboD3byu1xAPAkRHRA6DoqvofKsMKAXwN+O917GNdn8P/BkZHxMYR0QsYAjyxXlV3ch5xdHKZuTgiHikuS/wtlS6CRyMCYBlwLLADcFFEvA2sAL5VbD4JmBIRL3hyXC2Rmc9ExL8Av4+IVVS6SScA10bEP1OcHF/Hbp4GVkXEU8B1VE6ON3Uble6vp6icHP9OZr4UEX1b6310Nt45Lkkqxa4qSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTikNuTorNoQGRzSeoiIcyLi2YiYFhGTI+LMiHgoIn4aEfXAqcVzHx6IiKcjYmpEbFdse11EHNFkX8uK70Mj4uGI+K9i31dGhL+rajf8MEotFBGDgcOpjLr6FSrDfa+2aWYOysx/By6jMtrrAOBXVEZ7XZe9qdxBvQvwSSrDhEvtgsEhtdx+wB2Z2ZiZbwB3NVnW9HkPHzTa69o8kZnzMnMVMLmZ20hVYXBIbaM5o7OupPgdLLqiNm2y7P1jATk2kNoNg0NquUeAgyOiazGi8Ii1rLe20V7nA3sV0yOBTZpss3dE9CsCZTQwrTULl9aHV3xILZSZf4iIO6mMzvpXKg+8WtNDrtY22ut/AHcUo7pO4b1HKX8ALqcyMvGDVEZ4ldoFR8eV1kNEfCgzl0XEFsDDwPjMnLGe+xxKk0ehSu2NRxzS+pkUEbsAXalcObVeoSF1BB5xSJJK8eS4JKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEml/H9HdgFONa5wbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "is_asian = df[\"asian\"].astype(bool)\n",
    "is_asian = is_asian.map({True: 1, False: 0})\n",
    "\n",
    "is_black = df[\"black\"].astype(bool)\n",
    "is_black = is_black.map({True: 1, False: 0})\n",
    "\n",
    "is_white = df[\"white\"].astype(bool)\n",
    "is_white = is_white.map({True: 1, False: 0})\n",
    "\n",
    "is_hispanic = df[\"hispanic\"].astype(bool)\n",
    "is_hispanic = is_hispanic.map({True: 1, False: 0})\n",
    "\n",
    "converted = df[\"converted\"].astype(bool)\n",
    "converted = converted.map({True: 1, False: 0})\n",
    "\n",
    "df[\"asian_converted\"] = is_asian & converted\n",
    "df[\"black_converted\"] = is_black & converted\n",
    "df[\"white_converted\"] = is_white & converted\n",
    "df[\"hispanic_converted\"] = is_hispanic & converted\n",
    "sns.countplot(x=\"group\", hue=\"asian_converted\", data=df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As we see from the above graph the chance of either test or control converting is quiet low.  Since the demographic condition is a prior piece of information we can treat the above graph as:\n",
    "\n",
    "$$ P(Converted | Asian) $$\n",
    "\n",
    "However, recall that this would only be true in the real world.  Let's compare this to the other groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd73b2d3550>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGF5JREFUeJzt3X+8VXWd7/HXJ37IiBiCJ39wcLD0pqhAcORCFjr+jmE0GiwYSkR7cL3D9drtpnKvTVZTj8kfU5PR1MVfYBlRTIn9kDILTdIU0BSwktL0MP4AE0MNgePn/rEXeDSQvYC99zmc1/Px2I+91nd91zqffdjwZv36rshMJEmq1psaXYAkqXMxOCRJpRgckqRSDA5JUikGhySpFINDklSKwSFJKsXgkCSVYnBIkkrp3ugCamH//ffPQYMGNboMSepUli5dujYzm3bUb48MjkGDBrFkyZJGlyFJnUpE/KGafh6qkiSVUrPgiIjrI+KZiFjerq1fRNwWEY8U7/sV7RERV0fEqoh4MCKGt1tnStH/kYiYUqt6JUnVqeUex2zg9Ne1zQBuz8zDgduLeYD3AIcXr2nAV6ASNMBlwH8FRgKXbQkbSVJj1OwcR2beGRGDXtd8JnBCMT0HWARcUrTfmJUx3u+JiL4RcVDR97bM/CNARNxGJYzm1qpuSY2zadMmWltb2bBhQ6NL2aP16tWL5uZmevTosVPr1/vk+AGZ+WQx/RRwQDE9AHiiXb/Wom177ZL2QK2trfTp04dBgwYREY0uZ4+UmTz77LO0trZy6KGH7tQ2GnZyvNi72G1PkYqIaRGxJCKWrFmzZndtVlIdbdiwgf79+xsaNRQR9O/ff5f26uodHE8Xh6Ao3p8p2lcDA9v1ay7attf+FzJzVma2ZGZLU9MOL0OW1EEZGrW3q7/jegfHLcCWK6OmAAvatZ9dXF01Cni+OKT1I+DUiNivOCl+atEmSWqQmp3jiIi5VE5u7x8RrVSujvoc8K2IOA/4A/D+ovsPgbHAKuAlYCpAZv4xIv4ZuK/o9+ktJ8olSY1Ry6uqJm1n0Unb6JvA9O1s53rg+t1YmtSpjbjoxkaXUDNXjj+SfGJt1f0HD9yfffbZhxdeeOEvln31q19l77335uyzz2b27NmceuqpHHzwwbuz3Lramc/w2GOPMW7cOJYvX77jziXskUOOSNL555+/dXr27NkcffTRnTY42traOtRncMgRSR3a9V+dydevnwXA5z71caZOHA/APYt/zuTJkwG49NJLGTp0KKNGjeLpp58G4JOf/CRXXXUV8+fPZ8mSJUyePJlhw4bx5z//maVLl3L88cczYsQITjvtNJ588slt/3Bg1apVnHzyyQwdOpThw4fzu9/9jszkoosu4uijj+aYY45h3rx5ACxatIgTTjiBCRMmcMQRRzB58mQyk4ULF3LWWWdt3eaiRYsYN24cAD/+8Y8ZPXo0w4cP56yzztq69zRo0CAuueQShg8fzty5c6v+DEuXLmXo0KEMHTqUL3/5y7vzj2Ir9zi2Y08+HFDW0ivPbnQJ6sJGjBzF7Fn/zgfPncaKBx9g48aNbNq0iWX33sOYMWP4xje+wahRo/jsZz/LxRdfzDXXXMPHP/7xretPmDCBmTNnctVVV9HS0sKmTZu44IILWLBgAU1NTcybN49LL72U66/f9hHxyZMnM2PGDMaPH8+GDRt45ZVX+M53vsMDDzzAr371K9auXcuxxx7LmDFjALj//vtZsWIFBx98MMcddxyLFy/m5JNPZtq0abz44ov07t2befPmMXHiRNauXctnPvMZfvKTn9C7d28uv/xyPv/5z/OJT3wCgP79+7Ns2TIArr322qo+w9SpU5k5cyZjxozhoosuqsmficEhqUMbfMxQVj70IC+sX0/Pnnsx+OghrHjwAZbeew//+OEp9OzZc+v/3keMGMFtt932htv7zW9+w/LlyznllFOAymGggw46aJt9169fz+rVqxk/vrKX06tXLwDuuusuJk2aRLdu3TjggAM4/vjjue+++9h3330ZOXIkzc3NAAwbNozHHnuMd73rXZx++ul873vfY8KECfzgBz/giiuu4I477mDlypUcd9xxAGzcuJHRo0dv/fkf+MAHSn2GdevWsW7duq0h9qEPfYhbb711x7/kkgwOSR1ajx49GDDwEG7+9lyGtRzLfzliMPfefReP/+FRjjzySHr06LH1voRu3bqxefPmN9xeZnLUUUdx991316Tevfbaa+t0+3omTpzIzJkz6devHy0tLfTp04fM5JRTTmHu3G2PotS7d+9ttm/vM6xbt243fYo35jkOSR3eiJGjuGHWv9MycjQjRo5i3tfncORRx1R9I1ufPn1Yv349AG9/+9tZs2bN1n90N23axIoVK7a7XnNzMzfffDMAL7/8Mi+99BLvfve7mTdvHm1tbaxZs4Y777yTkSNHvmENxx9/PMuWLeOaa65h4sSJAIwaNYrFixezatUqAF588UV++9vf7vRn6Nu3L3379uWuu+4C4Kabbqrq91OWwSGpwxsxchRrn3maoSNa2L/pLey1114MHzmq6vXPOecczj//fIYNG0ZbWxvz58/nkksuYejQoQwbNoxf/OIX2133a1/7GldffTVDhgzhne98J0899RTjx49nyJAhDB06lBNPPJErrriCAw888A1r6NatG+PGjePWW2/demitqamJ2bNnM2nSJIYMGcLo0aP59a9/vUuf4YYbbmD69OkMGzaMyp0Ou1/UasON1NLSkrv6BEBPjr/Kk+Mdy5783bxy/JEcOLD6gfcGD9y/htXs2R5++GGOPPLI17RFxNLMbNnRuu5xSJJK8eS4JAHTp09n8eLFr2m78MILmTp1aoMq6rgMDkmCmt0styfyUJUkqRSDQ5JUisEhSSrFcxySurTdfXlzNZevL1y4kAsvvJC2tjY+/OEPM2PGjN1aQ625xyFJddTW1sb06dO59dZbWblyJXPnzmXlypWNLqsUg0OS6ujee+/lsMMO461vfSs9e/Zk4sSJLFiwYMcrdiAGhyTV0erVqxk4cODW+ebmZlavXt3AisozOCRJpRgcklRHAwYM4Iknntg639rayoABAxpYUXkGhyTV0bHHHssjjzzCo48+ysaNG/nmN7/JGWec0eiySvFyXEldWr1Hf+7evTszZ87ktNNOo62tjXPPPZejjjqqrjXsKoNDkups7NixjB07ttFl7DQPVUmSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIqX40rq0h7/9DG7dXuHfOKhHfY599xz+f73v89b3vIWli9fvlt/fj24xyFJdXbOOeewcOHCRpex0wwOSaqzMWPG0K9fv0aXsdMMDklSKQ0Jjoj4XxGxIiKWR8TciOgVEYdGxC8jYlVEzIuInkXfvYr5VcXyQY2oWZJUUffgiIgBwP8EWjLzaKAbMBG4HPhCZh4GPAecV6xyHvBc0f6Fop8kqUEadaiqO/BXEdEd2Bt4EjgRmF8snwO8t5g+s5inWH5SREQda5UktVP3y3Ezc3VEXAU8DvwZ+DGwFFiXmZuLbq3AliebDACeKNbdHBHPA/2BtXUtXNIeqZrLZ3e3SZMmsWjRItauXUtzczOf+tSnOO+883a8YgdR9+CIiP2o7EUcCqwDvg2cvhu2Ow2YBnDIIYfs6uYkqWbmzp3b6BJ2SSMOVZ0MPJqZazJzE/Ad4Digb3HoCqAZ2PL09tXAQIBi+ZuBZ1+/0cyclZktmdnS1NRU688gSV1WI4LjcWBUROxdnKs4CVgJ/AyYUPSZAiwopm8p5imW/zQzs471SpLaqXtwZOYvqZzkXgY8VNQwC7gE+GhErKJyDuO6YpXrgP5F+0eBGfWuWVJ9ZIL/L6y9Xf0dN2Ssqsy8DLjsdc2/B0Zuo+8G4Kx61CWpsVrXbaBf//X03LsPXjxZG5nJs88+S69evXZ6Gw5yKKnDuOHux5kKNPftRTW5ES+sqXlNe6JevXrR3Ny80+sbHJI6jPUvt3H1oker7r/0yrNrWI22x7GqJEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJXSkOCIiL4RMT8ifh0RD0fE6IjoFxG3RcQjxft+Rd+IiKsjYlVEPBgRwxtRsySpolF7HF8EFmbmEcBQ4GFgBnB7Zh4O3F7MA7wHOLx4TQO+Uv9yJUlb1D04IuLNwBjgOoDM3JiZ64AzgTlFtznAe4vpM4Ebs+IeoG9EHFTnsiVJhUbscRwKrAFuiIj7I+LaiOgNHJCZTxZ9ngIOKKYHAE+0W7+1aJMkNUAjgqM7MBz4Sma+A3iRVw9LAZCZCWSZjUbEtIhYEhFL1qxZs9uKlSS9ViOCoxVozcxfFvPzqQTJ01sOQRXvzxTLVwMD263fXLS9RmbOysyWzGxpamqqWfGS1NXVPTgy8yngiYh4e9F0ErASuAWYUrRNARYU07cAZxdXV40Cnm93SEuSVGfdG/RzLwBuioiewO+BqVRC7FsRcR7wB+D9Rd8fAmOBVcBLRV9JUoM0JDgy8wGgZRuLTtpG3wSm17woSVJVvHNcklRKVcEREbdX0yZJ2vO94aGqiOgF7A3sXwwBEsWiffFeCknqknZ0juO/AR8BDgaW8mpw/AmYWcO6JEkd1BsGR2Z+EfhiRFyQmV+qU02SpA6sqquqMvNLEfFOYFD7dTLzxhrVJUnqoKoKjoj4GvA24AGgrWhOwOCQpC6m2vs4WoDBxT0VkqQurNr7OJYDB9ayEElS51DtHsf+wMqIuBd4eUtjZp5Rk6okSR1WtcHxyVoWIUnqPKq9quqOWhciSeocqr2qaj2vPlipJ9ADeDEz961VYZKkjqnaPY4+W6YjIqg8B3xUrYqSJHVcpUfHzYqbgdNqUI8kqYOr9lDV+9rNvonKfR0balKRJKlDq/aqqr9rN70ZeIzK4SpJUhdT7TkOH9cqSQKqf5BTc0R8NyKeKV7/ERHNtS5OktTxVHty/AbgFirP5TgY+F7RJknqYqoNjqbMvCEzNxev2UBTDeuSJHVQ1QbHsxHxwYjoVrw+CDxby8IkSR1TtcFxLvB+4CngSWACcE6NapIkdWDVXo77aWBKZj4HEBH9gKuoBIokqQupdo9jyJbQAMjMPwLvqE1JkqSOrNrgeFNE7LdlptjjqHZvRZK0B6n2H/9/Be6OiG8X82cBn61NSZKkjqzaO8dvjIglwIlF0/syc2XtypIkdVRVH24qgsKwkKQurvSw6pKkrs3gkCSVYnBIkkoxOCRJpRgckqRSGhYcxWCJ90fE94v5QyPilxGxKiLmRUTPon2vYn5VsXxQo2qWJDV2j+NC4OF285cDX8jMw4DngPOK9vOA54r2LxT9JEkN0pDgKJ4e+LfAtcV8ULm5cH7RZQ7w3mL6zGKeYvlJRX9JUgM0ao/j34CLgVeK+f7AuszcXMy3AgOK6QHAEwDF8ueL/q8REdMiYklELFmzZk0ta5ekLq3uwRER44BnMnPp7txuZs7KzJbMbGlq8uGEklQrjRjh9jjgjIgYC/QC9gW+CPSNiO7FXkUzsLrovxoYCLRGRHfgzfj0QUlqmLrvcWTm/8nM5swcBEwEfpqZk4GfUXmyIMAUYEExfUsxT7H8p5mZdSxZktROR7qP4xLgoxGxiso5jOuK9uuA/kX7R4EZDapPkkSDH8aUmYuARcX074GR2+izgcrzPyRJHUBH2uOQJHUCBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpdQ+OiBgYET+LiJURsSIiLiza+0XEbRHxSPG+X9EeEXF1RKyKiAcjYni9a5YkvaoRexybgf+dmYOBUcD0iBgMzABuz8zDgduLeYD3AIcXr2nAV+pfsiRpi7oHR2Y+mZnLiun1wMPAAOBMYE7RbQ7w3mL6TODGrLgH6BsRB9W5bElSoaHnOCJiEPAO4JfAAZn5ZLHoKeCAYnoA8ES71VqLNklSAzQsOCJiH+A/gI9k5p/aL8vMBLLk9qZFxJKIWLJmzZrdWKkkqb2GBEdE9KASGjdl5neK5qe3HIIq3p8p2lcDA9ut3ly0vUZmzsrMlsxsaWpqql3xktTFNeKqqgCuAx7OzM+3W3QLMKWYngIsaNd+dnF11Sjg+XaHtCRJdda9AT/zOOBDwEMR8UDR9n+BzwHfiojzgD8A7y+W/RAYC6wCXgKm1rdcSVJ7dQ+OzLwLiO0sPmkb/ROYXtOiJElV885xSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJK6d7oAtTxPf7pYxpdQodxyCceanQJUsO5xyFJKsXgkCSV0mkOVUXE6cAXgW7AtZn5uQaXJKnBPIz6qnoeRu0UexwR0Q34MvAeYDAwKSIGN7YqSeqaOkVwACOBVZn5+8zcCHwTOLPBNUlSl9RZgmMA8ES7+daiTZJUZ53mHMeORMQ0YFox+0JE/KaR9exJ/hr2B9Y2uo4O4bJodAVqx+9mO7vnu/nX1XTqLMGxGhjYbr65aNsqM2cBs+pZVFcREUsys6XRdUiv53ezMTrLoar7gMMj4tCI6AlMBG5pcE2S1CV1ij2OzNwcEf8D+BGVy3Gvz8wVDS5LkrqkThEcAJn5Q+CHja6ji/IQoDoqv5sNEJnZ6BokSZ1IZznHIUnqIAyOLi4i+kbEP+7kuh+JiL13d01SGRExKCL+YSfXW16LmvZ0Bof6AjsVHMBHAINDjTYI2GZwRESnOY/bmfhL1eeAt0XEA8BtwDPA+4G9gO9m5mUR0Rv4FpX7Z7oB/wwcABwM/Cwi1mbm3zSkenV6EXE28DEggQeBfwKup3Jz3xpgamY+HhGzgT8BLcCBwMWZOZ/Kd/jI4js8B3gOeB+wD9AtIk4ArqAy1l0Cn8nMeXX7gHsgg0MzgKMzc1hEnApMoDI2WAC3RMQYoAn4z8z8W4CIeHNmPh8RHwX+JjO9c1c7JSKOAj4OvDMz10ZEPyr/+M/JzDkRcS5wNfDeYpWDgHcBR1C5l2s+le/wxzJzXLHNc4DhwJDM/GNE/D0wDBhKJYzui4g76/UZ90QeqlJ7pxav+4FlVP5yHg48BJwSEZdHxLsz8/kG1qg9y4nAt7f85yMz/wiMBr5RLP8alaDY4ubMfCUzV1LZ692e24ptUaw/NzPbMvNp4A7g2N35Iboa9zjUXgD/kpn/7y8WRAwHxgKfiYjbM/PTda9OgpfbTb/R4Ewv1rqQrsw9Dq0H+hTTPwLOjYh9ACJiQES8JSIOBl7KzK8DV1I5DPD6daWd8VPgrIjoD1AcqvoFlWGFACYDP9/BNnb0Pfw58IGI6BYRTcAY4N5dqrqLc4+ji8vMZyNicXFZ4q1UDhHcHREALwAfBA4DroyIV4BNwH8vVp8FLIyI//TkuHZGZq6IiM8Cd0REG5XDpBcAN0TERRQnx3ewmQeBtoj4FTCbysnx9r5L5fDXr6icHL84M5+KiEG763N0Nd45LkkqxUNVkqRSDA5JUikGhySpFINDklSKwSFJKsXgkCSVYnBINeTorNoTGRzSLoiIf4qI30TEXRExNyI+FhGLIuLfImIJcGHx3IefRsSDEXF7RBxSrDs7Iia029YLxfsJEXFnRPyg2PZXI8K/q+ow/DJKOykijgX+nsqoq++hMtz3Fj0zsyUz/xX4EpXRXocAN1EZ7XVHRlK5g3ow8DYqw4RLHYLBIe2844AFmbkhM9cD32u3rP3zHt5otNftuTczf5+ZbcDcKteR6sLgkGqjmtFZN1P8HSwORfVst+z1YwE5NpA6DIND2nmLgb+LiF7FiMLjttNve6O9PgaMKKbPAHq0W2dkRBxaBMoHgLt2Z+HSrvCKD2knZeZ9EXELldFZn6bywKttPeRqe6O9XgMsKEZ1Xchr91LuA2ZSGZn4Z1RGeJU6BEfHlXZBROyTmS9ExN7AncC0zFy2i9s8gXaPQpU6Gvc4pF0zKyIGA72oXDm1S6EhdQbucUiSSvHkuCSpFINDklSKwSFJKsXgkCSVYnBIkkoxOCRJpfx/5lPLNx8P8aUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=\"group\", hue=\"white_converted\", data=df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd73b21e588>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGcdJREFUeJzt3XuYVmW9//H3Vw7yM1SOmTIolF47QcUQiFIRZYeGB8wTmm0VD/zaWj/Lbep2m6bV3paV20NXbpIMy8A0TfKAkYqYpgZ5RkvyAEMeEFGBNibw/f3xLMZBQWbBPPPMwPt1XXPNWve613q+Aw98Zt1rPfeKzESSpKbarNYFSJLaFoNDklSKwSFJKsXgkCSVYnBIkkoxOCRJpRgckqRSDA5JUikGhySplPa1LqAaevTokX369Kl1GZLUpsyaNeu1zOy5rn4bZXD06dOHmTNn1roMSWpTIuLFpvRzqEqSVErVgiMifhIRr0bEk43aukXEtIh4tvjetWiPiLg8IuZExOMRMbDRPscX/Z+NiOOrVa8kqWmqecbxU+CA97SdA9yVmTsBdxXrAJ8Fdiq+xgE/gkrQABcAnwSGABesChtJUm1U7RpHZs6IiD7vaR4NDC+WJwLTgbOL9muzMsf7gxHRJSK2LfpOy8zXASJiGpUwmlStuiVtuHfeeYf6+nqWLVtW61K0Bp06daKuro4OHTqs1/4tfXF8m8x8qVh+GdimWO4FzGvUr75oW1u7pFasvr6eLbfckj59+hARtS5HjWQmCxcupL6+nr59+67XMWp2cbw4u2i2p0hFxLiImBkRMxcsWNBch5W0HpYtW0b37t0NjVYoIujevfsGnQ22dHC8UgxBUXx/tWifD/Ru1K+uaFtb+/tk5vjMHJSZg3r2XOdtyJKqzNBovTb076alg2MKsOrOqOOBWxq1H1fcXTUUeLMY0roTGBkRXYuL4iOLNklSjVTtGkdETKJycbtHRNRTuTvqYuCXEXES8CJwVNH9dmAUMAf4OzAWIDNfj4hvAn8s+l206kK5JKk2qnlX1TFr2TRiDX0TOG0tx/kJ8JNmLE1q0/b42rW1LmGdLvnczuS811Zrmz9vLqeOPZZbfnffau1XfP9iBg35FJ/ae5/Sr9Ovd49S/U8++WTOOOMM+vXrV/q1am369Ol07NiRT3/606X2WzWTRo8e5f6sPshGOeWIpLbjy/92zro7NZOrr766xV6rOS1fvpzp06fTuXPn0sFRDU45IqnFrFy5gvPP+iqHjNiLU449kmXL/pdzz/gSd942BYAf/NdFHLzfnnxu5D5c8q0LADj3jC9x4b+fyVEH/jOj9vkk03/3W6ByBrP33nszcOBABg4cyAMPPABUfjMfPnw4RxxxBB//+Mc59thjqQxqwPDhwxvmsZs6dSoDBw5kwIABjBjxvoGQBkuWLGHs2LHsuuuu7LbbbvzqV78CYNKkSey6667ssssunH322Q39O3fuzH/8x38wYMAAhg4dyiuvvMKbb77JDjvswMqVKwFYunQpvXv35p133uGvf/0rBxxwAHvssQd77703zzzzDAAnnHACX/ziF/nkJz/JUUcdxVVXXcWll17K7rvvzn333ceCBQs4/PDDGTx4MIMHD+b+++8HYOHChYwcOZL+/ftz8sknN/zszckzjrVoC8MBLWXWJcfVugRtJF58/jm+e8X/cNF3L+WMfz2Jabff2rDtjUWvc9edt3PrPX8gInjrzTcbts2vn8vk3/yWeS8+z9gxn+NTez1Mtx49mDZtGp06deLZZ5/lmGOOaQiFRx55hKeeeortttuOPffck/vvv5+99tqr4XgLFizglFNOYcaMGfTt25fXX1/7pdNvfvObbL311jzxxBMALFq0iL/97W+cffbZzJo1i65duzJy5Eh+/etfc+ihh7J06VKGDh3Kt7/9bc466yx+/OMfc95557H77rtz7733su+++3Lrrbey//7706FDB8aNG8dVV13FTjvtxEMPPcSpp57K3XffDVQ+D/PAAw/Qrl07vvGNb9C5c2fOPPNMAD7/+c/z1a9+lb322ou5c+ey//778/TTT3PhhRey1157cf7553PbbbcxYcKE5vsLLBgcklpMr97bs3P/XQHot+sA5tfPbdjWecut6Lh5J77+tdPZZ8RI9hkxsmHbAQeNZrPNNmOHvh+jbvsdeO6vz1LXewdOOeUUHn30Udq1a8df/vKXhv5Dhgyhrq4OgN13350XXnhhteB48MEHGTZsWMMH4Lp167bWmn/3u98xefLkhvWuXbsyY8YMhg8fzqpb/4899lhmzJjBoYceSseOHTnooIMA2GOPPZg2bRoAY8aM4frrr2ffffdl8uTJnHrqqSxZsoQHHniAI488suH4b7/9dsPykUceSbt27dZa1+zZsxvW33rrLZYsWcKMGTO46aabADjwwAPp2rX5Z2kyOCS1mI4dN29YbteuHW83+hBa+/btuX7KnTx4/wx+e/tv+MXECVwz+Wbg/Z87iAiuvfoqttlmGx577DFWrlxJp06dGrZvvvnqr7N8+fJq/Ujv06FDh4Z6G7/2IYccwrnnnsvrr7/OrFmz2G+//Vi6dCldunTh0UcfXeOxPvShD631dVauXMmDDz642s/dUrzGIalVWLp0CYsXv8Ww/T7D2ed/iz/Pfqph2523TWHlypXMfeF56ue+SN+P7sjixW+x7bbbstlmm/Gzn/2MFStWNPm1hg4dyowZM3j++ecBPnCo6jOf+Qw//OEPG9YXLVrEkCFDuPfee3nttddYsWIFkyZNYp99PviusM6dOzN48GBOP/10DjroINq1a8dWW21F3759ueGGG4DKdCCPPfbYGvffcsstWbx4ccP6yJEjueKKKxrWV4XPsGHD+MUvfgHAHXfcwaJFiz6wrvVhcEhqFf6+ZAmnjj2Wz43ch385/CDO+vpFDdu23a6Oow8eyRePP5rz//N7bN6pE8ccN5aJEycyYMAAnnnmmQ/87fy9evbsyfjx4znssMMYMGAAY8aMWWvf8847j0WLFrHLLrswYMAA7rnnHrbddlsuvvhi9t13XwYMGMAee+zB6NGj1/m6Y8aM4ec///lqr3fdddcxYcIEBgwYQP/+/bnlllvWuO/BBx/MzTff3HBx/PLLL2fmzJnstttu9OvXj6uuugqACy64gBkzZtC/f39uuukmtt9++yb/uTRVVOOKe60NGjQoN/QJgF4cf5cXx1uXtvDevORzO/OR3us3gd57nXvGl9hnxEj2P/CQ920r+zkOvevpp59m5513Xq0tImZl5qB17esZhySpFC+OS2rV/vMHV7bI61xzzTVcdtllq7Xtueeeq13fUIXBIUnA2LFjGTt2bK3LaBMcqpIklWJwSJJKMTgkSaV4jUPSRq25b19u6u3pU6dO5fTTT2fFihWcfPLJnHNOy80CXG2ecUhSM1uxYgWnnXYad9xxB7Nnz2bSpEmrzSvV1hkcktTMHn74YXbccUc++tGP0rFjR44++ui1fiK8LTI4JKmZzZ8/n969ezes19XVMX/+/BpW1LwMDklSKQaHJDWzXr16MW/evIb1+vp6evXqVcOKmpfBIUnNbPDgwTz77LM8//zz/OMf/2Dy5Mkccsj7J2lsq7wdV9JGrRazO7dv354rr7yS/fffnxUrVnDiiSfSv3//Fq+jWgwOSaqCUaNGMWrUqFqXURUOVUmSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIq340raqM29aNdmPd725z/RpH4nnngit956Kx/+8Id58sknm7WGWvOMQ5Kq4IQTTmDq1Km1LqMqDA5JqoJhw4bRrVu3WpdRFQaHJKmUmgRHRHw1Ip6KiCcjYlJEdIqIvhHxUETMiYjrI6Jj0XfzYn1Osb1PLWqWJFW0eHBERC/g/wGDMnMXoB1wNPAd4NLM3BFYBJxU7HISsKhov7ToJ0mqkVoNVbUH/k9EtAe2AF4C9gNuLLZPBA4tlkcX6xTbR0REtGCtkqRGWvx23MycHxHfA+YC/wv8FpgFvJGZy4tu9cCqp570AuYV+y6PiDeB7sBrLVq4pDapqbfPNrdjjjmG6dOn89prr1FXV8eFF17ISSedtO4d24AWD46I6ErlLKIv8AZwA3BAMxx3HDAOYPvtt9/Qw0nSBpk0aVKtS6iaWgxV/TPwfGYuyMx3gJuAPYEuxdAVQB2w6snu84HeAMX2rYGF7z1oZo7PzEGZOahnz57V/hkkaZNVi+CYCwyNiC2KaxUjgNnAPcARRZ/jgVuK5SnFOsX2uzMzW7BeSVIjLR4cmfkQlYvcfwKeKGoYD5wNnBERc6hcw5hQ7DIB6F60nwGc09I1SyonE/z9rvXa0L+bmsxVlZkXABe8p/k5YMga+i4DjmyJuiQ1j/o3ltGt+2I6brEl3gTZumQmCxcupFOnTut9DCc5lNTsrvnDXMYCdV06Uc3ciCULqnfwjVinTp2oq6tb7/0NDknNbvHbK7h8+vNVf51ZlxxX9dfQ+zlXlSSpFINDklSKwSFJKsXgkCSVYnBIkkoxOCRJpRgckqRSDA5JUikGhySpFINDklSKwSFJKsXgkCSVYnBIkkoxOCRJpRgckqRSDA5JUikGhySpFINDklSKwSFJKsXgkCSVYnBIkkoxOCRJpRgckqRSDA5JUikGhySpFINDklSKwSFJKsXgkCSVYnBIkkoxOCRJpRgckqRSDA5JUik1CY6I6BIRN0bEMxHxdER8KiK6RcS0iHi2+N616BsRcXlEzImIxyNiYC1qliRV1OqM4zJgamZ+HBgAPA2cA9yVmTsBdxXrAJ8Fdiq+xgE/avlyJUmrtHhwRMTWwDBgAkBm/iMz3wBGAxOLbhOBQ4vl0cC1WfEg0CUitm3hsiVJhVqccfQFFgDXRMQjEXF1RHwI2CYzXyr6vAxsUyz3AuY12r++aFtNRIyLiJkRMXPBggVVLF+SNm21CI72wEDgR5n5CWAp7w5LAZCZCWSZg2bm+MwclJmDevbs2WzFSpJWV4vgqAfqM/OhYv1GKkHyyqohqOL7q8X2+UDvRvvXFW2SpBpo8eDIzJeBeRHxT0XTCGA2MAU4vmg7HrilWJ4CHFfcXTUUeLPRkJYkqYW1r9Hrfhm4LiI6As8BY6mE2C8j4iTgReCoou/twChgDvD3oq8kqUZqEhyZ+SgwaA2bRqyhbwKnVb0oSVKTNGmoKiLuakqbJGnj94FnHBHRCdgC6FF8kjuKTVuxhltiJUkbv3UNVf1f4CvAdsAs3g2Ot4Arq1iXJKmV+sDgyMzLgMsi4suZeUUL1SRJasWadHE8M6+IiE8DfRrvk5nXVqkuSVIr1aTgiIifAR8DHgVWFM0JGByStIlp6u24g4B+xa2xkqRNWFM/Of4k8JFqFiJJahuaesbRA5gdEQ8Db69qzMxDqlKVJKnVampwfKOaRUiS2o6m3lV1b7ULkSS1DU29q2ox7z4foyPQAViamVtVqzBJUuvU1DOOLVctR0RQeZzr0GoVJUlqvUo/j6N49vevgf2rUI8kqZVr6lDVYY1WN6PyuY5lValIktSqNfWuqoMbLS8HXqAyXCVJ2sQ09RqHT92TJAFNf5BTXUTcHBGvFl+/ioi6ahcnSWp9mnpx/BpgCpXncmwH/KZokyRtYpoaHD0z85rMXF58/RToWcW6JEmtVFODY2FEfCEi2hVfXwAWVrMwSVLr1NTgOBE4CngZeAk4AjihSjVJklqxpt6OexFwfGYuAoiIbsD3qASKJGkT0tQzjt1WhQZAZr4OfKI6JUmSWrOmBsdmEdF11UpxxtHUsxVJ0kakqf/5fx/4Q0TcUKwfCXy7OiVJklqzpn5y/NqImAnsVzQdlpmzq1eWJKm1avJwUxEUhoUkbeJKT6suSdq0GRySpFIMDklSKQaHJKkUg0OSVErNgqOYLPGRiLi1WO8bEQ9FxJyIuD4iOhbtmxfrc4rtfWpVsySptmccpwNPN1r/DnBpZu4ILAJOKtpPAhYV7ZcW/SRJNVKT4CieHnggcHWxHlQ+XHhj0WUicGixPLpYp9g+ougvSaqBWp1x/DdwFrCyWO8OvJGZy4v1eqBXsdwLmAdQbH+z6L+aiBgXETMjYuaCBQuqWbskbdJaPDgi4iDg1cyc1ZzHzczxmTkoMwf17OnDCSWpWmoxw+2ewCERMQroBGwFXAZ0iYj2xVlFHTC/6D8f6A3UR0R7YGt8+qAk1UyLn3Fk5r9nZl1m9gGOBu7OzGOBe6g8WRDgeOCWYnlKsU6x/e7MzBYsWZLUSGv6HMfZwBkRMYfKNYwJRfsEoHvRfgZwTo3qkyRR44cxZeZ0YHqx/BwwZA19llF5/ockqRVoTWcckqQ2wOCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqpcWDIyJ6R8Q9ETE7Ip6KiNOL9m4RMS0ini2+dy3aIyIuj4g5EfF4RAxs6ZolSe+qxRnHcuDfMrMfMBQ4LSL6AecAd2XmTsBdxTrAZ4Gdiq9xwI9avmRJ0iotHhyZ+VJm/qlYXgw8DfQCRgMTi24TgUOL5dHAtVnxINAlIrZt4bIlSYWaXuOIiD7AJ4CHgG0y86Vi08vANsVyL2Beo93qizZJUg3ULDgiojPwK+ArmflW422ZmUCWPN64iJgZETMXLFjQjJVKkhqrSXBERAcqoXFdZt5UNL+yagiq+P5q0T4f6N1o97qibTWZOT4zB2XmoJ49e1aveEnaxNXirqoAJgBPZ+YPGm2aAhxfLB8P3NKo/bji7qqhwJuNhrQkSS2sfQ1ec0/gX4AnIuLRou1c4GLglxFxEvAicFSx7XZgFDAH+DswtmXLlSQ11uLBkZm/B2Itm0esoX8Cp1W1KElSk/nJcUlSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSmlf6wLU+s29aNdal9BqbH/+E7UuQao5zzgkSaUYHJKkUtrMUFVEHABcBrQDrs7Mi2tckqQacxj1XS05jNomzjgioh3wQ+CzQD/gmIjoV9uqJGnT1CaCAxgCzMnM5zLzH8BkYHSNa5KkTVJbCY5ewLxG6/VFmySphbWZaxzrEhHjgHHF6pKI+HMt69mY7AA9gNdqXUercEHUugI14nuzkeZ5b+7QlE5tJTjmA70brdcVbQ0yczwwviWL2lRExMzMHFTrOqT38r1ZG21lqOqPwE4R0TciOgJHA1NqXJMkbZLaxBlHZi6PiC8Bd1K5HfcnmflUjcuSpE1SmwgOgMy8Hbi91nVsohwCVGvle7MGIjNrXYMkqQ1pK9c4JEmthMGxiYuILhFx6nru+5WI2KK5a5LKiIg+EfH59dzvyWrUtLEzONQFWK/gAL4CGByqtT7AGoMjItrMddy2xD9UXQx8LCIeBaYBrwJHAZsDN2fmBRHxIeCXVD4/0w74JrANsB1wT0S8lpn71qR6tXkRcRxwJpDA48DXgZ9Q+XDfAmBsZs6NiJ8CbwGDgI8AZ2XmjVTewzsX7+GJwCLgMKAz0C4ihgPfpTLXXQLfyszrW+wH3AgZHDoH2CUzd4+IkcARVOYGC2BKRAwDegJ/y8wDASJi68x8MyLOAPbNTD+5q/USEf2B84BPZ+ZrEdGNyn/+EzNzYkScCFwOHFrssi2wF/BxKp/lupHKe/jMzDyoOOYJwEBgt8x8PSIOB3YHBlAJoz9GxIyW+hk3Rg5VqbGRxdcjwJ+o/OPcCXgC+ExEfCci9s7MN2tYozYu+wE3rPrlIzNfBz4F/KLY/jMqQbHKrzNzZWbOpnLWuzbTimNR7D8pM1dk5ivAvcDg5vwhNjWecaixAP4rM//nfRsiBgKjgG9FxF2ZeVGLVyfB242WP2hypqXVLmRT5hmHFgNbFst3AidGRGeAiOgVER+OiO2Av2fmz4FLqAwDvHdfaX3cDRwZEd0BiqGqB6hMKwRwLHDfOo6xrvfhfcCYiGgXET2BYcDDG1T1Js4zjk1cZi6MiPuL2xLvoDJE8IeIAFgCfAHYEbgkIlYC7wD/Wuw+HpgaEX/z4rjWR2Y+FRHfBu6NiBVUhkm/DFwTEV+juDi+jsM8DqyIiMeAn1K5ON7YzVSGvx6jcnH8rMx8OSL6NNfPsanxk+OSpFIcqpIklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ6oiZ2fVxsjgkDZARHw9Iv4cEb+PiEkRcWZETI+I/46ImcDpxXMf7o6IxyPirojYvtj3pxFxRKNjLSm+D4+IGRFxW3HsqyLCf6tqNXwzSuspIgYDh1OZdfWzVKb7XqVjZg7KzO8DV1CZ7XU34Doqs72uyxAqn6DuB3yMyjThUqtgcEjrb0/glsxclpmLgd802tb4eQ8fNNvr2jycmc9l5gpgUhP3kVqEwSFVR1NmZ11O8W+wGIrq2Gjbe+cCcm4gtRoGh7T+7gcOjohOxYzCB62l39pme30B2KNYPgTo0GifIRHRtwiUMcDvm7NwaUN4x4e0njLzjxExhcrsrK9QeeDVmh5ytbbZXn8M3FLM6jqV1c9S/ghcSWVm4nuozPAqtQrOjittgIjonJlLImILYAYwLjP/tIHHHE6jR6FKrY1nHNKGGR8R/YBOVO6c2qDQkNoCzzgkSaV4cVySVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFL+P1c39nBIlpeoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=\"group\", hue=\"hispanic_converted\", data=df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fd73b1f2828>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGMNJREFUeJzt3XmYVuWZ5/HvHRSJK4rEGMBAkLggKZCSuIxLx3GJneDSAcyiIhp61DE6mSQyY7rtJJ1rdLCTGM3lDGkX7DCIQgskURMjajRuFAqoqBGESNGI4IK4IIL3/PEesCAgdYB63yrq+7muuuqc5yx1F77w8zzPOc+JzESSpOb6WK0LkCS1LQaHJKkUg0OSVIrBIUkqxeCQJJVicEiSSjE4JEmlGBySpFIMDklSKTvUuoCWsPfee2fPnj1rXYYktSkzZsxYlpldN7ffdhkcPXv2pKGhodZlSFKbEhF/ac5+LdZVFRE3RsQrEfF0k7a9IuKeiHih+L5n0R4R8fOImBsRsyPi0CbHnFPs/0JEnNNS9UqSmqclxzhuBk7eoG0UcG9m9gHuLdYBvgj0Kb5GAtdDJWiAK4DPA4OAK9aGjSSpNlosODLzj8BrGzSfCowtlscCpzVpvyUrHgU6R8S+wEnAPZn5Wma+DtzDX4eRJKmKqj3GsU9mLi6WXwb2KZa7AQub7NdYtG2qXdJ26v3336exsZGVK1fWupTtVqdOnejevTs77rjjFh1fs8HxzMyI2GYvA4mIkVS6udhvv/221WklVVljYyO77bYbPXv2JCJqXc52JzN59dVXaWxspFevXlt0jmo/x7Gk6IKi+P5K0b4I6NFkv+5F26ba/0pmjsnM+sys79p1s3eTSWqlVq5cSZcuXQyNFhIRdOnSZauu6KodHFOBtXdGnQNMadJ+dnF31eHA8qJL63fAiRGxZzEofmLRJmk7Zmi0rK39822xrqqIGA8cB+wdEY1U7o66ErgtIs4D/gIMLXa/EzgFmAu8A5wLkJmvRcSPgOnFfj/MzA0H3CVJVdRiwZGZX93EpuM3sm8CF23iPDcCN27D0iRJW2G7fHJc2p4N/O4ttS6hRY0+/SBy4bL12hYtfIkLz/06U/7w4HrtF571Fa6++mrq6+tL/Yybb76ZhoYGrrvuuq2ut1omT57MZz/7WQ4++OBSx+2666689dZb27QWJzmUpFZu9erVTJ48mTlz5tS6FMDgkNRGrFmzmu9967/w5S8cyaV/fy7vvvvOetsvuOAC6uvr6du3L1dcccW69unTp3PkkUdSV1fHoEGDWLFixXrH/fa3v+WII45g2bL1r3LWWrJkCaeffjp1dXXU1dXx8MMPA/CTn/yEQw45hEMOOYSf/exnACxYsICDDjqIb37zm/Tt25cTTzyRd999l+eee45BgwatO+eCBQvo168fADNmzODYY49l4MCBnHTSSSxeXHnU7bjjjuPSSy+lvr6eq666iqlTp/Ld736X/v37M2/ePObNm8fJJ5/MwIEDOfroo3nuuecAmD9/PkcccQT9+vXj+9///tb8kW+SwSGpTZg/by5nnnUuv572MLvuthu33nLTett//OMf09DQwOzZs3nggQeYPXs2q1atYtiwYVxzzTXMmjWLP/zhD3z84x9fd8wdd9zBlVdeyZ133snee++90Z/7rW99i2OPPZZZs2bxxBNP0LdvX2bMmMFNN93EY489xqOPPsovf/lLnnzySQBeeOEFLrroIp555hk6d+7MpEmTOPDAA1m1ahXz588HYMKECQwbNoz333+fiy++mIkTJzJjxgxGjBjB5Zdfvu5nr1q1ioaGBi6//HIGDx7M6NGjmTlzJr1792bkyJFce+21zJgxg6uvvpoLL7wQgEsuuYQLLriAp556in333Xeb/jdYyzGOTdje+5HLmDH67FqXIPHJT3Xj0MM+D8CXTh/CuJt+ud722267jTFjxrB69WoWL17MnDlziAj23XdfDjvsMAB23333dftPmzaNhoYGfv/736/XvqFp06Zxyy2Vfw86dOjAHnvswUMPPcTpp5/OLrvsAsAZZ5zBgw8+yODBg+nVqxf9+/cHYODAgSxYsACAoUOHMmHCBEaNGsWECROYMGECzz//PE8//TQnnHACAGvWrFnvH/thw4ZttKa33nqLhx9+mCFDhqxre++99wD405/+xKRJkwA466yzuOyyyz7qj3WLGByS2oQNnz1ouj5//nyuvvpqpk+fzp577snw4cM3+4Bb7969efHFF/nzn/9cenD9o+y0007rljt06MC7774LVEJgyJAhnHHGGUQEffr04amnnqJv37488sgjGz3X2mDa0AcffEDnzp2ZOXPmRre39HMwdlVJahMWL2pk5ozKI113Tpm07uoD4M0332SXXXZhjz32YMmSJdx1110AHHDAASxevJjp0yvHrVixgtWrVwPw6U9/mkmTJnH22WfzzDPPbPLnHn/88Vx//fVA5Ypg+fLlHH300UyePJl33nmHt99+mzvuuIOjjz76I+vv3bs3HTp04Ec/+tG6K4kDDjiApUuXrguO999/f5O17LbbbuvGZ3bffXd69erF7bffDlSmEZk1axYARx11FLfeeisA48aN+8iatpTBIalN6NV7f8bfciNf/sKRLF/+BsPOGr5uW11dHQMGDODAAw/ka1/7GkcddRQAHTt2ZMKECVx88cXU1dVxwgknrHclcuCBBzJu3DiGDBnCvHnzNvpzr7nmGu677z769evHwIEDmTNnDoceeijDhw9n0KBBfP7zn+f8889nwIABm/0dhg0bxq9+9SuGDh26rr6JEydy2WWXUVdXR//+/dcNvm/ozDPPZPTo0QwYMIB58+Yxbtw4brjhBurq6ujbty9TpkxZV+8vfvEL+vXrx6JFG52haatF5dm77Ut9fX1u7RsAHeP4kGMcrcv2/tkcffpBfLJH8ybfO7jHxge0tXnPPvssBx100HptETEjMzfbb+cVhySpFAfHJYnK7bxrxwzWGjJkyHq3x6rC4JAk4PLLLzckmsmuKklSKQaHJKkUg0OSVIpjHJLatW19e3Nzb1+/++67ueSSS1izZg3nn38+o0aN2qZ1tCSvOCSpytasWcNFF13EXXfdxZw5cxg/fnyrmTK9OQwOSaqyxx9/nP3335/PfOYzdOzYkTPPPHPdk99tgcEhSVW2aNEievTosW69e/fuLTY9SEswOCRJpRgcklRl3bp1Y+HChevWGxsb6datWw0rKsfgkKQqO+yww3jhhReYP38+q1at4tZbb2Xw4MG1LqvZvB1XUrtWi9mfd9hhB6677jpOOukk1qxZw4gRI+jbt2/V69hSBock1cApp5zCKaecUusytohdVZKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAkleLtuJLatZd+2G+bnm+/f3xqs/uMGDGC3/zmN3ziE5/g6aef3qY/vxq84pCkKhs+fDh33313rcvYYgaHJFXZMcccw1577VXrMraYwSFJKqUmwRER/y0inomIpyNifER0ioheEfFYRMyNiAkR0bHYd6difW6xvWctapYkVVQ9OCKiG/AtoD4zDwE6AGcCVwE/zcz9gdeB84pDzgNeL9p/WuwnSaqRWnVV7QB8PCJ2AHYGFgNfACYW28cCpxXLpxbrFNuPj4ioYq2SpCaqfjtuZi6KiKuBl4B3gd8DM4A3MnN1sVsjsPatJt2AhcWxqyNiOdAFWNb0vBExEhgJsN9++7X0ryFpO9Gc22e3ta9+9avcf//9LFu2jO7du/ODH/yA8847b/MHthJVD46I2JPKVUQv4A3gduDkrT1vZo4BxgDU19fn1p5PklrK+PHja13CVqlFV9V/BuZn5tLMfB/4d+AooHPRdQXQHVj75vZFQA+AYvsewKvVLVmStFYtguMl4PCI2LkYqzgemAPcB3yl2OccYEqxPLVYp9g+LTO9opCkGql6cGTmY1QGuZ8AnipqGANcBnw7IuZSGcO4oTjkBqBL0f5tYFS1a5ZUPZng/xu2rK39863JXFWZeQVwxQbNLwKDNrLvSmBINeqSVHuNb6xkry4r6LjzbngD5baXmbz66qt06tRpi8/hJIeSWpWbHnmJc4HunTuxudyIt5ZWpabtTadOnejevfsWH29wSGpVVry3hp/fP79Z+84YfXYLV6ONca4qSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaXUJDgionNETIyI5yLi2Yg4IiL2ioh7IuKF4vuexb4RET+PiLkRMTsiDq1FzZKkilpdcVwD3J2ZBwJ1wLPAKODezOwD3FusA3wR6FN8jQSur365kqS1qh4cEbEHcAxwA0BmrsrMN4BTgbHFbmOB04rlU4FbsuJRoHNE7FvlsiVJhVpccfQClgI3RcSTEfGvEbELsE9mLi72eRnYp1juBixscnxj0SZJqoFaBMcOwKHA9Zk5AHibD7ulAMjMBLLMSSNiZEQ0RETD0qVLt1mxkqT11SI4GoHGzHysWJ9IJUiWrO2CKr6/UmxfBPRocnz3om09mTkmM+szs75r164tVrwktXdVD47MfBlYGBEHFE3HA3OAqcA5Rds5wJRieSpwdnF31eHA8iZdWpKkKtuhRj/3YmBcRHQEXgTOpRJit0XEecBfgKHFvncCpwBzgXeKfSVJNVKT4MjMmUD9RjYdv5F9E7ioxYuSJDWLT45LkkppVnBExL3NaZMkbf8+sqsqIjoBOwN7F1OARLFpd3yWQpLapc2Ncfw9cCnwKWAGHwbHm8B1LViXJKmV+sjgyMxrgGsi4uLMvLZKNUmSWrFm3VWVmddGxJFAz6bHZOYtLVSXJKmValZwRMS/Ab2BmcCaojkBg0OS2pnmPsdRDxxcPFMhSWrHmvscx9PAJ1uyEElS29DcK469gTkR8Tjw3trGzBzcIlVJklqt5gbHP7VkEZKktqO5d1U90NKFSJLahubeVbWCD1+s1BHYEXg7M3dvqcIkSa1Tc684dlu7HBFB5T3gh7dUUZKk1qv07LhZMRk4qQXqkSS1cs3tqjqjyerHqDzXsbJFKpIktWrNvavqy02WVwMLqHRXSZLameaOcfi6VkkS0PwXOXWPiDsi4pXia1JEdG/p4iRJrU9zB8dvAqZSeS/Hp4BfF22SpHamucHRNTNvyszVxdfNQNcWrEuS1Eo1NzhejYhvRESH4usbwKstWZgkqXVqbnCMAIYCLwOLga8Aw1uoJklSK9bc23F/CJyTma8DRMRewNVUAkWS1I4094rjc2tDAyAzXwMGtExJkqTWrLnB8bGI2HPtSnHF0dyrFUnSdqS5//j/C/BIRNxerA8BftwyJUmSWrPmPjl+S0Q0AF8oms7IzDktV5YkqbVqdndTERSGhSS1c6WnVZcktW8GhySpFINDklSKwSFJKqVmwVHMefVkRPymWO8VEY9FxNyImBARHYv2nYr1ucX2nrWqWZJU2yuOS4Bnm6xfBfw0M/cHXgfOK9rPA14v2n9a7CdJqpGaBEfxEqi/Bf61WA8qz4hMLHYZC5xWLJ9arFNsP77YX5JUA7W64vgZ8D3gg2K9C/BGZq4u1huBbsVyN2AhQLF9ebG/JKkGqh4cEfEl4JXMnLGNzzsyIhoiomHp0qXb8tSSpCZqccVxFDA4IhYAt1LporoG6BwRa59k7w4sKpYXAT0Aiu17sJGXSGXmmMysz8z6rl19OaEktZSqB0dm/o/M7J6ZPYEzgWmZ+XXgPioviAI4B5hSLE8t1im2T8vMrGLJkqQmWtNzHJcB346IuVTGMG4o2m8AuhTt3wZG1ag+SRI1fqdGZt4P3F8svwgM2sg+K6lM4y5JagVa0xWHJKkNMDgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaVUPTgiokdE3BcRcyLimYi4pGjfKyLuiYgXiu97Fu0RET+PiLkRMTsiDq12zZKkD9XiimM18N8z82DgcOCiiDgYGAXcm5l9gHuLdYAvAn2Kr5HA9dUvWZK0VtWDIzMXZ+YTxfIK4FmgG3AqMLbYbSxwWrF8KnBLVjwKdI6IfatctiSpUNMxjojoCQwAHgP2yczFxaaXgX2K5W7AwiaHNRZtG55rZEQ0RETD0qVLW6xmSWrvahYcEbErMAm4NDPfbLotMxPIMufLzDGZWZ+Z9V27dt2GlUqSmqpJcETEjlRCY1xm/nvRvGRtF1Tx/ZWifRHQo8nh3Ys2SVIN1OKuqgBuAJ7NzJ802TQVOKdYPgeY0qT97OLuqsOB5U26tCRJVbZDDX7mUcBZwFMRMbNo+5/AlcBtEXEe8BdgaLHtTuAUYC7wDnBudcuVJDVV9eDIzIeA2MTm4zeyfwIXtWhRkqRm88lxSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSJJKMTgkSaUYHJKkUgwOSVIpBockqRSDQ5JUisEhSSplh1oXoNbvpR/2q3UJrcZ+//hUrUuQas4rDklSKQaHJKkUg0OSVIpjHJLaLMffPlTN8bc2c8URESdHxPMRMTciRtW6Hklqr9pEcEREB+AXwBeBg4GvRsTBta1KktqnNhEcwCBgbma+mJmrgFuBU2tckyS1S20lOLoBC5usNxZtkqQq224GxyNiJDCyWH0rIp6vZT3bk0/D3sCyWtfRKlwRta5ATfjZbGLbfDY/3Zyd2kpwLAJ6NFnvXrStk5ljgDHVLKq9iIiGzKyvdR3Shvxs1kZb6aqaDvSJiF4R0RE4E5ha45okqV1qE1ccmbk6Iv4r8DugA3BjZj5T47IkqV1qE8EBkJl3AnfWuo52yi5AtVZ+NmsgMrPWNUiS2pC2MsYhSWolDI52LiI6R8SFW3jspRGx87auSSojInpGxNe28LinW6Km7Z3Boc7AFgUHcClgcKjWegIbDY6IaDPjuG2Jf6i6EugdETOBe4BXgKHATsAdmXlFROwC3Ebl+ZkOwI+AfYBPAfdFxLLM/JuaVK82LyLOBr4DJDAb+AfgRioP9y0Fzs3MlyLiZuBNoB74JPC9zJxI5TN8UPEZHgu8DpwB7Ap0iIjjgP9NZa67BP45MydU7RfcDhkcGgUckpn9I+JE4CtU5gYLYGpEHAN0Bf4jM/8WICL2yMzlEfFt4G8y0yd3tUUioi/wfeDIzFwWEXtR+cd/bGaOjYgRwM+B04pD9gX+E3AglWe5JlL5DH8nM79UnHM4cCjwucx8LSL+DugP1FEJo+kR8cdq/Y7bI7uq1NSJxdeTwBNU/nL2AZ4CToiIqyLi6MxcXsMatX35AnD72v/5yMzXgCOA/1ds/zcqQbHW5Mz8IDPnULnq3ZR7inNRHD8+M9dk5hLgAeCwbflLtDdecaipAP5XZv7fv9oQcShwCvDPEXFvZv6w6tVJ8F6T5Y+anOntli6kPfOKQyuA3Yrl3wEjImJXgIjoFhGfiIhPAe9k5q+A0VS6ATY8VtoS04AhEdEFoOiqepjKtEIAXwce3Mw5Nvc5fBAYFhEdIqIrcAzw+FZV3c55xdHOZearEfGn4rbEu6h0ETwSEQBvAd8A9gdGR8QHwPvABcXhY4C7I+I/HBzXlsjMZyLix8ADEbGGSjfpxcBNEfFdisHxzZxmNrAmImYBN1MZHG/qDirdX7OoDI5/LzNfjoie2+r3aG98clySVIpdVZKkUgwOSVIpBockqRSDQ5JUisEhSSrF4JAklWJwSC3I2Vm1PTI4pK0QEf8QEc9HxEMRMT4ivhMR90fEzyKiAbikeO/DtIiYHRH3RsR+xbE3R8RXmpzrreL7cRHxx4j4bXHu/xMR/l1Vq+GHUdpCEXEY8HdUZl39IpXpvtfqmJn1mfkvwLVUZnv9HDCOymyvmzOIyhPUBwO9qUwTLrUKBoe05Y4CpmTmysxcAfy6ybam73v4qNleN+XxzHwxM9cA45t5jFQVBofUMpozO+tqir+DRVdUxybbNpwLyLmB1GoYHNKW+xPw5YjoVMwo/KVN7Lep2V4XAAOL5cHAjk2OGRQRvYpAGQY8tC0Ll7aGd3xIWygzp0fEVCqzsy6h8sKrjb3kalOzvf4SmFLM6no361+lTAeuozIz8X1UZniVWgVnx5W2QkTsmplvRcTOwB+BkZn5xFae8ziavApVam284pC2zpiIOBjoROXOqa0KDakt8IpDklSKg+OSpFIMDklSKQaHJKkUg0OSVIrBIUkqxeCQJJXy/wFibbsiHAut6QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=\"group\", hue=\"black_converted\", data=df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "From these graphs it looks like all of the proportions are the same.  Next let's look at the raw counts and percentages to get a better sense of this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prob(Converted | Asian) = 0.0385\n",
      "Prob(Converted | White) = 0.026\n",
      "Prob(Converted | Black) = 0.032\n",
      "Prob(Converted | Hispanic) = 0.026\n"
     ]
    }
   ],
   "source": [
    "print(\"Prob(Converted | Asian) =\", df[\"asian_converted\"].sum()/len(df))\n",
    "print(\"Prob(Converted | White) =\", df[\"white_converted\"].sum()/len(df))\n",
    "print(\"Prob(Converted | Black) =\",df[\"black_converted\"].sum()/len(df))\n",
    "print(\"Prob(Converted | Hispanic) =\", df[\"hispanic_converted\"].sum()/len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "As you can see, the probabilities confirm that the probability of converting is higher assuming the person is Asian.  Just to be sure, let's rerun the test with Asian removed to ensure there is actually a difference between test and control across all demographics.  It could be the case that the difference is being driven just by Asian people who viewed the advertisement, in which case we could only segment the change by this demographic to capture all the gains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18.605286113439544, 0.13585945130256297)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demographics_no_asians = demographics[:]\n",
    "demographics_no_asians.remove(\"asian\")\n",
    "table = generate_contingency_table(df, demographics_no_asians)\n",
    "table_transpose = table.T\n",
    "chi2, pvalue, degrees_of_freedom, expected = chi2_contingency(table_transpose)\n",
    "\n",
    "chi2, pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "It seems as though the p value has fallen but it is still above the 95% confidence interval so we fail to reject the null hypothesis.  Let's look at the contingency table just to verify this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>male</th>\n",
       "      <th>female</th>\n",
       "      <th>18-24</th>\n",
       "      <th>24-35</th>\n",
       "      <th>35-50</th>\n",
       "      <th>50-70</th>\n",
       "      <th>70+</th>\n",
       "      <th>white</th>\n",
       "      <th>black</th>\n",
       "      <th>hispanic</th>\n",
       "      <th>high</th>\n",
       "      <th>medium</th>\n",
       "      <th>low</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>92</td>\n",
       "      <td>499.0</td>\n",
       "      <td>501.0</td>\n",
       "      <td>216</td>\n",
       "      <td>202</td>\n",
       "      <td>206</td>\n",
       "      <td>195</td>\n",
       "      <td>181</td>\n",
       "      <td>251</td>\n",
       "      <td>250</td>\n",
       "      <td>234</td>\n",
       "      <td>319.0</td>\n",
       "      <td>311.0</td>\n",
       "      <td>370.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>153</td>\n",
       "      <td>488.0</td>\n",
       "      <td>512.0</td>\n",
       "      <td>192</td>\n",
       "      <td>204</td>\n",
       "      <td>203</td>\n",
       "      <td>199</td>\n",
       "      <td>202</td>\n",
       "      <td>242</td>\n",
       "      <td>242</td>\n",
       "      <td>232</td>\n",
       "      <td>318.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>358.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         converted   male  female  18-24  24-35  35-50  50-70  70+  white  \\\n",
       "group                                                                       \n",
       "control         92  499.0   501.0    216    202    206    195  181    251   \n",
       "test           153  488.0   512.0    192    204    203    199  202    242   \n",
       "\n",
       "         black  hispanic   high  medium    low  \n",
       "group                                           \n",
       "control    250       234  319.0   311.0  370.0  \n",
       "test       242       232  318.0   324.0  358.0  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Clearly the control converted is far below test, even without the asians who have the highest conversion rate.  Therefore this campaign definitely would improve from the change.  Of course, in the real world at least a few more evaluations should occur using the above analysis on some validation sets before you can or should conclude anything.  But this should give you some confidence that this change might be a good one!\n",
    "\n",
    "## Dealing With Continuous Data\n",
    "\n",
    "So far we've assumed that our data was entirely categorical.  But what if our scenario was changed slightly.  First, let's talk about what is the same:\n",
    "\n",
    "* We have a test and control group, representing the change to an email for a sale\n",
    "* We still have information about our potential customers\n",
    "\n",
    "The difference is going to be the kind of information we have.  Specifically this time we'll have the following variables:\n",
    "\n",
    "* Age - as an integer\n",
    "* Salary\n",
    "\n",
    "Now let's go ahead and generate our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "\n",
    "test_size = 1000\n",
    "control_size = 1000\n",
    "test_probability = 0.15\n",
    "control_probability = 0.1\n",
    "test_dist = stats.bernoulli(test_probability)\n",
    "control_dist = stats.bernoulli(control_probability)\n",
    "\n",
    "test_df = pd.DataFrame()\n",
    "test_df[\"converted\"] = test_dist.rvs(size=test_size)\n",
    "test_df[\"group\"] = \"test\"\n",
    "control_df = pd.DataFrame()\n",
    "control_df[\"converted\"] = control_dist.rvs(size=control_size)\n",
    "control_df[\"group\"] = \"control\"\n",
    "df = test_df.append(control_df)\n",
    "df[\"Age\"] = np.random.normal(50, 25, size=len(df))\n",
    "df[\"Age\"] = df[\"Age\"].astype(int)\n",
    "df[\"Salary\"] = np.random.normal(75000, 15000, size=len(df))\n",
    "df[\"Salary\"] = df[\"Salary\"].apply(lambda x: round(x, 2))\n",
    "\n",
    "test= df[df[\"group\"] == \"test\"]\n",
    "control = df[df[\"group\"] == \"control\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>converted</th>\n",
       "      <th>group</th>\n",
       "      <th>Age</th>\n",
       "      <th>Salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>test</td>\n",
       "      <td>66</td>\n",
       "      <td>66084.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>test</td>\n",
       "      <td>40</td>\n",
       "      <td>65270.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>test</td>\n",
       "      <td>15</td>\n",
       "      <td>84882.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>test</td>\n",
       "      <td>70</td>\n",
       "      <td>85411.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>test</td>\n",
       "      <td>51</td>\n",
       "      <td>76544.84</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   converted group  Age    Salary\n",
       "0          0  test   66  66084.42\n",
       "1          1  test   40  65270.42\n",
       "2          1  test   15  84882.97\n",
       "3          1  test   70  85411.60\n",
       "4          0  test   51  76544.84"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Notice that we convert `Age` to only include the years, that `Salary` is rounded to two decimal places and that both of these variables are normally distributed.  \n",
    "\n",
    "Notice also, that we assume that our population of `Age` and `Salary` is homogenous.  This means, that both of our samples must come from the same population of people.  It may be the case that in fact different shape parameters capture the distribution of those who choose to convert or those who choose not to.  We will model that explicitly next!  But first, let's carry out our hypothesis test!\n",
    "\n",
    "For this set up, we'll make use of the one-way ANOVA test, via the F-test.  Before we carry out the test, let's just check to see if there is a difference in the percentage of those who converted in test and in control:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.845\n",
       "1    0.155\n",
       "Name: converted, dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test[\"converted\"].value_counts()/len(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.885\n",
       "1    0.115\n",
       "Name: converted, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "control[\"converted\"].value_counts()/len(control)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Looks like there is some difference, with test converting at 15% whereas control converts with 11%.  Let's see if Age can help us to explain this difference with an F-test!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "F_onewayResult(statistic=0.4133544105058655, pvalue=0.5208215094686701)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "test_converted = test[test[\"converted\"] == 1]\n",
    "control_converted = control[control[\"converted\"] == 1]\n",
    "stats.f_oneway(test_converted[\"Age\"], control_converted[\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Recall the null and alternative hypotheses for ANOVA:\n",
    "\n",
    "* Null Hypothesis: the two samples have the same population mean\n",
    "* Alternative Hypothesis: the two samples have different population mean's and are therefore different.\n",
    "\n",
    "Going through the set up:\n",
    "\n",
    "First we look at those in the test and control set that converted.  We want to know if Age is a factor as to why they may have converted.  Because the p value is 0.5, we are far above most thresholds.  So we fail to reject the null hypothesis.  Therefore, Age is not a factor as to whether or not someone converted and bought something due to the change in language in the email.\n",
    "\n",
    "Going a bit deeper into the set up:\n",
    "\n",
    "The reason we first segregated our data into those who converted, is because that is our dependent variable.  We are explicitly trying to understand who converted and why.  Since we don't explicitly control for age, we take it as our independent variable and therefore it is what we run our analysis over.  Since we don't control for it explicitly, we want to know if it makes a difference.\n",
    "\n",
    "Next let's look at Salary to see if this affects anything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "F_onewayResult(statistic=0.13355076287967246, pvalue=0.7150667844356584)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "test_converted = test[test[\"converted\"] == 1]\n",
    "control_converted = control[control[\"converted\"] == 1]\n",
    "stats.f_oneway(test_converted[\"Salary\"], control_converted[\"Salary\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Here we even more strongly fail to reject the null hypothesis, therefore Salary doesn't effect whether someone converted or not.  \n",
    "\n",
    "All of this makes sense because our data came from the same population for both Age and Salary.  Now, let's try regenerating our data, except let's assume that our salary ranges will be different!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "test= df[df[\"group\"] == \"test\"]\n",
    "control = df[df[\"group\"] == \"control\"]\n",
    "\n",
    "test_salary_mean = 150000\n",
    "test_salary_variance = 30000 \n",
    "\n",
    "control_salary_mean = 55000\n",
    "control_salary_variance = 2000\n",
    "\n",
    "test[\"Age\"] = np.random.normal(50, 25, size=len(test))\n",
    "test[\"Age\"] = test[\"Age\"].astype(int)\n",
    "test[\"Salary\"] = np.random.normal(test_salary_mean, test_salary_variance, size=len(test))\n",
    "test[\"Salary\"] = test[\"Salary\"].apply(lambda x: round(x, 2))\n",
    "\n",
    "control[\"Age\"] = np.random.normal(50, 25, size=len(control))\n",
    "control[\"Age\"] = control[\"Age\"].astype(int)\n",
    "control[\"Salary\"] = np.random.normal(control_salary_mean, control_salary_variance, size=len(control))\n",
    "control[\"Salary\"] = control[\"Salary\"].apply(lambda x: round(x, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Notice that we've made a very big shift in test and control salary, but kept Age the same.  Let's see how this effects our hypothesis tests!  First let's look at Age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "F_onewayResult(statistic=1.6348240431702024, pvalue=0.20214323774160542)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "test_converted = test[test[\"converted\"] == 1]\n",
    "control_converted = control[control[\"converted\"] == 1]\n",
    "stats.f_oneway(test_converted[\"Age\"], control_converted[\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Here again we fail to reject the null hypothesis, because our shape parameters didn't change.  Therefore both distributions should be from the same population.  Now let's look at salary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "F_onewayResult(statistic=1043.8123596703554, pvalue=2.0508424416521246e-94)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "test_converted = test[test[\"converted\"] == 1]\n",
    "control_converted = control[control[\"converted\"] == 1]\n",
    "stats.f_oneway(test_converted[\"Salary\"], control_converted[\"Salary\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "As you can see, now we strongly reject the null hypothesis, so clearly Salary has an effect!  This is probably pretty close to the how the real world works - one's salary will determine how likely they are to buy something.  Since the `Salary` in general was higher for those in the test set, it makes way more sense they will convert and purchase something.  This means that in this case, we can conclude that our change to the email was effective at all.  Therefore, we should explicitly control for `Salary` in future experiments.\n",
    "\n",
    "How can we do that?  \n",
    "\n",
    "One strategy is by explicitly forcing some of those in the higher salary category and some of those in the lower salary category into each group.  Then we can use statistical matching to the balance the dataset to ensure high wage earners and low wage earners are both represented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}