Example Project - Baby Names.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## USA Baby Names 1880-2016\n",
    "\n",
    "The United States Social Security Administration maintains an interesting data set of (almost) all names given to babies born in the United States, by sex and year, going back to 1880. This data set is available at [https://www.ssa.gov/oact/babynames/limits.html](https://www.ssa.gov/oact/babynames/limits.html)\n",
    "\n",
    "This data set is interesting and fun to explore and we'll use it as the basis of a simple data analysis project with the end goal of creating a script that can be called to output a plot of a single name's popularity over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we will assume that this dataset has already been downloaded and <font color=\"red\"><b> unzipped </b></font> into a subfolder called names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform # some of the subsequent code depends on operating system\n",
    "\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up some variables for use later\n",
    "dataset_path_windows = 'data\\\\names'   # Windows\n",
    "dataset_path_nix = './data/names'   # UNIX, Linux\n",
    "\n",
    "begin_year = 1880\n",
    "end_year = 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The subsequent non-python commands depend on your operating system\n",
    "platform.system()  # show name of operating system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if platform.system()=='Windows':\n",
    "    dataset_path = dataset_path_windows\n",
    "else:\n",
    "    dataset_path = dataset_path_nix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first examine the data files to see what we're working with. Note the `type` command on Windows is equivalent to `cat` on MacOS or Linux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List files folder using the inbuilt commands of your operating system\n",
    "# jupyter notebooks can call the operating system by using the exclamation mark\n",
    "if platform.system()=='Windows':\n",
    "    !dir $dataset_path\n",
    "else:\n",
    "    !ls $dataset_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read a single file into a python variable and print out the first five lines\n",
    "if platform.system()=='Windows':\n",
    "    sample = !type $dataset_path\\\\yob1880.txt\n",
    "else:\n",
    "    sample = !cat $dataset_path/yob1880.txt\n",
    "\n",
    "sample[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need a function to read in all of these files one by one and combine them into a single dataframe. Note that Pandas will correctly intepret the paths to the files, irrespectively of whether they were formulated in a Windows or Unix-based manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataframe(begin_year, end_year):\n",
    "    columns = ('name', 'sex', 'births')\n",
    "    pieces = []\n",
    "    for year in range(begin_year, end_year + 1):\n",
    "        filename = '%s/yob%d.txt' % (dataset_path, year)\n",
    "        piece = pd.read_csv(filename, names=columns)\n",
    "        piece['year'] = year\n",
    "        pieces.append(piece)\n",
    "        \n",
    "    return pd.concat(pieces, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now call our new function to get the dataset loaded into a Dataframe.\n",
    "df = create_dataframe(begin_year, end_year)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets explore this data a little, first, how many records do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets look at a specific name, lets make a new dataframe that includes only your name and look at the first 5 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets now look at some stats for your name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When was your name at peak popularity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we convert the raw birth numbers into percent of births that year? Lets make a new column for that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, some of these percentages are really small, why dont we change it to number of births of a given name per million births that year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why dont we make a graph of how common your name is over the years"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your name is like mine, there is actually a bunch of shading indicating variance, why would that be?\n",
    "\n",
    "\n",
    "Its because this data is also split on gender, so there is a chance to have the name listed twice because of gender. The gender split could be interesting though, so lets look at it graphically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a actually a really good breakdown of different name trends by Tim Urban at https://waitbutwhy.com/2013/12/how-to-name-baby.html\n",
    "\n",
    "so lets look quickly at a couple of the interesting trends he found with our code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Name Fads\n",
    "\n",
    "A name fad is when a specific name gets really popular for a specific generation, causing a person's age to be reasonable guessed based on their name alone.\n",
    "\n",
    "Check out Jennifer, Ashley, or Shirley for some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gender Takeovers\n",
    "\n",
    "Sometimes a name that is uncommon but solely one gender becomes extremely popular for the other gender, to the point that the original gender stops using it\n",
    "\n",
    "Check out Lynn or Aubrey"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}