notebooks/4_webapi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing data from a web API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the best things about \"web 2.0\" is more formal, external facing API's to allow developers to build tools on top of data. These APIs provide a reliable connection to data over a reliable protocol returning a reliable response in an easy, machine-readable format. There are tons of APIs out there. Sometimes you will need to register for a developer key (see the St. Louis Fed's API [here](https://research.stlouisfed.org/docs/api/)) and other times the APIs are free (see Coindesk's API [here](https://www.coindesk.com/api)). ProgrammableWeb has a great [directory](https://www.programmableweb.com/apis/directory).\n",
    "\n",
    "## With great power comes great responsibility. We should be responsible and polite internet citizens when hitting websites programmatically\n",
    "\n",
    "Below we will use Coindesk's API to get daily prices for bitcoin. Coindesk's API, like many others, returns a JSON (javascript object notation) object, which we can easily turn into a dataframe. Here we will need the `json` and `requests` libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pandas as pd\n",
    "import json\n",
    "import requests\n",
    "\n",
    "coindeskURL = 'https://api.coindesk.com/v1/bpi/historical/close.json?'\n",
    "\n",
    "# from API\n",
    "start = datetime.date(2019, 1 ,1)\n",
    "end = datetime.date(2019, 7, 1)\n",
    "\n",
    "url = f'{coindeskURL}start={start:%Y-%m-%d}&end={end:%Y-%m-%d}'\n",
    "\n",
    "print(f'Hitting this url:{url}')\n",
    "\n",
    "result = requests.get(url)\n",
    "result.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Digging into this a little bit, there are a few critical components to accessing data. Not all APIs will return the same structure of data. Many times you can look at what is returned by just hitting the URL with your browser. Let's try that\n",
    "\n",
    "[https://api.coindesk.com/v1/bpi/historical/close.json?start=2019-01-01&end=2019-07-01](https://api.coindesk.com/v1/bpi/historical/close.json?start=2019-01-01&end=2019-07-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we get:\n",
    "```\n",
    "{\n",
    "    \"bpi\":\n",
    "    {\n",
    "        \"2019-01-01\":3869.47,\n",
    "        \"2019-01-02\":3941.2167,\n",
    "        ...\n",
    "    },\n",
    "    \"disclaimer\": \"This data was produced from the CoinDesk Bitcoin Price Index.\",\n",
    "    \"time\":\n",
    "    {\n",
    "        \"updated\":\"Jul 2, 2019 00:03:00 UTC\",\n",
    "        \"updatedISO\":\"2019-07-02T00:03:00+00:00\"\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`bpi`, `disclaimer`, and `time` are all data that we can reference. The data we're primarily interested in is `bpi`. Different APIs can break up data differently, we will see examples in other APIs later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the json result in python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jsondata = json.loads(result.content)\n",
    "jsondata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then wrap that up in to a pandas dataframe! Note: pandas does have a `read_json` helper as well, but sometimes it is not well suited given the structure of the json output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note the json gets read with the disclaimer and time outputs as well\n",
    "data = pd.read_json(result.content)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({'Bitcoin Price Index': jsondata['bpi']})\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "data.plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}