Skip to content

Commit

Permalink
cleaning things up
Browse files Browse the repository at this point in the history
  • Loading branch information
EricSchles committed Jan 15, 2022
1 parent cb6b47e commit 7be89df
Show file tree
Hide file tree
Showing 29 changed files with 10,129 additions and 47,482 deletions.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -3599,7 +3599,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
"version": "3.9.5"
}
},
"nbformat": 4,
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,9 @@
" * triangle inequality\n",
" * euclidean distance\n",
" * manhattan distance\n",
" * L1 regularization with OLS\n",
" * L2 regularization with OLS\n",
" * Lasso Regression\n",
" * Ridge Regression\n",
" * ElasticNet Regression\n",
" * L1 regularization with OLS - Lasso Regression\n",
" * L2 regularization with OLS - Ridge Regression\n",
" * L1 + L2 regularization with OLS - ElasticNet Regression\n",
"* Non-Linear Regression"
]
},
Expand Down Expand Up @@ -4343,6 +4341,60 @@
"print(mdf.summary())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regularization\n",
"\n",
"In order to understand regularization we must first understand another problem. In the method of linear regression that we looked at above we had the following set up:\n",
"\n",
"```python\n",
"def gradient_descent(X, y, learning_rate=0.001, tolerance=1e-2, max_iter=1e5):\n",
" m = 0\n",
" b = 0\n",
" error = []\n",
" count = 0\n",
" while True:\n",
" m, b = update_weights(m, b, X, y, learning_rate)\n",
" cur_cost = cost_function(m, b, X, y)\n",
" error.append(cur_cost)\n",
" count += 1\n",
" if cur_cost < tolerance:\n",
" break\n",
" if count > max_iter:\n",
" break\n",
" return m, b, error\n",
"```\n",
"\n",
"One of the things to note in the above function is - there are a number of places where the training loop can terminate. If any of those things are triggered then the model 'should' be optimized. However, it's not always the case, even if you have a process following the underlying distribution in question, that you will end up with a model that reaches what statisticians call 'convergence'. Convergence is a state in which the model parameters are the best they could be within some reasonable tolerance from the 'best model' given the structure of the model. Models fail to reach convergence, sometimes because of bad initial values, because of the limitations of floating point numbers. \n",
"\n",
"Next we'll cover a technique to help our model learn a good fit between independent and dependent variables. Suppose you are trying to fit your data to a model and you are having trouble with convergence. That's what regularization is there for. It imposes a 'penalty' on the cost function. These extra terms in the cost function mean that any deviances from the dependent variable are extra expensive. So our model will try to fit even closer to the targets. \n",
"\n",
"When would you want this, versus when would you want?\n",
"\n",
"Well, if your only goal is accuracy of your model in learning from the examples you have, then always. That means if you are dealing with a closed system, and all the data that will be fed into the system will look _exactly_ like the training or testing data, then this is _always_ desirable. However, many statistical systems are _not_ this. If we are in any of the following cases:\n",
"\n",
", even in the presence of data that doesn't completely follow the distribution in question. In may also be the case that the underlying process follows the distribution in question, but you still may not have enough data in your data fully realizes the underlying process. Or perhaps the distribution you think you are modeling may actually be a different one, as we saw in chapter 1. \n",
"\n",
"In any of these 'real world' cases, we need a way to handle non-structural conformity.\n",
"\n",
"### distance functions - digression\n",
"\n",
"#### triangle inequality\n",
"#### euclidean distance\n",
"#### manhattan distance\n",
"\n",
"### A General Regularization Function\n",
"\n",
"#### L1 regularization with OLS - Lasso Regression\n",
"\n",
"#### L2 regularization with OLS - Ridge Regression\n",
"\n",
"#### L1 + L2 regularization with OLS - ElasticNet Regression\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
File renamed without changes
File renamed without changes.
File renamed without changes.
File renamed without changes
Original file line number Diff line number Diff line change
Expand Up @@ -760,7 +760,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 2,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -870,7 +870,7 @@
"17 137.13 KY "
]
},
"execution_count": 1,
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -923,7 +923,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 6,
"metadata": {},
"outputs": [
{
Expand All @@ -932,7 +932,7 @@
"3.768732178474927"
]
},
"execution_count": 2,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -967,22 +967,21 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1.5176955469568277e-05"
"1.5176955469736423e-05"
]
},
"execution_count": 3,
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# To Do: fix this - should only be for probability distributions\n",
"from sklearn import ensemble\n",
"from sklearn import metrics\n",
"from scipy import stats\n",
Expand All @@ -996,25 +995,6 @@
"stats.entropy(y, qk=y_pred)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.00755828784864156\n"
]
}
],
"source": [
"from sklearn.metrics import mean_squared_error\n",
"\n",
"print(mean_squared_error(y, y_pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -38638,7 +38618,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
"version": "3.9.5"
}
},
"nbformat": 4,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -726,15 +726,16 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[10, 7]\n",
"[22, 15]\n"
"[10, 7, 0]\n",
"[22, 15, 0]\n",
"[0, 0, 0]\n"
]
}
],
Expand Down Expand Up @@ -839,8 +840,8 @@
" final[i + new_size][j + new_size] = c22.matrix[i][j]\n",
" return Matrix(final)\n",
" \n",
"A = Matrix([[1,2], [3,4]])\n",
"B = Matrix([[2, 1], [4, 3]])\n",
"A = Matrix([[1,2,4], [3,4,7], [1,12,15]])\n",
"B = Matrix([[2, 1, 6], [4, 3, 4], [1,1,1]])\n",
"(A*B).pprint()"
]
},
Expand Down Expand Up @@ -1015,14 +1016,14 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.2024725964825778\n"
"0.1892145506579259\n"
]
}
],
Expand Down Expand Up @@ -2908,7 +2909,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
"version": "3.9.5"
}
},
"nbformat": 4,
Expand Down
Binary file added 5/NewtonIteration_Ani.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes.
File renamed without changes.
15 changes: 15 additions & 0 deletions chapter_summaries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Part 1. Computer Science

In this part of the book we will introduce the Python programming language. In order to motivate this we will introduce algorithms, data structures, discrete mathematics, calculus, linear algebra, combinatorics, graphs, differential equations.

Part 2. Data Science

This is the content that has been written, and will be expanded.

Part 3. From Computer Science to Data Science

In this part of the book we will introduce a number of NP-Complete algorithms and then approximate them using data science. The central tenets of this section will be built off of:

* https://arxiv.org/abs/2009.01398
* https://en.wikipedia.org/wiki/Predictability
* https://mathworld.wolfram.com/Predictability.html
13 changes: 1 addition & 12 deletions intro_to_python/engineering_basics/Introduction To Python.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -242,17 +242,6 @@
"Now then, we can carry out the same exercise with floating point numbers, we'll use `y` this time, instead of `x` just to keep things fresh:"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"Now then, we can carry out the same exercise with floating point numbers, we'll use y this time, instead of x just to keep things fresh:"
]
},
{
"cell_type": "code",
"execution_count": 4,
Expand Down Expand Up @@ -1584,7 +1573,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
"version": "3.9.5"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit 7be89df

Please sign in to comment.