2021-04-19_intro_to_deep_learning_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Course\n",
    "\n",
    "- Notebook Author: [Trenton McKinney][1]\n",
    "- Course: **[DataCamp: Introduction to Deep Learning in Python][2]**\n",
    " - This [notebook][3] was created as a reproducible reference.\n",
    " - The material is from the course\n",
    "   - The course website uses `tensorflow v2.6.0`, `scikit-learn v1.0`, `pandas v1.3.4`, and `numpy v1.19.5`\n",
    "   - This notebook uses `v2.6.0`, `v1.3.0`, `v2.1.4`, and `v1.26.3` respectively, so there are differences in model performance and parameters compared to the course.\n",
    " - I completed the exercises with my PC GPU: NVIDIA GeForce RTX 3080 Ti\n",
    " - If you find the content beneficial, consider a [DataCamp Subscription][4].\n",
    " - I added a function (**`create_dir_save_file`**) to automatically download and save the required data (`data/course_name`) and image (`Images/course_name`) files.\n",
    "\n",
    "  [1]: https://trenton3983.github.io/\n",
    "  [2]: https://learn.datacamp.com/courses/introduction-to-deep-learning-in-python\n",
    "  [3]: https://github.com/trenton3983/DataCamp/blob/master/2021-04-19_intro_to_deep_learning_in_python.ipynb\n",
    "  [4]: https://www.datacamp.com/join-me/MjQwNzMyMg=="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Course Description\n",
    "\n",
    "Deep learning is the machine learning technique behind the most exciting capabilities in diverse areas like robotics, natural language processing, image recognition, and artificial intelligence, including the famous AlphaGo. In this course, you'll gain hands-on, practical knowledge of how to use deep learning with Keras 2.0, the latest version of a cutting-edge library for deep learning in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pprint import pprint as pp\n",
    "from itertools import combinations\n",
    "from pathlib import Path\n",
    "import requests\n",
    "import numpy as np\n",
    "import sys\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mtick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import datasets\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "from tensorflow.keras.callbacks import EarlyStopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.config.list_physical_devices('GPU')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enable the last line to print device placement logging when running `.fit`\n",
    "# example output: Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0\n",
    "# tf.debugging.set_log_device_placement(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set tf logging levels - 0: Info, 1: Warning, 2: Error, 3: None\n",
    "# %env TF_CPP_MIN_LOG_LEVEL=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create some tensors\n",
    "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
    "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
    "c = tf.matmul(a, b)\n",
    "\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration Options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('max_columns', 200)\n",
    "# pd.set_option('max_rows', 300)\n",
    "# pd.set_option('display.expand_frame_repr', True)\n",
    "# plt.rcParams[\"patch.force_edgecolor\"] = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dir_save_file(dir_path: Path, url: str):\n",
    "    \"\"\"\n",
    "    Check if the path exists and create it if it does not.\n",
    "    Check if the file exists and download it if it does not.\n",
    "    \"\"\"\n",
    "    if not dir_path.parents[0].exists():\n",
    "        dir_path.parents[0].mkdir(parents=True)\n",
    "        print(f'Directory Created: {dir_path.parents[0]}')\n",
    "    else:\n",
    "        print('Directory Exists')\n",
    "        \n",
    "    if not dir_path.exists():\n",
    "        r = requests.get(url, allow_redirects=True)\n",
    "        open(dir_path, 'wb').write(r.content)\n",
    "        print(f'File Created: {dir_path.name}')\n",
    "    else:\n",
    "        print('File Exists')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = Path('data/2021-04-19_intro_to_deep_learning_in_python')\n",
    "images_dir = Path('Images/2021-04-19_intro_to_deep_learning_in_python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_1 = 'https://assets.datacamp.com/production/repositories/654/datasets/8a57adcdb5bfb3e603dad7d3c61682dfe63082b8/hourly_wages.csv'\n",
    "file_2 = 'https://assets.datacamp.com/production/repositories/654/datasets/24769dae9dc51a77b9baa785d42ea42e3f8f7538/mnist.csv'\n",
    "file_3 = 'https://assets.datacamp.com/production/repositories/654/datasets/92b75b9bc0c0a8a30999d76f4a1ee786ef072a9c/titanic_all_numeric.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = [file_1, file_2, file_3]\n",
    "data_paths = list()\n",
    "\n",
    "for data in datasets:\n",
    "    file_name = data.split('/')[-1].replace('?raw=true', '')\n",
    "    data_path = data_dir / file_name\n",
    "    create_dir_save_file(data_path, data)\n",
    "    data_paths.append(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `hw`: Hourly Wages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hw = pd.read_csv(data_paths[0])\n",
    "hw.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `mnist`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = pd.read_csv(data_paths[1], header=None)\n",
    "mnist.iloc[:2, :6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `titanic`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = pd.read_csv(data_paths[2])\n",
    "titanic.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These are the usual ipython objects, including this one you are creating\n",
    "ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']  # list a variables\n",
    "\n",
    "# Get a sorted list of the objects and their sizes\n",
    "sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics of deep learning and neural networks\n",
    "\n",
    "In this chapter, you'll become familiar with the fundamental concepts and terminology used in deep learning, and understand why deep learning techniques are so powerful today. You'll build simple neural networks and generate predictions with them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to deep learning\n",
    "\n",
    "1. Imagine you work for a bank\n",
    "   - Imagine you work for a bank, and you need to build a model predicting how many transactions each customer will make next year. You have predictive data or features like: \n",
    "1. Example as seen by linear regression\n",
    "   - each customer’s age,\n",
    "   - bank balance,\n",
    "   - whether they are retired, and so on. \n",
    "   - We'll get to deep learning in a moment, but for comparison, consider how a simple linear regression model works for this problem.\n",
    "   - The linear regression embeds an assumption that the outcome, in this case how many transactions a user makes, is the sum of individual parts.\n",
    "   - It starts by saying, \"what is the average?\" \n",
    "   - Then it adds the effect of age.\n",
    "   - Then the effect of bank balance. And so on.\n",
    "   - ![][1]\n",
    "   - So the linear regression model isn't identifying the interactions between these parts, and how they affect banking activity.\n",
    "   - Say we plot predictions from this model.\n",
    "   - We draw one line with the predictions for retired people, and another with the predictions for those still working.\n",
    "   - We put current bank balance on the horizontal axis, and the vertical axis is the predicted number of transactions.\n",
    "   - The left graph shows predictions from a model with no interactions.\n",
    "   - In that model we simply add up the effect of the retirement status, and current bank balance.\n",
    "   - The lack of interactions is reflected by both lines being parallel.\n",
    "   - That's probably unrealistic, but it's an assumption of the linear regression model.\n",
    "   - The graph on the right shows the predictions from a model that allows interactions, and the lines don't need to be parallel.\n",
    "   - ![][2]\n",
    "1. Interactions\n",
    "   - Neural networks are a powerful modeling approach that accounts for interactions like this especially well.\n",
    "   - Deep learning, the focus of this course, is the use of especially powerful neural networks.\n",
    "   - Because deep learning models account for these types of interactions so well, they perform great on most prediction problems you've seen before.\n",
    "   - But their ability to capture extremely complex interactions also allow them to do amazing things with text, images, videos, audio, source code and almost anything else you could imagine doing data science with.\n",
    "1. Course structure\n",
    "   - The first two chapters of this course focus on conceptual knowledge about deep learning.\n",
    "   - This part will be hard, but it will prepare you to debug and tune deep learning models on conventional prediction problems, and it will lay the foundation for progressing towards those new and exciting applications.\n",
    "   - You'll see this pay off in the third and fourth chapter.\n",
    "1. Build and tune deep learning models using keras\n",
    "   - You will write code that looks like this, to build and tune deep learning models using keras, to solve many of the same modeling problems you might have previously solved with scikit-learn.\n",
    "```python\n",
    "import numpy as np\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "predictors = np.loadtxt('predictors_data.csv', delimiter=',')\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "```\n",
    "   - As a start to how deep learning models capture interactions and achieve these amazing results, we'll modify the diagram you saw a moment ago.\n",
    "1. Deep learning models capture interactions\n",
    "   - Here there is an interaction between retirement status and bank balance.\n",
    "   - Instead of having them separately affect the outcome, we calculate a function of these variables that accounts for their interaction, and use that to predict the outcome.\n",
    "   - Even this graphic oversimplifies reality, where most things interact with each in some way, and real neural network models account for far more interactions.\n",
    "   - So the diagram for a simple neural network looks like this.\n",
    "   - ![][3]\n",
    "1. Interactions in neural network\n",
    "   - On the far left, we have something called an input layer. This represents our predictive features like age or income.\n",
    "   - On the far right we have the output layer. The prediction from our model, in this case, the predicted number of transactions.\n",
    "   - All layers that are not the input or output layers are called hidden layers.\n",
    "   - They are called hidden layers because, while the inputs and outputs correspond to visible things that happened in the world, and they can be stored as data, the values in the hidden layer aren't something we have data about, or anything we observe directly from the world.\n",
    "   - Nevertheless, each dot, called a node, in the hidden layer, represents an aggregation of information from our input data, and each node adds to the model's ability to capture interactions.\n",
    "   - So the more nodes we have, the more interactions we can capture.\n",
    "   - ![][4]\n",
    "\n",
    "\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl1.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl2.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl3.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl4.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing neural network models to classical regression models\n",
    "\n",
    "Which of the models in the diagrams has greater ability to account for interactions?\n",
    "\n",
    "![][1]\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~Model 1.~~\n",
    "- **Model 2.**\n",
    "  - Model 2 has more nodes in the hidden layer, and therefore, greater ability to capture interactions.\n",
    "- ~~They are both the same.~~\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl5.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward propagation\n",
    "\n",
    "- We’ll start by showing how neural networks use data to make predictions. This is called the forward propagation algorithm.\n",
    "\n",
    "1. Bank transactions example\n",
    "   - Let's revisit our example predicting how many transactions a user will make at our bank.\n",
    "   - For simplicity, we'll make predictions based on only the number of children and number of existing accounts.\n",
    "1. Forward propagation\n",
    "   - This graph shows a customer with two children and three accounts.\n",
    "   - The forward-propagation algorithm will pass this information through the network to make a prediction in the output layer.\n",
    "   - Lines connect the inputs to the hidden layer.\n",
    "   - Each line has a weight indicating how strongly that input effects the hidden node that the line ends at.\n",
    "   - These are the first set of weights.\n",
    "   - We have one weight from the top input into the top node of the layer, and one weight from the bottom input to the top node of the hidden layer.\n",
    "   - These weights are the parameters we train or change when we fit a neural network to data, so these weights will be a focus throughout this course.\n",
    "   - To make predictions for the top node of the hidden layer, we take the value of each node in the input layer, multiply it by the weight that ends at that node, and then sum up all the values.\n",
    "   - In this case, we get (2 times 1) plus (3 times 1), which is 5.\n",
    "   - Now do the same to fill in the value of this node on the bottom.\n",
    "   - That is (two times (minus one)) plus (three times one).\n",
    "   - That's one.\n",
    "   - Finally, repeat this process for the next layer, which is the output layer.\n",
    "   - That is (five times two) plus (one times -1).\n",
    "   - That gives an output of 9.\n",
    "   - We predicted nine transactions.\n",
    "   - ![][1]\n",
    "   - That's forward-propagation.\n",
    "   - We moved from the inputs on the left, to the hidden layer in the middle, and then from the hidden layers to the output on the right.\n",
    "   - ![][2]\n",
    "   - ![][3]\n",
    "   - ![][4]\n",
    "   - We always use that same multiply then add process.\n",
    "   - **If you're familiar with vector algebra or linear algebra, that operation is a dot product.**\n",
    "   - If you don't know about dot products, that's fine too.\n",
    "   - That was forward propagation for a single data point.\n",
    "   - In general, we do forward propagation for one data point at a time.\n",
    "   - The value in that last layer is the model's prediction for that data point.\n",
    "1. Forward propagation code\n",
    "   - Let's see the code for this.\n",
    "   - We import Numpy for some of the mathematical operations.\n",
    "   - We've stored the input data as an array. We then have weights into each node in the hidden layer and to the output.\n",
    "   - We store the weights going into each node as an array, and we use a dictionary to store those arrays.\n",
    "   - Let’s start forward propagating. We fill in the top hidden node here, which is called node zero.\n",
    "   - We multiply the inputs by the weights for that node, and then sum both of those terms together.\n",
    "   - Notice that we had two weights for node_0. That matches the two items in the array it is multiplied by, which is the input_data.\n",
    "   - These get converted to a single number by the sum function at the end of the line.\n",
    "   - We then do the same thing for the bottom node of the hidden layer, which is called node 1.\n",
    "   - Now, both node zero and node one have numeric values.\n",
    "1. Forward propagation code\n",
    "   - To simplify multiplication, we put those in an array here.\n",
    "   - If we print out the array, we confirm that those are the values from the hidden layer you saw a moment ago.\n",
    "   - It can also be instructive to verify this by hand with pen and paper.\n",
    "   - To get the output, we multiply the values in the hidden layer by the weights for the output.\n",
    "   - Summing those together gives us 10 minus 1, which is 9.\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl6.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl7.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl8.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl9.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = np.array([2, 3])\n",
    "weights = {'node_0': np.array([1, 1]),\n",
    "'node_1': np.array([-1, 1]),\n",
    "'output': np.array([2, -1])}\n",
    "node_0_value = (input_data * weights['node_0']).sum()\n",
    "node_1_value = (input_data * weights['node_1']).sum()\n",
    "\n",
    "hidden_layer_values = np.array([node_0_value, node_1_value])\n",
    "print(hidden_layer_values)\n",
    "\n",
    "output = (hidden_layer_values * weights['output']).sum()\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding the forward propagation algorithm\n",
    "\n",
    "In this exercise, you'll write code to do forward propagation (prediction) for your first neural network:\n",
    "\n",
    "![][1]\n",
    "\n",
    "Each data point is a customer. The first input is how many accounts they have, and the second input is how many children they have. The model will predict how many transactions the user makes in the next year. You will use this data throughout the first 2 chapters of this course.\n",
    "\n",
    "The input data has been pre-loaded as `input_data`, and the weights are available in a dictionary called `weights`. The array of weights for the first node in the hidden layer are in `weights['node_0']`, and the array of weights for the second node in the hidden layer are in `weights['node_1']`.\n",
    "\n",
    "The weights feeding into the output node are available in `weights['output']`.\n",
    "\n",
    "NumPy will be pre-imported for you as `np` in all exercises.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Calculate the value in node 0 by multiplying `input_data` by its weights `weights['node_0']` and computing their sum. This is the 1st node in the hidden layer.\n",
    "- Calculate the value in node 1 using `input_data` and `weights['node_1']`. This is the 2nd node in the hidden layer.\n",
    "- Put the hidden layer values into an array. This has been done for you.\n",
    "- Generate the prediction by multiplying `hidden_layer_outputs` by `weights['output']` and computing their sum.\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl10.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = np.array([3, 5])\n",
    "weights = {'node_0': np.array([2, 4]), 'node_1': np.array([ 4, -5]), 'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate node 0 value: node_0_value\n",
    "node_0_value = (input_data * weights['node_0']).sum()\n",
    "\n",
    "# Calculate node 1 value: node_1_value\n",
    "node_1_value = (input_data * weights['node_1']).sum()\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_value, node_1_value])\n",
    "\n",
    "# Calculate output: output\n",
    "output = (hidden_layer_outputs * weights['output']).sum()\n",
    "\n",
    "# Print output\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation functions\n",
    "\n",
    "- But creating this multiply-add-process is only half the story for hidden layers. For neural networks to achieve their maximum predictive power, we must apply something called an activation function in the hidden layers.\n",
    "1. Linear vs Nonlinear Functions\n",
    "   - ![][1]\n",
    "   - An activation function allows the model to capture non-linearities.\n",
    "   - Non-linearities, as shown on the right here, capture patterns like how going from no children to one child may impact your banking transactions differently than going from three children to four.\n",
    "   - We have examples of linear functions, straight lines on the left, and non-linear functions on the right.\n",
    "   - If the relationships in the data aren’t straight-line relationships, we will need an activation function that captures non-linearities.\n",
    "1. Activation functions\n",
    "   - An activation function is something applied to the value coming into a node, which then transforms it into the value stored in that node, or the node output.\n",
    "1. Improving our neural network\n",
    "   - ![][2]\n",
    "   - Let's go back to the previous diagram. The top hidden node previously had a value of 5.\n",
    "   - For a long time, an s-shaped function called $tanh$ was a popular activation function.\n",
    "1. Activation functions\n",
    "   - ![][3]\n",
    "   - If we used the $tanh$ activation function, this node's value would be $tanh(5)$, which is very close to 1.\n",
    "   - Today, the standard in both industry and research applications is something called\n",
    "1. ReLU (Rectified Linear Activation)\n",
    "   - ![][4]\n",
    "   - the ReLU or rectified linear activation function.\n",
    "   - That's depicted here. Though it has two linear pieces, it's surprisingly powerful when composed together through multiple successive hidden layers, which you will see soon.\n",
    "   - The code that incorporates activation functions is shown here. \n",
    "1. Activation functions\n",
    "   - It is the same as the code you saw previously, but we've distinguished the input from the output in each node, which is shown in these lines and then again here.\n",
    "   - And we've applied the $tanh$ function to convert the input to the output.\n",
    "   - That gives us a prediction of 1-point-2 transactions.\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl11.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl12.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl13.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl14.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = np.array([-1, 2])\n",
    "weights = {'node_0': np.array([3, 3]), 'node_1': np.array([1, 5]), 'output': np.array([2, -1])}\n",
    "\n",
    "node_0_input = (input_data * weights['node_0']).sum()\n",
    "node_0_output = np.tanh(node_0_input)\n",
    "\n",
    "node_1_input = (input_data * weights['node_1']).sum()\n",
    "node_1_output = np.tanh(node_1_input)\n",
    "\n",
    "hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "\n",
    "output = (hidden_layer_outputs * weights['output']).sum()\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Rectified Linear Activation Function\n",
    "\n",
    "As Dan explained to you in the video, an \"activation function\" is a function applied at each node. It converts the node's input into some output.\n",
    "\n",
    "The rectified linear activation function (called ReLU) has been shown to lead to very high-performance networks. This function takes a single number as an input, returning 0 if the input is negative, and the input if the input is positive.\n",
    "\n",
    "Here are some examples:\n",
    "\n",
    "$relu(3) = 3$\n",
    "\n",
    "$relu(-3) = 0$\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Fill in the definition of the `relu()` function:\n",
    "- Use the `max()` function to calculate the value for the output of `relu()`.\n",
    "- Apply the `relu()` function to `node_0_input` to calculate `node_0_output`.\n",
    "- Apply the `relu()` function to `node_1_input` to calculate `node_1_output`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def relu`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(input_):\n",
    "    '''Define your relu activation function here'''\n",
    "    # Calculate the value for the output of the relu function: output\n",
    "    output = max(0, input_)\n",
    "    \n",
    "    # Return the value just calculated\n",
    "    return(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = np.array([3, 5])\n",
    "weights = {'node_0': np.array([2, 4]), 'node_1': np.array([ 4, -5]), 'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate node 0 value: node_0_output\n",
    "node_0_input = (input_data * weights['node_0']).sum()\n",
    "node_0_output = relu(node_0_input)\n",
    "\n",
    "# Calculate node 1 value: node_1_output\n",
    "node_1_input = (input_data * weights['node_1']).sum()\n",
    "node_1_output = relu(node_1_input)\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "\n",
    "# Calculate model output (do not apply relu)\n",
    "model_output = (hidden_layer_outputs * weights['output']).sum()\n",
    "\n",
    "# Print model output\n",
    "print(model_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You predicted 52 transactions. Without this activation function, you would have predicted a negative number! The real power of activation functions will come soon when you start tuning model weights.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the network to many observations/rows of data\n",
    "\n",
    "You'll now define a function called `predict_with_network()` which will generate predictions for multiple data observations, which are pre-loaded as `input_data`. As before, `weights` are also pre-loaded. In addition, the `relu()` function you defined in the previous exercise has been pre-loaded.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Define a function called `predict_with_network()` that accepts two arguments - `input_data_row` and `weights` - and returns a prediction from the network as the output.\n",
    "- Calculate the input and output values for each node, storing them as: `node_0_input`, `node_0_output`, `node_1_input`, and `node_1_output`.\n",
    "  - To calculate the input value of a node, multiply the relevant arrays together and compute their sum.\n",
    "  - To calculate the output value of a node, apply the `relu()` function to the input value of the node.\n",
    "- Calculate the model output by calculating `input_to_final_layer` and `model_output` in the same way you calculated the input and output values for the nodes.\n",
    "- Use a `for loop` to iterate over `input_data`:\n",
    "  - Use your `predict_with_network()` to generate predictions for each row of the `input_data` - `input_data_row`. Append each prediction to `results`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def predict_with_network1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define predict_with_network()\n",
    "def predict_with_network1(input_data_row, weights):\n",
    "\n",
    "    # Calculate node 0 value\n",
    "    node_0_input = (input_data_row * weights['node_0']).sum()\n",
    "    node_0_output = relu(node_0_input)\n",
    "\n",
    "    # Calculate node 1 value\n",
    "    node_1_input = (input_data_row * weights['node_1']).sum()\n",
    "    node_1_output = relu(node_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_layer_outputs\n",
    "    hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "    \n",
    "    # Calculate model output\n",
    "    input_to_final_layer = (hidden_layer_outputs * weights['output']).sum()\n",
    "    model_output = relu(input_to_final_layer)\n",
    "    \n",
    "    # Return model output\n",
    "    return(model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = [np.array([3, 5]), np.array([ 1, -1]), np.array([0, 0]), np.array([8, 4])]\n",
    "weights = {'node_0': np.array([2, 4]), 'node_1': np.array([ 4, -5]), 'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create empty list to store prediction results\n",
    "results = []\n",
    "for input_data_row in input_data:\n",
    "    # Append prediction to results\n",
    "    results.append(predict_with_network1(input_data_row, weights))\n",
    "\n",
    "# Print results\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deeper networks\n",
    "\n",
    "- The difference between modern deep learning and the historical neural networks that didn't deliver these amazing results, is the use of models with not just one hidden layer, but with many successive hidden layers.\n",
    "- We forward propagate through these successive layers in a similar way to what you saw for a single hidden layer.\n",
    "\n",
    "1. Multiple hidden layers\n",
    "   - Here is a network with two hidden layers. We first fill in the values for hidden layer one as a function of the inputs.\n",
    "   - ![][1]\n",
    "   - ![][2]\n",
    "   - Then apply the activation function to fill in the values in these nodes.\n",
    "   - Then use values from the first hidden layer to fill in the second hidden layer.\n",
    "   - ![][3]\n",
    "   - Then we make a prediction based on the outputs of hidden layer two.\n",
    "   - ![][4]\n",
    "   - In practice, it's becoming common to have neural networks that have many, many layers; five layers, ten layers.\n",
    "   - A few years ago 15 layers was state of the art but this can scale quite naturally to even a thousand layers.\n",
    "   - You use the same forward propagation process, but you apply that iterative process more times.\n",
    "   - ![][5]\n",
    "   - Let's walk through the first steps of that.\n",
    "   - Assume all layers here use the ReLU activation function.\n",
    "   - We'll start by filling in the top node of the first hidden layer.\n",
    "   - ![][6]\n",
    "   - That will use these two weights.\n",
    "   - The top weights contributes 3 times 2, or 6.\n",
    "   - ![][7]\n",
    "   - The bottom weight contributes 20.\n",
    "   - ![][8]\n",
    "   - The ReLU activation function on a positive number just returns that number.\n",
    "   - So we get 26.\n",
    "   - ![][9]\n",
    "   - Now let's do the bottom node of that first hidden layer.\n",
    "   - ![][10]\n",
    "   - We use these two nodes.\n",
    "   - Using the same process, we get 4 times 3, or 12 from this weight.\n",
    "   - ![][11]\n",
    "   - And -25 from the bottom weight.\n",
    "   - ![][12]\n",
    "   - So the input to this node is 12 minus 25.\n",
    "   - Recall that, when we apply ReLU to a negative number, we get 0.\n",
    "   - ![][13]\n",
    "   - So this node is 0.\n",
    "   - We've shown the values for the subsequent layers here.\n",
    "   - ![][14]\n",
    "   - Pause this video, and verify you can calculate the same values at each node.\n",
    "   - At this point, you understand the mechanics for how neural networks make predictions.\n",
    "   - Let's close this chapter with an interesting and important fact about these deep networks.\n",
    "1. Representation learning\n",
    "   - That is, they internally build up representations of the patterns in the data that are useful for making predictions.\n",
    "   - And they find increasingly complex patterns as we go through successive hidden layers of the network.\n",
    "   - In this way, neural networks partially replace the need for feature engineering, or manually creating better predictive features.\n",
    "   - Deep learning is also sometimes called representation learning, because subsequent layers build increasingly sophisticated representations of the raw data, until we get to a stage where we can make predictions.\n",
    "   - This is easiest to understand from an application to images, which you will see later in this course.\n",
    "   - Even if you haven't worked with images, you may find it useful to think through this example heuristically.\n",
    "   - ![][15]\n",
    "1. Representation learning\n",
    "   - When a neural network tries to classify an image, the first hidden layers build up patterns or interactions that are conceptually simple.\n",
    "   - A simple interaction would look at groups of nearby pixels and find patterns like diagonal lines, horizontal lines, vertical lines, blurry areas, etc.\n",
    "   - Once the network has identified where there are diagonal lines and horizontal lines and vertical lines, subsequent layers combine that information to find larger patterns, like big squares.\n",
    "   - A later layer might put together the location of squares and other geometric shapes to identify a checkerboard pattern, a face, a car, or whatever is in the image.\n",
    "   - The cool thing about deep learning is that the modeler doesn't need to specify those interactions.\n",
    "1. Deep learning\n",
    "   - We never tell the model to look for diagonal lines.\n",
    "   - Instead, when you train the model, which you'll learn to do in the next chapter, the network gets weights that find the relevant patterns to make better predictions.\n",
    "   - Working with images may still seem abstract, but this idea of finding increasingly complex or abstract patterns is a recurring theme when people talk about deep learning, and it will feel more concrete as you work with these networks more.\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl15.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl16.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl17.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl18.jpg\n",
    "  [5]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl19.jpg\n",
    "  [6]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl20.jpg\n",
    "  [7]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl21.jpg\n",
    "  [8]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl22.jpg\n",
    "  [9]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl23.jpg\n",
    "  [10]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl24.jpg\n",
    "  [11]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl25.jpg\n",
    "  [12]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl26.jpg\n",
    "  [13]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl27.jpg\n",
    "  [14]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl28.jpg\n",
    "  [15]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl29.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward propagation in a deeper network\n",
    "You now have a model with 2 hidden layers. The values for an input data point are shown inside the input nodes. The weights are shown on the edges/lines. What prediction would this model make on this data point?\n",
    "\n",
    "Assume the activation function at each node is the _identity function_. That is, each node's output will be the same as its input. So the value of the bottom node in the first hidden layer is -1, and not 0, as it would be if the ReLU activation function was used.\n",
    "\n",
    "![][1]\n",
    "\n",
    "Possible Answers\n",
    "\n",
    "- **0.**\n",
    "- ~~7.~~\n",
    "- ~~9.~~\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl30.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer neural networks\n",
    "\n",
    "In this exercise, you'll write code to do forward propagation for a neural network with 2 hidden layers. Each hidden layer has two nodes. The input data has been preloaded as `input_data`. The nodes in the first hidden layer are called `node_0_0` and `node_0_1`. Their weights are pre-loaded as `weights['node_0_0']` and `weights['node_0_1']` respectively.\n",
    "\n",
    "The nodes in the second hidden layer are called node_1_0 and node_1_1. Their weights are pre-loaded as `weights['node_1_0']` and `weights['node_1_1']` respectively.\n",
    "\n",
    "We then create a model output from the hidden nodes using weights pre-loaded as `weights['output']`.\n",
    "\n",
    "![][1]\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Calculate `node_0_0_input` using its weights `weights['node_0_0']` and the given `input_data`. Then apply the `relu()` function to get `node_0_0_output`.\n",
    "- Do the same as above for `node_0_1_input` to get `node_0_1_output`.\n",
    "- Calculate `node_1_0_input` using its weights `weights['node_1_0']` and the outputs from the first hidden layer - `hidden_0_outputs`. Then apply the `relu()` function to get `node_1_0_output`.\n",
    "- Do the same as above for `node_1_1_input` to get `node_1_1_output`.\n",
    "- Calculate `model_output` using its weights `weights['output']` and the outputs from the second hidden layer `hidden_1_outputs` array. Do not apply the `relu()` function to this output.\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl31.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def predict_with_network2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_with_network2(input_data, weights):\n",
    "    # Calculate node 0 in the first hidden layer\n",
    "    node_0_0_input = (input_data * weights['node_0_0']).sum()\n",
    "    node_0_0_output = relu(node_0_0_input)\n",
    "\n",
    "    # Calculate node 1 in the first hidden layer\n",
    "    node_0_1_input = (input_data * weights['node_0_1']).sum()\n",
    "    node_0_1_output = relu(node_0_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_0_outputs\n",
    "    hidden_0_outputs = np.array([node_0_0_output, node_0_1_output])\n",
    "    \n",
    "    # Calculate node 0 in the second hidden layer\n",
    "    node_1_0_input = (hidden_0_outputs * weights['node_1_0']).sum()\n",
    "    node_1_0_output = relu(node_1_0_input)\n",
    "\n",
    "    # Calculate node 1 in the second hidden layer\n",
    "    node_1_1_input = (hidden_0_outputs * weights['node_1_1']).sum()\n",
    "    node_1_1_output = relu(node_1_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_1_outputs\n",
    "    hidden_1_outputs = np.array([node_1_0_output, node_1_1_output])\n",
    "\n",
    "    # Calculate model output: model_output\n",
    "    model_output = relu((hidden_1_outputs * weights['output']).sum())\n",
    "    \n",
    "    # Return model_output\n",
    "    return(model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data = np.array([3, 5])\n",
    "weights = {'node_0_0': np.array([2, 4]),\n",
    "           'node_0_1': np.array([ 4, -5]),\n",
    "           'node_1_0': np.array([-1,  2]),\n",
    "           'node_1_1': np.array([1, 2]),\n",
    "           'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = predict_with_network2(input_data, weights)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representations are learned\n",
    "\n",
    "How are the weights that determine the features/interactions in Neural Networks created?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~A user chooses them when creating the model.~~\n",
    "- **The model training process sets them to optimize predictive accuracy.**\n",
    "- ~~The weights are random numbers.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Levels of representation\n",
    "\n",
    "Which layers of a model capture more complex or \"higher level\" interactions?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~The first layers capture the most complex interactions.~~\n",
    "- **The last layers capture the most complex interactions.**\n",
    "- ~~All layers capture interactions of similar complexity.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing a neural network with backward propagation\n",
    "\n",
    "Learn how to optimize the predictions generated by your neural networks. You'll use a method called backward propagation, which is one of the most important techniques in deep learning. Understanding how it works will give you a strong foundation to build on in the second half of the course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The need for optimization\n",
    "\n",
    "- You've seen the forward-propagation algorithm that neural networks use to make predictions.\n",
    "- However, **the mere fact that a model has the structure of a neural network does not guarantee that it will make good predictions.**\n",
    "1. A baseline neural network\n",
    "   - To see the importance of model weights, we'll go back to a network you saw in the previous chapter.\n",
    "   - We'll use a simple example for the sake of explanation.\n",
    "   - For the moment, we won't use an activation function in this example, or if you prefer, you might think of an activation function that returns the input, sometimes called the identity function.\n",
    "   - We have values of 2 and 3 for the inputs, and the true value of the target is 13.\n",
    "   - So, the closer our prediction is to 13, the more accurate this model is for this data point.\n",
    "   - We use forward propagation to fill in the values of hidden layer.\n",
    "   - That gives us hidden node values of 5 and 1.\n",
    "   - Continuing forward propagation, we use those hidden node values to make a prediction of 9.\n",
    "   - Since the true target value is 13, our error is 13 minus 9, which is 4.\n",
    "   - ![][1]\n",
    "   - Changing any weight will change our prediction.\n",
    "   - Let's see what happens if we change the two weights from the hidden layer to the output.\n",
    "   - In this case, we make the top weight 3 and the bottom weight -2.\n",
    "   - Now forward propagation gives us a prediction of 13.\n",
    "   - ![][3]\n",
    "   - That is exactly the value we wanted to predict.\n",
    "   - So, this change in weights improved the model for this data point.\n",
    "1. Predictions with multiple points\n",
    "   - Making accurate predictions gets harder with multiple points.\n",
    "   - First of all, at any set of weights, we have many values of the error, corresponding to the many points we make predictions for. \n",
    "1. Loss function\n",
    "   - We use something called a loss function to aggregate all the errors into a single measure of the model's predictive performance.\n",
    "1. Squared error loss function\n",
    "   - For example, a common loss function for regression tasks is mean-squared error.\n",
    "   - You square each error, and take the average of that as a measure of model quality.\n",
    "   - The loss function aggregates all of the errors into a single score.\n",
    "   - ![][4]\n",
    "1. Loss function\n",
    "   - For an illustration, consider a model with only two weights, we could plot the model's performance for each set of weights like this.\n",
    "   - The values of the weights are plotted on the x and y axis, and the loss function is on the vertical or z axis.\n",
    "   - Lower values mean a better model, so our goal is to find the weights giving the lowest value for the loss function.\n",
    "   - **We do this with an algorithm called gradient descent.**\n",
    "   - An analogy may be helpful.\n",
    "   - ![][5]\n",
    "1. Gradient descent\n",
    "   - Imagine you are in a pitch dark field, and you want to find the lowest point.\n",
    "   - You might feel the ground to see how it slopes, and take a small step downhill.\n",
    "   - This gives an improvement, but not necessarily the lowest point yet.\n",
    "   - So you repeat this process until it is uphill in every direction.\n",
    "   - This is roughly how gradient descent works.\n",
    "1. Gradient descent steps\n",
    "   - The steps are: Start at a random point, until you are somewhere flat, find the slope, and take a step downhill. \n",
    "1. Optimizing a model with a single weight \n",
    "   - Let's look at optimizing a model with a single weight, and then we'll scale up to optimizing multiple weights.\n",
    "   - We have a curve showing the loss function on the vertical axis, at different values of the weight, which is on the horizontal axis.\n",
    "   - We are looking for the low point on this curve, because that means our model is as accurate as possible.\n",
    "   - We have drawn this tangent line to the curve at our current point.\n",
    "   - The slope of that tangent line captures the slope of the loss function at the our current weight.\n",
    "   - That slope corresponds to something called the derivative from calculus. We use this slope to decide what direction we step.\n",
    "   - In this case, the slope is positive.\n",
    "   - So if we want to go downhill, we go in the direction opposite the slope, towards lower numbers.\n",
    "   - If we repeatedly take small steps opposite the slope, recalculating the slope each time, we will eventually get to the minimum value.\n",
    "   - ![][6]\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl33.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl35.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl36.jpg\n",
    "  [5]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl37.jpg\n",
    "  [6]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl38.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating model errors\n",
    "\n",
    "For the exercises in this chapter, you'll continue working with the network to predict transactions for a bank.\n",
    "\n",
    "What is the error (predicted - actual) for the following network using the ReLU activation function when the input data is `[3, 2]` and the actual value of the target (what you are trying to predict) is 5? It may be helpful to get out a pen and piece of paper to calculate these values.\n",
    "\n",
    "![][1]\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~5.~~\n",
    "- ~~6.~~\n",
    "- **11.**\n",
    "  - _The network generates a prediction of `16`, which results in an error of `11`._\n",
    "- ~~16.~~\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl39.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding how weights change model accuracy\n",
    "\n",
    "Imagine you have to make a prediction for a single data point. The actual value of the target is 7. The weight going from `node_0` to the output is 2, as shown below. If you increased it slightly, changing it to 2.01, would the predictions become more accurate, less accurate, or stay the same?\n",
    "\n",
    "![][1]\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~More accurate.~~\n",
    "- **Less accurate.**\n",
    "  - _Increasing the weight to `2.01` would increase the resulting error from `9` to `9.08`, making the predictions less accurate._\n",
    "- ~~Stay the same.~~\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl40.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding how weight changes affect accuracy\n",
    "\n",
    "Now you'll get to change weights in a real network and see how they affect model accuracy!\n",
    "\n",
    "Have a look at the following neural network:\n",
    "\n",
    "![][1]\n",
    "\n",
    "Its weights have been pre-loaded as `weights_0`. Your task in this exercise is to update a **single** weight in `weights_0` to create `weights_1`, which gives a perfect prediction (in which the predicted value is equal to `target_actual`: 3).\n",
    "\n",
    "Use a pen and paper if necessary to experiment with different combinations. You'll use the `predict_with_network()` function, which takes an array of data as the first argument, and weights as the second argument.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create a dictionary of weights called `weights_1` where you have changed 1 weight from `weights_0` (You only need to make 1 edit to `weights_0` to generate the perfect prediction).\n",
    "- Obtain predictions with the new weights using the `predict_with_network()` function with `input_data` and `weights_1`.\n",
    "- Calculate the error for the new weights by subtracting `target_actual` from `model_output_1`.\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl41.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The data point you will make a prediction for\n",
    "input_data = np.array([0, 3])\n",
    "\n",
    "# Sample weights\n",
    "weights_0 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 2],\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# The actual target value, used to calculate the error\n",
    "target_actual = 3\n",
    "\n",
    "# Make prediction using original weights\n",
    "model_output_0 = predict_with_network1(input_data, weights_0)\n",
    "\n",
    "# Calculate error: error_0\n",
    "error_0 = model_output_0 - target_actual\n",
    "error_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create weights that cause the network to make perfect prediction (3): weights_1\n",
    "weights_1 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 0],\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# Make prediction using new weights: model_output_1\n",
    "model_output_1 = predict_with_network1(input_data, weights_1)\n",
    "\n",
    "# Calculate error: error_1\n",
    "error_1 = model_output_1 - target_actual\n",
    "\n",
    "# Print error_1\n",
    "error_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling up to multiple data points\n",
    "\n",
    "You've seen how different weights will have different accuracies on a single prediction. But usually, you'll want to measure model accuracy on many points. You'll now write code to compare model accuracies for two different sets of weights, which have been stored as `weights_0` and `weights_1`.\n",
    "\n",
    "`input_data` is a list of arrays. Each item in that list contains the data to make a single prediction. `target_actuals` is a list of numbers. Each item in that list is the actual value we are trying to predict.\n",
    "\n",
    "In this exercise, you'll use the `mean_squared_error()` function from `sklearn.metrics`. It takes the true values and the predicted values as arguments.\n",
    "\n",
    "You'll also use the preloaded `predict_with_network()` function, which takes an array of data as the first argument, and weights as the second argument.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `mean_squared_erro`r from `sklearn.metrics`.\n",
    "- Using a `for loop` to iterate over each row of `input_data`:\n",
    "  - Make predictions for each row with `weights_0` using the `predict_with_network()` function and append it to `model_output_0`.\n",
    "  - Do the same for `weights_1`, appending the predictions to `model_output_1`.\n",
    "- Calculate the mean squared error of `model_output_0` and then `model_output_1` using the `mean_squared_error()` function. The first argument should be the actual values (`target_actuals`), and the second argument should be the predicted values (`model_output_0` or `model_output_1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_0 = {'node_0': np.array([2, 1]), 'node_1': np.array([1, 2]), 'output': np.array([1, 1])}\n",
    "weights_1 = {'node_0': np.array([2, 1]), 'node_1': np.array([1. , 1.5]), 'output': np.array([1. , 1.5])}\n",
    "input_data = [np.array([0, 3]), np.array([1, 2]), np.array([-1, -2]), np.array([4, 0])]\n",
    "target_actuals = [1, 3, 5, 7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# Create model_output_0 \n",
    "model_output_0 = []\n",
    "# Create model_output_1\n",
    "model_output_1 = []\n",
    "\n",
    "# Loop over input_data\n",
    "for row in input_data:\n",
    "    # Append prediction to model_output_0\n",
    "    model_output_0.append(predict_with_network1(row, weights_0))\n",
    "    \n",
    "    # Append prediction to model_output_1\n",
    "    model_output_1.append(predict_with_network1(row, weights_1))\n",
    "\n",
    "# Calculate the mean squared error for model_output_0: mse_0\n",
    "mse_0 = mean_squared_error(target_actuals, model_output_0)\n",
    "\n",
    "# Calculate the mean squared error for model_output_1: mse_1\n",
    "mse_1 = mean_squared_error(target_actuals, model_output_1)\n",
    "\n",
    "# Print mse_0 and mse_1\n",
    "print(f\"Mean squared error with weights_0: %{round(mse_0, 2)}\")\n",
    "print(f\"Mean squared error with weights_1: %{round(mse_1, 2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`model_output_1` has a higher mean squared error.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient descent\n",
    "\n",
    "- With gradient descent, you repeatedly repeatedly found a slope capturing how your loss function changes as a weight changes.\n",
    "- You then made a small change to the weight to get to a lower point, and you repeated this until you couldn't go downhill any more.\n",
    "- If the slope is positive:\n",
    "  - going opposite the slope means moving to lower numbers.\n",
    "  - Subtracting the slope from the current value achieves this.\n",
    "  - Too big a step might lead us far astray.\n",
    "- So, instead of directly subtracting the slope, we multiply the slope by a small number, called the learning rate, and we change the weight by the product of that multiplication.\n",
    "- Learning rate are frequently around point-01.\n",
    "- This ensures we take small steps, so we reliably move towards the optimal weights.\n",
    "- But how do we find the relevant slope for each weight we need to update? Working this out for yourself involves calculus, especially the application of the chain rule.\n",
    "- Don't worry if you don't remember or don't know the underlying calculus.\n",
    "- We'll explain some basic concepts here, and Keras and TensorFlow do the calculus for us.\n",
    "- ![][1]\n",
    "- [Gradient Descent animation: 1. Simple linear Regression][5]\n",
    "1. Slope calculation example\n",
    "   - Here is a first example to calculate a slope for a weight, and in this example we will look at a single data point.\n",
    "   - Weights feed from one node into another, and you always get the slope you need by multiplying three things.\n",
    "     - First, the slope of the loss function with respect to the value at the node we feed into.\n",
    "     - Second, the value of the node that feeds into our weight.\n",
    "     - Third, the slope of the activation function with respect to the value we feed into.\n",
    "   - Let's start with the slope of the loss function with respect to the value of the node our weight feeds into. \n",
    "   - In this case, that node is the model's prediction.\n",
    "   - If you work through some calculus, you will find that the slope of the mean-squared loss function with respect to the prediction is $2 * (predicted value - actual value)$.\n",
    "   - Which is $2 * error$.\n",
    "   - Here, the prediction from forward propagation was $6$.\n",
    "   - The actual target value is $10$, so the error is $6 - 10$, which is $-4$.\n",
    "   - The second thing we multiply is the value at the node we are feeding from. Here, that is 3.\n",
    "   - Finally, the slope of the activation function at the value we feed into.\n",
    "   - Since we don't have an activation function here, we can leave that out.\n",
    "   - So our final result for the slope of the loss if we graphed it against this weight is $2 * -4 * 3$, or $-24$.\n",
    "   - We would now improve this weight by subtracting the learning rate times that slope, $-24$.\n",
    "   - If the learning rate were $0.01$, we would update this weight to be $2.24$.\n",
    "   - That gives us a better model.\n",
    "   - And it would continue improving if we repeated this process.\n",
    "   - For multiple weights feeding to the output, we repeat this calculation separately for each weight.\n",
    "   - Then we update both weights simultaneously using their respective derivatives.\n",
    "   - ![][2]\n",
    "   - ![][3]\n",
    "1. Network with two inputs affecting prediction\n",
    "   - Here is a network with two weights going directly to an output, and again with no activation function.\n",
    "   - Let's see the code to calculate slopes and update the weights.\n",
    "   - First, we set up the weights, input data, and a target value to predict.\n",
    "   - ![][4]\n",
    "1. Code to calculate slopes and update weights\n",
    "   - Here is the slope calculation.\n",
    "   - We uses numpy broadcasting, which multiplies an array by a number so that each entry in the array is multiplied by that number.\n",
    "   - We multiply the two times the error times the array with the input nodes.\n",
    "   - This gives us an array that used the 1st node value for the first calculated slope, and the second node value for the 2nd calculated slope.\n",
    "   - This is exactly what we wanted. Incidentally, the mathematical term for this array of slopes is a \"gradient\", and this is where the name gradient descent comes from.\n",
    "   - We update the weights by some small step in that direction, where the step size is partially determined by the learning rate.\n",
    "   - And the new error is $2.5$, which is an improvement over the old error, which was $5$.\n",
    "   - Repeating that process from the new values would give further improvements.\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/gradient_descent.gif\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl42.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl43.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl44.jpg\n",
    "  [5]: https://towardsdatascience.com/gradient-descent-animation-1-simple-linear-regression-e49315b24672"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array([1, 2])\n",
    "input_data = np.array([3, 4])\n",
    "target = 6\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = (weights * input_data).sum()\n",
    "error = preds - target\n",
    "error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gradient = 2 * input_data * error\n",
    "gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_updated = weights - learning_rate * gradient\n",
    "preds_updated = (weights_updated * input_data).sum()\n",
    "error_updated = preds_updated - target\n",
    "error_updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating slopes\n",
    "\n",
    "You're now going to practice calculating slopes. When plotting the mean-squared error loss function against predictions, the slope is:\n",
    "- $2 * x * (xb-y)$\n",
    "- $2 * input\\_data * error$.\n",
    "\n",
    "Note that $x$ and $b$ may have multiple numbers ($x$ is a vector for each data point, and $b$ is a vector). In this case, the output will also be a vector, which is exactly what you want.\n",
    "\n",
    "You're ready to write the code to calculate this slope while using a single data point. You'll use pre-defined weights called `weights` as well as data for a single point called `input_data`. The actual value of the target you want to predict is stored in `target`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Calculate the predictions, `preds`, by multiplying `weights` by the `input_data` and computing their sum.\n",
    "- Calculate the error, which is `preds` minus `target`. Notice that this error corresponds to $xb-y$ in the gradient expression.\n",
    "- Calculate the slope of the loss function with respect to the prediction. To do this, you need to take the product of `input_data` and `error` and multiply that by $2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def get_slope`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_slope(input_data, target, weights):\n",
    "    # Calculate the predictions: preds\n",
    "    preds = (weights * input_data).sum()\n",
    "\n",
    "    # Calculate the error: error\n",
    "    error = preds - target\n",
    "\n",
    "    # Calculate the slope: slope\n",
    "    slope = 2 * input_data * error\n",
    "    \n",
    "    return slope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array([0, 2, 1])\n",
    "input_data = np.array([1, 2, 3])\n",
    "target = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_slope(input_data, target, weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improving model weights\n",
    "\n",
    "You've just calculated the slopes you need. Now it's time to use those slopes to improve your model. If you add the slopes to your weights, you will move in the right direction. However, it's possible to move too far in that direction. So you will want to take a small step in that direction first, using a lower learning rate, and verify that the model is improving.\n",
    "\n",
    "The weights have been pre-loaded as `weights`, the actual value of the target as `target`, and the input data as `input_data`. The predictions from the initial weights are stored as `preds`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Set the learning rate to be $0.01$ and calculate the error from the original predictions. This has been done for you.\n",
    "- Calculate the updated weights by subtracting the product of `learning_rate` and `slope` from `weights`.\n",
    "- Calculate the updated predictions by multiplying `weights_updated` with `input_data` and computing their sum.\n",
    "- Calculate the error for the new predictions. Store the result as `error_updated`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate: learning_rate\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the predictions: preds\n",
    "preds = (weights * input_data).sum()\n",
    "\n",
    "# Calculate the error: error\n",
    "error = preds - target\n",
    "\n",
    "# Calculate the slope: slope\n",
    "slope = 2 * input_data * error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update the weights: weights_updated\n",
    "weights_updated = weights - learning_rate * slope\n",
    "\n",
    "# Get updated predictions: preds_updated\n",
    "preds_updated = (weights_updated * input_data).sum()\n",
    "\n",
    "# Calculate updated error: error_updated\n",
    "error_updated = preds_updated - target\n",
    "\n",
    "# Print the original error\n",
    "print(error)\n",
    "\n",
    "# Print the updated error\n",
    "print(error_updated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making multiple updates to weights\n",
    "\n",
    "You're now going to make multiple updates so you can dramatically improve your model weights, and see how the predictions improve with each update.\n",
    "\n",
    "To keep your code clean, there is a pre-loaded `get_slope()` function that takes `input_data`, `target`, and `weights` as arguments. There is also a `get_mse()` function that takes the same arguments. The `input_data`, `target`, and `weights` have been pre-loaded.\n",
    "\n",
    "This network does not have any hidden layers, and it goes directly from the input (with 3 nodes) to an output node. Note that `weights` is a single array.\n",
    "\n",
    "We have also pre-loaded `matplotlib.pyplot`, and the error history will be plotted after you have done your gradient descent steps.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Using a `for loop` to iteratively update weights:\n",
    "- Calculate the slope using the `get_slope()` function.\n",
    "- Update the weights using a learning rate of $0.01$.\n",
    "- Calculate the mean squared error (`mse`) with the updated weights using the `get_mse()` function.\n",
    "- Append `mse` to `mse_hist`.\n",
    "- What trend do you notice?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def get_mse`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mse(input_data, target, weights):\n",
    "    \n",
    "    preds = (weights * input_data).sum()\n",
    "    mse = mean_squared_error([target], [preds])\n",
    "    return mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.array([0, 2, 1])\n",
    "input_data = np.array([1, 2, 3])\n",
    "target = 0\n",
    "\n",
    "n_updates = 20\n",
    "mse_hist = []\n",
    "\n",
    "# Iterate over the number of updates\n",
    "for i in range(n_updates):\n",
    "    # Calculate the slope: slope\n",
    "    slope = get_slope(input_data, target, weights)\n",
    "    \n",
    "    # Update the weights: weights\n",
    "    weights = weights - 0.01 * slope\n",
    "    \n",
    "    # Calculate mse with new weights: mse\n",
    "    mse = get_mse(input_data, target, weights)\n",
    "    \n",
    "    # Append the mse to mse_hist\n",
    "    mse_hist.append(mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the mse history\n",
    "plt.plot(mse_hist)\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Mean Squared Error')\n",
    "plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**As you can see, the mean squared error decreases as the number of iterations go up.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation\n",
    "\n",
    "- You've used gradient descent to optimize weights in a simple model.\n",
    "- Now we'll add a technique called \"back propagation\" to calculate the slopes you need to optimize more complex deep learning models.\n",
    "- Just as forward propagation sends input data through the hidden layers and into the output layer, back propagation takes the error from the output layer and propagates it backward through the hidden layers, towards the input layer.\n",
    "- It calculates the necessary slopes sequentially from the weights closest to the prediction, through the hidden layers, eventually back to the weights coming from the inputs.\n",
    "- We then use these slopes to update our weights as you've seen.\n",
    "- Back propagation is tricky, so you should focus on the general structure of the algorithm, rather than trying to memorize every mathematical detail.\n",
    "- ![][1]\n",
    "- ![][2]\n",
    "- ![][3]\n",
    "1. Backpropagation process\n",
    "   - In the big picture, we are trying to estimate the slope of the loss function with respect to each weight in our network.\n",
    "   - You've already seen that we use prediction errors to calculate some of those slopes.\n",
    "   - So we always do forward propagation to make a prediction and calculate an error before we do back propagation.\n",
    "   - Here are the results of forward propagation. Node values are in white and weights are in black.\n",
    "   - We need to be at this step before we can start back-propagation.\n",
    "   - Notice, we are using the \"relu\" activation function.\n",
    "     - So any node whose input is negative takes a value of 0, and that happens in the top node of the first hidden layer.\n",
    "   - For back-propagation, we go back one layer at a time, and each time we go back a layer, we'll use a formula for slopes that you saw in the last video.\n",
    "   - Every weight feeds from some input node into some output node.\n",
    "   - The three things we multiply to get the slope for that weight are\n",
    "      1. the value at the weights input node\n",
    "      1. the slope from plotting the loss function against that weight's output node\n",
    "      1. the slope of the activation function at the weight's output.\n",
    "   - We know the value at the node feeding into this weight.\n",
    "   - Either it is in an input layer, in which case we have it from the data. Or that node is in a hidden layer, in which case we calculated its value when we did forward propagation.\n",
    "   - The second item on this list is the slope of the loss function with respect to the output node.\n",
    "   - We do backward propagation from the right side of our diagram to the left.\n",
    "   - So we already calculated that slope by the time we to plug it into the current calculation.\n",
    "   - Finally we need the slope of the activation function at the node it feeds into.\n",
    "   - ![][4]\n",
    "   - ![][5]\n",
    "1. ReLU Activation Function\n",
    "   - You can see from this diagram that, for the ReLU function, the slope is 0 if the input into a node is negative.\n",
    "   - If the input into the node is positive, the output is the same as the input.\n",
    "   - So the slope would be 1.\n",
    "   - ![][6]\n",
    "1. Backpropagation process\n",
    "   - So far, we have focused on calculating slopes of the loss function with respect to weights.\n",
    "   - We also keep track of the slopes of the loss function with respect to node values, because we use those slopes in our calculations of slopes at weights.\n",
    "   - The slope of the loss function with respect to any node value is the sum of the slopes for every weight coming into that node.\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl45.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl46.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl47.jpg\n",
    "  [4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl48.jpg\n",
    "  [5]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl49.jpg\n",
    "  [6]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl50.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The relationship between forward and backward propagation\n",
    "\n",
    "If you have gone through 4 iterations of calculating slopes (using backward propagation) and then updated weights, how many times must you have done forward propagation?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~0.~~\n",
    "- ~~1.~~\n",
    "- **4.**\n",
    "  - Each time you generate predictions using forward propagation, you update the weights using backward propagation.\n",
    "- ~~8.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thinking about backward propagation\n",
    "\n",
    "If your predictions were all exactly right, and your errors were all exactly 0, the slope of the loss function with respect to your predictions would also be 0. In that circumstance, which of the following statements would be correct?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- **The updates to all weights in the network would also be 0.**\n",
    "  - In this situation, the updates to all weights in the network would indeed also be 0\n",
    "- ~~The updates to all weights in the network would be dependent on the activation functions.~~\n",
    "- ~~The updates to all weights in the network would be proportional to values from the input data.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation in practice\n",
    "\n",
    "- Let's see this back propagation in a deeper network.\n",
    "1. Backpropagation\n",
    "   - Start at the last set of weights.\n",
    "   - Those are currently 1 and 2.\n",
    "   - We multiply 3 things.\n",
    "     - The node values feeding into these weights are 1 and 3.\n",
    "     - The relevant slope for the output node is 2 times the error.\n",
    "     - That's 6. And the slope of the activation function is 1, since the output node is positive.\n",
    "   - So, we have a slope for the top weight of 6, and a slope for the bottom weight of 18.\n",
    "   - Those slopes we just calculated feed into the formula associated with weights further back in the network.\n",
    "   - Let's do that calculation one layer back now. We've hidden the earlier and later layers, since we don't need them to calculate the slopes for this layer of the network.\n",
    "   - This graph uses white to denotes node values, black to denote weight values, and the red shows the calculated slopes of the loss function with respect to that node, which we just finished calculating.\n",
    "   - This is all the information we need to calculate the slopes of the loss function with respect to the weights in this diagram.\n",
    "   - ![][1]\n",
    "   - ![][2]\n",
    "   - ![][3]\n",
    "1. Calculating slopes associated with any weight\n",
    "   - Recall, the three things we multiply to get slopes associated with any weight:\n",
    "      1. value at the node feeding into the weight\n",
    "      1. the slope of the activation function for the node being fed into (that slope is 1 in all cases here)\n",
    "      1. the slope of the loss function with respect to the output node\n",
    "1. Backpropagation\n",
    "   - Let's start with the slopes related to the weights going into the top node.\n",
    "   - For the top weight going into the top node, we multiply 0 for the input node's value, which is in white.\n",
    "   - Times 6 for the output node's slope, which is in red.\n",
    "   - Times the derivative of the ReLU activation function.\n",
    "   - That output node has a positive value for the input, so the ReLU activation has a slope of 1 (e.g. 0 times 6 times 1 is 0.)\n",
    "   - For the other weight going into this node, we have 1 times 6 times the slope of the ReLU activation function at the output node's value.\n",
    "   - The slope of the activation function is still 1.\n",
    "   - So, we have 1 times 6 times 1, which is 6.\n",
    "   - Here we also show slopes associated with the other two weights.\n",
    "   - We would multiply them all by a learning rate, and use the results to update the weights in gradient descent.\n",
    "   - Pause the video and make sure you understand how these last two weights were calculated.\n",
    "   - You are through the hardest concepts in this course, which are gradient descent and back-propagation.\n",
    "1. Backpropagation: Recap\n",
    "   - As a recap, we start at some random set of weights.\n",
    "   - We then go through the following iterative process\n",
    "      1. Use forward propagation to make a prediction.\n",
    "      1. Use backward propagation to calculate the slope of the loss function with respect to each weight.\n",
    "      1. Multiply that slope by the learning rate, and subtract that from the current weights.\n",
    "   - Keep going with that cycle until we get to a flat part. \n",
    "1. Stochastic gradient descent\n",
    "   - For computational efficiency, it is common to calculate slopes on only a subset of the data, called a batch, for each update of the weights.\n",
    "   - You then use a different batch of data to calculate the next update.\n",
    "   - Once we have used all our data, we start over again at the beginning of the data.\n",
    "   - Each time through the full training data is called an epoch.\n",
    "   - So if we're going through our data for the 3rd time, we'd say we are on the 3rd epoch.\n",
    "   - **When slopes are calculated on one batch at a time, rather than on the full data, that is called _stochastic gradient descent_**, rather than gradient descent, which uses all of the data for each slope calculation.\n",
    "   - The process will be partially automated for you, but understanding the process will help fix any surprises that come up when building your models.\n",
    "   \n",
    "   \n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl51.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl52.jpg\n",
    "  [3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl53.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A round of backpropagation\n",
    "\n",
    "In the network shown below, we have done forward propagation, and node values calculated as part of forward propagation are shown in white. The weights are shown in black. Layers after the question mark show the slopes calculated as part of back-prop, rather than the forward-prop values. Those slope values are shown in purple.\n",
    "\n",
    "This network again uses the ReLU activation function, so the slope of the activation function is 1 for any node receiving a positive value as input. Assume the node being examined had a positive value (so the activation function's slope is 1).\n",
    "\n",
    "![][1]\n",
    "\n",
    "What is the slope needed to update the weight with the question mark?\n",
    "\n",
    "![][2]\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~0.~~\n",
    "- ~~2.~~\n",
    "- **6.**\n",
    "- ~~Not enough information.~~\n",
    "\n",
    "\n",
    "  [1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl54.jpg\n",
    "  [2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl55.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building deep learning models with keras\n",
    "\n",
    "In this chapter, you'll use the Keras library to build deep learning models for both regression and classification. You'll learn about the Specify-Compile-Fit workflow that you can use to make predictions, and by the end of the chapter, you'll have all the tools necessary to build deep neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a keras model\n",
    "\n",
    "1. You've learned the theory of back-propagation, which is core to understanding deep learning. Now you'll learn how to create and optimize these networks using the Keras interface to the TensorFlow deep learning library.\n",
    "\n",
    "2. Model building steps\n",
    "The Keras workflow has 4 steps. First, you specify the architecture, which is things like: how many layers do you want? how many nodes in each layer? What activation function do you want to use in each layer? Next, you compile the model. This specifies the loss function, and some details about how optimization works. Then you fit the model. Which is that cycle of back-propagation and optimization of model weights with your data. And finally you will want to use your model to make predictions. We'll go through these steps sequentially. The first step is creating or specifying your model.\n",
    "    - Specify Architecture\n",
    "    - Compile\n",
    "    - Fit\n",
    "    - Predict\n",
    "\n",
    "3. Model specification\n",
    "Here is the code to do that. This code has three blocks. First we import what we will need. Numpy is here only for reading some data. The other two imports are used for building our model. The second block of two lines reads the data. We read the data here so we can find the number of nodes in the input layer. That is stored as the variable n_cols. We always need to specify how many columns are in the input when building a keras model, because that is the number of nodes in the input layer. We then start building the model. The first line of model specification is model equals Sequential. There are two ways to build up a model, and we will focus on sequential, which is the easier way to build a model. Sequential models require that each layer has weights or connections only to the one layer coming directly after it in the network diagram. There are more exotic models out there with complex patterns of connections, but Sequential will do the trick for everything we need here. We start adding layers using the add method of the model. he type of layer you have seen, that standard layer type, is called a Dense layer. It is called Dense because all of the nodes in the previous layer connect to all of the nodes in the current layer. As you advance in deep learning, you may start using layers that aren't Dense. In each layer, we specify the number of nodes as the first positional argument, and the activation function we want to use in that layer using the keyword argument activation. Keras supports every activation function you will want in practice. In the first layer, we need to specify input shapes as shown here. That says the input will have n_cols columns, and there is nothing after the comma, meaning it can have any number of rows, that is, any number of data points. You'll notice the last layer has 1 node. That is the output layer, and it matches those diagrams where we ended with only a single node as the output or prediction of the model. This model has 2 hidden layers, and an output layer. You may be struck that each hidden layers has 100 nodes. Keras and TensorFlow do the math for us, so don't feel afraid to use much bigger networks than we've seen before. It's quite common to use 100 or 1000s nodes in a layer. You'll learn more about choosing an appropriate number of nodes later.\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "predictors = np.loadtxt('predictors_data.csv', delimiter=',')\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape=(n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding your data\n",
    "You will soon start building models in Keras to predict wages based on various professional and demographic factors. Before you start building a model, it's good to understand your data by performing some exploratory analysis.\n",
    "\n",
    "The data is pre-loaded into a pandas DataFrame called df. Use the .head() and .describe() methods in the IPython Shell for a quick overview of the DataFrame.\n",
    "\n",
    "The target variable you'll be predicting is wage_per_hour. Some of the predictor variables are binary indicators, where a value of 1 represents True, and 0 represents False.\n",
    "\n",
    "Of the 9 predictor variables in the DataFrame, how many are binary indicators? The min and max values as shown by .describe() will be informative here. How many binary indicator predictors are there?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~0.~~\n",
    "- ~~5.~~\n",
    "- **6.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hourly_wages.csv\n",
    "hw = pd.read_csv(data_paths[0])\n",
    "\n",
    "display(hw.head(3))\n",
    "\n",
    "hw.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying a model\n",
    "Now you'll get to work with your first model in Keras, and will immediately be able to run more complex neural network models on larger datasets compared to the first two chapters.\n",
    "\n",
    "To start, you'll take the skeleton of a neural network and add a hidden layer and an output layer. You'll then fit that model and see Keras do the optimization so your model continually gets better.\n",
    "\n",
    "As a start, you'll predict workers wages based on characteristics like their industry, education and level of experience. You can find the dataset in a pandas dataframe called df. For convenience, everything in df except for the target has been converted to a NumPy matrix called predictors. The target, wage_per_hour, is available as a NumPy matrix called target.\n",
    "\n",
    "For all exercises in this chapter, we've imported the Sequential model constructor, the Dense layer constructor, and pandas.\n",
    "\n",
    "**Instructions**\n",
    "- Store the number of columns in the `predictors` data to `n_col`. This has been done for you.\n",
    "- Start by creating a `Sequential` model called `model`.\n",
    "- Use the `.add()` method on `model` to add a `Dense` layer.\n",
    "  - Add `50` units, specify `activation='relu'`, and the `input_shape` parameter to be the tuple `(n_cols,)` which means it has `n_cols` items in each row of data, and any number of rows of data are acceptable as inputs.\n",
    "- Add another `Dense` layer. This should have `32` units and a `'relu'` activation.\n",
    "- Finally, add an output layer, which is a `Dense` layer with a single node. Don't use any activation function here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "# import keras\n",
    "# from keras.layers import Dense\n",
    "# from keras.models import Sequential\n",
    "\n",
    "predictors = hw.iloc[:, 1:].values\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# Set up the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "model.add(Dense(50, activation='relu', input_shape=(n_cols,)))\n",
    "\n",
    "# Add the second layer\n",
    "model.add(Dense(30, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compiling and fitting a model\n",
    "\n",
    "1. After you've specified a model, the next task is to compile it, which sets up the network for optimization, for instance creating an internal function to do back-propagation efficiently. The compile methods\n",
    "\n",
    "2. Why you need to compile your model\n",
    "has two important arguments for you to choose. The first is what optimizer to use, which controls the learning rate. In practice, the right choice of learning rate can make a big difference for how quickly our model finds good weights, and even how good a set of weights it can find. There are a few algorithms that automatically tune the learning rate. Even many experts in the field don't know all the details of all the optimization algorithms. So the pragmatic approach is to choose a versatile algorithm and use that for most problems. Adam is an excellent choice as your go-to optimizer. Adam adjusts the learning rate as it does gradient descent, to ensure reasonable values throughout the weight optimization process. The second thing you specify is the loss function. Mean squared error is the most common choice for regression problems. When we use keras for classification, you will learn a new default metric.\n",
    "\n",
    "    - Specify the optimizer\n",
    "      - Many options and mathematically complex\n",
    "      - \"Adam\" is usually a good choice\n",
    "    - Loss function\n",
    "      - \"mean_squared_error\" common for regression\n",
    "\n",
    "4. Compiling a model\n",
    "Here is an example of the code to compile a model. It builds a model, as you've already seen, and then we add a compile command after building the model. After compiling the model, you can\n",
    "\n",
    "```python\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "```\n",
    "\n",
    "6. What is fitting a model\n",
    "fit it. That is applying back-propagation and gradient descent with your data to update the weights. The fit step looks similar to what you've seen in scikit-learn, though it has more options which we will explore soon. Even with the Adam optimizer, which is pretty smart, it can improve your optimization process if you scale all the data so each feature is, on average, about similar sized values. One common approach is to subtract each feature by that features mean, and divide it by it's standard deviation.\n",
    "\n",
    "     - Applying back-propagation and gradient descent with your data to update the weights\n",
    "     - Scaling data before fitting can ease optimization\n",
    "\n",
    "8. Fitting a model\n",
    "You can see what the code looks like here. After the compile step, we run fit, with the predictors as the first argument. When you run this, you will see some output showing the optimizations progress as it fits the data. We'll go into more detail about this output soon, but for now,\n",
    "\n",
    "```python\n",
    "...\n",
    "model.fit(predictors, target)\n",
    "```\n",
    "\n",
    "10. Let's practice!\n",
    "just think of it as a log showing model performance on the training data as we update model weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the model\n",
    "You're now going to compile the model you specified earlier. To compile the model, you need to specify the optimizer and loss function to use. In the video, Dan mentioned that the Adam optimizer is an excellent choice. You can read more about it as well as other keras optimizers [here][1], and if you are really curious to learn more, you can read the [original paper][2] that introduced the Adam optimizer.\n",
    "\n",
    "In this exercise, you'll use the Adam optimizer and the mean squared error loss function. Go for it!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Compile the model using `model.compile()`. Your `optimizer` should be `'adam'` and the loss should be `'mean_squared_error'`.\n",
    "\n",
    "\n",
    "  [1]: https://keras.io/optimizers/#adam\n",
    "  [2]: https://arxiv.org/abs/1412.6980v8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "# import keras\n",
    "# from keras.layers import Dense\n",
    "# from keras.models import Sequential\n",
    "\n",
    "# Specify the model\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Verify that model contains information from compiling\n",
    "print(\"Loss function: \" + model.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the model\n",
    "\n",
    "You're at the most fun part. You'll now fit the model. Recall that the data to be used as predictive features is loaded in a NumPy matrix called `predictors` and the data to be predicted is stored in a NumPy matrix called `target`. Your `model` is pre-written and it has been compiled with the code from the previous exercise.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Fit the `model`. Remember that the first argument is the predictive features (`predictors`), and the data to be predicted (`target`) is the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "# import keras\n",
    "# from keras.layers import Dense\n",
    "# from keras.models import Sequential\n",
    "\n",
    "target = hw.wage_per_hour.values\n",
    "\n",
    "# Specify the model\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Fit the model\n",
    "model.fit(x=predictors, y=target, epochs=5, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification models\n",
    "\n",
    "- So far we have focused on regression models. But deep learning works similarly for classification, that is for predicting outcomes from a set of discrete options.\n",
    "\n",
    "2. Classification\n",
    "For classification, you do a couple of things differently. The biggest changes are:1, set the loss function as `'categorical_crossentropy'` instead of `'mean_squared_error'`. This isn't the only possible loss function for classification problems, but it is by far the most common. You may have heard of this before under the name LogLoss. We won't go into the mathematics of categorical crossentropy here. For categorical crossentropy loss function, a lower score is better. But it's still hard to interpret. So I've added this argument \"metrics equals accuracy\". This means I want to print out the accuracy score at the end of each epoch, which makes it easier to see and understand the models progress. Second you need to modify the last layer, so it has a separate node for each potential outcome. You will also change the activation function to softmax. The softmax activation function ensures the predictions sum to 1, so they can be interpreted like probabilities.\n",
    "\n",
    "    - `categorical_crossentropy` loss function\n",
    "    - Similar to log loss: Lower is better\n",
    "    - Add `metrics=['accuracy']` to compile step for easy-to-understand diagnostics\n",
    "    - Output layer has separate node for each possible outcome, and uses `'softmax'` activation\n",
    "\n",
    "3. Quick look at the data\n",
    "Here is some data for a binary classification problem. We have data from the NBA basketball league. It includes facts about each shot, and the"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| shot_clock | dribbles | touch_time | shot_dis | close_def_dis | shot_result |\r\n",
    "|------------|----------|------------|----------|---------------|-------------|\r\n",
    "| 10.8       | 2        | 1.9        | 7.7      | 1.3           | 1           |\r\n",
    "| 3.4        | 0        | 0.8        | 28.2     | 6.1           | 0           |\r\n",
    "| 0          | 3        | 2.7        | 10.1     | 0.9           | 0           |\r\n",
    "| 10.3       | 2        | 1.9        | 17.2     | 3.4           | 0        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Quick look at the data\n",
    "shot result is either 0 or 1, indicating whether the shot went in or not. The outcome here is in a single column, which is not uncommon. But in general, we'll want to convert categoricals in Keras to a format with a separate column for each output. Keras includes a function to do that, which you will see in the code soon. This setup is consistent with the fact that your model will have a separate node in the output for each possible class.\n",
    "\n",
    "5. Transforming to categorical\n",
    "We have a new column for each value of shot_result. A 1 in any column indicates that this column corresponds to the value from the original data. This is sometimes called one-hot encoding. If the original data had 3 or 4 or 100 different values, the new array for our data would have 3 or 4 or 100 columns respect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| shot_result |  | Outcome 0 | Outcome 1 |\r\n",
    "|-------------|---|-----------|-----------|\r\n",
    "| 1           | → | 0         | 1         |\r\n",
    "| 0           | → | 1         | 0         |\r\n",
    "| 0           | → | 1         | 0         |\r\n",
    "| 0           | → | 1    | 0         |     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "data = pd.read_csv('basketball_shot_log.csv')\n",
    "predictors = data.drop(['shot_result'], axis=1).values\n",
    "target = to_categorical(data['shot_result'])\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape=(n_cols,)))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(predictors, target)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Classification\n",
    "Lets look at the results now. Both accuracy and loss improve measurably for the first 3 epochs, and then the improvement slows down. Sometimes it gets a little worse for an epoch, sometimes it gets a little better. We will soon see a more sophisticated way to determine how long to train, but training for 10 epochs got us to that flat part of the loss function, so this worked well in this case.\n",
    "\n",
    "Epoch 1/10\r\n",
    "128069/128069 [==============================] - 4s - loss: 0.7706 - acc: 0.575\r\n",
    "\r\n",
    "Epoch 2/10\r\n",
    "128069/128069 [==============================] - 5s - loss: 0.6656 - acc: 0.003\r\n",
    "\r\n",
    "Epoch 3/10\r\n",
    "128069/128069 [==============================] - 6s - loss: 0.6611 - acc:0.6094\r\n",
    "\r\n",
    "Epoch 4/10\r\n",
    "128069/128069 [==============================] - 7s - loss: 0.6584 - ac: 0.6106\r\n",
    "\r\n",
    "Epoch 5/10\r\n",
    "128069/128069 [==============================] - 7s - loss: 0.6561  acc: 0.6150\r\n",
    "\r\n",
    "Epoch 6/10\r\n",
    "128069/128069 [==============================] - 9s - loss: 0.653 - acc: 0.6158\r\n",
    "\r\n",
    "Epoch 7/10\r\n",
    "128069/128069 [==============================] - 9s - loss: 06543 - acc: 0.6162\r\n",
    "\r\n",
    "Epoch 8/10\r\n",
    "128069/128069 [==============================] - 9s - loss 0.6538 - acc: 0.6158\r\n",
    "\r\n",
    "Epoch 9/10\r\n",
    "128069/128069 [==============================] - 10s - lss: 0.6535 - acc: 0.6157\r\n",
    "\r\n",
    "Epoch 10/10\r\n",
    "128069/128069 [==============================] - 10s  loss: 0.6531 - acc: 0.6166\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding your classification data\n",
    "\n",
    "Now you will start modeling with a new dataset for a classification problem. This data includes information about passengers on the Titanic. You will use predictors such as `age`, `fare` and where each passenger embarked from to predict who will survive. This data is from a tutorial on data science competitions. Look here for descriptions of the features.\n",
    "\n",
    "The data is pre-loaded in a pandas DataFrame called `df`.\n",
    "\n",
    "It's smart to review the maximum and minimum values of each variable to ensure the data isn't misformatted or corrupted. What was the maximum age of passengers on the Titanic? Use the `.describe()` method in the IPython Shell to answer this question.\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~29.699~~.\n",
    "- **80.**\n",
    "- ~~891~~.\n",
    "- ~~It is not listed.~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# titanic data\n",
    "df = pd.read_csv(data_paths[2])\n",
    "df.age_was_missing = df.age_was_missing.astype(int)  # convert Boolean to a number\n",
    "display(df.head(2))\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last steps in classification models\n",
    "You'll now create a classification model using the titanic dataset, which has been pre-loaded into a DataFrame called `df`. You'll take information about the passengers and predict which ones survived.\n",
    "\n",
    "The predictive variables are stored in a NumPy array `predictors`. The target to predict is in `df.survived`, though you'll have to manipulate it for keras. The number of predictive features is stored in `n_cols`.\n",
    "\n",
    "Here, you'll use the `'sgd'` optimizer, which stands for [Stochastic Gradient Descent][1]. You'll learn more about this in the next chapter!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Convert `df.survived` to a categorical variable using the `to_categorical()` function.\n",
    "- Specify a `Sequential` model called `model`.\n",
    "- Add a `Dense` layer with `32` nodes. Use `'relu'` as the `activation` and `(n_cols,)` as the `input_shape`.\n",
    "- Add the `Dense` output layer. Because there are two outcomes, it should have 2 units, and because it is a classification model, the `activation` should be `'softmax'`.\n",
    "- Compile the model, using `'sgd'` as the `optimizer`, `'categorical_crossentropy'` as the loss function, and `metrics=['accuracy']` to see the accuracy (what fraction of predictions were correct) at the end of each epoch.\n",
    "- Fit the model using the `predictors` and the `target`.\n",
    "\n",
    "\n",
    "  [1]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "# import keras\n",
    "# from keras.layers import Dense\n",
    "# from keras.models import Sequential\n",
    "# from keras.utils import to_categorical\n",
    "\n",
    "predictors = df.iloc[:, 1:].values\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# Convert the target to categorical: target\n",
    "target = to_categorical(df.survived)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "model.add(Dense(32, activation='relu', input_shape=(n_cols,)))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors, target, epochs=5, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using models\n",
    "\n",
    "- Now that you can build basic deep learning models, I'll show you how to use them. Then we'll go into some finer details on fine tuning model architectures. The things you'll want to do in order to use these models are\n",
    "  - save a model after you've trained it\n",
    "  - reload that model\n",
    "  - make predictions with the model\n",
    "\n",
    "- Saving, reloading and using your Model\n",
    "  - ```python\n",
    "    from tensorflow.keras.models import load_model\n",
    "    model.save('model_file.h5')\n",
    "    my_model = load_model('model_file.h5')\n",
    "    predictions = my_model.predict(data_to_predict_with)\n",
    "    probability_true = predictions[:, 1]\n",
    "    ```\n",
    "  - Here is the code to save a model, reload it, and make predictions. We've imported a `load_model` function here. Once I have a model I want to save, I can save it with the `save` method. I supply a filename. Models are saved in a format called hdf5, for which h5 is the common extension. I then load the model back into memory with the `load_model` function here. I then make predictions. The model I've loaded here is a classification model. The predictions come in the same format as the prediction target. You may recall that this had 1 column for whether the shot was missed, and then a 2nd column for whether the shot was made. In practice, I probably only want the probability that the shot is made. So, I'll extract that second column with `numpy` indexing, and I called that `probability_true`. Lastly, sometimes I'll want to verify that the model I loaded has the same structure I expect.\n",
    "\n",
    "- Verifying model structure\n",
    "  - You can print out a summary of the model architecture with the `summary` method. You can see the output here. Now that you can save your model, reload it, make predictions, and verify its structure, you have most of what you need to not just build models, but to work with them in practical situations.\n",
    "  - ![][1]\n",
    " \n",
    "[1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/model_summary.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions\n",
    "\n",
    "The trained network from your previous coding exercise is now stored as `model`. New data to make predictions is stored in a NumPy array as `pred_data`. Use `model` to make predictions on your new data.\n",
    "\n",
    "In this exercise, your predictions will be probabilities, which is the most common way for data scientists to communicate their predictions to colleagues.\n",
    "\n",
    "**Instructions**\n",
    "- Create your predictions using the model's `.predict()` method on `pred_data`.\n",
    "- Use NumPy indexing to find the column corresponding to predicted probabilities of survival being True. This is the second column (index `1`) of `predictions`. Store the result in `predicted_prob_true` and print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify, compile, and fit the model\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape=(n_cols,)))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.compile(optimizer='sgd', \n",
    "              loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(predictors[:800, :], target[:800, :])\n",
    "\n",
    "# Calculate predictions: predictions\n",
    "pred_data = predictors[800:, :]\n",
    "predictions = model.predict(pred_data)\n",
    "\n",
    "# Calculate predicted probability of survival: predicted_prob_true\n",
    "predicted_prob_true = predictions[:, 1]\n",
    "\n",
    "# Print predicted_prob_true\n",
    "predicted_prob_true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning keras models\n",
    "\n",
    "Learn how to optimize your deep learning models in Keras. Start by learning how to validate your models, then understand the concept of model capacity, and finally, experiment with wider and deeper networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding model optimization\n",
    "\n",
    "- At this point, you have a good understanding of how neural networks work, and how to build them in Keras. But you probably don't yet have a great intuition for how to choose things like model architecture and model optimization arguments. You'll learn that in this last chapter.\n",
    "\n",
    "2. Why optimization is hard\n",
    "In practice, optimization is a hard problem. The optimal value for any one weight depends on the values of the other weights, and we are optimizing many weights at once. Even if the slope tells us which weights to increase, and which to decrease, our updates may not improve our model meaningfully. A small learning rate might cause us to make such small updates to the model's weights that our model doesn't improve materially. A very large learning rate might take us too far in the direction that seemed good. A smart optimizer like Adam helps, but optimization problems can still occur. The easiest way to see the effect of different learning rates is to use the simplest optimizer,\n",
    "  - Simultaneously optimizing 1000s of parameters with complex relationships\n",
    "  - Updates may not improve model meaningfully\n",
    "  - Updates too small (if learning rate is slow) or too large (if learning rate is high)\n",
    "\n",
    "3. Stochastic gradient descent\n",
    "_Stochastic Gradient Descent_, sometimes abbreviated to _SGD_. This optimizer uses a fixed learning rate. Learning rates around point-01 are common. But you can specify the learning rate you need with `lr` argument as shown here. We have a function that creates a new model here. We create models in a for loop, and each time around we compile the model using SGD with a different learning rate. We pass in the optimizer with the same argument where we previously passed the string for \"Adam\". In an exercise, you will compare the results of training models trained with low, medium and high learning rates. Even if your learning rate is well tuned, you can run into the so-called\n",
    "\n",
    "4. The dying neuron problem\n",
    "\"dying-neuron\" problem. This problem occurs when a neuron takes a value less than 0 for all rows of your data. Recall that, with the ReLU activation function, any node with a negative input value produces an output of 0, and it also has a slope of 0 as you see in this graph. Because the slope is 0, the slope of any weights flowing into that node are also 0. So those weights don't get updated. In other words, once the node starts always getting negative inputs, it may continue only getting negative inputs. It's contributing nothing to the model at this point, and hence the claim that the node or neuron is \"dead.\" At first, this might suggest using an activation function whose slope is never exactly zero. However, those types of functions were used for many years.\n",
    "  - ![][1]\n",
    "\n",
    "5. Vanishing gradients\n",
    "For example, in an earlier video we used an s-shaped function called tanh. However, values that were outside the middle of the S were\n",
    "  - ![][2]\n",
    "\n",
    "6. Vanishing gradients\n",
    "relatively flat, or had small slopes. A small but non-zero slope might work in a network with only a few hidden layers. But in a deep network, one with many layers, the repeated multiplication of small slopes causes the slopes to get close to 0, which meant updates in backprop were close to 0. This is called the vanishing gradient problem. This in turn might suggest using an activation function that isn't even close to flat anywhere. There is research in this area, including variations on ReLU. Those aren't widely used though. For now, it's a phenomenon worth keeping in mind if you are ever pondering why your model isn't training better. If it happens, changing the activation function may be the solution.\n",
    "  - Occurs when many layers have very small slopes (e.g. due to being on flat part of tanh curve\n",
    "  - In deep networks, updates to backprop were close to 0\n",
    "\n",
    "[1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl56.jpg\n",
    "[2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl57.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `def get_new_model`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_new_model(input_shape):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "    model.add(Dense(100, activation='relu'))\n",
    "    model.add(Dense(2, activation='softmax'))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "lr_to_test = [0.000001, 0.01, 1]\n",
    "\n",
    "# loop over learning rates\n",
    "for lr in lrt_to_test:\n",
    "    model = get_new_model()\n",
    "    my_optimizer = SGD(lr=lr)\n",
    "    model.compile(optimizer=my_optimizer, loss='categorical_crossentropy')\n",
    "    model.fit(predictors, target)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Diagnosing optimization problems\n",
    "\n",
    "Which of the following could prevent a model from showing an improved loss in its first few epochs?\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- ~~Learning rate too low~~\n",
    "- ~~Learning rate too high~~\n",
    "- ~~Poor choice of activation function~~\n",
    "- **All of the above**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing optimization parameters\n",
    "\n",
    "It's time to get your hands dirty with optimization. You'll now try optimizing a model at a very low learning rate, a very high learning rate, and a \"just right\" learning rate. You'll want to look at the results after running this exercise, remembering that a low value for the loss function is good.\n",
    "\n",
    "For these exercises, we've pre-loaded the predictors and target values from your previous classification models (predicting who would survive on the Titanic). You'll want the optimization to start from scratch every time you change the learning rate, to give a fair comparison of how each learning rate did in your results. So we have created a function `get_new_model()` that creates an unoptimized model to optimize.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `SGD` from `keras.optimizers`.\n",
    "- Create a list of learning rates to try optimizing with called `lr_to_test`. The learning rates in it should be `.000001`, `0.01`, and `1`.\n",
    "- Using a `for` loop to iterate over `lr_to_test`:\n",
    "  - Use the `get_new_model()` function to build a new, unoptimized model.\n",
    "  - Create an optimizer called `my_optimizer` using the `SGD()` constructor with keyword argument `lr=lr`.\n",
    "  - Compile your model. Set the optimizer parameter to be the SGD object you created above, and because this is a classification problem, use `'categorical_crossentropy'` for the `loss` parameter.\n",
    "  - Fit your model using the `predictors` and `target`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup data\n",
    "# titanic data\n",
    "df = pd.read_csv(data_paths[2])\n",
    "df.age_was_missing = df.age_was_missing.astype(int)  # convert Boolean to a number\n",
    "\n",
    "predictors = df.iloc[:, 1:].values\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Convert the target to categorical: target\n",
    "target = to_categorical(df.survived)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the SGD optimizer, which is done in the Imports section\n",
    "# from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "# Create list of learning rates: lr_to_test\n",
    "lr_to_test = [0.000001, 0.01, 0.1]\n",
    "\n",
    "# Loop over learning rates\n",
    "for lr in lr_to_test:\n",
    "    print('\\n\\nTesting model with learning rate: %f\\n'%lr )\n",
    "    \n",
    "    # Build new model to test, unaffected by previous models\n",
    "    model = get_new_model(input_shape)\n",
    "    \n",
    "    # Create SGD optimizer with specified learning rate: my_optimizer\n",
    "    my_optimizer = SGD(learning_rate=lr)\n",
    "    \n",
    "    # Compile the model\n",
    "    model.compile(optimizer=my_optimizer, loss='categorical_crossentropy')\n",
    "    \n",
    "    # Fit the model\n",
    "    model.fit(predictors, target)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model validation\n",
    "\n",
    "1. You may recall from previous machine learning classes that your model's performance on the training data is not a good indication of how it will perform on new data. For this reason, we use validation data to test model performance. Validation data is data that is explicitly held out from training, and used only to test model performance.\n",
    "\n",
    "2. Validation in deep learning\n",
    "You may already be familiar with k-fold cross validation. In practice, few people run k-fold cross validation on deep learning models because deep learning is typically used on large datasets. So the computational expense of running k-fold validation would be large, and we usually trust a score from a single validation run because those validation runs are reasonably large. Keras makes it easy to use some of your data as validation data, and we see that\n",
    "  - Commonly use validation split rather than cross-validation\n",
    "  - Deep learning widely used on large datasets\n",
    "  - Single validation score is based on large amount of data, and is reliable\n",
    "\n",
    "3. Model validation\n",
    "in this code, where we specify the split using the keyword argument validation_split when calling the fit method. Here, we have already specified a model, and we'll make small changes in both the compile and fit steps to see model validation information. This is a classification problem, and we'd like to see measures of accuracy. So, we include metrics equals 'accuracy' in the compile step. In the fit step, we specify what fraction of the data is used for validation. In this case, we'll use 30%.Our goal is to have the best validation score possible, so we should keep training while validation score is improving, and then stop training when the validation score isn't improving. We do this with something called\n",
    "  - ![][58]\n",
    "\n",
    "4. Early Stopping\n",
    "\"early stopping.\" We can use early stopping with only some small changes to the code. See here, we've imported something called EarlyStopping. We then create an \"early stopping monitor\" before fitting the model. That monitor takes an argument called patience, which is how many epochs the model can go without improving before we stop training. 2 or 3 are reasonable values for patience. Sometimes you'll get a single epoch with no improvement, but the model will start improving again after that epoch. But if you see 3 epochs with no improvement, it's unlikely to turn around and start improving again. We pass early_stopping_monitor to the fit function under an argument called callbacks. Notice that callbacks takes a list. You may consider adding other callbacks as you become very advanced. But early stopping is all you want for now. By default, keras trains for 10 epochs. Now that we have smart logic for determining when to stop, we can set a high maximum number of epochs. This happens with the nb_epoch argument, as you see here. Keras will go until this number of epochs, unless the validation loss stops improving, in which case it will stop earlier. This is smarter training logic than relying on a fixed number of epochs without looking at the validation scores.\n",
    "  - ![][59]\n",
    "\n",
    "5. Output from early stopping\n",
    "Let's look at the output. In epoch 9 we had a validation loss score of point-6513. We didn't beat that score in the next 2 epochs, so we stopped training. Now that you have a reliable way of measuring model performance, namely through scores, you should feel free\n",
    "  - ![][60]\n",
    "\n",
    "6. Experimentation\n",
    "to experiment with different architectures. More layers, fewer layers. Layers with more nodes, layers with fewer nodes. And so on. Creating a great model requires some experimentation. Before we finish, we'll give a little bit of insight into how to choose where you experiment.\n",
    "  - More layers\n",
    "  - Fewer layers\n",
    "  - Layers with more nodes\n",
    "  - Layers with fewer nodes\n",
    "  - Creating a great model requires experimentation\n",
    "\n",
    "7. Let's practice!\n",
    "But, now that you can get validation scores, you are poised to run those experiments and figure out what works best for your data.\n",
    "\n",
    "[58]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl58.jpg\n",
    "[59]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl59.jpg\n",
    "[60]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl60.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating model accuracy on validation dataset\n",
    "\n",
    "Now it's your turn to monitor model accuracy with a validation data set. A model definition has been provided as `model`. Your job is to add the code to compile it and then fit it. You'll check the validation score in each epoch.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Compile your model using `'adam'` as the `optimizer` and `'categorical_crossentropy'` for the `loss`. To see what fraction of predictions are correct (the `accuracy`) in each epoch, specify the additional keyword argument `metrics=['accuracy']` in `model.compile()`.\n",
    "- Fit the model using the `predictors` and `target`. Create a validation split of 30% (or `0.3`). This will be reported in each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "hist = model.fit(predictors, target, validation_split=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Early stopping: Optimizing the optimization\n",
    "\n",
    "Now that you know how to monitor your model performance throughout optimization, you can use early stopping to stop optimization when it isn't helping any more. Since the optimization stops automatically when it isn't helping, you can also set a high value for `epochs` in your call to `.fit()`, as Dan showed in the video.\n",
    "\n",
    "The model you'll optimize has been specified as `model`. As before, the data is pre-loaded as `predictors` and `target`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `EarlyStopping` from `tensorflow.keras.callbacks`.\n",
    "- Compile the model, once again using `'adam'` as the `optimizer`, `'categorical_crossentropy'` as the loss function, and `metrics=['accuracy']` to see the accuracy at each epoch.\n",
    "- Create an `EarlyStopping` object called `early_stopping_monitor`. Stop optimization when the validation loss hasn't improved for 2 epochs by specifying the `patience` parameter of `EarlyStopping()` to be `2`.\n",
    "- Fit the model using the `predictors` and `target`. Specify the number of `epochs` to be `30` and use a validation split of `0.3`. In addition, pass `[early_stopping_monitor]` to the `callbacks` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import EarlyStopping\n",
    "# from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Fit the model\n",
    "hist = model.fit(predictors, target, validation_split=0.3, epochs=30, callbacks=[early_stopping_monitor])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Wonderful work! Because optimization will automatically stop when it is no longer helpful, it is okay to specify the maximum number of epochs as 30 rather than using the default of 10 that you've used so far. Here, it seems like the optimization stopped after 4 epochs.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimenting with wider networks\n",
    "\n",
    "Now you know everything you need to begin experimenting with different models!\n",
    "\n",
    "A model called `model_1` has been pre-loaded. You can see a summary of this model printed in the IPython Shell. This is a relatively small network, with only 10 units in each hidden layer.\n",
    "\n",
    "In this exercise you'll create a new model called `model_2` which is similar to `model_1`, except it has 100 units in each hidden layer.\n",
    "\n",
    "After you create `model_2`, both models will be fitted, and a graph showing both models loss score at each epoch will be shown. We added the argument `verbose=False` in the fitting commands to print out fewer updates, since you will look at these graphically instead of as text.\n",
    "\n",
    "Because you are fitting two models, it will take a moment to see the outputs after you hit run, so be patient.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create `model_2` to replicate `model_1`, but use `100` nodes instead of `10` for the first two `Dense` layers you add with the `'relu'` activation. Use `2` nodes for the `Dense` output layer with `'softmax'` as the `activation`.\n",
    "- Compile `model_2` as you have done with previous models: Using `'adam'` as the `optimizer`, `'categorical_crossentropy'` for the loss, and `metrics=['accuracy']`.\n",
    "- Hit 'Submit Answer' to fit both the models and visualize which one gives better results! Notice the keyword argument `verbose=False` in `model.fit()`: This prints out fewer updates, since you'll be evaluating the models graphically instead of through text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the model\n",
    "model_1 = Sequential()\n",
    "model_1.add(Dense(10, activation='relu', input_shape=input_shape))\n",
    "model_1.add(Dense(10, activation='relu'))\n",
    "model_1.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model_1.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "model_1.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first and second layers\n",
    "model_2.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "model_2.add(Dense(100, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit model_1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model_2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', label='model_1')\n",
    "plt.plot(model_2_training.history['val_loss'], 'b', label='model_2')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "_ = plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The blue model is the one you made, the red is the original model. Your model had a lower loss value, so it is the better model. Nice job!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding layers to a network\n",
    "\n",
    "You've seen how to experiment with wider networks. In this exercise, you'll try a deeper network (more hidden layers).\n",
    "\n",
    "Once again, you have a baseline model called `model_1` as a starting point. It has 1 hidden layer, with `10` units. You can see a summary of that model's structure printed out. You will create a similar network with 3 hidden layers (still keeping 10 units in each layer).\n",
    "\n",
    "This will again take a moment to fit both models, so you'll need to wait a few seconds to see the results after you run your code.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Specify a model called `model_2` that is like `model_1`, but which has 3 hidden layers of 50 units instead of only 1 hidden layer.\n",
    "  - Use `input_shape` to specify the input shape in the first hidden layer.\n",
    "  - Use `'relu'` activation for the 3 hidden layers and `'softmax'` for the output layer, which should have 2 units.\n",
    "- Compile `model_2` as you have done with previous models: Using `'adam'` as the `optimizer`, `'categorical_crossentropy'` for the `loss`, and `metrics=['accuracy']`.\n",
    "- Hit 'Submit Answer' to fit both the models and visualize which one gives better results! For both models, you should look for the best val_loss and val_acc, which won't be the last epoch for that model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the model\n",
    "model_1 = Sequential()\n",
    "model_1.add(Dense(10, activation='relu', input_shape=input_shape))\n",
    "model_1.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model_1.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "model_1.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The input shape to use in the first hidden layer\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first, second, and third hidden layers\n",
    "model_2.add(Dense(10, activation='relu', input_shape=input_shape))\n",
    "model_2.add(Dense(10, activation='relu'))\n",
    "model_2.add(Dense(10, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit model 1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model 2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', label='model_1')\n",
    "plt.plot(model_2_training.history['val_loss'], 'b', label='model_2')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "_ = plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Great work! The blue model is the one you made and the red is the original model. The model with the lower loss value is the better model.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thinking about model capacity\n",
    "\n",
    "1. At this point, you know how to run experiments, and compare different models performance. However, it takes some practice to get an intuition for what experiments or architectures to try. There is still a little more art to finding good deep learning architectures than there is for tuning other machine learning algorithms. But something called \"model capacity\" should be one of the key considerations you think about when deciding what models to try.\"Model capacity\" or \"network capacity\" is closely related to the terms overfitting and underfitting.\n",
    "\n",
    "2. Overfitting\n",
    "You may recall overfitting and a graphic like this from a previous DataCamp course. Overfitting is the ability of a model to fit oddities in your training data that are there purely due to happenstance, and that won't apply in a new dataset.\n",
    "   - When you are overfitting, your model will make accurate predictions on training data, but it will make inaccurate predictions on validation data and new datasets.\n",
    "   - Underfitting is the opposite. That is when your model fails to find important predictive patterns in the training data. So it is accurate in neither the training data nor validation data.\n",
    "   - Because we want to do well on new datasets that weren't used for training the model, our validation score is the ultimate measure of a model's predictive quality.\n",
    "   - Let's get back to model capacity. Model capacity is a model's ability to capture predictive patterns in your data. So, the more capacity a model, the further to the right we will be on this graph. If you had a network, and you increased the number of nodes or neurons in a hidden layer, that would increase model capacity. And if you add layers, that increases capacity. Said another way, making larger layers or increasing the number of layers moves you further to the right of this graph. So, with that in mind,\n",
    "   - ![][61]\n",
    "\n",
    "3. Workflow for optimizing model capacity\n",
    "here is a good workflow for you. Start with a simple network, and get the validation score. Then keep adding capacity as long as the score keeps improving. Once it stops improving, you can decrease capacity slightly, but you are probably near the ideal.\n",
    "   - Start with a small network\n",
    "   - Gradually increase capacity\n",
    "   - Keep increasing capacity until validation score is no longer improving\n",
    "\n",
    "4. Sequential experiments\n",
    "Let's walk through that process once. Here, I've started a model that has one hidden layer and 100 units. That's a relatively simple, or low capacity, model. I get a mean squared error\n",
    "   - ![][62]\n",
    "\n",
    "7. Sequential experiments\n",
    "of 5-point-4. Since I started with a simple model, I now try increasing capacity. I could increase the number of layers or use more hidden nodes. I'll start by using more nodes in the one hidden layer. That improved the model, so I'll keep increasing capacity.\n",
    "\n",
    "8. Sequential experiments\n",
    "This time I'll switch to using 2 hidden layers. Each layer has 250 nodes. That improved the error more. So, I try 3 layers, continuing to add capacity as long as it helps.\n",
    "\n",
    "9. Sequential experiments\n",
    "This hurt the score. So, the model with 2 layers and 250 nodes is about perfect. I'll try another model that reduces capacity slightly from the last model I built.\n",
    "\n",
    "10. Sequential experiments\n",
    "That is 3 hidden layers with 200 nodes each. That seems the best model yet. So I'll stick with that. Should you change capacity by adding layers or by adding nodes to an existing layer? There isn't a universal answer to that. You can experiment. But you should generally be thinking about whether you are trying to increase or decrease capacity, ideally honing in on the right capacity by looking at validation scores.\n",
    "\n",
    "[61]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl61.jpg\n",
    "[62]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/dl62.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimenting with model structures\n",
    "You've just run an experiment where you compared two networks that were identical except that the 2nd network had an extra hidden layer. You see that this 2nd network (the deeper network) had better performance. Given that, which of the following would be a good experiment to run next for even better performance?\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- ~~Try a new network with fewer layers than anything you have tried yet.~~\n",
    "- **Use more units in each hidden layer.**\n",
    "- ~~Use fewer units in each hidden layer.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stepping up to images\n",
    "\n",
    "1. To close, you are going to apply your knowledge to a new and different dataset.\n",
    "\n",
    "2. Recognizing handwritten digits\n",
    "This is the MNIST dataset, which contains images of handwritten digits. This is a very popular dataset for getting started working with images. There is an image of each handwritten digit, and each image is composed of a 28 pixel by 28 pixel grid. The image is represented by showing how dark each pixel is. So, 0 would be as light as possible, and 255 is as dark as possible. I've flattened the 28 x 28 grid for you into a 784 x 1 array for each image. Each image shows a digit like 0, 1, 2, 3, 4, all the way up to 9. Your model will predict which digit it is that was written. So you will create a deep learning model taking in those 784 features for each image as inputs, and predicting digits from among 10 possible values for the output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building your own digit recognition model\n",
    "\n",
    "You've reached the final exercise of the course - you now know everything you need to build an accurate model to recognize handwritten digits!\n",
    "\n",
    "We've already done the basic manipulation of the MNIST dataset shown in the video, so you have `X` and `y` loaded and ready to model with. `Sequential` and `Dense` from `tensorflow.keras` are also pre-imported.\n",
    "\n",
    "To add an extra challenge, we've loaded only 2500 images, rather than 60000 which you will see in some published results. Deep learning models perform better with more data, however, they also take longer to train, especially when they start becoming more complex.\n",
    "\n",
    "If you have a computer with a CUDA compatible GPU, you can take advantage of it to improve computation time. If you don't have a GPU, no problem! You can set up a deep learning environment in the cloud that can run your models on a GPU. Here is a [blog post][1] by Dan that explains how to do this - check it out after completing this exercise! It is a great next step as you continue your deep learning journey.\n",
    "\n",
    "Ready to take your deep learning to the next level? Check out [Advanced Deep Learning with Keras][2] to see how the Keras functional API lets you build domain knowledge to solve new types of problems. Once you know how to use the functional API, take a look at [Image Processing with Keras in Python][3] to learn image-specific applications of Keras..\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create a Sequential object to start your model. Call this model.\n",
    "- Add the first `Dense` hidden layer of `50` units to your model with `'relu'` activation. For this data, the `input_shape` is `(784,)`.\n",
    "- Add a second `Dense` hidden layer with `50` units and a `'relu'` activation function.\n",
    "- Add the output layer. Your activation function should be `'softmax'`, and the number of nodes in this layer should be the same as the number of possible outputs in this case: `10`.\n",
    "- Compile `model` as you have done with previous models: Using `'adam'` as the `optimizer`, `'categorical_crossentropy'` for the `loss`, and `metrics=['accuracy']`.\n",
    "- Fit the model using `X` and `y` using a `validation_split` of `0.3`.\n",
    "\n",
    "\n",
    "[1]: https://www.datacamp.com/community/tutorials/deep-learning-jupyter-aws\n",
    "[2]: https://www.datacamp.com/courses/advanced-deep-learning-with-keras\n",
    "[3]: https://www.datacamp.com/courses/image-processing-with-keras-in-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert x_train.shape == (60000, 28, 28)\n",
    "assert x_test.shape == (10000, 28, 28)\n",
    "assert y_train.shape == (60000,)\n",
    "assert y_test.shape == (10000,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = x_test.reshape((10000, 784))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.zeros((y_test.size, y_test.max() + 1))\n",
    "y[np.arange(y_test.size), y_test] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first hidden layer\n",
    "model.add(Dense(50, activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Add the second hidden layer\n",
    "model.add(Dense(50, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X, y, epochs=10, validation_split=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Thoughts\n",
    "\n",
    "1. Congrats. You are on your way to using the most fun and powerful modeling technique around.\n",
    "\n",
    "2. Next steps - There is still an immense amount to learn as you become a deep learning master. But it's like riding a bike. The hardest part is getting to the point where you can practice on your own, and you are there. You'll probably enjoy experimenting with deep learning using the same types of data you've used before for predictive modeling, tables of numeric data like what you commonly see in a pandas DataFrame or numpy array. From there, many people start working with images, using something called convolutional neural networks. But you could decide you want to focus on text. Or sound. Or you might get creative and apply deep learning to data others haven't even thought of. You'll find it's a powerful technique as you get the hang of it. Kaggle is a great place to find datasets to work with, and their forums are a great place to keep learning. As you get more advanced, start checking out the wikipedia page titled \"List of datasets for machine learning research.\" It includes datasets from some very interesting domains. Keras has excellent documentation. When there's something specific to look up, keras-dot-io is likely to have the answer. The keras and tensorflow repositories on github also have nice examples to work from. As you start working with larger datasets and more complex deep learning models, you may find these models can take a long time to fit. If you have a computer with a graphical processing unit, or GPU, you may be able to set up TensorFlow to use that GPU for computation. This generally requires a GPU that meets a standard called CUDA compatibility. Most GPU's made by NVIDIA are CUDA compatible. If you don't have a computer with a CUDA compatible GPU, here is a link with instructions about setting up a deep learning environment in the cloud that can run your models on a GPU. More than anything, share your work online, and have fun. If you keep experimenting and build out on the tools you've learned, you'll soon be amazed at what you can do.\n",
    "   - Start with standard prediction problems on tables of numbers\n",
    "   - Images (with convolutional neural networks) are common next steps\n",
    "   - keras.io for excellent documentation\n",
    "   - Graphical processing unit (GPU) provides dramatic speedups in model training times\n",
    "   - Need a CUDA compatible GPU\n",
    "   - FOr training on GPUs in the cloud look here:\n",
    "     - [Deep Learning with Jupyter Notebooks in the Cloud](https://www.datacamp.com/community/tutorials/deep-learning-jupyter-aws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Certificate\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-04-19_intro_to_deep_learning_in_python/2024-01-18_DataCamp_intro_to_deep_learning_in_python_certificate.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf-gpu",
   "language": "python",
   "name": "tf-gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}