gradient-boosted-tree-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Ensemble Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will explore the use of boosting. We will use the pre-implemented gradient boosted trees in GraphLab Create. We will:\n",
    "\n",
    "* Use SFrames to do some feature engineering.\n",
    "* Train a boosted ensemble of decision-trees (gradient boosted trees) on the LendingClub dataset.\n",
    "* Predict whether a loan will default along with prediction probabilities (on a validation set).\n",
    "* Evaluate the trained model and compare it with a baseline.\n",
    "* Find the most positive and negative loans using the learned model.\n",
    "* Explore how the number of trees influences classification performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fire up Graphlab Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load LendingClub dataset\n",
    "\n",
    "We will be using the [LendingClub](https://www.lendingclub.com/) data. The [LendingClub](https://www.lendingclub.com/) is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors. \n",
    "\n",
    "We will build a classification model to predict whether or not a loan provided by lending club is likely to default.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: C:\\Users\\Nan\\AppData\\Local\\Temp\\graphlab_server_1530244734.log.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to nanlee_89@yahoo.com and will expire on December 07, 2018.\n"
     ]
    }
   ],
   "source": [
    "loans = graphlab.SFrame('lending-club-data.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'member_id', 'loan_amnt', 'funded_amnt', 'funded_amnt_inv', 'term', 'int_rate', 'installment', 'grade', 'sub_grade', 'emp_title', 'emp_length', 'home_ownership', 'annual_inc', 'is_inc_v', 'issue_d', 'loan_status', 'pymnt_plan', 'url', 'desc', 'purpose', 'title', 'zip_code', 'addr_state', 'dti', 'delinq_2yrs', 'earliest_cr_line', 'inq_last_6mths', 'mths_since_last_delinq', 'mths_since_last_record', 'open_acc', 'pub_rec', 'revol_bal', 'revol_util', 'total_acc', 'initial_list_status', 'out_prncp', 'out_prncp_inv', 'total_pymnt', 'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int', 'total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'last_pymnt_d', 'last_pymnt_amnt', 'next_pymnt_d', 'last_credit_pull_d', 'collections_12_mths_ex_med', 'mths_since_last_major_derog', 'policy_code', 'not_compliant', 'status', 'inactive_loans', 'bad_loans', 'emp_length_num', 'grade_num', 'sub_grade_num', 'delinq_2yrs_zero', 'pub_rec_zero', 'collections_12_mths_zero', 'short_emp', 'payment_inc_ratio', 'final_d', 'last_delinq_none', 'last_record_none', 'last_major_derog_none']\n"
     ]
    }
   ],
   "source": [
    "print loans.column_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the target column\n",
    "\n",
    "The target column (label column) of the dataset that we are interested in is called `bad_loans`. In this column **1** means a risky (bad) loan **0** means a safe  loan.\n",
    "\n",
    "In order to make this more intuitive, we reassign the target to be:\n",
    "* **+1** as a safe  loan, \n",
    "* **-1** as a risky (bad) loan. \n",
    "\n",
    "We put this in a new column called `safe_loans`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans = loans.remove_column('bad_loans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting features\n",
    "\n",
    "We will be using a subset of features (categorical and numeric). The features we will be using are **described in the code comments** below. \n",
    "\n",
    "The features we will be using are described in the code comments below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "target = 'safe_loans'\n",
    "features = ['grade',                     # grade of the loan (categorical)\n",
    "            'sub_grade_num',             # sub-grade of the loan as a number from 0 to 1\n",
    "            'short_emp',                 # one year or less of employment\n",
    "            'emp_length_num',            # number of years of employment\n",
    "            'home_ownership',            # home_ownership status: own, mortgage or rent\n",
    "            'dti',                       # debt to income ratio\n",
    "            'purpose',                   # the purpose of the loan\n",
    "            'payment_inc_ratio',         # ratio of the monthly payment to income\n",
    "            'delinq_2yrs',               # number of delinquincies \n",
    "            'delinq_2yrs_zero',          # no delinquincies in last 2 years\n",
    "            'inq_last_6mths',            # number of creditor inquiries in last 6 months\n",
    "            'last_delinq_none',          # has borrower had a delinquincy\n",
    "            'last_major_derog_none',     # has borrower had 90 day or worse rating\n",
    "            'open_acc',                  # number of open credit accounts\n",
    "            'pub_rec',                   # number of derogatory public records\n",
    "            'pub_rec_zero',              # no derogatory public records\n",
    "            'revol_util',                # percent of available credit being used\n",
    "            'total_rec_late_fee',        # total late fees received to day\n",
    "            'int_rate',                  # interest rate of the loan\n",
    "            'total_rec_int',             # interest received to date\n",
    "            'annual_inc',                # annual income of borrower\n",
    "            'funded_amnt',               # amount committed to the loan\n",
    "            'funded_amnt_inv',           # amount committed by investors for the loan\n",
    "            'installment',               # monthly payment owed by the borrower\n",
    "           ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skipping observations with missing values\n",
    "\n",
    "One common approach to coping with missing values is to **skip** observations that contain missing values.\n",
    "\n",
    "We run the following code to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dropping 29 observations; keeping 122578 \n"
     ]
    }
   ],
   "source": [
    "loans, loans_with_na = loans[[target] + features].dropna_split()\n",
    "\n",
    "# Count the number of rows with missing data\n",
    "num_rows_with_na = loans_with_na.num_rows()\n",
    "num_rows = loans.num_rows()\n",
    "print 'Dropping %s observations; keeping %s ' % (num_rows_with_na, num_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, there are not too many missing values. We are retaining most of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure the classes are balanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will undersample the larger class (safe loans) in order to balance out our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.502247166849\n",
      "Percentage of risky loans                : 0.497752833151\n",
      "Total number of loans in our new dataset : 46503\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == 1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "\n",
    "# Undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed = 1)\n",
    "risky_loans = risky_loans_raw\n",
    "loans_data = risky_loans.append(safe_loans)\n",
    "\n",
    "print \"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data))\n",
    "print \"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data))\n",
    "print \"Total number of loans in our new dataset :\", len(loans_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data into training data and validation data. We will use the validation data to help us select model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data, validation_data = loans_data.random_split(.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient boosted tree classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient boosted trees are a powerful variant of boosting methods; they have been used to win many [Kaggle](https://www.kaggle.com/) competitions, and have been widely used in industry.  We will explore the predictive power of multiple decision trees as opposed to a single decision tree.\n",
    "\n",
    "We will now train models to predict `safe_loans` using the features above. In this section, we will experiment with training an ensemble of 5 trees. To cap the ensemble classifier at 5 trees, we call the function with **max_iterations=5** (recall that each iterations corresponds to adding a tree)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_5 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, verbose = False,\n",
    "        target = target, features = features, max_iterations = 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making predictions\n",
    "\n",
    "Just like we did in previous sections, let us consider a few positive and negative examples **from the validation set**. We will do the following:\n",
    "* Predict whether or not a loan is likely to default.\n",
    "* Predict the probability with which the loan is likely to default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">29.44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.97</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">payment_inc_ratio</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">inq_last_6mths</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">open_acc</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.30496</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.4952</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.96736</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.90524</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">int_rate</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_int</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">annual_inc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt_inv</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">93.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9.91</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">823.48</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">92000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1622.21</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">25000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.77</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">719.11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">50004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">62.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">696.99</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">100000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">installment</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">483.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">281.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">123.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">158.77</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[4 rows x 25 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsafe_loans\tint\n",
       "\tgrade\tstr\n",
       "\tsub_grade_num\tfloat\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tpayment_inc_ratio\tfloat\n",
       "\tdelinq_2yrs\tint\n",
       "\tdelinq_2yrs_zero\tint\n",
       "\tinq_last_6mths\tint\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\topen_acc\tint\n",
       "\tpub_rec\tint\n",
       "\tpub_rec_zero\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tint_rate\tfloat\n",
       "\ttotal_rec_int\tfloat\n",
       "\tannual_inc\tint\n",
       "\tfunded_amnt\tint\n",
       "\tfunded_amnt_inv\tint\n",
       "\tinstallment\tfloat\n",
       "\n",
       "Rows: 4\n",
       "\n",
       "Data:\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "|     1      |   B   |      0.2      |     0     |       3        |    MORTGAGE    |\n",
       "|     1      |   B   |      0.6      |     1     |       1        |      RENT      |\n",
       "|     -1     |   D   |      0.4      |     0     |       3        |      RENT      |\n",
       "|     -1     |   A   |      1.0      |     0     |       11       |    MORTGAGE    |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "|  dti  |      purpose       | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "| 29.44 |    credit_card     |      6.30496      |      0      |        1         |\n",
       "| 12.19 |    credit_card     |      13.4952      |      0      |        1         |\n",
       "| 13.97 |       other        |      2.96736      |      3      |        0         |\n",
       "| 16.33 | debt_consolidation |      1.90524      |      0      |        1         |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "|       0        |        1         |           1           |    8     |    0    |\n",
       "|       0        |        1         |           1           |    8     |    0    |\n",
       "|       0        |        0         |           1           |    14    |    0    |\n",
       "|       0        |        1         |           1           |    17    |    0    |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "|      1       |    93.9    |        0.0         |   9.91   | ... |\n",
       "|      1       |    59.1    |        0.0         |  11.71   | ... |\n",
       "|      1       |    59.5    |        0.0         |  16.77   | ... |\n",
       "|      1       |    62.1    |        0.0         |   8.9    | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "[4 rows x 25 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select all positive and negative examples.\n",
    "validation_safe_loans = validation_data[validation_data[target] == 1]\n",
    "validation_risky_loans = validation_data[validation_data[target] == -1]\n",
    "\n",
    "# Select 2 examples from the validation set for positive & negative loans\n",
    "sample_validation_data_risky = validation_risky_loans[0:2]\n",
    "sample_validation_data_safe = validation_safe_loans[0:2]\n",
    "\n",
    "# Append the 4 examples into a single dataset\n",
    "sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)\n",
    "sample_validation_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting on sample validation data\n",
    "\n",
    "For each row in the **sample_validation_data**, use **model_5** to predict whether or not the loan is classified as a **safe loan**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: int\n",
       "Rows: 4\n",
       "[1L, 1L, -1L, 1L]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_5.predict(sample_validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Prediction probabilities\n",
    "\n",
    "For each row in the **sample_validation_data**, what is the probability (according **model_5**) of a loan being classified as **safe**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: float\n",
       "Rows: 4\n",
       "[0.7045905590057373, 0.5963408946990967, 0.44925159215927124, 0.6119099855422974]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_5.predict(sample_validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Evaluating the model on the validation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the accuracy is defined as follows:\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified examples}}{\\mbox{# total examples}}\n",
    "$$\n",
    "\n",
    "Evaluate the accuracy of the **model_5** on the **validation_data**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.66813873330461"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_5.evaluate(validation_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the number of **false positives** made by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of false positives on the validation data: 1618\n"
     ]
    }
   ],
   "source": [
    "validation_data_prediction = model_5.predict(validation_data)\n",
    "## number of false positives\n",
    "false_positives = ((validation_data_prediction == 1) & (validation_data['safe_loans']==-1)).sum()\n",
    "print \"number of false positives on the validation data:\", false_positives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the number of **false negatives** made by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of false negatives on the validation data: 1463\n"
     ]
    }
   ],
   "source": [
    "false_negatives  = ((validation_data_prediction == -1) & (validation_data['safe_loans']==1)).sum()\n",
    "print \"number of false negatives on the validation data:\", false_negatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most positive & negative loans.\n",
    "\n",
    "In this section, we will find the loans that are most likely to be predicted **safe**. We can do this in a few steps:\n",
    "\n",
    "* **Step 1**: Use the **model_5** (the model with 5 trees) and make **probability predictions** for all the loans in the **validation_data**.\n",
    "* **Step 2**: Similar to what we did in the very first assignment, add the probability predictions as a column called **predictions** into the validation_data.\n",
    "* **Step 3**: Sort the data (in descreasing order) by the probability predictions.\n",
    "\n",
    "Start here with **Step 1** & **Step 2**. Make predictions using **model_5** for examples in the **validation_data**. Use `output_type = probability`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "validation_data['predictions'] = model_5.predict(validation_data, output_type='probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to go to **Step 3**. We can now use the `prediction` column to sort the loans in **validation_data** (in descending order) by prediction probability. Find the top 5 loans with the highest probability of being predicted as a **safe loan**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "validation_data = validation_data.sort('predictions', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "|     1      |   A   |      0.2      |     0     |       11       |    MORTGAGE    |\n",
      "|     1      |   A   |      0.4      |     0     |       4        |    MORTGAGE    |\n",
      "|     1      |   A   |      0.6      |     0     |       11       |    MORTGAGE    |\n",
      "|     1      |   A   |      0.2      |     0     |       11       |    MORTGAGE    |\n",
      "|     1      |   A   |      0.2      |     0     |       11       |      OWN       |\n",
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "+-------+----------------+-------------------+-------------+------------------+\n",
      "|  dti  |    purpose     | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
      "+-------+----------------+-------------------+-------------+------------------+\n",
      "|  4.21 |  credit_card   |      0.955726     |      0      |        1         |\n",
      "| 12.76 |      car       |       1.7376      |      0      |        1         |\n",
      "|  2.4  |      car       |      2.49545      |      0      |        1         |\n",
      "|  6.47 | major_purchase |       1.131       |      1      |        0         |\n",
      "|  0.69 |      car       |      3.17372      |      0      |        1         |\n",
      "+-------+----------------+-------------------+-------------+------------------+\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "|       2        |        1         |           1           |    9     |    0    |\n",
      "|       2        |        1         |           1           |    11    |    0    |\n",
      "|       0        |        1         |           1           |    6     |    0    |\n",
      "|       0        |        0         |           1           |    9     |    0    |\n",
      "|       0        |        1         |           1           |    4     |    0    |\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "|      1       |    7.9     |        0.0         |   6.39   | ... |\n",
      "|      1       |    5.5     |        0.0         |   6.76   | ... |\n",
      "|      1       |    0.0     |        0.0         |   6.17   | ... |\n",
      "|      1       |    9.6     |        0.0         |   5.42   | ... |\n",
      "|      1       |    9.1     |        0.0         |   6.03   | ... |\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "[5 rows x 26 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print validation_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the top 5 loans (in the **validation_data**) with the **lowest probability** of being predicted as a **safe loan**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "|     -1     |   B   |      1.0      |     0     |       5        |      RENT      |\n",
      "|     -1     |   C   |      0.4      |     0     |       4        |      RENT      |\n",
      "|     -1     |   D   |      0.8      |     0     |       3        |      RENT      |\n",
      "|     -1     |   C   |      0.8      |     1     |       0        |    MORTGAGE    |\n",
      "|     -1     |   C   |      0.2      |     0     |       5        |      RENT      |\n",
      "+------------+-------+---------------+-----------+----------------+----------------+\n",
      "+-------+--------------------+-------------------+-------------+------------------+\n",
      "|  dti  |      purpose       | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
      "+-------+--------------------+-------------------+-------------+------------------+\n",
      "| 29.42 | debt_consolidation |      14.3733      |      0      |        1         |\n",
      "|  8.4  |    credit_card     |      11.8779      |      0      |        1         |\n",
      "|  8.95 |   small_business   |       16.727      |      0      |        1         |\n",
      "| 17.37 |  home_improvement  |      12.5753      |      0      |        1         |\n",
      "| 30.17 | debt_consolidation |      13.5391      |      0      |        1         |\n",
      "+-------+--------------------+-------------------+-------------+------------------+\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "|       0        |        0         |           1           |    14    |    0    |\n",
      "|       0        |        1         |           1           |    9     |    0    |\n",
      "|       2        |        1         |           1           |    7     |    0    |\n",
      "|       0        |        1         |           1           |    8     |    0    |\n",
      "|       1        |        1         |           1           |    7     |    0    |\n",
      "+----------------+------------------+-----------------------+----------+---------+\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "|      1       |    57.5    |      20.9132       |  14.09   | ... |\n",
      "|      1       |    60.0    |       34.64        |  15.31   | ... |\n",
      "|      1       |    41.6    |      16.7025       |   15.2   | ... |\n",
      "|      1       |    46.1    |       18.86        |  15.31   | ... |\n",
      "|      1       |    80.7    |      19.7362       |  14.33   | ... |\n",
      "+--------------+------------+--------------------+----------+-----+\n",
      "[5 rows x 26 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print validation_data.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Effect of adding more trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will train 5 different ensemble classifiers in the form of gradient boosted trees. We will train models with 10, 50, 100, 200, and 500 trees.  We use the **max_iterations** parameter in the boosted tree module. \n",
    "\n",
    "Let's get sarted with a model with **max_iterations = 10**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_10 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "        target = target, features = features, max_iterations = 10, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, train 4 models with **max_iterations** to be:\n",
    "* `max_iterations = 50`, \n",
    "* `max_iterations = 100`\n",
    "* `max_iterations = 200`\n",
    "* `max_iterations = 500`. \n",
    "\n",
    "Let us call these models **model_50**, **model_100**, **model_200**, and **model_500**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_50 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "                                                    target = target, features = features, max_iterations = 50, verbose=False)\n",
    "model_100 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "                                                    target = target, features = features, max_iterations = 100, verbose=False)\n",
    "model_200 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "                                                    target = target, features = features, max_iterations = 200, verbose=False)\n",
    "model_500 = graphlab.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "                                                    target = target, features = features, max_iterations = 500, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare accuracy on entire validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will compare the predicitve accuracy of our models on the validation set. Evaluate the **accuracy** of the 10, 50, 100, 200, and 500 tree models on the **validation_data**. Use the `.evaluate` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.672770357604\n",
      "0.690758293839\n",
      "0.691727703576\n",
      "0.684510986644\n",
      "0.671800947867\n"
     ]
    }
   ],
   "source": [
    "print model_10.evaluate(validation_data)['accuracy']\n",
    "\n",
    "print model_50.evaluate(validation_data)['accuracy']\n",
    "\n",
    "print model_100.evaluate(validation_data)['accuracy']\n",
    "\n",
    "print model_200.evaluate(validation_data)['accuracy']\n",
    "\n",
    "print model_500.evaluate(validation_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the training and validation error vs. number of trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the classification error is defined as\n",
    "\n",
    "$$\n",
    "\\mbox{classification error} = 1 - \\mbox{accuracy} \n",
    "$$\n",
    "\n",
    "In this section, we will plot the **training and validation errors versus the number of trees** to get a sense of how these models are performing. We will compare the 10, 50, 100, 200, and 500 tree models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "def make_figure(dim, title, xlabel, ylabel, legend):\n",
    "    plt.rcParams['figure.figsize'] = dim\n",
    "    plt.title(title)\n",
    "    plt.xlabel(xlabel)\n",
    "    plt.ylabel(ylabel)\n",
    "    if legend is not None:\n",
    "        plt.legend(loc=legend, prop={'size':15})\n",
    "    plt.rcParams.update({'font.size': 16})\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to plot the classification errors (on the **train_data** and **validation_data**) versus the number of trees, we will need lists of these accuracies, which we get by applying the method `.evaluate`. \n",
    "\n",
    "**Steps to follow:**\n",
    "\n",
    "* **Step 1:** Calculate the classification error for model on the training data (**train_data**).\n",
    "* **Step 2:** Store the training errors into a list (called `training_errors`) that looks like this:\n",
    "```\n",
    "[train_err_10, train_err_50, ..., train_err_500]\n",
    "```\n",
    "* **Step 3:** Calculate the classification error of each model on the validation data (**validation_data**).\n",
    "* **Step 4:** Store the validation classification error into a list (called `validation_errors`) that looks like this:\n",
    "```\n",
    "[validation_err_10, validation_err_50, ..., validation_err_500]\n",
    "```\n",
    "Once that has been completed, the rest of the code should be able to evaluate correctly and generate the plot.\n",
    "\n",
    "\n",
    "Let us start with **Step 1**. Following code computes the classification error on the **train_data** for models **model_10**, **model_50**, **model_100**, **model_200**, and **model_500**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_err_10 = 1 - model_10.evaluate(train_data)['accuracy']\n",
    "\n",
    "train_err_50 = 1 - model_50.evaluate(train_data)['accuracy']\n",
    "\n",
    "train_err_100 = 1 - model_100.evaluate(train_data)['accuracy']\n",
    "\n",
    "train_err_200 = 1 - model_200.evaluate(train_data)['accuracy']\n",
    "\n",
    "train_err_500 = 1 - model_500.evaluate(train_data)['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us run **Step 2**. Save the training errors into a list called **training_errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_errors = [train_err_10, train_err_50, train_err_100, \n",
    "                   train_err_200, train_err_500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, onto **Step 3**. Compute the classification error on the **validation_data** for models **model_10**, **model_50**, **model_100**, **model_200**, and **model_500**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "validation_err_10 = 1 - model_10.evaluate(validation_data)['accuracy']\n",
    "\n",
    "validation_err_50 = 1 - model_50.evaluate(validation_data)['accuracy']\n",
    "\n",
    "validation_err_100 = 1 - model_100.evaluate(validation_data)['accuracy']\n",
    "                     \n",
    "validation_err_200 = 1 - model_200.evaluate(validation_data)['accuracy']\n",
    "    \n",
    "validation_err_500 = 1 - model_500.evaluate(validation_data)['accuracy'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us run **Step 4**. Save the training errors into a list called **validation_errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "validation_errors = [validation_err_10, validation_err_50, validation_err_100, \n",
    "                     validation_err_200, validation_err_500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will plot the **training_errors** and **validation_errors** versus the number of trees. We will compare the 10, 50, 100, 200, and 500 tree models. We provide some plotting code to visualize the plots within this notebook. \n",
    "\n",
    "Run the following code to visualize the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAELCAYAAAAP/iu7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VGX2wPHvyaQSaiCFIiUUgQAiHRWRGkQQZaWJK6Ki\niK7rWvgBIoIu6i5rwYqKLuqqSBVUBAWx0kFKQu81CRA6CWnv74+ZhJlkkkySmUzK+TzPPJn73nbm\nBubk3vue+4oxBqWUUsqTfLwdgFJKqbJPk41SSimP02SjlFLK4zTZKKWU8jhNNkoppTxOk41SSimP\n02SjVAkiIgdFpKeX9h0uIr+KyAURedUbMaiyy9fbAaiySUQOAuFAul3zLGPMY96JSLngIeAUUNk4\nKcATkVnAUWPMxOIOTJV+mmyUJ/U3xizPbyER8TXGpOXXVtBtlGeFPB71gO3OEo0H96nKCb2Mpoqd\niNwnIn+IyOsichqYnEubj4hMFJFDIpIgIp+KSBXbNuqLiBGRB0TkMPCTk/3sEJF+dtO+InJSRNqI\nSKCI/E9ETovIWRFZLyLhucR7UESeFpGtInJORL4SkUC7z/J7tuWNiDSyvZ8lIu+KyPcictH2GSNE\n5A0ROSMiO0Xk+my7bC8i223z/5u5L9v2+onIZlvMq0SkVbY4/09EtgKXRCTHH5MicoPts56z/bwh\nM05gBDDWFmfPbOs9BAy3m/9NbvsUkVoiMt92rA+IyON22/ERkXEiss927OeISIhtnsu/E1X6aLJR\n3tIR2I/1UtvUXNrus726AZFAReDtbNvpCjQDop3s40tgmN10NHDKGLMJ6xdrFeAaoDowGkjKI97B\nQB+gAdDKFperBgMTgRrAFWA1sMk2PQ94Ldvyw22xNgSa2NbFlpQ+Bh62xfw+sFhEAuzWHQbcBlR1\ncrYYAnwHvGlb/zXgOxGpboy5D/gc+LcxpmL2M1JjzAfZ5vd3tk8gA/gG2ALUBnoAT4hI5u/nb8Ad\nWH9vtYAzwDu2eQX9nahSRJON8qSvbX+hZr5G2c07box5yxiTZoxJyqVtOPCaMWa/MeYiMB4Ymu0v\n9snGmEt227D3BXC7iFSwTd+NNQEBpGL9QmtkjEk3xmw0xpzP47O8aYw5boxJxPpl2roAx2GhbfvJ\nwEIg2RjzqTEmHfgKyH5m87Yx5ohtX1O5mjAfAt43xqy1xfwJ1uTVKVucR3I5HrcBe4wxn9mO8ZfA\nTqC/k2ULwn6f7YFQY8wLxpgUY8x+4ENgqG3Z0cCzxpijxpgrwGTgLtvvtKC/E1WK6D0b5Ul35HHP\n5ogLbbWAQ3bTh7D+m7W/tOJsOwAYY/aKyA6gv+2yz+1c/WL/DOtf0LNFpCrwP6xfgqm5bC7O7v1l\nW2yuird7n+RkumK25e0/0yG7fdUDRojI3+zm+2eLJdfjQc7jmbn92nms4wr7fdYDaonIWbs2C/Cb\n3fyFIpJhNz8d6++0oL8TVYromY3yFmc3obO3Hcf65ZSpLpCG45d1fjezMy+lDcB683svgDEm1Rgz\nxRjTHLgB6Afc63r4WS4BmWdOiEhEIbaR3TV27+tiPQ5g/VKfaoypaveqYDtDyZTX8ch+PDO3f8zF\nuHLbtn37EeBAthgrGWP62s2/Ndv8QGPMMTf+TlQJpMlGlWRfAv8QkQYiUhF4CfiqgD2eZgO9gUew\nXlYDQES6iUhLEbEA57Fewslwvok8bQGiRKS17Ub+5EJsI7tHRaSO7R7Ls1gvtYH1ctRoEekoVsEi\ncpuIVHJxu0uAJiJyt+1G/hCgOfCti+vHY713lpd1wAVbp4EgEbGISAsRaW+bPwOYKiL1AEQkVEQG\n2N6763eiSiBNNsqTvrH1XMp8LSzg+h9jvbTyK3AASMZ6g9llxpgTWG/I38DVL22ACKw3588DO4Bf\nbPsqEGPMbuAFYDmwB/g97zVc8gXwA9bOEvuAf9r2tQEYhbWTxBlgLwXoqGCMOY31bOEp4DQwFuhn\njDnl4iY+Aprb7r99ncs+0m37aI31d3YKmIn1xj/AdGAx8IOIXADWYO0YAm76naiSSXTwNKWUUp6m\nZzZKKaU8TpONUkopj9Nko5RSyuM02SillPK4UlfUWaNGDVO/fn1vh6GUUgrYuHHjKWNMaH7Llbpk\nU79+fTZs2ODtMJRSSgEikv2pFE7pZTSllFIep8lGKaWUx2myUUop5XGabJRSSnmcJhullFIeV+p6\noxXJhTio5I4nwCulVCliDKQlQ8plSLV7ZU77WKBhd4+GUH6STWoyvHcjVKsPnR6B5gPA4uftqJRS\nCjIyIC0JUpMg5VK2ZJAEqZeyJQr75TLfJ9nWuXR1ndSkq+vlNdRR1XrwxFaPfsTyk2xi5sHlU9bX\n/Afgx0nQYRS0GQEVQrwdnVKqJMtId/Jlbp8Qsn2x57pcLkkkzdko3sUo1fP7Lx/JxhhY/a5j2/lj\nsHwy/PJvuG6Y9WynRmOvhKeUKqKM9CKeEVxyXCcrcdjepyV7+xN6Vuplj++ifCSb5HNQuRYkxOac\nl3oZNnxkfTXubU06kd1ApPjjVKqsSk914xmBk/fpKd7+hCWfJQD8gsA/GPwq2L0PAv+KHt99+Ug2\nQVXhnnlwchesnQGbv3R+2rrnB+srtJk16bQabP1FKFWWGWP9snb1iz2/S0LOzg4yCjKSdznlG2hN\nApkJwK+CbbpC/u8dkkjmvCDwC7a+9w0Ci3e/7kvdSJ3t2rUzRX422uVE2DgL1n0IF47nvlyF6tDu\nfmj/oPZiU95jDKRdyePmb+b7/G4SOzuLsP006d7+lCWfX7Yv8Mz3fkG2afv3FXImjqz3TtbxqwA+\npbMSRUQ2GmPa5btcuUw2mdJTYfsiWP0OHN+U+3I+ftDyLuvZTs3r3LNvVXYY49pln3zPCHK5r5B6\nGUyGtz9lCSeFPAsIckwCOZKIbTnfwFKbDDxNk01BGANH1sGad2HH4rz/Y9e7ETqNgWtvtfZNVyWf\nMZ47I8h8qbyJT+HPCLKSRR7L+QbqfVYvcTXZlI97NvkRgbodra+zh2HdB7DxU7hyLueyh/6wvqrV\nh46jofVwCKxc7CGXSxkZ1t9J0pmrr8tnHKdze+llorz5+ObyZV6A+wLOzggy31v8NRmUc+XuzObU\nxSv4+fhQpUI+BZ1XLlg7Eqx9DxL3575cQGW4/q/Q8SFrAlL5S0+z9hBMOgNJia4li6QzkHSWPAvT\nyjKLfx5nBM7uC7h4RpA57evv7U+oSim9jJZNRobhqw1HeOX7nfRtGcHLA1u5uGK6tYfa6nfg4G+5\nLyc+0PQ26PQo1O1UPv6KS0spQKKwSxjOzhhLO0tAEc4I8ruXUEGfdqFKLE02duLPJ/Po55vYcOhM\nVtvc0Z1pX7+ATw6I2wZr3oNtc/Pu11+zNXR+FJrfUTr+YkxNKnjCSDoDKRe9HbnrnHUrdccZQeY8\nvX+nyilNNnaupKVz6/Tf2H/yUlZbk/CKfPu3Lvj7FqKHyYV42PAxrJ9pffxNbsRivfxh8bN+Gfn4\n2d772n76Wfu++9i3OZmX2Z7rPNv2nc1Lu5J7ssi8hFWaqqP9K0FQNWvtVFA166OGgqrl/QqsCn6B\n3o5cqTJJk002q/edZtiHaxzanom+lke7NSp8MKnJ1meurX7X+dMJVC4EAqvknyRyvKrq5SSlShjt\njZZN54bVuattHeZtPJrV9uaKPfRvVYu61SsUbqN+gXD9PdYeaQd+tV5i272UcnMTW3wKkTCqWRON\nXnZSqlwpN8kGYELfZqzYEc+Zy6kAXEnLYOKiGD4Z2R4pyg19EYjsan2d3md9JM62udZLVKWBj1/+\nZxTOLlf5V9JCN6WUS8rNZbRMczcc4Zl5juM2vDXsevpfV6uooTkyxvqEgoxU28806yurLc0N8zKn\n03KfZ/GFoHzua/gHl4/ec0optysRl9FEpA8wHbAAM40xr2SbPwB4EcgA0oAnjDG/ezKmzEtpaw8k\nZrVN+WY7NzcJpUqQG+8HiNh6opWC3mhKKeVhHrsGIiIW4B3gVqA5MExEmmdbbAVwnTGmNXA/MNNT\n8djFxdQ7W+JnufqX/KmLV5i2bKend62UUuWWJy+4dwD2GmP2G2NSgNnAAPsFjDEXzdXreMEU0531\nRmEVeaRrQ4e2z9ceZtPhUnKPRSmlShlPJpvawBG76aO2NgcicqeI7AS+w3p2k4OIPCQiG0Rkw8mT\nJ90S3Jhujahv1wvNGJiwYBup6fp0XaWUcjevdyUyxiw0xjQF7sB6/8bZMh8YY9oZY9qFhoa6Zb+B\nfhb+eUdLh7adcRf4+PcDbtm+UkqpqzyZbI4B19hN17G1OWWM+RWIFJEaHozJwU2Na3Dn9Y4nW28s\n38ORRH1kvFJKuZMnk816oLGINBARf2AosNh+ARFpJLYCFxFpAwQApz0YUw7P3tbMoRdaUmo6zy+O\npbR1CVdKqZLMY8nGGJMGPAYsA3YAc4wxsSIyWkRG2xb7CxAjIpux9lwbYor5W75GxQDG39rUoe2n\nnQksjYkrzjCUUqpMK3dFnc5kZBgGv7/a4anQ4ZUDWP5kVyoF6rO4lFIqN64WdXq9g0BJ4OMjvDSw\nJb4+V2tv4s9f4dUfdnsxKqWUKjs02dg0Ca/EQzdHOrR9svogW46c9U5ASilVhmiysfO37o2pG5Kt\n9mbhNtK09kYppYpEk42dIH8LL97RwqEt9vh5Zq066J2AlFKqjNBkk03XJqE5ngD92o+7OX42yUsR\nKaVU6afJxonn+jWjUuDVB2JfTrHW3iillCocTTZOhFUK5P/6ONbe/Lg9nmWxWnujlFKFockmF3d3\nqMv1das6tE1eHMvFK2leikgppUovTTa58PERXrqzJRa72psT55J5/UetvVFKqYLSZJOHZjUr8+BN\nDRza/vvHAWKOnfNSREopVTppssnH33s2pnbVoKzpDFvtTXpG6XrMj1JKeZMmm3xU8PflxTuiHNq2\nHj3HZ6sPeiUepZQqjTTZuKB703D6toxwaPvPD7uJO5fspYiUUqp00WTjouf7R1Ex4GrtzcUraUz5\nRmtvlFLKFZpsXBReOZBnoq91aPs+Jo4VO+K9FJFSSpUemmwK4J5O9WhVp4pD26RFsVxO0dobpZTK\niyabArDYam/sSm84djaJ6cv3eC8opZQqBTTZFFCL2lUYeaNj7c3M3w+w/fh5L0WklFIlnyabQniy\nVxNqVQnMmk7PMExYuI0Mrb1RSimnNNkUQnCAL5Nvd6y92XzkLJ+vO+yliJRSqmTTZFNIvaMi6N08\n3KHt30t3knBea2+UUio7TTZFMPn2KIL9LVnTF5LTeOHb7V6MSCmlSiZNNkVQq2oQT/Z2rL35dusJ\nft6V4KWIlFKqZMoz2YjVNcUVTGk0onM9ompVdmh7blEMSSnpXopIKaVKnjyTjTHGAEuKKZZSydfi\nw8sDHWtvjiQm8dZPWnujlFKZXLmMtklE2ns8klKsVZ2q3Nu5vkPbB7/uZ1fcBe8EpJRSJYwryaYj\nsFpE9onIVhHZJiJbPR1YafNU7yaEVw7Imk7LMDyrtTdKKQW4lmyigYZAd6A/0M/2U9mpFOjH5P6O\ntTcbDp1h1qqD3glIKaVKkHyTjTHmEFAVa4LpD1S1tals+rSIoEfTMIe2F77dzrs/78V6+0sppcqn\nfJONiPwd+BwIs73+JyJ/83RgpZGIMGVAFEF+Fof2fy/dxfOLY3UoaaVUueXKZbQHgI7GmEnGmElA\nJ2CUZ8MqvepUq8DrQ1rjb3E8tJ+uPsSjn28iOVW7RCulyh9Xko0A9t+Q6bY2lYs+LSKYdX97KtmN\n7AmwNDaOv360lrOXU7wUmVJKeYcryea/wFoRmSwik4E1wEcejaoMuKFhDeY+0tmhhxrA+oNnuGvG\nao6dTfJSZEopVfxc6SDwGjASSLS9Rhpj3vB0YGVB04jKLBhzI43DKjq07024yMB3/2DHCR0DRylV\nPuT3uBqLiOw0xmwyxrxpe/1ZXMGVBbWrBjF3dGfa16/m0B5//gqDZ6xm1b5TXopMKaWKT36Pq0kH\ndolI3WKKp0yqWsGfzx7oyK0tIhzaL1xJ476P1/PNluNeikwppYqHK/dsqgGxIrJCRBZnvjwdWFkT\n6Gfh7bvbMKJzPYf2lPQM/vbln8z8bb+XIlNKKc/zzX8RnvN4FOWExUeYfHsUNasG8cr3Ox3m/fO7\nHcSdS2ZC32b4+GhnP6VU2ZJnshERCzDZGNOtmOIp80SE0V0bElYpgLHztpJmV+g58/cDxF+4wn8G\ntSLA15LHVpRSqnRx5Z5NhohUKaZ4yo2Bberw35HtHUb6BPhmy3FGfLyO88mpXopMKaXcz5V7NheB\nbSLykYi8mfnydGDlQZfGoXz1cGdqVHSsxVmzP5HBM1YTdy7ZS5EppZR7uZJsFmC9b/MrsNHupdyg\nRe0qLBxzA5E1gh3ad8ZdYOC7f7AnXsfEUUqVfuLK04hFJAioa4zZ5fmQ8tauXTuzYcMGb4fhdomX\nUnjgk/X8efisQ3vlQF8+uq897euHeCkypZTKnYhsNMa0y285V5763B/YDCy1TbfWrs/uFxLszxcP\ndqJnM8chCs4npzF85lqWxpzwUmRKKVV0rlxGmwx0AM4CGGM2A5EejKncCvK3MOOetgzrcI1De0pa\nBo98volPVx/0SlxKKVVUriSbVGPMuWxtGZ4IRoGvxYeX7mzJk72aOLQbA5MWxfLvpTt1IDalVKnj\nSrKJFZG7AYuINBaRt4BVHo6rXBMRHu/RmH/9pSWWbAWe7/68j6fmbiE1XfO9Uqr0cCXZ/A2IAq4A\nXwDngCc8GZSyGtK+LjPvbZdj5M8Fm45x/6z1XLyS5qXIlFKqYFzqjVaSlNXeaHnZfOQs989aT+Il\nx0HXWtSuzMf3tSesUqCXIlNKlXdu642mvK/1NVVZ8MgN1A2p4NAec+w8f3lvFftPXvRSZEop5RpN\nNqVE/RrBzH/kBlrVcXxy0JHEJO6asZo/D5/xUmRKKZU/TTalSGilAL4c1YmuTUId2hMvpTDswzWs\n2BHvpciUUipvrhR1horIBBH5QEQ+znwVR3Aqp+AAX2aOaMddbes4tCenZjDq0w18ue6wlyJTSqnc\nuTKezSLgN2A5kO7ZcJQr/Cw+TLurFRGVA3l75d6s9gwD4xdsI+5cMk/0bIyIjoujlCoZXEk2FYwx\n/+fxSFSBiAhPR19LeJVAnl8Ug92wOExfsYe4c8lMvbMFvha9UqqU8j5Xvom+FZG+Ho9EFcpfO9Xj\nvXvaEuDr+Kv8asMRHvpsI5dTtBZHKeV9riSbv2NNOMkicsH2Ou/KxkWkj4jsEpG9IjLOyfzhIrJV\nRLaJyCoRua6gH0BBdFQEnz/YkSpBfg7tP+1MYNiHazl54YqXIlNKKat8k40xppIxxscYE2h7X8kY\nUzm/9WxDSr8D3Ao0B4aJSPNsix0AuhpjWgIvAh8U/CMogHb1Q5j/SGdqVw1yaN9y5Cy9Xv+FORuO\n6DPVlFJe49IFfRG5XUT+Y3v1c3HbHYC9xpj9xpgUYDYwwH4BY8wqY0xmgcgaoA6q0BqFVWLBmBto\nVtPxb4Gzl1MZO28rQz9Yw94ELQBVShU/V7o+v4L1Utp22+vvIvKyC9uuDRyxmz5qa8vNA8D3ucTw\nkIhsEJENJ0+edGHX5Vd45UDmPNyJGxtVzzFv7YFE+k7/jdd+3E1yqnYsVEoVH1fObPoCvYwxHxtj\nPgb6ALe5MwgR6YY12Tjt9WaM+cAY084Y0y40NNTZIspOpUA/Zo3swNg+1+boOJCSnsGbK/bQd/pv\nrNp3yksRKqXKG1f7xVa1e18l16UcHQPsRwGrY2tzICKtgJnAAGPMaRe3rfLhZ/FhzC2N+PEfXbm5\nSc4Evf/UJe7+cC1PzdmS4wGfSinlbq4km5eBP0Vkloh8AmwEprqw3nqgsYg0EBF/YCjgMJy0iNQF\nFgB/NcbsLljoyhV1q1fgk5HteXPY9dSoGJBj/vxNR+nx6s/M1Q4ESikPcmmIARGpCbS3Ta4zxsS5\ntHFrfc4bgAX42BgzVURGAxhjZojITOAvwCHbKmn5Paq6PA4x4C7nLqfyr2U7+WKt80fadIoMYeqd\nLWkYWrGYI1NKlVauDjGQa7IRkabGmJ0i0sbZfGPMpiLGWCiabIpuw8FEJizcxu74nD3T/C0+PHJL\nQ8Z0a0iAr8XJ2kopdZU7ks0HxpiHRGSlk9nGGNO9qEEWhiYb90hJy2Dm7/uZvnwPV9JyDjEdWSOY\nqXe2pHPDnL3alFIqU5GTjd2GAo0xyfm1FRdNNu516PQlJn4dw297nPdMu6ttHSb0bUZIsH8xR6aU\nKg3cOVLnKhfbVClUr3own97fgelDW1OjYs6EMm+jtQPBvI1HtQOBUqrQck02IhIhIm2BIBG5XkTa\n2F63ABVyW0+VPiLCgNa1WfHkLQzrUDfH/DOXU3l67haGfbiGfToEtVKqEPK6ZzMCuA9oB9hft7oA\nzDLGLPB4dE7oZTTP23AwkfELtrHHyaNt/C0+PNqtEaNvidQOBEopt96z+YsxZr7bIisiTTbFIyUt\ngw9/28+bK3LpQBAazEt3tqRTpHYgUKo8c1uysW3sNiAKCMxsM8a8UKQIC0mTTfE6eOoSzy3KvQPB\nIFsHgmragUCpcsltHQREZAYwBPgbIMAgoF6RI1SlQv0aeXcgmLvxKD1e+4X52oFAKZUHV3qj3WCM\nuRc4Y4yZAnQGmng2LFWSZHYgWP5kV4Z1uCbH/MRLKTw1dwvDZ65lv3YgUEo54UqySbL9vCwitYBU\noKbnQlIlVdUK/rw8sBVzR3emcVjOR9qs2neaPtN/s93n0SEMlFJXuZJsvhWRqsA0YBNwEPjSk0Gp\nkq19/RC+e7wLT/dugn/2IQzSMnjtx930nf4ba/frQ7yVUlYudRDIWlgkAAg0xpzzXEh50w4CJcvB\nU9YnEPy+13kHgsHt6jD+Vu1AoFRZ5c4OAo/azmwwxlwBfERkjBtiVGVA/RrBfPZAB94Y0prqThLK\nnA3WDgQLNmkHAqXKM1cuo40yxpzNnDDGnAFGeS4kVdqICHdcX5sVT3VlaHvnHQienLOFez5ay4FT\nl7wQoVLK21xJNhYRkcwJEbEAek1E5VC1gj+v/KUVcx7uTCMnHQj+2Hua6Dd+5S3tQKBUueNKslkK\nfCUiPUSkB9bOAUs9G5YqzTo0CGHJ4114qpfzDgSv/rib2978nXUHEr0UoVKquLnyuBof4GGgh63p\nR2CmMcYrf5pqB4HS5cCpS0z8eht/7HXeM21Iu2sY37cpVSvoybJSpZFbH1dTkmiyKX2MMXy9+Rgv\nfruDxEspOeZXD/bnuX7NGdC6FnZXbJVSpUCRe6OJyBzbz20isjX7y53BqrJNRLjz+jqseLIrQ9rl\n7EBw+lIKT3y1mb9+tI6D2oFAqTIpryEGahljjouI0+egGWMOeTSyXOiZTem3dv9pJizcxr6TOROL\nv68Pj3dvxEM3N8xxv0cpVfK4o87mW9vPfxpjDmV/uSdMVR51jKzOkr934clcOhD854fd9H3zN9Yf\n1A4ESpUVeSUbfxG5G7hBRAZmfxVXgKpsCvC18HiPxiz9exduaJhzTJy9CRcZNGM14+Zv5ezlnPd5\nlFKlS16X0W4ChgODgcXZZhtjzP0ejs0pvYxW9hhjWPjnMf75nXYgUKq0cedInQ8YYz5yW2RFpMmm\n7DpzKYWXv9/BnA1Hnc7v0rgG/7yjBfWqBxdzZEqp3BQ52YhId2PMT7ldMjPGLChijIWiyabsW2Pr\nQLDfSQeCAF8fHu/RmFFdIrUDgVIlgDs6CHS1/ezv5NWvyBEqlYtOkdX5/u9d+EfPJvhbHP+JXknL\nYNqyXdymHQiUKlW0qFOVaPtPXuTZhTGszmVsnGEdrmFcn2ZUqeBXzJEppcC9Qwz8XUQqi9VMEdkk\nIr3dE6ZSeYsMrcgXozry6qDrqOYkoXy57gg9XvuZRZuP6RAGSpVgrlz0vt8Ycx7oDVQH/gq84tGo\nlLIjIvylbR1WPHULg9rWyTH/1MUU/j57M/d+vI5Dp/UJBEqVRK4km8y+pn2BT40xsXZtShWbkGB/\npg26ji9HdSIyNGePtN/2nKL367/yzsq9pKRleCFCpVRuXEk2G0XkB6zJZpmIVAL0f7Lyms4NrR0I\nnujZONcOBP3e+o0N2oFAqRLD1SEGWgP7jTFnRSQEqGOM8crDOLWDgLK37+RFnl24jTX7nSeWYR3q\nMq5PU+1AoJSHuK2DANAZ2GVLNPcAE4FzRQ1QKXdoGFqRL0d14j+5diA4TI/XfmHxluPagUApL3Ll\nzGYrcB3QCpgFzAQGG2O65rWep+iZjcpN4qUUXlqyg3kbnT+BIDI0mD5REfRpEUHL2lX00TdKuYE7\nH1ezyRjTRkQmAceMMR9ltrkr2ILQZKPys2rfKSYujGF/HmPj1KoSSG9b4mlfPwSLjyYepQrDncnm\nF2ApMBK4GUgAthhjWroj0ILSZKNckZyazns/7+O9n/eRkp53f5bqwf70ah5OdIsIbmhYnQBfSzFF\nqVTp585kEwHcDaw3xvwmInWBW4wxn7on1ILRZKMKYm/CRf61dCc/70ogNT3/ezaVAnzp3iyMPlER\ndL02lAr+vsUQpVKll9uSTUmjyUYVxrmkVFbuTGBpTBw/704gOTX/3vsBvj50bRJKnxYR9Ggarj3a\nlHLCnWc2nYC3gGaAP2ABLhpjqrgj0ILSZKOKKiklnV92n2RZbBzLd8RzITkt33V8fYTODasTHRVB\n76hwwioFFkOkSpV87kw2G4ChwFygHXAv0MQYM94dgRaUJhvlTilpGazef5qlMXH8uD2OUxfzHxVU\nBNrWrUafFhFER0VwTUiFYohUqZLJrcnGGNNORLYaY1rZ2v40xlzvplgLRJON8pT0DMOmw2dYGhPH\n0pg4jp1Ncmm9qFqVs7pUNwqrqF2qVbnizmTzK9ATa31NHHACuM8Yc507Ai0oTTaqOBhjiD1+3pp4\nYuPYm3DRpfUiawQT3SKCPlERtKqjtTyq7HNnsqmHtbuzH/APoArwrjFmrzsCLShNNsob9iZcYFls\nPEtj4tiri4x3AAAgAElEQVR2zLUHaGgtjyoPtDeaUh5y9MxlfoiNZ2lsHOsPJuLKfyGt5VFlVZGT\njYhsA3L9b5R5/6a4abJRJcnJC1dYvsN6xrNq3ymXankqBvjSvWkYfVpE0LVJKMEBWsujSi93JJt6\nea1ojDlUyNiKRJONKqkKW8tzc5NQ+kRF0LOZ1vKo0scdyaYREG6M+SNb+41AnDFmn1siLSBNNqo0\nyKzl+SE2jh8LU8vTPJywylrLo0o+dySbb4Hxxpht2dpbAi8ZY/q7JdIC0mSjSpuUtAzW7D/N0tg4\nfojVWh5Vtrgj2aw3xrTPZd42fRCnUgVX2Fqe5jUr06eFtWdbY63lUSWIO5LNHmNM41zm7TXGNCpi\njIWiyUaVFZm1PMtirYlnj9byqFLIHcnmS+AnY8yH2dofBHoZY4a4JdIC0mSjyqq9CRdZFhvHstg4\nth51rZanZpVAoqOsl9ra16+Gr8WVwXeVch93JJtwYCGQAmy0NbfD+jDOO40xcW6KtUBcSTbnz58n\nISGB1NTUYopKqYLx8/MjLCyMypUrO51fmFqekGB/ejULp0+LCG5opLU8qni48wkC3YAWtslYY8xP\nboiv0PJLNufPnyc+Pp7atWsTFBSklxhUiWOMISkpiWPHjhEeHp5rwsmktTyqJCu3TxDYu3cvtWrV\nokIF7b2jSrbLly9z/PhxGjVy/fbn+WS7Wp5dJ0lKTc93Hftanh7Nwqhawb8oYSvlwNVkU+b+3ElN\nTSUoKMjbYSiVr6CgoAJf6q0c6MeA1rUZ0Lo2SSnp/LrnJMtirOPynM+lludKWgY/bo/nx+3xWbU8\nvaMiiNZaHlWMPJpsRKQPMB3rgGszjTGvZJvfFPgv0AZ41hjzHzft1x2bUcqjivrvNMjfktU5wLGW\nJ55TF684XSctw/DbnlP8tucUkxbF0KZuNfrYtlG3ul4NUJ7jsctoImIBdgO9gKPAemCYMWa73TJh\nQD3gDuCMK8kmv8toO3bsoFmzZkWMXqni4Yl/r+kZhj8za3li4zh6Rmt5lOeUhMtoHYC9xpj9toBm\nAwOArGRjjEkAEkTkNg/GoVS5YvER2tUPoV39EJ69rZnLtTzbT5xn+4nzvPbjbiJrBGcNj3Cd1vIo\nN/Bkp/zawBG76aO2tgITkYdEZIOIbDh58qRbgiupRCTf188//1zk/URERDBx4sQCrZOcnIyIMHPm\nzCLvXxUPEaFF7So81ftafnyyK8uf7Moz0dfSqk6VPNfbf+oSM37Zxx3v/MENr/zE5MWxrN53mrT0\n/B8uqpQzpaKDgDHmA+ADsF5G83I4HrV69eqs90lJSXTv3p2JEydy221XT/6aN29e5P0sWbKEsLCw\nAq0TEBDA6tWradiwYZH3r7yjUVhFGoU14tFujTh2NokfbGc86w8mkpHL/6wT55KZteogs1Yd1Foe\nVWieTDbHgGvspuvY2lQeOnXqlPX+4kXrJY+GDRs6tOcmOTmZwEDXehe1adOmwLGJiEtxeJsxhpSU\nFAICAnLMS0pKKnRvxZSUFHx9ffHxKRtV+rWrBjHyxgaMvLEBpy5eYfl2axHpH3tzr+VJvJTCVxuO\n8NWGI1rLowrEk/9r1gONRaSBiPgDQ4HFHtxfuTJjxgxEhE2bNtGlSxeCgoJ46623MMbw1FNP0aJF\nC4KDg7nmmmsYMWIE2S8/Zr+MNnToUG666SaWLFlCVFQUFStWpGvXruzatStrGWeX0Tp16sQ999zD\nJ598QmRkJJUrV6Z///7ExTk+YGL//v306tWLoKAgGjZsyBdffEG/fv3o06dPvp913rx5tGnThsDA\nQGrVqsWzzz5LevrV+pJx48ZRp04dVq5cSZs2bQgICGDx4sUsXboUEeGnn36ib9++BAcH8/TTTwPW\nRD5mzBjCwsIIDAykY8eOrFy50mG/mZ/t7bffpkGDBgQFBXH69GkXfjulT42KAQztUJdZIzuw8ble\nTB/amltbRBDkl/uZy8UraSzecpwxn2+izYs/8uAnG5i38ShnL+f/VGtV/njsTxFjTJqIPAYsw9r1\n+WNjTKyIjLbNnyEiEcAGoDKQISJPAM2NMefdFUf9cd+5a1NFcvAVz/SBGDJkCI8++igvvPACISEh\nZGRkcOrUKcaPH0/t2rVJSEhg2rRp9OrViz///DPPG7179+5l4sSJTJ48GT8/P5588knuvvtuNm7c\nmOs6AL/++iuHDx/mjTfe4Pz58zzxxBOMGTOGBQsWAJCRkUG/fv1ISUlh1qxZ+Pr6MmXKFBITE2nR\nokWe2/70008ZOXIkjz32GK+88gq7du1iwoQJiAj//Oc/s5Y7d+4cDz74IOPHjycyMpK6deuyd+9e\nAO677z4eeOABnn766axi3xEjRrB8+XJeeeUV6tWrx3vvvUd0dDS///47HTp0yNruihUr2L17N6++\n+ir+/v7loljYaS1PbBzLt+ddy7N8RzzLd8Rj8RE6R1YnuoXW8qirPHrea4xZAizJ1jbD7n0c1str\nqpCefvppHn74YYe2Tz75JOt9eno6bdu2pVGjRqxfv97hizS7xMRE1q5dS7161kFak5OTGTZsGAcP\nHqR+/fq5rnfp0iW+++47KlWqBMDRo0eZOHEiaWlp+Pr6snDhQnbs2MGWLVto1co6mnibNm1o1KhR\nnskmPT2d//u//+Ohhx5i+vTpAPTu3RuLxcLYsWMZO3Zs1qNeLl68yLx584iOjs5aPzPZDB8+nOef\nfz6rffPmzSxYsIDZs2czZIj1ebLR0dE0bdqUqVOnsmjRoqxlL1y4wPfff0/16tVzjbMss6/lSU23\n1fLExLEsj1qe9AzD73tP8ftereVRV5WNi8/lmH3HgUyLFy+mU6dOVKlSBV9f36zHoezevTvPbTVp\n0iQr0cDVjghHjx7Nc73OnTtnJZrM9dLT07Mupa1fv5769etnJRqABg0a0LJl3kMixcTEEBcXx6BB\ng0hLS8t6de/enUuXLrFjx46sZf38/OjVq5fT7WQ/RuvWrcNisTBw4MCsNovFwl133cXvv//usGyn\nTp3KbaLJzs/iQ5fGoUy9syVrJ/Rg3ujOPHhTA+pUy/0emDGw8dAZpi7Zwc3TVnLr9N+YvnwPu+Iu\nUNoelaWKRu/olXLh4eEO03/88Qd33nknQ4cO5dlnnyU0NJTU1FRuvvlmkpOT89xW1apVHab9/a3P\n0CrqenFxcYSGhuZYz1mbvVOnTgHQo0cPp/OPHDlCx44ds7aV24377MfoxIkTVKtWDT8/vxzLnTlz\nJs91lZWzWp4fYq1FpLvjc6/l2XHiPDtOnOf15btpUCOYaK3lKTfKfLLx1L2SkiL7f9D58+dTt25d\nPv/886w2+5v83hAREcEvv/ySo/3kyZNERETkul5ISAhgvSzorLu3fRfsvL6oss+rWbMmZ86cITU1\n1SHhxMfHU61atTzXVTll1vK0qF2FJ3tfy76TtnF5YuLYkse4PAdstTwzftmn4/KUA2U+2ZQ3SUlJ\nWWcWmewTjze0b9+ef/3rX2zdujXrUtqBAwfYtm1bnsmmZcuWhIaGcujQIe699163xdOhQwfS09NZ\nuHAhgwcPBqz3h+bPn89NN93ktv2UVw1DKzLmlkaMuaXwtTw9m1m7VN/YqIbW8pQRmmzKmF69ejFj\nxgyeeeYZ+vTpw6+//srs2bO9GtOdd95J06ZNGThwIC+99BK+vr5MnjyZiIiIPGtWfH19mTZtGqNG\njSIxMZHevXvj6+vLvn37WLhwIUuWLMFiKfgXUevWrRk4cCAPP/wwiYmJWb3RDh486PXEXNYUtpZn\nzoajzNlwlIoBvnRrGkafqAhuuVZreUoz/c2VMQMHDuTFF1/k3Xff5d1336VLly58/fXXREVFeS0m\nHx8fvvvuOx566CHuvfdeIiIieP755/nvf/+b78BhI0aMICQkhJdffpn3338/q8ND//79i1Rc+ckn\nn/DMM8/w3HPPceHCBa677jqWLl1K+/btC71NlbfMWp6hHepmjcuzLDaOlTtzH5fn4pU0vtlynG+2\nHMff14ebG4fSp0UEPXVcnlKnzA2epk99Lh1Onz5NZGQk48aNY/z48d4Ox2v03yskp6bz6+6TLM2n\nlsee1vKUHCXhqc9KZXn77bcJDAykUaNGxMfHM23aNMB65qLKt0A/C72jIuidrZbnh+3xnLyQfy3P\nc1/H0KZuVevwCFE1tZanhNJko4qFv78/06ZN4/Dhw1gsFjp27MiKFSuoVauWt0NTJUhmLU+XxqG8\nOKAFfx6xjsvzfUze4/JsOnyWTYfP8tKSnTSrWZk+ti7VTcJ1XJ6SQi+jKeVF+u/VNcYYtp84z7KY\n/Gt57GXW8kRHhXNdnar4+GjicTe9jKaUKjNEhKhaVYiqVfhanojKgURHhRPdIoIO9UO0lqeYabJR\nSpU69rU8xzNreWLjWHcg91qeuPPJfLL6EJ+sPkS1Cn70ah6utTzFSJONUqpUq1U1iPtubMB9Nzbg\n9MUrLN8Rz9KYOH7Po5bnzOVUreUpZnpUlVJlRvWKAQxpX5ch7YtWyxMdFU7PZuFUC9ZaHnfRZKOU\nKpPsx+Wxr+VZsSOBc0mpTtdJyTYuT6fIEPrYumWHay1PkWiyUUqVedlredbuT2Rp7AmWxeZdy/PH\n3tP8sfc0zy2KzarliY6KoF714GL+BKWfdscoYfr375/nOC+PPfYYVatW5coV5/9Bstu7dy8iwtKl\nS7Pa6tSpw7hx4/Jcb/PmzYhIjvFd8jNjxgwWL845+rcr+1SqOPhZfLipcQ3+eUdL1o7vwfxHOjOq\nSwOuCcl9XB4gq46n67Sf6fPGr7yxfDc7487ruDwu0jObEmbYsGEMHz6c7du353isfnp6OvPmzWPg\nwIEEBAQUeh/ffPMNNWrUKGqoTs2YMYN27dpx++23F9s+lSosHx+hbb0Q2tYLYULfZtZanth4lsXE\nsSv+Qq7r7Yy7wM64C7yxfA/1q1cgukUEfaIitJYnD5psSpgBAwZQoUIFvvzyS1588UWHeStXriQ+\nPp5hw4YVaR/XX399kdYvLfssqJSUFCwWS44nSRtjSElJKXSCT0pKIigo77+alfc51PL0asL+kxdZ\nFmt9SvWWI2dzXe/g6cu8/8t+3v9lv9by5EGPRAkTHBxM//79+eqrr3LMmz17NmFhYXTv3h2AY8eO\nMXLkSBo0aEBQUBBNmjTh+eefJzXV+c3PTM4uab311ltcc801BAcHM2DAgKwhne1NmzaNdu3aUbly\nZcLDwxkwYAD79u3Lmn/TTTexZcsWPvroI0QEEeF///tfrvucPXs2LVq0ICAggLp16zJp0iTS06/2\nGJo5cyYiQmxsLD179iQ4OJhmzZqxaNGifI6i9Sxw6tSpNGzYkICAAK699lo+++wzh2Vuuukmhg4d\nynvvvUdkZCRBQUEkJCQwceJEIiIi+PXXX2nbti2BgYEsWLAAgH379jFgwAAqVapEpUqVGDBgAPv3\n78/aZlpaGiLC9OnTefzxxwkNDS0ViVblFBlakUduaciiR29k1bjuTO7fnE6RIeR14pJZy3P3h2tp\nP3U5Y+dt4aed8STn0hOuPCn7ZzaTq3g7AqvJuVc5Zzds2DC++uorNm7cSNu2bQFITU1lwYIFDB8+\nPOsv75MnTxISEsKrr75K9erV2blzJ1OmTOHUqVO88847Lu9v/vz5PP744zz66KP079+flStXMmrU\nqBzLHTlyhDFjxtCgQQPOnz/Pe++9x4033siePXuoVKkSH3zwAXfccQfNmjXLepJzo0aNnO5zyZIl\nDBs2jJEjR/Kf//yHzZs3M2nSJBITE3n77bdzHI+HHnqIsWPH8sYbbzBkyBAOHDhAzZo1c/1MY8aM\n4YsvvuD555+ndevWLFu2jBEjRhAaGkqfPn2ylvvll1/Ys2cP06ZNIzAwkEqVKgFw4cIFRo4cybhx\n42jYsCF16tQhOTmZHj16EBQUxEcffYSPjw+TJk2ia9eubNu2zWF47FdeeYVu3brx2Wef6TX9MsBZ\nLc+y2Hh+33OKlPQMp+vY1/IE+1ustTwtIrjl2jAqlsNanvL3iUuBW2+9lapVqzJ79uysZLNs2TLO\nnDnjcAmtdevWtG7dOmv6xhtvJCgoiNGjRzN9+nR8fV379U6dOpV+/fplfclHR0cTHx/PrFmzHJZ7\n8803s96np6fTq1cvQkND+eabb7j77rtp3rw5FSpUIDQ0lE6dOuW5z0mTJtGzZ08+/vhjAPr06UNG\nRgaTJk3i2WefdUgkTz/9dNZIna1btyYiIoLvvvuOBx980Om2d+3axQcffMD//vc/hg8fDkDPnj05\nduwYU6ZMcUg2586dY+vWrYSGhjps4/Lly7z55pvcdtvVYcXffvttjh07xt69e6lXrx5gHYW0UaNG\nfPjhhzzzzDNZy9apU4cvvvgiz2OgSif7Wp4Lyams3HWSZTFxrNyVwOUU52cwl1LS+XbrCb7desJW\ny1OD6KiIclXLo5fRSiB/f38GDhzInDlzsv4q/uqrr6hXrx6dO3fOWi4jI4NXX32VZs2aERQUhJ+f\nHyNGjCApKYmjR4+6tK+UlBS2bNnCgAEDHNoHDhyYY9lVq1bRs2dPqlevjq+vL8HBwVy+fJndu3cX\n6POlpqayefNmBg0a5NA+ZMgQ0tPTWbNmjUN77969s96HhYVRo0aNPD/f8uXL8fPzY8CAAaSlpWW9\nevTowZ9//klGxtW/RDt06JAj0QBYLBaHpASwbt062rdvn5VoAOrVq0enTp1y9NqzT1Kq7KoU6Mft\n19XineFt2PRcLz68tx1/aVOHKkF+ua5jreVJ4Jl5W2k3dTnDZ67hs9UHiT+fXHyBe4Ge2ZRQw4YN\n4+OPP2b16tW0adOGRYsWMWbMGIfHpb/66quMHz+eCRMm0KVLF6pWrcqaNWt4/PHHSU527R9uQkIC\nGRkZhIWFObRnnz5w4ADR0dHccMMNfPDBB9SsWRN/f3+io6Nd3pf9PtPT0wkPD3doz5xOTEx0aLe/\nPAXWZJzXPk+dOkVqamrWJTFn+4+IiHDYZ3bVq1fP0VHgxIkTTpcPDw8nPj7e6WdR5Uegn4VezcPp\n1Ty80LU819etmjU8Qlmr5Sn7yaYA90pKkm7duhEeHs7s2bM5ceIEFy5cyNELbe7cuQwdOpQXXngh\nq23r1q0F2k9YWBg+Pj4kJCQ4tGef/v7777ly5Qpff/11Vs+qlJQUzp7NvZdOXvu0WCw59pH5hR0S\nElLgbdoLCQnB39+f33//3elYJtWrV896n9tYJ87aa9as6dAhIlN8fHyOmHUMlfIts5bnpsY1eOH2\nFvx55CzLYuP4PuYERxJzH5fnz8Nn+fPwWV7+fidNIypZB4RrEcG14ZVK/b+psp9sSimLxcLgwYOZ\nO3cux44do1mzZlx33XUOyyQlJeXojvv5558XaD/+/v60atWKRYsWOdwDyex9Zb8vi8XicB9o9uzZ\nDpekMreX35mOn58f119/PXPnznXoiDBnzhwsFku+93vy0717d1JSUrh48SLdunUr0rbsdezYkS+/\n/JLDhw9Tt25dAA4fPsyaNWt46aWX3LYfVbZYa3mq0bZeNcbf2pQdJy6w1DY8Qnmq5dFkU4INGzaM\nt956i4ULFzJlypQc83v16sV7771Hu3btiIyM5NNPP+XgwYMF3s+ECRMYPHgwjz32GLfffjs//fQT\ny5cvd1imR48ejB07lpEjRzJy5Ei2bdvG66+/TuXKlR2Wa9q0KStXruSHH34gJCSEyMhIp2cqU6ZM\n4bbbbuPBBx9k0KBBbNmyhcmTJzN69Og8e5m5IioqilGjRjFo0CDGjh1L27ZtSUpKIjY2lv379/P+\n++8XarsPPPAA//73v7n11luZPHkyIsLzzz9PRESE0957SmUnIjSvVZnmtSoXqZand1Q4faIi6NCg\n9NTylI4oy6nOnTtTv359jDFOCzmnTJnC4MGDmTBhAsOGDSM4OJjXX3+9wPsZNGgQb7zxBgsXLuSO\nO+4gJiaGDz/80GGZ1q1b89FHH7Fq1Sr69evHnDlzmD9/fo77IpMmTaJJkyYMGjSI9u3bs2TJEqf7\n7Nu3L1988QVr1qyhf//+vPnmm4wdO5bp06cXOH5nZsyYwYQJE5g1axZ9+/Zl5MiRfP/993Tp0qXQ\n2wwMDOSnn36iYcOG3H///YwcOZKGDRvy888/57ivpJQr7Gt5Vo/vzpTbo+gcWT3fWp5PVx/i7pnW\nWp5n5m5hxY6SX8ujw0Ir5UX671U5c/riFVbsSGBpbFyetTz2vFXLo8NCK6VUKVW9YgCD21/D4PbX\nFKmWp3dUBL1KSC2PJhullCrBMmt5br+uFsmp6fy25xTLYuP4cXt8PuPyJLB8RwIWH6FjgxD6tIig\nd/MIIqp4Z1weTTZKKVVKZK/lWXcgkaUxcSyLjSMhj1qeVftOs2rfaSbZ1fJER0VQv0bx1fJoslFK\nqVLIz+LDjY1qcGOjGky5PSqrlmdpTByHEy/nup6zWp7oqAiaRni2lqdMJhtjTKkvgFJlX2nrnKNK\nrtxqeX6IjWNnnGu1PP8d2Z5u14blumxRlblk4+fnR1JSEhUqVPB2KErlKSkpCT+/3J+hpVRhZK/l\nOXDqUtYZz+ZcanmC/Cx0jqzudJ67lLlkExYWxrFjx6hduzZBQUF6hqNKHGMMSUlJHDt2TJ+hpjyu\nQY1gRndtyOiuDTlxLokfYuNZGhPH2gOnybCdXHdrGkqgnyXvDRVRmUs2mRXtx48fz3cQMaW8xc/P\nj/Dw8BxPYFDKk2pWCWLEDfUZcUN9Ei+lWMfliYmjX6taHt93mSvqVEopVXxcLerUx9UopZTyOE02\nSimlPE6TjVJKKY/TZKOUUsrjNNkopZTyOE02SimlPK7UdX0WkZPAIRcWrQGc8nA4pYEeBys9DlZ6\nHK7SY2FV1ONQzxgTmt9CpS7ZuEpENrjS97us0+NgpcfBSo/DVXosrIrrOOhlNKWUUh6nyUYppZTH\nleVk84G3Aygh9DhY6XGw0uNwlR4Lq2I5DmX2no1SSqmSoyyf2SillCohNNkopZTyuDKXbESkj4js\nEpG9IjLO2/F4moh8LCIJIhJj1xYiIj+KyB7bz2p288bbjs0uEYn2TtTuJSLXiMhKEdkuIrEi8ndb\ne7k6DgAiEigi60Rki+1YTLG1l7tjASAiFhH5U0S+tU2Xu+MgIgdFZJuIbBaRDba24j8Oxpgy8wIs\nwD4gEvAHtgDNvR2Xhz/zzUAbIMau7d/AONv7ccC/bO+b245JANDAdqws3v4MbjgGNYE2tveVgN22\nz1qujoPtswlQ0fbeD1gLdCqPx8L2+Z4EvgC+tU2Xu+MAHARqZGsr9uNQ1s5sOgB7jTH7jTEpwGxg\ngJdj8ihjzK9AYrbmAcAntvefAHfYtc82xlwxxhwA9mI9ZqWaMeaEMWaT7f0FYAdQm3J2HACM1UXb\npJ/tZSiHx0JE6gC3ATPtmsvdcchFsR+HspZsagNH7KaP2trKm3BjzAnb+zggc6D7Mn98RKQ+cD3W\nv+jL5XGwXTraDCQAPxpjyuuxeAMYC2TYtZXH42CA5SKyUUQesrUV+3HwdcdGVMlljDEiUi76t4tI\nRWA+8IQx5ryIZM0rT8fBGJMOtBaRqsBCEWmRbX6ZPxYi0g9IMMZsFJFbnC1THo6DzU3GmGMiEgb8\nKCI77WcW13Eoa2c2x4Br7Kbr2NrKm3gRqQlg+5lgay+zx0dE/LAmms+NMQtszeXuONgzxpwFVgJ9\nKH/H4kbgdhE5iPVyencR+R/l7zhgjDlm+5kALMR6WazYj0NZSzbrgcYi0kBE/IGhwGIvx+QNi4ER\ntvcjgEV27UNFJEBEGgCNgXVeiM+txHoK8xGwwxjzmt2scnUcAEQk1HZGg4gEAb2AnZSzY2GMGW+M\nqWOMqY/1e+AnY8w9lLPjICLBIlIp8z3QG4jBG8fB2z0lPNDzoi/W3kj7gGe9HU8xfN4vgRNAKtbr\nqw8A1YEVwB5gORBit/yztmOzC7jV2/G76RjchPW69FZgs+3Vt7wdB9vnagX8aTsWMcAkW3u5OxZ2\nn+8WrvZGK1fHAWvP3C22V2zmd6I3joM+rkYppZTHlbXLaEoppUogTTZKKaU8TpONUkopj9Nko5RS\nyuM02SillPI4TTaqzBIRIyKv2k0/LSKT3bTtWSJylzu2lc9+BonIDhFZma29vojc7en9K+UummxU\nWXYFGCgiNbwdiD0RKchjoh4ARhljumVrrw84TTYF3L5SxUKTjSrL0rCOr/6P7DOyn5mIyEXbz1tE\n5BcRWSQi+0XkFREZbhsjZpuINLTbTE8R2SAiu23P4sp8COY0EVkvIltF5GG77f4mIouB7U7iGWbb\nfoyI/MvWNglrwepHIjIt2yqvAF1sY5T8Q0TuE5HFIvIT1mI9ROQZuzim2O3rHtvn2Swi79tittiO\nSYwtjhzHTKmi0L+AVFn3DrBVRP5dgHWuA5phHbphPzDTGNNBrIOy/Q14wrZcfazPmWoIrBSRRsC9\nwDljTHsRCQD+EJEfbMu3AVoY66Pbs4hILeBfQFvgDPCDiNxhjHlBRLoDTxtjNmSLcZytPTPJ3Wfb\nfitjTKKI9Mb6qJEOWMe4WSwiNwMngSHAjcaYVBF5FxiOtbq8tjGmhW17VQtwvJTKlyYbVaYZ69Of\nPwUeB5JcXG29sT1+XUT2AZnJYhtgfzlrjjEmA9gjIvuBplifPdXK7qypCtYv/RRgXfZEY9Me+NkY\nc9K2z8+xDor3tYvxZvrRGJM5tlFv2+tP23RFWxytsCa19banYgdhfQjjN0CkiLwFfGf3mZVyC002\nqjx4A9gE/NeuLQ3bZWQR8cE6smumK3bvM+ymM3D8P5P9WU8G61nE34wxy+xn2B5zf6lw4bvMfvsC\nvGyMeT9bHH8DPjHGjM++sohcB0QDo4HBwP0ejFWVM3rPRpV5tr/252C92Z7pINa/8AFuxzqiZUEN\nEpGWZdwAAAEESURBVBEf232cSKwPLlwGPGIb8gARaWJ72m5e1gFdRaSGiFiAYcAv+axzAesQ2LlZ\nBtwv1jF+EJHaYh3PZAVwl+195lj09WydKHyMMfOBiVgvySnlNnpmo8qLV4HH7KY/BBaJyBZgKYU7\n6ziMNVFUBkYbY5JFZCbWezmbbEMfnOTqkLtOGWNOiMg4rGPPCPCdMWZRXutgfapzui3+WVjv9dhv\n8wcRaQastl0uuwjcY4zZLiITsd4X8sH6tPBHsV5i/K+tDSDHmY9SRaFPfVZKKeVxehlNKaWUx2my\nUUop5XGabJRSSnmcJhullFIep8lGKaWUx2myUUop5XGabJRSSnnc/wMeQqhQcXpAUQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd82ba20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([10, 50, 100, 200, 500], training_errors, linewidth=4.0, label='Training error')\n",
    "plt.plot([10, 50, 100, 200, 500], validation_errors, linewidth=4.0, label='Validation error')\n",
    "\n",
    "make_figure(dim=(10,5), title='Error vs number of trees',\n",
    "            xlabel='Number of trees',\n",
    "            ylabel='Classification error',\n",
    "            legend='best')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}