jt-kaggle-plant-seedling-classification-1-Copy2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image classification  - Kaggle Plant Seedling Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put these at the top of every notebook, to get automatic reloading and inline plotting\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we import the libraries we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This file contains all the main external libs we'll use\n",
    "from fastai.imports import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.transforms import *\n",
    "from fastai.conv_learner import *\n",
    "from fastai.model import *\n",
    "from fastai.dataset import *\n",
    "from fastai.sgdr import *\n",
    "from fastai.plots import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = \"data/plant-seedlings-classification/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.backends.cudnn.enabled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Enable data augmentation, and precompute=True\n",
    "1. Use `lr_find()` to find highest learning rate where loss is still clearly improving\n",
    "1. Train last layer from precomputed activations for 1-2 epochs\n",
    "1. Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1\n",
    "1. Unfreeze all layers\n",
    "1. Set earlier layers to 3x-10x lower learning rate than next higher layer\n",
    "1. Use `lr_find()` again\n",
    "1. Train full network with cycle_mult=2 until over-fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The dataset \n",
    "This is provided from kaggle in the plant-seedlings-classfication competition. The .zip files provided are in test and train foldes with the train folder having subfolders for each class (Keras style).\n",
    "\n",
    "Lets first download the data. Im using the official kaggle CLI https://github.com/Kaggle/kaggle-api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !mkdir $PATH\n",
    "# !kaggle competitions download -c plant-seedlings-classification -p ~/github/fastai/courses/dl1/data/plant-seedlings-classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unzip all files provided by Kaggle in to directory and then remove the .zip files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !unzip -q '{PATH}*.zip' -d {PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !rm -r {PATH}*.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# os.listdir(PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When running this through previously i found that 2 classes were always getting lots of mis-classifications (Black-grass & Loose Silky-bent). I also noted that Black-grass was under represented compared to Loose Silky-bent (Black-grass=263 ; Loose Silky-bent=654. So i duplicated all the images in the Black-grass folder (appending _2 to the file name). I did this via bash before the next steps\n",
    "\n",
    "\n",
    "Lets create a csv reference file so we can use the much easier `ImageClassifierData.from_csv()` option"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from glob2 import glob\n",
    "# import pandas as pd\n",
    "\n",
    "# df = pd.DataFrame(columns=[\"file\", \"species\"])\n",
    "\n",
    "# for image in glob(f'{PATH}train/**/*.png'):\n",
    "#     dir_ = image.split('/')\n",
    "#     file_, species = dir_[-1], dir_[-2]\n",
    "\n",
    "#     df = df.append({\n",
    "#         \"file\": file_,\n",
    "#         \"species\": species.replace(\" \", \"_\")\n",
    "#         }, ignore_index=True)\n",
    "\n",
    "# df.to_csv(f'{PATH}labels.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in our dataframe and view the structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.to_csv(f'{PATH}labels.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>species</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d04eff450.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6d8b4041a.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ec48facfa.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8f4d248bd.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6a3339ef1.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>09f038ce4.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>67e37de9b.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>d733b32d8.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1fb63feb5.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>410598462.png</td>\n",
       "      <td>Charlock</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            file   species\n",
       "0  d04eff450.png  Charlock\n",
       "1  6d8b4041a.png  Charlock\n",
       "2  ec48facfa.png  Charlock\n",
       "3  8f4d248bd.png  Charlock\n",
       "4  6a3339ef1.png  Charlock\n",
       "5  09f038ce4.png  Charlock\n",
       "6  67e37de9b.png  Charlock\n",
       "7  d733b32d8.png  Charlock\n",
       "8  1fb63feb5.png  Charlock\n",
       "9  410598462.png  Charlock"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_csv = f'{PATH}labels.csv'\n",
    "labels_df = pd.read_csv(labels_csv)\n",
    "\n",
    "labels_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now move all the images from under the class subfolders into the root train folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !mv {PATH}train/**/*.png {PATH}train/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove all the empty class subfolders from train/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !rm -r {PATH}train/**/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets view the distribution of the classes a a pivot`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>species</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Loose_Silky-bent</th>\n",
       "      <td>654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Common_Chickweed</th>\n",
       "      <td>611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black-grass</th>\n",
       "      <td>526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scentless_Mayweed</th>\n",
       "      <td>516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Small-flowered_Cranesbill</th>\n",
       "      <td>496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fat_Hen</th>\n",
       "      <td>475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Charlock</th>\n",
       "      <td>390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sugar_beet</th>\n",
       "      <td>385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cleavers</th>\n",
       "      <td>287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shepherds_Purse</th>\n",
       "      <td>231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Common_wheat</th>\n",
       "      <td>221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maize</th>\n",
       "      <td>221</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           file\n",
       "species                        \n",
       "Loose_Silky-bent            654\n",
       "Common_Chickweed            611\n",
       "Black-grass                 526\n",
       "Scentless_Mayweed           516\n",
       "Small-flowered_Cranesbill   496\n",
       "Fat_Hen                     475\n",
       "Charlock                    390\n",
       "Sugar_beet                  385\n",
       "Cleavers                    287\n",
       "Shepherds_Purse             231\n",
       "Common_wheat                221\n",
       "Maize                       221"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_df.pivot_table(index = 'species', aggfunc = len).sort_values('file', ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the dataset distribution size of images. we first need to define the data loader and transformations. We can later change these or wrap in a data function before training begins\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initial Model parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "arch=resnet101 # model architecture to use\n",
    "sz=224 # image size\n",
    "bs=64 # mini-batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make validation indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create validation indexes\n",
    "n = len(list(open(labels_csv)))-1\n",
    "val_idxs = get_cv_idxs(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_idxs = [0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz, bz):\n",
    "    tfms = tfms_from_model(arch, # pass in architecture\n",
    "                       sz, # image size\n",
    "                       aug_tfms=transforms_top_down, # side_on for normal images top_down for satelite of photos taken from above\n",
    "                       max_zoom=1.15)\n",
    "\n",
    "    data = ImageClassifierData.from_csv(path = PATH,\n",
    "                                    folder='train',\n",
    "                                    csv_fname = labels_csv, # name of the labels reference file\n",
    "                                    val_idxs = val_idxs, # validation indexes\n",
    "                                    test_name = 'test',\n",
    "                                    tfms = tfms, # transformations to apply\n",
    "                                    bs = bs) #mini batch size\n",
    "    return data if sz < 300 else data.resize(340, 'tmp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_data(sz, bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data Augementation/Transformation Settings\n",
    "tfms = tfms_from_model(arch, # pass in architecture\n",
    "                       sz, # image size\n",
    "                       aug_tfms=transforms_top_down, # side_on for normal images top_down for satelite of photos taken from above\n",
    "                       max_zoom=1.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ImageClassifierData.from_csv(path = PATH,\n",
    "                                    folder='train',\n",
    "                                    csv_fname = labels_csv, # name of the labels reference file\n",
    "                                    val_idxs = val_idxs, # validation indexes\n",
    "                                    test_name = 'test',\n",
    "                                    tfms = tfms, # transformations to apply\n",
    "                                    bs = bs) #mini batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data set image resolutions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([2544., 1061.,  247.,  115.,   23.,    7.,    6.,    6.,    1.,    1.]),\n",
       " array([  49. ,  409.3,  769.6, 1129.9, 1490.2, 1850.5, 2210.8, 2571.1, 2931.4, 3291.7, 3652. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAER9JREFUeJzt3X3MnXV9x/H3Z4C4CBllFNZBXavpMnHZKumQhMW4sfHkH8VEk/KHNIysZoNME5esaDKYhgSXqZGEseDogM2JzIfQzG7YoYvxDx6KQyhW5BY6qW1oHYoaEzbwuz/O74ZDuZ973+fc9Pd+JSfnOt/rd871PRfn3J9eT4dUFZKk/vzCuBuQJI2HASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnq1LHjbmAmp5xySq1Zs2bcbUjSq8qDDz74g6paOdu4ZR0Aa9asYdeuXeNuQ5JeVZL891zGuQtIkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6NeuVwElWA7cDvwL8HLi5qj6Z5Frgj4FDbegHq2pHe87VwBXAC8CfVdXdrX4h8EngGODvq+r6xX07L7dm65eW8uWntff6d4xluZI0H3P5KYjngQ9U1TeSnAg8mGRnm/eJqvqb4cFJzgQ2AW8GfhX4jyS/3mbfCPwhsA94IMn2qvrWYrwRSdL8zBoAVXUAONCmf5JkD3D6DE/ZCNxRVc8BTyaZAM5u8yaq6gmAJHe0sQaAJI3BvI4BJFkDvAW4r5WuSvJwkm1JVrTa6cBTQ0/b12rT1Q9fxpYku5LsOnTo0OGzJUmLZM4BkOQE4PPA+6vqx8BNwBuB9Qy2ED42OXSKp9cM9ZcXqm6uqg1VtWHlyll/zVSStEBz+jnoJMcx+OP/6ar6AkBVPT00/1PAv7aH+4DVQ08/A9jfpqerS5JGbNYtgCQBbgH2VNXHh+qrhoa9E9jdprcDm5Icn2QtsA64H3gAWJdkbZLXMDhQvH1x3oYkab7msgVwLvAe4JEkD7XaB4FLk6xnsBtnL/BegKp6NMmdDA7uPg9cWVUvACS5CribwWmg26rq0UV8L5KkeZjLWUBfZ+r99ztmeM51wHVT1HfM9DxJ0uh4JbAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ2aNQCSrE7y1SR7kjya5H2tfnKSnUkeb/crWj1JbkgykeThJGcNvdbmNv7xJJuX7m1JkmYzly2A54EPVNWbgHOAK5OcCWwF7qmqdcA97THARcC6dtsC3ASDwACuAd4KnA1cMxkakqTRmzUAqupAVX2jTf8E2AOcDmwEbmvDbgMuadMbgdtr4F7gpCSrgAuAnVX1TFX9ENgJXLio70aSNGfzOgaQZA3wFuA+4LSqOgCDkABObcNOB54aetq+VpuuLkkagzkHQJITgM8D76+qH880dIpazVA/fDlbkuxKsuvQoUNzbU+SNE9zCoAkxzH44//pqvpCKz/ddu3Q7g+2+j5g9dDTzwD2z1B/maq6uao2VNWGlStXzue9SJLmYS5nAQW4BdhTVR8fmrUdmDyTZzNw11D9snY20DnAs20X0d3A+UlWtIO/57eaJGkMjp3DmHOB9wCPJHmo1T4IXA/cmeQK4HvAu9u8HcDFwATwM+BygKp6JslHgAfauA9X1TOL8i4kSfM2awBU1deZev89wHlTjC/gymleaxuwbT4NSpKWhlcCS1KnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjo1awAk2ZbkYJLdQ7Vrk3w/yUPtdvHQvKuTTCR5LMkFQ/ULW20iydbFfyuSpPmYyxbArcCFU9Q/UVXr220HQJIzgU3Am9tz/jbJMUmOAW4ELgLOBC5tYyVJY3LsbAOq6mtJ1szx9TYCd1TVc8CTSSaAs9u8iap6AiDJHW3st+bdsSRpURzJMYCrkjzcdhGtaLXTgaeGxuxrtenqkqQxWWgA3AS8EVgPHAA+1uqZYmzNUH+FJFuS7Eqy69ChQwtsT5I0mwUFQFU9XVUvVNXPgU/x0m6efcDqoaFnAPtnqE/12jdX1Yaq2rBy5cqFtCdJmoMFBUCSVUMP3wlMniG0HdiU5Pgka4F1wP3AA8C6JGuTvIbBgeLtC29bknSkZj0InOQzwNuBU5LsA64B3p5kPYPdOHuB9wJU1aNJ7mRwcPd54MqqeqG9zlXA3cAxwLaqenTR340kac7mchbQpVOUb5lh/HXAdVPUdwA75tWdJGnJeCWwJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6tSx427gaLRm65fGtuy9179jbMuW9OriFoAkdWrWAEiyLcnBJLuHaicn2Znk8Xa/otWT5IYkE0keTnLW0HM2t/GPJ9m8NG9HkjRXc9kCuBW48LDaVuCeqloH3NMeA1wErGu3LcBNMAgM4BrgrcDZwDWToSFJGo9ZA6CqvgY8c1h5I3Bbm74NuGSofnsN3AuclGQVcAGws6qeqaofAjt5ZahIkkZooccATquqAwDt/tRWPx14amjcvlabrv4KSbYk2ZVk16FDhxbYniRpNot9EDhT1GqG+iuLVTdX1Yaq2rBy5cpFbU6S9JKFBsDTbdcO7f5gq+8DVg+NOwPYP0NdkjQmCw2A7cDkmTybgbuG6pe1s4HOAZ5tu4juBs5PsqId/D2/1SRJYzLrhWBJPgO8HTglyT4GZ/NcD9yZ5Arge8C72/AdwMXABPAz4HKAqnomyUeAB9q4D1fV4QeWJUkjNGsAVNWl08w6b4qxBVw5zetsA7bNqztJ0pLxSmBJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjp1RAGQZG+SR5I8lGRXq52cZGeSx9v9ilZPkhuSTCR5OMlZi/EGJEkLsxhbAL9XVeurakN7vBW4p6rWAfe0xwAXAevabQtw0yIsW5K0QEuxC2gjcFubvg24ZKh+ew3cC5yUZNUSLF+SNAdHGgAFfDnJg0m2tNppVXUAoN2f2uqnA08NPXdfq0mSxuDYI3z+uVW1P8mpwM4k355hbKao1SsGDYJkC8DrX//6I2xPkjSdI9oCqKr97f4g8EXgbODpyV077f5gG74PWD309DOA/VO85s1VtaGqNqxcufJI2pMkzWDBAZDkdUlOnJwGzgd2A9uBzW3YZuCuNr0duKydDXQO8OzkriJJ0ugdyS6g04AvJpl8nX+uqn9P8gBwZ5IrgO8B727jdwAXAxPAz4DLj2DZkqQjtOAAqKongN+eov4/wHlT1Au4cqHLkyQtLq8ElqROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR16kj/p/BaZtZs/dJYlrv3+neMZbmSFs4tAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pRXAmtReAWy9OrjFoAkdcoAkKROGQCS1CkDQJI6ZQBIUqdGHgBJLkzyWJKJJFtHvXxJ0sBIAyDJMcCNwEXAmcClSc4cZQ+SpIFRXwdwNjBRVU8AJLkD2Ah8a8R96CgxrusPwGsQ9Oo36gA4HXhq6PE+4K0j7kFaFOMMn94Ytktj1AGQKWr1sgHJFmBLe/jTJI/N4/VPAX6wwN5GyT4Xl30uvmXVaz467axl1ecMRt3nr81l0KgDYB+weujxGcD+4QFVdTNw80JePMmuqtqw8PZGwz4Xl30uvldLr/Z5ZEZ9FtADwLoka5O8BtgEbB9xD5IkRrwFUFXPJ7kKuBs4BthWVY+OsgdJ0sDIfw20qnYAO5bo5Re062gM7HNx2efie7X0ap9HIFU1+yhJ0lHHn4KQpE4dFQGw3H5eIsneJI8keSjJrlY7OcnOJI+3+xWtniQ3tN4fTnLWEve2LcnBJLuHavPuLcnmNv7xJJtH1Oe1Sb7f1utDSS4emnd16/OxJBcM1Zf0s5FkdZKvJtmT5NEk72v1ZbVOZ+hzWa3TJK9Ncn+Sb7Y+/6rV1ya5r62bz7aTSEhyfHs80eavma3/Je7z1iRPDq3P9a0+tu/SjKrqVX1jcDD5u8AbgNcA3wTOHHNPe4FTDqv9NbC1TW8FPtqmLwb+jcE1EucA9y1xb28DzgJ2L7Q34GTgiXa/ok2vGEGf1wJ/PsXYM9t/9+OBte3zcMwoPhvAKuCsNn0i8J3Wz7JapzP0uazWaVsvJ7Tp44D72nq6E9jU6n8H/Emb/lPg79r0JuCzM/U/gj5vBd41xfixfZdmuh0NWwAv/rxEVf0vMPnzEsvNRuC2Nn0bcMlQ/fYauBc4KcmqpWqiqr4GPHOEvV0A7KyqZ6rqh8BO4MIR9DmdjcAdVfVcVT0JTDD4XCz5Z6OqDlTVN9r0T4A9DK54X1brdIY+pzOWddrWy0/bw+ParYDfBz7X6oevz8n1/DngvCSZof+l7nM6Y/suzeRoCICpfl5ipg/2KBTw5SQPZnBlM8BpVXUABl9G4NRWXw79z7e3cfZ8VduE3ja5W2WGfkbaZ9v98BYG/xpctuv0sD5hma3TJMckeQg4yOAP4neBH1XV81Ms88V+2vxngV8eR59VNbk+r2vr8xNJjj+8z8P6Gev3/2gIgFl/XmIMzq2qsxj86umVSd42w9jl2P+k6XobV883AW8E1gMHgI+1+tj7THIC8Hng/VX145mGTtPTSHqdos9lt06r6oWqWs/glwLOBt40wzKXTZ9JfhO4GvgN4HcY7Nb5i3H3OZOjIQBm/XmJUauq/e3+IPBFBh/ipyd37bT7g234cuh/vr2Npeeqerp96X4OfIqXNunH2meS4xj8Uf10VX2hlZfdOp2qz+W6TltvPwL+k8E+85OSTF63NLzMF/tp83+Jwa7DcfR5YdvVVlX1HPAPLKP1OZWjIQCW1c9LJHldkhMnp4Hzgd2tp8kj/JuBu9r0duCydpbAOcCzk7sORmi+vd0NnJ9kRdtlcH6rLanDjo28k8F6nexzUzsjZC2wDrifEXw22v7mW4A9VfXxoVnLap1O1+dyW6dJViY5qU3/IvAHDI5XfBV4Vxt2+PqcXM/vAr5Sg6Or0/W/lH1+eyj0w+A4xfD6XDbfpReN6mjzUt4YHGH/DoN9hR8acy9vYHD2wTeBRyf7YbBf8h7g8XZ/cr10NsGNrfdHgA1L3N9nGGzq/x+Df31csZDegD9icGBtArh8RH3+Y+vjYQZfqFVD4z/U+nwMuGhUnw3gdxlssj8MPNRuFy+3dTpDn8tqnQK/BfxX62c38JdD36v727r5F+D4Vn9tezzR5r9htv6XuM+vtPW5G/gnXjpTaGzfpZluXgksSZ06GnYBSZIWwACQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlT/w9ImLlNMrr74wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# # see distribution of all images\n",
    "# size_d = {k: Image.open(PATH + k).size for k in data.trn_ds.fnames}\n",
    "# row_sz, col_sz = list(zip(*size_d.values()))\n",
    "# row_sz = np.array(row_sz); col_sz = np.array(col_sz)\n",
    "# plt.hist(row_sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1041.,  858.,  280.,  457.,  403.,  270.,  204.,  126.,   78.,   64.]),\n",
       " array([ 49., 144., 239., 334., 429., 524., 619., 714., 809., 904., 999.]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAD+RJREFUeJzt3W2sXVWdx/Hvb6jAiKNt4UKwrXMhNj7ExIE0WHQyMdRBAWN5AQnEDA3TpG+YEcVEy8wLMjNvIDGiJBNiI2idGEYHiTRAJKRgJvPCjkUNgtXpFRl6pdJLeHCicbSZ/7w4q3Ist0/33N7T3vX9JCdnr7XXOXuts0/vr2ufffZJVSFJ6s8fjbsDkqTxMAAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnVoy7g4czllnnVWTk5Pj7oYknVQef/zxF6pq4kjtTugAmJycZOfOnePuhiSdVJL899G08xCQJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR16oT+JvCoJjc/OJbtPnPrFWPZriQdC2cAktQpA0CSOmUASFKnjhgASe5Osi/Jk0N1y5M8kmR3u1/W6pPkjiRTSZ5IcuHQYza09ruTbDg+w5EkHa2jmQF8GfjQQXWbge1VtRrY3soAlwGr220TcCcMAgO4BXgPcBFwy4HQkCSNxxEDoKr+HXjxoOr1wNa2vBW4cqj+KzXwHWBpknOBDwKPVNWLVfUS8AivDRVJ0gKa62cA51TVXoB2f3arXwHsGWo33eoOVS9JGpP5/hA4s9TVYepf+wTJpiQ7k+ycmZmZ185Jkl411wB4vh3aod3va/XTwKqhdiuB5w5T/xpVtaWq1lTVmomJI/6kpSRpjuYaANuAA2fybADuH6q/rp0NtBZ4pR0iehi4NMmy9uHvpa1OkjQmR7wURJJ7gPcDZyWZZnA2z63A15NsBJ4Frm7NHwIuB6aAXwPXA1TVi0n+Cfhua/ePVXXwB8uSpAV0xACoqmsPsWrdLG0LuOEQz3M3cPcx9U6SdNz4TWBJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOrVk3B1YjCY3PziW7T5z6xVj2a6kk5MzAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOjVSACT5RJKnkjyZ5J4kpyc5L8mOJLuTfC3Jqa3taa081dZPzscAJElzM+cASLIC+BiwpqreBZwCXAPcBtxeVauBl4CN7SEbgZeq6q3A7a2dJGlMRj0EtAT44yRLgNcDe4FLgHvb+q3AlW15fSvT1q9LkhG3L0maozkHQFX9HPgM8CyDP/yvAI8DL1fV/tZsGljRllcAe9pj97f2Z851+5Kk0YxyCGgZg//Vnwe8GTgDuGyWpnXgIYdZN/y8m5LsTLJzZmZmrt2TJB3BKIeAPgD8rKpmqup3wH3Ae4Gl7ZAQwErgubY8DawCaOvfBLx48JNW1ZaqWlNVayYmJkboniTpcEYJgGeBtUle347lrwN+BDwGXNXabADub8vbWpm2/tGqes0MQJK0MEb5DGAHgw9zvwf8sD3XFuDTwE1Jphgc47+rPeQu4MxWfxOweYR+S5JGNNLvAVTVLcAtB1U/DVw0S9vfAFePsj1J0vzxm8CS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHVqpABIsjTJvUl+nGRXkouTLE/ySJLd7X5Za5skdySZSvJEkgvnZwiSpLkYdQbweeBbVfV24N3ALmAzsL2qVgPbWxngMmB1u20C7hxx25KkEcw5AJK8EfgL4C6AqvptVb0MrAe2tmZbgSvb8nrgKzXwHWBpknPn3HNJ0khGmQGcD8wAX0ry/SRfTHIGcE5V7QVo92e39iuAPUOPn251fyDJpiQ7k+ycmZkZoXuSpMMZJQCWABcCd1bVBcCvePVwz2wyS129pqJqS1Wtqao1ExMTI3RPknQ4owTANDBdVTta+V4GgfD8gUM77X7fUPtVQ49fCTw3wvYlSSOYcwBU1S+APUne1qrWAT8CtgEbWt0G4P62vA24rp0NtBZ45cChIknSwlsy4uP/FvhqklOBp4HrGYTK15NsBJ4Frm5tHwIuB6aAX7e2kqQxGSkAquoHwJpZVq2bpW0BN4yyPUnS/PGbwJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1asm4O6DFYXLzg2PZ7jO3XjGW7UqLgTMASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6NXIAJDklyfeTPNDK5yXZkWR3kq8lObXVn9bKU2395KjbliTN3XzMAG4Edg2VbwNur6rVwEvAxla/EXipqt4K3N7aSZLGZKQASLISuAL4YisHuAS4tzXZClzZlte3Mm39utZekjQGo14K4nPAp4A/aeUzgZeran8rTwMr2vIKYA9AVe1P8kpr/8LwEybZBGwCeMtb3jJi97TYjesSFOBlKHTym/MMIMmHgX1V9fhw9SxN6yjWvVpRtaWq1lTVmomJibl2T5J0BKPMAN4HfCTJ5cDpwBsZzAiWJlnSZgErgeda+2lgFTCdZAnwJuDFEbYvSRrBnGcAVXVzVa2sqkngGuDRqvoo8BhwVWu2Abi/LW9rZdr6R6vqNTMASdLCOB7fA/g0cFOSKQbH+O9q9XcBZ7b6m4DNx2HbkqSjNC+/B1BV3wa+3ZafBi6apc1vgKvnY3uSpNH5TWBJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ2al5+E1IlhcvOD4+6CpJOIMwBJ6pQzAGmOxjXjeubWK8ayXS0+zgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTcw6AJKuSPJZkV5KnktzY6pcneSTJ7na/rNUnyR1JppI8keTC+RqEJOnYjTID2A98sqreAawFbkjyTmAzsL2qVgPbWxngMmB1u20C7hxh25KkEc05AKpqb1V9ry3/D7ALWAGsB7a2ZluBK9vyeuArNfAdYGmSc+fcc0nSSOblYnBJJoELgB3AOVW1FwYhkeTs1mwFsGfoYdOtbu989EHqxTgv++2F6BaXkT8ETvIG4BvAx6vql4drOktdzfJ8m5LsTLJzZmZm1O5Jkg5hpABI8joGf/y/WlX3ternDxzaaff7Wv00sGro4SuB5w5+zqraUlVrqmrNxMTEKN2TJB3GKGcBBbgL2FVVnx1atQ3Y0JY3APcP1V/XzgZaC7xy4FCRJGnhjfIZwPuAvwJ+mOQHre7vgFuBryfZCDwLXN3WPQRcDkwBvwauH2HbkqQRzTkAquo/mP24PsC6WdoXcMNctydJml9+E1iSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSp+bl9wAk9WFcv0Xg7xAcH84AJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ3yi2CSTnh+Ae34cAYgSZ0yACSpUx4CkqRDGNehJ1iYw0/OACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdWvAASPKhJD9JMpVk80JvX5I0sKABkOQU4J+By4B3AtcmeedC9kGSNLDQM4CLgKmqerqqfgv8K7B+gfsgSWLhA2AFsGeoPN3qJEkLbKEvBpdZ6uoPGiSbgE2t+L9JnjzuvTpxnQW8MO5OjJHjd/zdjj+3jTT+Pz2aRgsdANPAqqHySuC54QZVtQXYApBkZ1WtWbjunVgcv+N3/I7/eG5joQ8BfRdYneS8JKcC1wDbFrgPkiQWeAZQVfuT/A3wMHAKcHdVPbWQfZAkDSz4D8JU1UPAQ0fZfMvx7MtJwPH3zfH37biPP1V15FaSpEXHS0FIUqdOyADo4XIRSVYleSzJriRPJbmx1S9P8kiS3e1+WatPkjvaa/JEkgvHO4L5keSUJN9P8kArn5dkRxv/19rJAiQ5rZWn2vrJcfZ7PiRZmuTeJD9u74OLe9r/ST7R3vtPJrknyemLff8nuTvJvuHT2+eyz5NsaO13J9kw1/6ccAHQ0eUi9gOfrKp3AGuBG9o4NwPbq2o1sL2VYfB6rG63TcCdC9/l4+JGYNdQ+Tbg9jb+l4CNrX4j8FJVvRW4vbU72X0e+FZVvR14N4PXoYv9n2QF8DFgTVW9i8FJIdew+Pf/l4EPHVR3TPs8yXLgFuA9DK6ucMuB0DhmVXVC3YCLgYeHyjcDN4+7Xwsw7vuBvwR+Apzb6s4FftKWvwBcO9T+9+1O1huD74FsBy4BHmDwRcEXgCUHvxcYnDl2cVte0tpl3GMYYexvBH528Bh62f+8elWA5W1/PgB8sIf9D0wCT851nwPXAl8Yqv+DdsdyO+FmAHR4uYg2nb0A2AGcU1V7Adr92a3ZYnxdPgd8Cvi/Vj4TeLmq9rfy8Bh/P/62/pXW/mR1PjADfKkdAvtikjPoZP9X1c+BzwDPAnsZ7M/H6Wf/DzvWfT5v74UTMQCOeLmIxSTJG4BvAB+vql8eruksdSft65Lkw8C+qnp8uHqWpnUU605GS4ALgTur6gLgV7w69Z/Nohp/O2SxHjgPeDNwBoNDHgdbrPv/aBxqzPP2WpyIAXDEy0UsFklex+CP/1er6r5W/XySc9v6c4F9rX6xvS7vAz6S5BkGV4W9hMGMYGmSA99PGR7j78ff1r8JeHEhOzzPpoHpqtrRyvcyCIRe9v8HgJ9V1UxV/Q64D3gv/ez/Yce6z+ftvXAiBkAXl4tIEuAuYFdVfXZo1TbgwKf6Gxh8NnCg/rp2ZsBa4JUD08aTUVXdXFUrq2qSwT5+tKo+CjwGXNWaHTz+A6/LVa39Sfs/wKr6BbAnydta1TrgR3Sy/xkc+lmb5PXt38KB8Xex/w9yrPv8YeDSJMvaTOrSVnfsxv2ByCE+JLkc+C/gp8Dfj7s/x2mMf85g2vYE8IN2u5zBcc3twO52v7y1D4Ozo34K/JDB2RNjH8c8vRbvBx5oy+cD/wlMAf8GnNbqT2/lqbb+/HH3ex7G/WfAzvYe+CawrKf9D/wD8GPgSeBfgNMW+/4H7mHwmcfvGPxPfuNc9jnw1+21mAKun2t//CawJHXqRDwEJElaAAaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmd+n+xvH4tmuPj9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# # see distribution of images less than 1000 pixels\n",
    "# plt.hist(row_sz[row_sz < 1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below if you need to reset your precomputed activations\n",
    "# shutil.rmtree(f'{PATH}tmp', ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shutil.rmtree(f'{PATH}model', ignore_errors=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/resnet101-5d3b4d8f.pth\" to /home/ubuntu/.torch/models/resnet101-5d3b4d8f.pth\n",
      "100%|██████████| 178728960/178728960 [00:01<00:00, 115185199.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 63/63 [00:52<00:00,  1.19it/s]\n",
      "100%|██████████| 16/16 [00:14<00:00,  1.07it/s]\n",
      "100%|██████████| 13/13 [00:10<00:00,  1.24it/s]\n"
     ]
    }
   ],
   "source": [
    "# Create Learner\n",
    "learn = ConvLearner.pretrained(arch,\n",
    "                               data,\n",
    "                               precompute=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find initial learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ac1b50bf53d46ef9ebb901512f0647d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 79%|███████▉  | 50/63 [00:01<00:00, 39.76it/s, loss=9.35]\n",
      "                                                          \r"
     ]
    }
   ],
   "source": [
    "# # use learn.lr_find to find the optimal learning rate\n",
    "lrf = learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOXd//H3dyYrJEQCYRGIYRFZlDUggopbXau0VsEqiIhb62O1+vRxa/urtYvWqrVuiKi4r6h1xxU31oDsAWRH1kDYE8h2//6YkcaYwARycmYyn9d1zZUzZ+45881NyCdnu29zziEiIgIQ8LsAERGJHgoFERHZR6EgIiL7KBRERGQfhYKIiOyjUBARkX0UCiIiso9noWBmKWY23czmmNkCM7ujmjbJZvaymS01s2lmluNVPSIicmBe7insBU5xzvUEegFnmtmAKm1GA1udc52A+4G7PaxHREQOwLNQcCG7wk8Tw4+qt08PAZ4OL78GnGpm5lVNIiKyfwlebtzMgsBMoBPwsHNuWpUmbYA1AM65MjPbDjQDNte0zebNm7ucnBxvChYRaaBmzpy52TmXdaB2noaCc64c6GVmhwFvmNnRzrn5lZpUt1fwo8GYzOwq4CqA7Oxs8vLyPKlXRKShMrNVkbSrl6uPnHPbgEnAmVVe+g5oB2BmCUAGUFjN+8c653Kdc7lZWQcMOhEROUheXn2UFd5DwMxSgdOARVWavQWMDC9fAHzqNGyriIhvvDx81Bp4OnxeIQC84px7x8z+DOQ5594CngCeNbOlhPYQLvKwHhEROQDPQsE5NxfoXc36P1Za3gNc6FUNIiJSO7qjWURE9lEoiIjIPnETCoW7S/hg/nqKSsr8LkVEJGp5ep9CNPl44Ub+b8JcUhIDnHhkFmd0b8VpXVuS0SjR79JERKJG3ITC+X3a0DYzlYnzNzBxwUY+XLiRhIAxoEMzzji6Fad3a0nLJil+lyki4iuLtdsCcnNz3aHe0VxR4Zi7djsTF2xg4vwNLN+8G4DuhzfhxM5ZDO6cRZ/spiQlxM3RNRFp4MxspnMu94Dt4jEUKnPOsXTTLj5cuJHPlxQwa9VWyiocjZOCDOzUnBM7Z3FS5yzaZTaq9bZ37y0jNTFIIKAx/kTEXwqFg7RzTymTl23h8yUFfL64gLXbigFol5nKcR2acVzHZhzXoTmtMn58qGnnnlJmrCxk8tItTF62hfwNO8hITaRfTibHts9kQIdmdG3dhKBCQkTqmUKhDjjnWL55N18uKWDysi1MW1HI9uJSADo0b8yAjs3ok92UFZt3MXnZFuZ+t53yCkdSQoC+2U3p1z6TDduLmbaikFVbigBIT0nYFxKDj8riqJbpaLRwEfGaQsED5RWO/PU7mLJsC1OWb2H6ikJ27S0jGDB6ts1gYMfmDOzYjD5HNCUlMfiD967fXsz0FYVMXV7ItBVbWF4QOo/RvnljzujeijOPbkXPthkKCBHxhEKhHpSVV7CsYDdtmqaSlly7C7k27tjDx/kb+WD+BqYs20JZhePwjBRO796Ks45uRW5Opg4ziUidUSjEkG1FJXycv4kP5m/gi28LKCmr4IhmjbjyhA5c0Lftj/Y6RERqS6EQo3bvLePj/I08+fVK5qzZRvO0JEYNas/wY4/QjXYictAUCjHOOcfU5YWM+XwZny8poHFSkIuPzWb08R2qvfJJRGR/FAoNyMJ1Oxj7xTLenruegEHv7KY0T0sis3ESmY3CX9OSyWyURGpSgJIyR1lFBaXlFZSWO0rLKygrd2SlJzOgQzOdqxCJQwqFBmhNYRFPfb2S+eu2U7i7hMLdJWwtKqE2/4RtDktlaG47hvZrS+uMVO+KFZGoolCIE+UVjm1FoYDYsruEvWUVJAaNpGCAhGCAxKCRGAyQEDAWrt/BS9PX8NXSzQQMBnfO4qL+2ZzSpQWJQQ3pIdKQKRSkRqu3FPFK3hpenbmGjTv2kpWezLDcdowalEOztGS/yxMRDygU5IDKyiuYtLiAl2as5pNFm0hJCDJ8QDZXntiBFuk6mS3SkCgUpFaWbtrJw58t4z+z15IYDPDL/tlcPbiDzjuINBAKBTkoKzfv5pFJS3l91loCZlyY25YrTuhATrNGGoJDJIYpFOSQrCks4tHPl/Fq3hpKyx0piQGyMxuRndmYI5o14ohmjcjObETHrDTaNk1VYIhEOYWC1In124v5eOFGVm0pYlVhEau3FLG6sIji0vJ9bY5o1oiTOmdxUpcWHNehmYblEIlCCgXxjHOOgp17WV1YxML1O/h8cQFfL9vMntIKkhMCDOzYjJOOasEpXVoc1OREIlL3FApSr/aUljNtRSGTFm9i0uICVmzeTcDgrvN7MLRfO7/LE4l7kYZC7cZ7FqlBSmKQweH5rf/fuaET1n/4z3xufn0ugIJBJEboNlbxRE7zxjx+aS4nHJnF/02Yy8szVvtdkohEQKEgnklJDDJ2RF8Gd87i5gnzeGm6gkEk2ikUxFMpiUEeG9GXk47K4pbX5/HCNAWDSDRTKIjnUhKDjBnel5OPyuK2N+bx/LRVfpckIjVQKEi9SEkMMmZEX07p0oLb35jPc1MVDCLRSKEg9SY5Icijw/twapcW/P7N+Tw7ZaXfJYlIFQoFqVfJCUEeGd6H07q25A//WcDTk1f6XZKIVKJQkHqXnBDkkUv68JNuLfl/by3gya9W+F2SiIR5Fgpm1s7MPjOzfDNbYGbXV9Mmw8zeNrM54TajvKpHoktSQoCHL+7DGd1b8ud3FjLuy+V+lyQieLunUAbc5JzrCgwArjWzblXaXAssdM71BE4C7jWzJA9rkiiSlBDgoYv7cPYxrfjLu/mM/WKZ3yWJxD3Phrlwzq0H1oeXd5pZPtAGWFi5GZBuoXGX04BCQmEicSIxGOCBi3pjNpu/vbeICgfXDO7od1kicatexj4ysxygNzCtyksPAW8B64B0YJhzrqI+apLokRgM8MCwXgTMuOv9RZRXOK49uZPfZYnEJc9DwczSgAnADc65HVVePgOYDZwCdAQ+MrMvq7Yzs6uAqwCys7O9Lll8kBAMcP/QngQM7pm4mLTkBEYOzPG7LJG44+nVR2aWSCgQnnfOvV5Nk1HA6y5kKbAC6FK1kXNurHMu1zmXm5WV5WXJ4qOEYID7hvbitK4tuePtBXyxpMDvkkTijpdXHxnwBJDvnLuvhmargVPD7VsCRwG6DCWOBQPGAxf14qhWTbj2+Vks3bTT75JE4oqXewqDgBHAKWY2O/w428yuMbNrwm3uBAaa2TzgE+Bm59xmD2uSGNA4OYFxI3NJTgxw+fg8tu4u8bskkbihmdckas1avZWLxk6ld7vDeHb0sSQl6F5LkYMV6cxr+l8mUatPdlPuuaAH01YU8oc35xNrf8CIxCJNxylRbUivNizdtIsHP13KkS3TuOKEDn6XJNKgKRQk6v32tM4sK9jFX9/Lp33zxpzataXfJYk0WDp8JFEvEDDuvbAXRx+ewW9e/IZFG6re7iIidUWhIDEhNSnI45fmkpaSwOjxeRTs3Ot3SSINkkJBYkarjBTGXdqPLbv3ctWzeewpLfe7JJEGR6EgMeWYthn8a1gvvlm9jd+9NldXJInUMYWCxJwzj27NzWd24e0563jgk2/9LkekQdHVRxKTrhncgWUFu/jXx9/SvnljhvRq43dJIg2C9hQkJpkZf/v5MfRvn8nvXpvLzFVb/S5JpEFQKEjMSkoIMGZ4X1pnpHD1s3msKSzyuySRmKdQkJiW2TiJJ0b2o6SsgiuezmPnnlK/SxKJaQoFiXmdWqTx6PC+LC3YxY2vzNEVSSKHQKEgDcKgTs257eyufLRwI2M+15QcIgdLoSANxuWDcvhpj9bcM3ERk5dqWg6Rg6FQkAbDzLj7Fz3okJXGdS9+w/rtxX6XJBJzFArSoDROTmDM8L7sKS3n2udnUVJW4XdJIjFFoSANTqcWafzjgp7MWr2Nv72X73c5IjFFoSAN0jk9WjP6+PaMn7yS/8xe63c5IjFDoSAN1i1ndaFfTlNumTCPxRt2+l2OSExQKEiDlRgM8PDFfUhLSeBXz83UjW0iEVAoSIPWokkKD/2yN6sKi7j2hW/YW6Y5GET2R6EgDd6xHZrx9/OP4YslBVz3wjeUluuKJJGaKBQkLgzNbccd53Xnw4UbufGVOZRXaCgMkepoPgWJGyMH5lBcWs5d7y8iJSHA3b/oQSBgfpclElUUChJXrhnckaKScv79ybekJgW547zumCkYRL6nUJC489vTjqS4pIzHv1xBalKQW87somAQCVMoSNwxM247uyvFpeU89vlyGiUmcP1pR/pdlkhUUChIXDIz/nze0ewpreD+j5fQJDWBUYPa+12WiO909ZHErUAgNKrqaV1b8td385n33Xa/SxLxnUJB4lowYPzzwh40T0vm+pe+oaikzO+SRHylUJC4d1ijJO4b1pMVW3Zz5zsL/S5HxFcKBRFgYMfmXHViB16cvoYP5m/wuxwR3ygURMJu+slRHN2mCbe8PpcN2/f4XY6ILzwLBTNrZ2afmVm+mS0ws+traHeSmc0Ot/ncq3pEDiQpIcADF/Vmb2kFN706mwoNhSFxyMs9hTLgJudcV2AAcK2ZdavcwMwOAx4BznPOdQcu9LAekQPqmJXGH8/txtdLtzDuq+V+lyNS7zwLBefceufcrPDyTiAfaFOl2cXA68651eF2m7yqRyRSF/VrxxndW3LPxMXMX6vLVCW+1Ms5BTPLAXoD06q81BloamaTzGymmV1aH/WI7I+Zcdf5PchsnMRvXvqG4hLNwSDxw/NQMLM0YAJwg3NuR5WXE4C+wDnAGcAfzKxzNdu4yszyzCyvoKDA65JFaNo4ifuG9mJ5wW5GPjmdV/PWULi7xO+yRDxnznl3Ms3MEoF3gInOufuqef0WIMU596fw8yeAD5xzr9a0zdzcXJeXl+dRxSI/9MyUlYyZtIx12/cQMOjfPpPTu7Xi9O4tadu0kd/liUTMzGY653IP2M6rULDQsJNPA4XOuRtqaNMVeIjQXkISMB24yDk3v6btKhSkvjnnmL92Bx8u3MDEBRtYsnEXAN0Pb8KoQe25oG9bnysUObBIQ8HLAfEGASOAeWY2O7zuNiAbwDk3xjmXb2YfAHOBCmDc/gJBxA9mxjFtMzimbQY3nX4UKzbv5sMFG3hrzjr+99U5JAaNIb2qXkMhEps8PXzkBe0pSLTYW1bOiCemM3v1Np674lj6t8/0uySRGkW6p6A7mkUOUnJCkLEj+tI2M5Wrns1jecEuv0sSOWQKBZFDcFijJJ66rB8BM0aNn8GWXXv9LknkkCgURA7REc0a8/iluazfvoernp3JnlLd1yCxS6EgUgf6HtGU+4f2Yuaqrdz06hyNmyQxS6EgUkfO6dGaW87qwrtz13PPh4v9LkfkoGiOZpE6dPWJHVi1pYhHJy2jXdNGXHxstt8lidSKQkGkDpkZdw7pzrptxdz2xjx27Cnl6hM7ELqXUyT66fCRSB1LCAZ4bERfzu15OHe9v4jb3phPWXmF32WJRER7CiIeSEkM8sCwXmRnpvLwZ8tYu62Yhy/uTXpKot+liexXRHsKZna9mTWxkCfMbJaZne51cSKxLBAwfndGF+7+xTF8vXQzF46ZwrptxX6XJbJfkR4+ujw87PXpQBYwCrjLs6pEGpBh/bIZP6ofa7cW8/NHvtbEPRLVIg2F78+SnQ085ZybU2mdiBzACUdm8dqvBhI0Y+hjU/gkf6PfJYlUK9JQmGlmHxIKhYlmlk5oVFMRidBRrdJ589pBdMhqzBXP5HHPxEU6AS1RJ9JQGA3cAvRzzhUBiYQOIYlILbRoksKrVw9kaN92PPzZMi4aO1XnGSSqRBoKxwGLnXPbzGw48HtAB0ZFDkJqUpC7L+jBAxf1In/9Ds564Es+WqjDSRIdIg2FR4EiM+sJ/B+wCnjGs6pE4sCQXm145zcn0C4zlSufyeOOtxewt0yD6Ym/Ig2FMheajWcI8IBz7gEg3buyROJD++aNmfCrgVw2MIenvl7JLx6dzMrNu/0uS+JYpKGw08xuJTS95rtmFiR0XkFEDlFyQpA/ndedsSP6sqawmHMf+opFG3b4XZbEqUhDYRiwl9D9ChuANsA9nlUlEodO796Kd647nkZJQS5/agYbd+zxuySJQxGFQjgIngcyzOynwB7nnM4piNSxdpmNeGJkP7YVlzL66Rns3lvmd0kSZyId5mIoMB24EBgKTDOzC7wsTCReHd0mg4cv7sPCdTv4zYvfUK4Je6QeRXr46HZC9yiMdM5dCvQH/uBdWSLx7eQuLbjjvO58smgTf357AaHrPES8F+koqQHn3KZKz7egYbdFPDXiuBxWFxbx+JcryG7WmNHHt/e7JIkDkYbCB2Y2EXgx/HwY8J43JYnI9249qytrCov5y7sLads0lTO6t/K7JGngIj3R/DtgLNAD6AmMdc7d7GVhIhIafvv+Yb3o2fYwrn/pG+as2eZ3SdLARXwIyDk3wTl3o3Put865N7wsSkT+KzUpyLiRuWSlJzP66Rms0M1t4qH9hoKZ7TSzHdU8dpqZ7q4RqSfN05J56rL+OAcXPz6VNYVFfpckDdR+Q8E5l+6ca1LNI90516S+ihQR6NQijeeuOJbi0nIuGjuVtRpdVTygK4hEYkjX1k149vJj2bGnlIsfn8qG7brrWeqWQkEkxhzTNoOnL+/P5p17uXjcVAp27vW7JGlAFAoiMahPdlPGX96f9dv2MHzcNAp3l/hdkjQQCgWRGNUvJ5MnRuaycstuho+bxrYiBYMcOoWCSAwb2Kk5Yy/NZemmXVz65HSKSjSAnhwaz0LBzNqZ2Wdmlm9mC8zs+v207Wdm5RpkT6T2BnfO4uFL+jD3u+088Mm3fpcjMc7LPYUy4CbnXFdgAHCtmXWr2ig8Yc/dwEQPaxFp0H7SrSUX9m3LE1+uYPGGnX6XIzHMs1Bwzq13zs0KL+8E8glNzlPVdcAEYFM1r4lIhG49uytpKQn84c35GlVVDlq9nFMwsxygNzCtyvo2wM+BMfVRh0hDltk4iVvP6sL0lYW8NvM7v8uRGOV5KJhZGqE9gRucc1WHxvgXcLNzrvwA27jKzPLMLK+goMCrUkVi3oV929H3iKb87b18tuoyVTkInoaCmSUSCoTnnXOvV9MkF3jJzFYCFwCPmNnPqjZyzo11zuU653KzsrK8LFkkpgUCxl9+djQ79pRx9weL/C5HYpCXVx8Z8ASQ75y7r7o2zrn2zrkc51wO8Brwa+fcm17VJBIPurZuwujj2/PSjDXMXFXodzkSY7zcUxgEjABOMbPZ4cfZZnaNmV3j4eeKxL3rTz2SwzNSuP2N+ZSWV/hdjsSQSGdeqzXn3FeA1aL9ZV7VIhJvGicn8Mdzu3PNczMZ//VKrjyxg98lSYzQHc0iDdQZ3VtySpcW3P/xEtZpmG2JkEJBpIEyM+44rzsVzvHntxf6XY7ECIWCSAPWLrMR151yJB8s2KB7FyQiCgWRBu7KEzowsGMzbpkwly+/1X0+sn8KBZEGLikhwJgRfenUIo1fPTeLBeu2+12SRDGFgkgcaJKSyFOj+pGeksCop2bw3dYiv0uSKKVQEIkTrTNSefry/hSXlnPZUzM0KY9US6EgEkc6t0xn7IhcVm8p4qpnZrKndL/DjkkcUiiIxJnjOjbjn0N7Mn1lITe+MpuKCg2zLf+lUBCJQ+f1PJzbz+7Ke/M28Jd38/0uR6KIZ8NciEh0u+KE9qzbXsyTX6+geXoSvz6pk98lSRRQKIjEKTPj9+d0Y8uuEv7xwWJSE4OMGtTe77LEZwoFkTgWDBj3Du3JntJy7nh7IY2Sggzrl+13WeIjnVMQiXOJwQAPXtybwZ2zuOX1efxn9lq/SxIfKRREhOSEIGOG96V/TiY3vjKHD+Zv8Lsk8YlCQUQASE0K8sRl/ejRNoPrXpzFpMWb/C5JfKBQEJF90pITGD+qP0e2SOfqZ2cyZdkWv0uSeqZQEJEfyEhN5NnR/cnObMTop2cwa/VWv0uSeqRQEJEfaZaWzPNXHEuL9GRGPjmd+Ws1smq8UCiISLVaNEnh+SsH0CQlkRFPTGPxhp1+lyT1QKEgIjVqc1gqL1x5LInBAJeMm8bygl1+lyQeUyiIyH4d0awxL1x5LM45Lhk3jTWFmouhIVMoiMgBdWqRznNXHEtRSTkXj5vK+u3FfpckHlEoiEhEurZuwrOj+7NtdymXPD6NTTv3+F2SeEChICIR69H2MJ4a1Y8NO/YwfNw0Cndr9raGRqEgIrWSm5PJuEtzWbWliGufn0W5JulpUBQKIlJrAzs1568/P4Ypy7dw/0dL/C5H6pBCQUQOygV923JRv3Y89NlSPl200e9ypI4oFETkoP3pvO50a92E3748h++26lLVhkChICIHLSUxyCOX9KGiwnHt87PYW1bud0lyiBQKInJIcpo35p4LezDnu+387d18v8uRQ6RQEJFDdubRrbni+PY8PWUVb81Z53c5cggUCiJSJ24+qwt9j2jKLRPmsnSTBs+LVQoFEakTicEAD13cm5TEIL96bhZFJWV+lyQHwbNQMLN2ZvaZmeWb2QIzu76aNpeY2dzwY7KZ9fSqHhHxXuuMVB64qBdLC3Zx48tzKCuv8LskqSUv9xTKgJucc12BAcC1ZtatSpsVwGDnXA/gTmCsh/WISD044cgsfn9ONz5YsIHb3piHc7rjOZYkeLVh59x6YH14eaeZ5QNtgIWV2kyu9JapQFuv6hGR+jP6+PZsLyrh358upUlKIref0xUz87ssiYBnoVCZmeUAvYFp+2k2Gni/PuoREe/99ied2bGnjHFfrSAjNZHrTj3S75IkAp6HgpmlAROAG5xzO2poczKhUDi+htevAq4CyM7O9qhSEalLZsYff9qNHcWl3PvREjIaJXLpcTl+lyUH4OnVR2aWSCgQnnfOvV5Dmx7AOGCIc25LdW2cc2Odc7nOudysrCzvChaROhUIGHdf0IPTurbkj/9ZwJvfrPW7JDkAL68+MuAJIN85d18NbbKB14ERzjkNtSjSAH1/qeqADpnc9OocPl6owfOimZd7CoOAEcApZjY7/DjbzK4xs2vCbf4INAMeCb+e52E9IuKTlMQg40b2o/vhTfj1C7P4JF/BEK0s1i4Xy83NdXl5yg6RWFS4u4Rfjp3K4o07OfmoLG49uyudW6b7XVZcMLOZzrncA7XTHc0iUm8yGyfxn/8ZxK1ndSFv1VbO/NcX3PzaXDbu0HzP0UJ7CiLii627S3jw06U8O3UlwYBx5QkduHpwR9KS6+VK+bijPQURiWpNGyfxx3O78cmNJ3Fa15Y8+OlSBv/jM12hVI3yCsf/vDCLj+rhJL1CQUR8ld2sEQ9d3Ic3rx1ETvPG3PDybMZ8vszvsqLKX95dyDtz17OhHg6zKRREJCr0ancYL145gHN7Hs5d7y/irvcXadwk4NkpK3nq65VcPqg9IwYc4fnn6eCdiESNpIQA/xrWi4zUBMZ8voztxSX85WfHEAzE57hJkxZv4k9vL+TULi24/Zyu9fKZCgURiSrBgHHnkKNp2iiJBz9dyvbiUu4f1ovkhKDfpdWrxRt28j8vfEPnluk88Mve9RaMCgURiTpmxk2nH0VGaiJ/eTefnXvyGDO8L43j5Mqkgp17uXz8DBolBXliZG69XpGlcwoiErWuOKED91zQg6+XbuaScdPYVlTid0me21NazpXP5LFl917Gjczl8MNS6/XzFQoiEtUuzG3Ho8P7snDdDoY+NoVNDfhGt4oKx/++Ooc5323jX8N606PtYfVeg0JBRKLeGd1bMX5UP77bWswFY6awprDI75I8cf/HS3hn7npuPrMLZx7dypcaFAoiEhMGdmrO81ccy/biUi4YM5lvN+70u6Q6NX/tdh78dCkX9m3L1Sd28K0OhYKIxIze2U15+eoBVDgY+tgU5n233e+S6sy9Hy4mIzWRP5zbzdepSxUKIhJTurRqwqtXH0ejpAR++fhUpi2vdm6umDJzVSGfLS7g6sEdaJKS6GstCgURiTk5zRvz2q+Oo2WTZC59cjqfLdrkd0mH5N4Pl9A8LYnLBub4XYpCQURiU+uMVF65+jiObJnGlc/k8dacdX6XdFAmL93M5GVb+PVJnWiU5P99GAoFEYlZzdKSeeHKAfTJbspvXvyGWybMZdfeMr/Liphzjns/WkKrJilcfGy23+UACgURiXFNUhJ59or+XDO4Iy/nreHsB75kxspCv8uKyKQlBcxctZXrTu1ESmJ0DOOhUBCRmJecEOSWs7rwytXH4XAMfWwKd72/iL1l5X6XViPnHPd+uJh2malc2Led3+Xso1AQkQajX04m719/Ihf1a8eYz5cx5KGvWbRhh99lVWvigo3MX7uD60/tTFJC9Pwqjp5KRETqQFpyAn8/vwdPjMxl864Sznvwax77fBkVFdEzN0N5heO+jxbTIasxP+t1uN/l/IBCQUQapFO7tmTiDSdwSpcW/P39RVz5TB7bi0r9LguAd+auY8nGXdz4k84kBKPr13B0VSMiUoeapSXz6PA+3HFed774toCfPvQl89f6exd0WXkF93+0hC6t0jn76Na+1lIdhYKINGhmxsiBObx89XGUlzvOf3QyL01f7dtUn6/PWsvKLUXcdPpRBKJwRjn/75QQEakHfbKb8s5vTuD6l77hltfnkbdqK3cOOZrUpLq7FNQ5xyt5a3jos6WUlztSkoI0SgqSmhgkNSmB1MQAM1dto2fbDE7r2qLOPrcuKRREJG5kNk5i/Kj+PPDJtzz46bcsWLeDRy/pQ07zxoe87TWFRdz6+jy+WrqZPtmH0TErjaLScvaUlFNUUs6O4lI27SgnIzWB28/xd9C7/TG/dqEOVm5ursvLy/O7DBGJcZMWb+KGl2dTUlbB8AFHcMUJ7WmRnlLr7VRUOJ6ZspJ/TFyMAbec3ZVL+mdH3aEhM5vpnMs9YDuFgojEq++2FnHPxMW8PWcdCcEAw3LbcfXgDrRt2iii9y8r2MXNr80lb9VWBnfO4m/nH0Obep4+M1IKBRGRCK3cvJsxny9jwqzvcA6G9Goou6FtAAAKN0lEQVTDr0/uSMestB+021tWzuZdJRTs3MtX3xbw70+XkpoY5I8/7cb5fdpE7SEhUCiIiNTa+u3FjP1iOS9OX83esgqO79ScsnJHwa69FOzcy/biH97ncNbRrbhjSPeDOuxU3xQKIiIHafOuvTz19Qo+XLCRjNREstKTQ4+0ZFo0CS23OawRR7VK97vUiCkURERkn0hDQTeviYjIPgoFERHZx7NQMLN2ZvaZmeWb2QIzu76aNmZm/zazpWY218z6eFWPiIgcmJd3NJcBNznnZplZOjDTzD5yzi2s1OYs4Mjw41jg0fBXERHxgWd7Cs659c65WeHlnUA+0KZKsyHAMy5kKnCYmUXfsIEiInGiXs4pmFkO0BuYVuWlNsCaSs+/48fBISIi9cTzUDCzNGACcINzruq8eNXd/veja2TN7CozyzOzvIKCAi/KFBERPA4FM0skFAjPO+der6bJd0DlGavbAuuqNnLOjXXO5TrncrOysrwpVkREvLt5zUKDgDwNFDrnbqihzTnA/wBnEzrB/G/nXP8DbLcAWAVkAFWnUIpkXeXnzYHNB/xmaq+6OurqPftrV9Nrte2Xqs+96qeaaquL9/jRTxB7P1MHanOw/8+qW6f/e9Wvq69+OsI5d+C/qp1znjyA4wkdCpoLzA4/zgauAa4JtzHgYWAZMA/IrcX2xx7MusrPgTyPvvcf1VFX79lfu5peq22/1Fc/edlXfvSTl33lRz9F2icH03ex1k8HaheL//dqenh2Sapz7iuqP2dQuY0Drj3Ij3j7INdV16auHcxnRPqe/bWr6bWD6Zf66KeD/ZxI3qN+OvR+qul1/d+L/LVo/pmqVsyNfVSXzCzPRTAWSLxTP0VOfRUZ9VNk/OineB/mYqzfBcQI9VPk1FeRUT9Fpt77Ka73FERE5IfifU9BREQqUSiIiMg+CgUREdlHobAfZtbYzGaa2U/9riVamVlXMxtjZq+Z2a/8ridamdnPzOxxM/uPmZ3udz3RzMw6mNkTZvaa37VEm/DvpKfDP0uXePEZDTIUzOxJM9tkZvOrrD/TzBaH52+4JYJN3Qy84k2V/quLfnLO5TvnrgGGAg3yEsM66qc3nXNXApcBwzws11d11FfLnXOjva00etSyz84HXgv/LJ3nRT0NMhSA8cCZlVeYWZDQ3dNnAd2AX5pZNzM7xszeqfJoYWanAQuBjfVdfD0azyH2U/g95wFfAZ/Ub/n1Zjx10E9hvw+/r6EaT931VbwYT4R9Rmh8uO9Hli73ohgvJ9nxjXPui/Bw3ZX1B5Y655YDmNlLwBDn3N+BHx0eMrOTgcaE/kGKzew951yFp4XXs7rop/B23gLeMrN3gRe8q9gfdfTzZMBdwPsuPM9IQ1RXP1PxpDZ9RmgQ0baEhg3y5I/6BhkKNahu7oYaZ3lzzt0OYGaXAZsbWiDsR636ycxOIrRLmwy852ll0aVW/QRcB5wGZJhZJ+fcGC+LizK1/ZlqBvwV6G1mt4bDI97U1Gf/Bh4KDybqyXAY8RQKEc3d8KMGzo2v+1KiWq36yTk3CZjkVTFRrLb99G9C/6HjUW37aguhgTPjWbV95pzbDYzy8oMb6jmF6kQ0d4OonyKkfoqc+qr2fOuzeAqFGcCRZtbezJKAi4C3fK4pGqmfIqN+ipz6qvZ867MGGQpm9iIwBTjKzL4zs9HOuTJCE/pMBPKBV5xzC/ys02/qp8ionyKnvqq9aOszDYgnIiL7NMg9BREROTgKBRER2UehICIi+ygURERkH4WCiIjso1AQEZF9FAriOTPbVQ+fcV6Ew6HX5WeeZGYDD+J9vc1sXHj5MjN7qO6rqz0zy6k6fHM1bbLM7IP6qknqn0JBYkZ4OOFqOefecs7d5cFn7m98sJOAWocCcBvw4EEV5DPnXAGw3swG+V2LeEOhIPXKzH5nZjPMbK6Z3VFp/ZsWmuVugZldVWn9LjP7s5lNA44zs5VmdoeZzTKzeWbWJdxu31/cZjbezP5tZpPNbLmZXRBeHzCzR8Kf8Y6Zvff9a1VqnGRmfzOzz4HrzexcM5tmZt+Y2cdm1jI81PE1wG/NbLaZnRD+K3pC+PubUd0vTjNLB3o45+ZU89oRZvZJuG8+MbPs8PqOZjY1vM0/V7fnZaEZud41szlmNt/MhoXX9wv3wxwzm25m6eE9gi/DfTirur0dMwua2T2V/q2urvTym4Ans35JFHDO6aGHpw9gV/jr6cBYQiNABoB3gBPDr2WGv6YC84Fm4ecOGFppWyuB68LLvwbGhZcvAx4KL48HXg1/RjdC49IDXEBoeO8A0ArYClxQTb2TgEcqPW/Kf+/+vwK4N7z8J+B/K7V7ATg+vJwN5Fez7ZOBCZWeV677bWBkePly4M3w8jvAL8PL13zfn1W2+wvg8UrPM4AkYDnQL7yuCaGRkRsBKeF1RwJ54eUcYH54+Srg9+HlZCAPaB9+3gaY5/fPlR7ePOJp6Gzx3+nhxzfh52mEfil9AfzGzH4eXt8uvH4LodmlJlTZzuvhrzMJzeVQnTddaA6MhWbWMrzueODV8PoNZvbZfmp9udJyW+BlM2tN6BftihrecxrQzWzfqMdNzCzdObezUpvWQEEN7z+u0vfzLPCPSut/Fl5+AfhnNe+dB/zTzO4G3nHOfWlmxwDrnXMzAJxzOyC0V0FoTP5ehPq3czXbOx3oUWlPKoPQv8kKYBNweA3fg8Q4hYLUJwP+7px77AcrQxP1nAYc55wrMrNJQEr45T3OuarTDu4Nfy2n5p/hvZWWrcrXSOyutPwgcJ9z7q1wrX+q4T0BQt9D8X62W8x/v7cDiXhgMufcEjPrC5wN/N3MPiR0mKe6bfyW0DSzPcM176mmjRHaI5tYzWsphL4PaYB0TkHq00TgcjNLAzCzNhaakzcD2BoOhC7AAI8+/yvgF+FzCy0JnSiORAawNrw8stL6nUB6pecfEhrZEoDwX+JV5QOdavicyYSGSIbQMfuvwstTCR0eotLrP2BmhwNFzrnnCO1J9AEWAYebWb9wm/TwifMMQnsQFcAIoLoT+BOBX5lZYvi9ncN7GBDas9jvVUoSuxQKUm+ccx8SOvwxxczmAa8R+qX6AZBgZnOBOwn9EvTCBEKTl8wHHgOmAdsjeN+fgFfN7Etgc6X1bwM///5EM/AbIDd8YnYh1cwe5pxbRGhKzvSqr4XfPyrcDyOA68PrbwBuNLPphA4/VVfzMcB0M5sN3A78xTlXAgwDHjSzOcBHhP7KfwQYaWZTCf2C313N9sYBC4FZ4ctUH+O/e2UnA+9W8x5pADR0tsQVM0tzzu2y0DzA04FBzrkN9VzDb4GdzrlxEbZvBBQ755yZXUTopPMQT4vcfz1fAEOcc1v9qkG8o3MKEm/eMbPDCJ0wvrO+AyHsUeDCWrTvS+jEsAHbCF2Z5AszyyJ0fkWB0EBpT0FERPbROQUREdlHoSAiIvsoFEREZB+FgoiI7KNQEBGRfRQKIiKyz/8HNbNb+hIl2a0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# #plot learing rate\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d15cd881442a467b8ce82ab705e15c34",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                 \n",
      "    0      1.102201   0.638188   0.783433  \n",
      "    1      0.6997     0.457675   0.853293                  \n",
      "    2      0.520555   0.417479   0.860279                  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.41748]), 0.8602794402849651]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Fit learner with a few epochs\n",
    "learn.fit(lr, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5fd5322cdb5a42109c12cbddd3972706",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=2), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                  \n",
      "    0      0.370049   0.369513   0.878244  \n",
      "    1      0.338812   0.341267   0.89022                   \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.34127]), 0.8902195609972149]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Fit learner with a few epochs\n",
    "learn.fit(5e-3, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0cae3832445340868485a19a55f40f85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=2), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                  \n",
      "    0      0.29358    0.337955   0.89022   \n",
      "    1      0.282165   0.333225   0.891218                  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.33322]), 0.8912175637995173]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(5e-4, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.save('inital_train')\n",
    "learn.save('resne101_inital_train_all_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.precompute=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a7e63b2028545e4939e2b378681b761",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                  \n",
      "    0      0.541432   0.359509   0.873253  \n",
      "    1      0.513547   0.327887   0.888224                  \n",
      "    2      0.505611   0.343209   0.876248                  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.34321]), 0.8762475052279627]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, 3, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.save('resnet50_224_ll') # training and validation sets\n",
    "learn.save('resnet101_224_ll') # training and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.load('resnet50_224_ll') # training and validation sets\n",
    "learn.load('resnet101_224_ll') # training and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.set_data(data = get_data(299, bs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df140c96a9754508a6936eeb4305b59a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                  \n",
      "    0      0.455418   0.308661   0.897206  \n",
      "    1      0.433453   0.286407   0.903194                  \n",
      "    2      0.40298    0.286503   0.893214                  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.2865]), 0.8932135730922341]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, 3, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.save('resnet50_299_ll') # training and validation sets\n",
    "learn.save('resnet101_299_ll_all_data') # training and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.load('resnet50_299_ll') # training and validation sets\n",
    "learn.load('resnet101_299_ll_all_data') # training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Swapped datasize back to 224 to continue training, not sure why really....just trial and error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.set_data(data = get_data(224, bs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()\n",
    "learn.bn_freeze(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dd8638e4a6c400ea8498383dc33a755",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/63 [00:00<?, ?it/s]\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.cu:58",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-32-4726d92214e5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlrf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlr_find\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/github/fastai/courses/dl1/fastai/learner.py\u001b[0m in \u001b[0;36mlr_find\u001b[0;34m(self, start_lr, end_lr, wds, linear, **kwargs)\u001b[0m\n\u001b[1;32m    328\u001b[0m         \u001b[0mlayer_opt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_layer_opt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_lr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    329\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLR_Finder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlayer_opt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrn_dl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_lr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlinear\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 330\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_gen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlayer_opt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'tmp'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    332\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/github/fastai/courses/dl1/fastai/learner.py\u001b[0m in \u001b[0;36mfit_gen\u001b[0;34m(self, model, data, layer_opt, n_cycle, cycle_len, cycle_mult, cycle_save_name, best_save_name, use_clr, use_clr_beta, metrics, callbacks, use_wd_sched, norm_wds, wds_sched_mult, use_swa, swa_start, swa_eval_freq, **kwargs)\u001b[0m\n\u001b[1;32m    232\u001b[0m             \u001b[0mmetrics\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreg_fn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreg_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclip\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfp16\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp16\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    233\u001b[0m             \u001b[0mswa_model\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mswa_model\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0muse_swa\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mswa_start\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mswa_start\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 234\u001b[0;31m             swa_eval_freq=swa_eval_freq, **kwargs)\n\u001b[0m\u001b[1;32m    235\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    236\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_layer_groups\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_layer_groups\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/github/fastai/courses/dl1/fastai/model.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(model, data, n_epochs, opt, crit, metrics, callbacks, stepper, swa_model, swa_start, swa_eval_freq, **kwargs)\u001b[0m\n\u001b[1;32m    126\u001b[0m             \u001b[0mbatch_num\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    127\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mcb\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mcb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 128\u001b[0;31m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_stepper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mV\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mV\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    129\u001b[0m             \u001b[0mavg_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavg_loss\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mavg_mom\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mavg_mom\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    130\u001b[0m             \u001b[0mdebias_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavg_loss\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mavg_mom\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mbatch_num\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/github/fastai/courses/dl1/fastai/model.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, xs, y, epoch)\u001b[0m\n\u001b[1;32m     46\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     47\u001b[0m         \u001b[0mxtra\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 48\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     49\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mxtra\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp16\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    356\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 357\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    358\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     65\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    356\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 357\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    358\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     65\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    356\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 357\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    358\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torchvision/models/resnet.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     79\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     80\u001b[0m         \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconv2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 81\u001b[0;31m         \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbn2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     82\u001b[0m         \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    356\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 357\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    358\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     35\u001b[0m         return F.batch_norm(\n\u001b[1;32m     36\u001b[0m             \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunning_mean\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunning_var\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 37\u001b[0;31m             self.training, self.momentum, self.eps)\n\u001b[0m\u001b[1;32m     38\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__repr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mbatch_norm\u001b[0;34m(input, running_mean, running_var, weight, bias, training, momentum, eps)\u001b[0m\n\u001b[1;32m   1011\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Expected more than 1 value per channel when training, got input size {}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1012\u001b[0m     \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_functions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mBatchNorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrunning_mean\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrunning_var\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmomentum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackends\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcudnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menabled\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1013\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1014\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1015\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.cu:58"
     ]
    }
   ],
   "source": [
    "lrf = learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEOCAYAAABiodtuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8lfXd//HXJ5sMAiQhzLBkiAyFIKAoUK274saFC6tUq71r96/t3WF7a9V6uwcqRa1bHIhVnIjsPUX2SlgJIYEkJCHJ9/fHOba5SYAk5JzrnJz38/HII2dc55w3X2Peua7vNcw5h4iISE1RXgcQEZHQo3IQEZFaVA4iIlKLykFERGpROYiISC0qBxERqUXlICIitagcRESkFpWDiIjUonIQEZFaYrwO0FDp6emua9euXscQEQkrixcvznfOZdR3+bArh65du7Jo0SKvY4iIhBUz29qQ5bVZSUREalE5iIhILSoHERGpReUgIiK1qBxERKQWlYOIiNSichARCQMfr9pJfnF50D5P5SAiEuJ2FZVx12tLefzz9UH7TJWDiEiImzR7M1XVjvEjugftM1UOIiIhrOjgIV6dv42LBnQgKy0xaJ+rchARCWH/nLeV4vJKbh8ZvLUGUDmIiISsskNV/GP2Zs7slcFJHVKD+tkqBxGREPX24hzyiyv40cgeQf9slYOISAiqrKpm4sxNDOzcimHd2wT981UOIiIh6KNVu9hWUMqPRnbHzIL++SoHEZEQ45zjma820j0jiXP6tvMkg8pBRCTEzNqQz+od+7n9zO5ERQV/rQFUDiIiIeeZrzaS2TKeS07p6FkGlYOISAhZkVPI7A17GT+iG/Ex0Z7lUDmIiISQZ77aSEpCDNecmuVpDpWDiEiI2JxfwkerdjFuWBdSEmI9zaJyEBEJERNnbiI2OoqbT+/mdRSVg4hIKNizv4wpi3O4cnAnMlLivY6jchARCQXTVuykoqqaW88I7gn2jkTlICISAlbkFNKuZQLd0pO8jgKoHEREQsKKnCIGdArumVePRuUgIuKxooOH2JRfEhnlYGaTzGyPma06wvOpZvaBmS03s9VmdnOgsoiIhLLVuUUADOjUyuMk/xHINYfJwHlHef5O4Bvn3EBgFPB3M4sLYB4RkZC0POe7coiANQfn3Eyg4GiLACnmOxdtsn/ZykDlEREJVStyCslqk0irxND5+zjGw89+ApgK7ABSgLHOuWoP84iIeGJFThGnZIXOJiXwdkL6XGAZ0AE4GXjCzFrWtaCZ3WZmi8xsUV5eXjAziogE1N7icnILDzIwhOYbwNtyuBl4x/lsADYDfepa0Dk30TmX7ZzLzsjICGpIEZFAWuGfb+gfQvMN4G05bAPOAjCzTKA3sMnDPCIiQbcipwgz6NcxtMohYHMOZvYavr2Q0s0sB/gDEAvgnHsGuBeYbGYrAQN+5ZzLD1QeEZFQtCKnkBMykkmO93IKuLaApXHOXXOM53cA5wTq80VEQp1zjuU5RZzZK93rKLXoCGkREY/s2l9GfnF5yE1Gg8pBRMQzy7eH3sFv31E5iIh4ZEVOITFRxont69yL31MqBxERj6zMLaJ3uxQSYqO9jlKLykFExAPOOf9pukNvvgFUDiIinti6t5Sig4dCcr4BVA4iIp5YkRu6k9GgchAR8cSK7YXEx0TRKzPF6yh1UjmIiHhgRU4RfTu0JDY6NH8Nh2YqEZFmrKrasWpHUUge/PYdlYOISJBtzCumtKIqZOcbQOUgIhJ0K0LwsqCHUzmIiATZipxCkuNj6J6e7HWUI1I5iIgE2fKcIvp1bElUlHkd5YhUDiIiQVRRWc2anftD9sjo76gcRESCaN3uA1RUVof0fAOoHEREgmp5TiFASO/GCioHEZGgWplTROvEWDq1buF1lKNSOYiIBNHynCL6d2qFWehORoPKQUQkaA5WVLFu9wEGhvh8A6gcRESC5pud+6mqdvTvqHIQERG/Fd9NRncO7cloUDmIiATNipwiMlvGk9kywesox6RyEBEJgsqqauZu3Bvyu7B+R+UgIhIEn63Zw679ZVw+uJPXUepF5SAiEgQvz9tCh9QEzurT1uso9aJyEBEJsA17ipm9YS/XDetCTIhe+e1w4ZFSRCSM/XPeVuKioxg7pLPXUepN5SAiEkAl5ZVMWZzDBf3bkZ4c73WcelM5iIgE0LtLczlQXsm44V29jtIgKgcRkQBxzvHy3K2c1KElg7LCYxfW76gcREQCZMHmAtbuPsANw7uE/In2DqdyEBEJkJfmbiW1RSwXD+zodZQGC1g5mNkkM9tjZquOsswoM1tmZqvN7KtAZRERCbbd+8uYvnoXV2V3okVctNdxGiyQaw6TgfOO9KSZtQKeAi52zp0EXBnALCIiQfXq/G1UOcf1w7p4HaVRAlYOzrmZQMFRFrkWeMc5t82//J5AZRERCaZDVdW8tmAbI3tl0CUtyes4jeLlnEMvoLWZzTCzxWZ2g4dZRESazPTVu9hzoJwbhofnWgNAjMefPRg4C2gBzDWzec65dYcvaGa3AbcBZGVlBTWkiEhDvTR3K53btGBkr/A4j1JdvFxzyAE+ds6VOOfygZnAwLoWdM5NdM5lO+eyMzIyghpSRKQhvt21nwWbC7h+aBeio8Jr99WavCyH94EzzCzGzBKBocAaD/OIiBy3l+duJT4miquyw+c8SnUJ2GYlM3sNGAWkm1kO8AcgFsA594xzbo2ZfQysAKqB551zR9ztVUQk1O0vO8S7S3P5wcAOtE6K8zrOcQlYOTjnrqnHMg8CDwYqg4hIML2/bAelFVWMC9PdV2vSEdIiIk3kjYXb6Nu+JQM6pXod5bipHEREmsCq3CJW5e5n7JDOYXcepbqoHEREmsCbi7YTFxPFJSeH33mU6qJyEBE5TmWHqnh3aS4X9GtHamKs13GahMpBROQ4fbRqJwfKKhk7pPkcpKtyEBE5Tq8v2E7XtESGdW/jdZQmo3IQETkOm/KKmb+5gKuayUT0d1QOIiLH4c1FOURHGVcM6uR1lCalchARaaRDVdW8vTiH7/VpS9uWCV7HaVIqBxGRRvri2z3kF5dz9ZDwPo9SXVQOIiKN9ObC7bRNiWdkr+Z3tmiVg4hII+wqKuPLtXu4MrsTMdHN71dp8/sXiYgEwduLt1PtCPtTcx+JykFEpIGqqx1vLNrOaT3SwvYa0ccSMeXgnGP97gNexxCRZmDupr1sLzjI2GY4Ef2diCmHKUtyOe/Rr3lj4Tavo4hImHt94XZSW8Ry7kntvI4SMBFTDuf3a8eIE9L51ZSV/O+n63DOeR1JRMLQvpIKpq/axaWndCQhNtrrOAETMeWQFB/D8zdmc8XgTjz6+Xp+PWUllVXVXscSkTDz7tJcKqqqm/UmJQjgZUJDUWx0FA9eMYAOqQk89sUG9hwo48nrBpEYF1HDICKNdKDsEM98tZHBXVpzYvuWXscJqIhZc/iOmXHPOb3566X9+GpdHtdMnEd+cbnXsUQkDDzy2Xryisv574v6eh0l4CKuHL5z3dAuPDsum7W7D3D503PYkl/idSQRCWHf7trP5DlbuObULAZ2buV1nICz+kzMmtlPgH8AB4DngVOAXzvnPglsvNqys7PdokWLmuz9lmzbx/jJC4ky48bTuhIfE0Xcd1/Rvu/xMVF0TU+iT7vmvRopInVzzjH22Xms33OAL342itZJcV5HajAzW+ycy67v8vXd2H6Lc+5RMzsXyABuxlcWQS+HpjYoqzVTfnQaP3xpEQ9/uu6Iy0UZvHDjEEb3aRvEdCISCt5ZksuCLQX87fL+YVkMjVHfcvjuChYXAP9wzi23ZnRVi+4ZyXx2z0gqqx0VldW+r6r/fC87VMUv317B3a8t5Z07TqNnZorXkUUkSIoOHuK+j9ZwSlYrrhzcvPdQqqm+cw6LzewTfOUw3cxSgGa1H6iZERsdRVJ8DK2T4shsmUDnNon0yEjmpA6pPHdDNvGx0dz60iL2lVR4HVdEguThT9ZSUFLBvWP6ERXVbP4mPqb6lsN44NfAEOdcKRCLb9NSxOjQqgXPjhvMzsIy7nx1CYd0jIRIs7cqt4iX521l3LAu9OuY6nWcoKpvOQwH1jrnCs3seuB3QFHgYoWmwV1ac99l/ZmzcS9//uAbr+OISABVVzt+//4q2iTFcc85vb2OE3T1LYengVIzGwj8EtgKvBSwVCHs8sGduP3M7rw8bysvz9vqdRwRCZC3Fm9n6bZCfnP+iaS2iPU6TtDVtxwqnW+f1zHAo865R4GInZX95Xl9+F6ftvxx6mrmbMj3Oo6INLF9JRXc/9G3DOnamssGdfQ6jifqWw4HzOw3wDjgQzOLxjfvEJGio4xHrz6Z7ulJ3PHqErbu1QF0Is3Jg5+sZX9ZJfde0o9mtGNmg9S3HMYC5fiOd9gFdAQeDFiqMJCSEMvzN/qOJxn/4iL2lx3yOJGINIUVOYW8tmAbNw7vGtEHvtarHPyF8AqQamYXAWXOuYicc6ipS1oST103iC35JVz73Dz2HCjzOpKIHIfqascfpq4mLSmen36/p9dxPFWvcjCzq4AFwJXAVcB8M7sikMHCxWk90nnuhmw27inh0ifnsGGPrjYnEq7eWZrL0m2F/Pr8PqQkROyWc6D+m5V+i+8YhxudczcApwK/P9oLzGySme0xs1XHWG6ImVWFc9mM7tOWN24fRnllFZc/PZcFmwu8jiQiDbS/7BD3f/Qtp2S14rJTInMSuqb6lkOUc25Pjft76/HaycB5R1vAP7H9N2B6PXOErAGdWvHuHaeTlhzH9S/M58MVO72OJCIN8Nhn69lbUs6fLj4poo6EPpL6lsPHZjbdzG4ys5uAD4F/He0FzrmZwLH+hL4LmALsOcZyYaFzm0SmTDiNAR1TufPVJTz/9SavI4lIPWzYc4DJc7YwNrszAzo1/9Nx10d9J6R/AUwEBgADgYnOuV8dzwebWUfgUuCZ43mfUNM6KY5/3jqU8/u14y8fruFPH6ymqlrXqxYJVc45/jj1GxLjovnFuZF3JPSR1Pv6mM65Kfj+ym8qjwC/cs5VHWs/YjO7DbgNICsrqwkjBEZCbDRPXjuIv3y4hkmzN5O77yB/v2pgxE9wiYSi6at3MWtDPn/8QV/SkuO9jhMyjnqxHzM7ANS1gAHOOXfUnYDNrCswzTnXr47nNvOfU4GnA6XAbc659472nk19sZ9AmzRrM3/91xq6tEnk6esH07tdxB5YLhJyyg5VcdbfvyIlIYZpd40gJrr5XhyzoRf7OepIOOdSnHMt6/hKOVYxHItzrptzrqtzrivwNnDHsYohHN0yohuv3jqUA+WVXPLkbN5dmuN1JBHxe+arjeQWHuQPPzipWRdDYwRsNMzsNWAu0NvMcsxsvJlNMLMJgfrMUDW0exof3jWC/p1S+ekby/ndeyspr6zyOpZIRNteUMrTMzZy4YD2DO+R5nWckFPvOYeGcs5d04BlbwpUjlDRtmUCr946lAenr+XZmZtYmVPEk9cNolPrRK+jiUSkv364higzfnvBiV5HCUlajwqimOgofnPBiTxz/WA25ZVw0eOz+HJts9iLVySszFyXx8erd3Hn6B50aNXC6zghSeXggfP6tWPqXSNo1zKBm/+xkPGTFzJ7Qz5H2zlARJrGjsKD/PSNZXTPSOLWM7p7HSdkqRw80i09iXfvOJ27z+rJsu2FXPf8fM575GveWLiNskOajxAJhLJDVdz+8mLKK6uZOC6bhNhoryOFrKPuyhqKwm1X1vooO1TF1OU7mDRrM9/uOkCbpDiuG5rFuGFdaNsywet4Is2Cc4573lzOu0tzef6GbM7um+l1pKBq6K6sKocQ4pxj3qYCJs3ezGdrdhMTZVxyckfuPqsnndto4lrkeLwwazP3TvuGe77fi7vPirzTcTe0HAK2t5I0nJkxvEcaw3uksXVvCf+YvYVXF2zjvWW5jB3Smbu+15NMrUmINNicDfn8z7/WcE7fTH48+gSv44QFrTmEuF1FZTzx5XpeX7Cd6Chj3LAu/GhUDx3mL1JP2wtKufiJWaQnx/PunaeTHB+ZfxNrs1Iztb2glEc/X887S3JIiI3mltO78cMzupOaqPM1iRzJwYoqLnt6Djn7Spn64xF0S0/yOpJnVA7N3IY9xTzy2TqmrdhJXHQUGSnxpCXHkZYUR1pyvP97HGlJ8Yzu05Y2SXFeRxbxhHOOu19fxrQVO5h00xBG927rdSRPac6hmTuhbTJPXDuIO0bt5/1lueQVl7O3uIK84nLW7jpAfkkFFZXVAHRNS+S124bRPlUH+Ujkee7rTXywfAe/OLd3xBdDY6gcwlTfDi3p26H2uQ+dcxSXV7J8exET/rmYqyfO47UfDtNRoBJRlm0v5G8fr+WC/u24Y1QPr+OEJR0E18yYGSkJsYzomc5L40+loLiCqyfOI7fwoNfRRILiYEUV97yxjMyUeO6/fADHul6M1E3l0IwNymrNy7cOZV9pBVdPnEvOvlKvI4kE3H0frWFTfgkPXTmQlrrAVqOpHJq5kzu34p/jh1JYeoirJ85je4EKQpqvr9bl8dLcrYwf0Y3TTkj3Ok5YUzlEgIGdW/HKrUPZf1AFIc1XYWkFv3hrOT3bJuta0E1A5RAhBnRqxas/HEZxeaUKQpod5xy/fW8VBSUV/O/Yk3VCvSagcogg/Tqm8sqtQympqOTSp+bw2oJtVFZVex1L5LhNXb6DD1fs5Kff70W/jqlex2kWVA4Rpl/HVF6/bRhZbVrwm3dWcs4jM/nXyp26loSErZ1FB/n9e6sYlNWK28/U9RmaisohAvVp15IpPzqN527IJibKuOOVJYx5cjazN+R7HU2kQaqrHT9/azmV1Y6HrzqZmGj9SmsqGskIZWZ8v28mH/3kTB66ciB7iyu47vn5jHthPitziryOJ1IvL87dwuwNe/ndhX3pGsHnTQoElUOEi44yrhjcic9/NpLfX9SX1Tv284MnZvH/3l1JSXml1/FEjmhTXjH3f/Qt3+vTlmtO7ex1nGZH5SAAJMRGM35EN776xSh+eEY3XluwjfMencmCzQVeRxOp04PT1xIbHcX9l/XXUdABoHKQ/yMlIZbfXtiXN28fjmGMnTiXv374ja5rLSFlVW4RH63axS0juulSugGicpA6Denaho9+cgbXDc3iua83c9Hjs1iRU+h1LBEA/v7JWlJbxHLrGd28jtJsqRzkiJLiY/jLJf156ZZTKS7zHRvxv5+u45COjRAPLd5awJdr85gwsofOnRRAKgc5pjN7ZTD9v85kzMAOPPr5ei5/eg6b8oq9jiURyDnHg9PXkp4cz42ndfE6TrOmcpB6SU2M5eGxJ/PM9YPYVlDKhY/N4o2F23TwnATVnI17mbepgDtH9yAxTpejCSSVgzTIef3a8/FPzuSUrFb8aspK7nhlCYWlFV7Hkgjw3VpDh9QErh2a5XWcZk/lIA3WLjWBf44fyq/P78On3+zm/Ee/Zu7GvV7Hkmbui2/3sGx7IXed1ZP4GJ1YL9BUDtIoUVHGhJE9ePeO00mIjeba5+fxwMffarJaAqK62vHQJ+vokpbIFYM7eR0nIqgc5Lj075TKtLtGcNXgzjw1YyOXPz2HtxfnsLe43Oto0ox8tGoXa3bu57/O7kmszp8UFJrRkeOWFB/D364YwKjeGfzpg2/4+VvLMfNdpvSsE9ty9omZ9GybrKNYpVGqqh0Pf7qWnm2TuXhgR6/jRAyVgzSZ8/u359yT2rFqRxGfrdnDF9/u5oGP1/LAx2vp3KYFZ/XJ5LqhWfTMTPE6qoSRd5fmsjGvhGeuH0R0lP7ACBYLt10Rs7Oz3aJFi7yOIfW0q6iMz7/dzedr9jB7Qz7RUcaT1w1idO+2XkeTMFBRWc1ZD88gtUUsH/x4hNY+j4OZLXbOZdd3+YBtvDOzSWa2x8xWHeH568xshf9rjpkNDFQW8U671ASuG9qFSTcNYeYvR9M1LYlbX1zE6wu2eR1NwsCbi7azveAgPzunt4ohyAI5szMZOO8oz28GRjrnBgD3AhMDmEVCQGbLBN6cMJzTT0jn1++s5OFP1uogOjmigxVVPP7FegZ3ac2oXhlex4k4ASsH59xM4Ijne3bOzXHO7fPfnQdo/7QIkBwfwws3ZjM2uzOPfbGBn725nIpK7f4qtb04dwu795fzq/P6aK3BA6EyIT0e+MjrEBIcsdFR3H95fzq2bsHDn65j94Eynr5+sE6iJv9WVHqIp77cwOjeGZzarY3XcSKS5zsMm9lofOXwq6Msc5uZLTKzRXl5ecELJwFjZtx9Vk8eunIg8zcVcNUzc9lZdNDrWBIinp25kf1llfzi3D5eR4lYnpaDmQ0AngfGOOeOeP4F59xE51y2cy47I0PbHpuTKwZ34h83DyFn30EufmI2r8zfqs1MEW73/jImzd7MmJM70LdDS6/jRCzPysHMsoB3gHHOuXVe5RDvndEzg7cmDKdjqxb89t1VjHrwS16eu4XySl19LhI99vl6KqscP/t+b6+jRLRA7sr6GjAX6G1mOWY23swmmNkE/yL/DaQBT5nZMjPTwQsR7MT2LXn3jtN46ZZTad+qBb9/fzUjH5jBi3O26BKlEWRzfgmvL9zOtUOzyEpL9DpORNNBcBJynHPM3rCXRz9fx8It+2ibEs+EkT24dmgWCbE6G2dzdtdrS/nsm9189ctRtE3RtaGbUsgcBCfSWGbGiJ7pvHn7cF794VC6pSfx52nfcNlTc9i2t9TreBIgq3KL+GD5DsaP6KZiCAEqBwlZZsZpPdJ54/bhPH9DNjn7Srno8a/58ts9XkeTAHhg+lpaJcZy28juXkcRVA4SJs7um8m0u86gY+tEbnlxIf/76Tqqq8Nrk6gc2ZyN+cxcl8cdo3roeJcQoXKQsJGVlsg7PzqNS0/pyKOfr2f8iwt1idJmwDnHAx+vpX1qAjcM7+p1HPFTOUhYaREXzd+vHMhfLunHrA35/OCJWazKLfI6lhyH6at3s2x7If91dk/tcBBCVA4SdsyM64d14c3bh3Oo0v376nMSfiqrqnnok7V0z0ji8kE6vVooUTlI2DolqzXT7h7BoKzW/Pyt5dw77RsqdQ3rsPLl2jw27Cnmnu/3IkaX/wwp+q8hYS09OZ6Xx5/KTad15YVZm7nlxUUUHTzkdSypp3eW5JCWFMe5J7XzOoocRuUgYS8mOoo/XnwS913Wnzkb8rn0qdlszi/xOpYcQ2FpBZ+v2cOYkzsSq7WGkKP/ItJsXHNqFq/cOpTC0kOMeWIWX6/XGXxD2QcrdlJRVc1lgzp6HUXqoHKQZmVo9zTev/N02qe24KZ/LOQfszfranMh6p0lOfTOTOEknXk1JKkcpNnp3CaRKXecxujebfnTB9/wm3dW6jTgIWZTXjFLtxVy+eCOuspbiFI5SLOUHB/DxHGDuXN0D15fuJ2xE3UxoVDyzpJcogzGnKxNSqFK5SDNVlSU8Ytz+/DUdYNYt+sAFz42i1nr872OFfGqqx3vLs1lRM8MMlvqBHuhSuUgzd4F/dsz9a4RpCXFMW7SfJ74Yr3Oy+Sh+ZsLyC08yOWaiA5pKgeJCD0yknnvztO5eGAHHvpkHbe+tIiiUh0P4YUpS3JIjo/hnL46tiGUqRwkYiTFx/DI2JO5d8xJfL0+jwsf/1rnZQqy0opKPlq5kwv6t6NFnM6jFMpUDhJRzIxxw7vy5u3Dqap2XPb0HJ7/epMuRRokn6zeTUlFlc6jFAZUDhKRTslqzbS7RjC8exp/+XANI/72JU/N2MD+Mm1qCqQpS3Lo1LoFQ7q28TqKHIPKQSJWWnI8k28ewuu3DaNvh5Y88PFaTr/vCx74+Fvyi8u9jtfs7CoqY9aGfC47pSNRUTq2IdTFeB1AxEtmxrDuaQzrnsaq3CKenrGRp7/ayAuzNjN2SGd+eEZ3OrdJ9Dpms/Deslycg8u0SSksqBxE/Pp1TOXJ6waxKa+YZ7/axGsLtvHq/G2MHdKZn5zVk7baJ7/RnHNMWZzD4C6t6Zqe5HUcqQdtVhI5TPeMZP52xQBm/nI01w7N4o2F2xn54Awemr5WcxKNtCp3P+v3FOske2FE5SByBO1TW/DnMf347J6RnN03kye+3MDIB77U3k2NMGVJDnExUVzUv4PXUaSeVA4ix9A1PYnHrzmFaXeNoF/HVP7y4RrO+vtXvL04hyodaX1Mh6qqmbp8B98/MZPUxFiv40g9qRxE6qlfx1ReHj+UV24dSlpyHD9/azmXPDlbB9Idw4y1eRSUVGiTUphROYg00OknpPP+nafz2DWnsLOojDFPzua+f63hYIU2NR3uUFU1T3y5gfTkOM7sleF1HGkAlYNII5gZFw/swOf3jOTKwZ14duYmzn1kpq4+d5iHP13H8u2F/OnifroUaJjRfy2R45CaGMv9lw/g9duGERNljHthAfe8uYyCkgqvo3lu1vp8nvlqI1cP6cyFA9p7HUcayMLtEorZ2dlu0aJFXscQqaXsUBVPfrmBp2dspGWLWH56dk9OaJtC66RY2iTG0SoxjriYyPh7LL+4nPMf/ZrUFrF88OMROsleCDCzxc657Pour4PgRJpIQmw0PzunNxcN6MCv31nB799fXWuZ5PgYWiXGkpYUx+g+bRk3rAtpyfEepA2c6mrHz99aTtHBQ7w8/lQVQ5hSOYg0sd7tUpgy4TTW7ylmb0k5+0oOsa+0gn0lFRSUVlBYeojcwoM88tl6np6xkSsGd2L8iG50z0j2OnqTmDR7MzPW5nHvmJPo066l13GkkVQOIgEQFWX0bpcCpBxxmQ17inlh1ibeWpzDqwu2cfaJmdx2Zneyu7TGLDxPTLcyp4i/ffwt5/TN5PphXbyOI8chYHMOZjYJuAjY45zrV8fzBjwKXACUAjc555Yc63015yDNTd6Bcl6eu4WX5m2lsPQQJ3duxYSR3Tn3pHZhVRLF5ZVc9NjXlFdW89FPzqBVYpzXkaSGhs45BHJ2bDJw3lGePx/o6f+6DXg6gFlEQlZGSjz3nNObOb/+HveOOYl9pRVM+OcSxjw5m6/X5xEuO4389/ur2FZQyiNjT1YxNAMBKwfn3Eyg4CiLjAFecj7zgFZmpv1IKPZPAAALaUlEQVTdJGIlxsUwbnhXvvjZKB66ciB7iysY98ICrn1uPku37fM63lG9uzSHd5bkctf3ejK0e5rXcaQJeDnn0BHYXuN+jv+xnd7EEQkN0VHGFYM78YOB7Xl1/jae+GIDlz41h3P6ZvLzc3vTK/PI8xiB4pwjt/Ag2wpKySnwfd9WUMr2faVsLyglv7iCU7u24a7vnRD0bBIYXpZDXRtT61x/NrPb8G16IisrK5CZREJGfEw0N5/ejSuzOzNp1mYmztzEeY/M5JJTOvK9Pm3pnZlC1/SkgB55XFFZzQfLd/D8rM2s2bn/349HRxntUxPIapPIWX0yyUpL5JpTs4jRUdDNRkAPgjOzrsC0I0xIPwvMcM695r+/FhjlnDvqmoMmpCVSFZRU8PSMDbw0dyvlldUAxEVH0T0jid7tUuiVmULvzBROzmpF+nEeO1FYWsEr87fx4pwt7DlQTq/MZK45NYtemSlktUmkXWqCTocRZsLpILipwI/N7HVgKFB0rGIQiWRtkuL47YV9+dk5vdmYV8y63QdYu8v3fdGWfby/bAcAUQan9Ujn4oEdOPekdg06TfaW/BImzd7MW4tyOHioijN6pvPglQM5s2d6WO05JccvkLuyvgaMAtKB3cAfgFgA59wz/l1Zn8C3R1MpcLNz7pirBFpzEKnbgbJDrNt9gBlr85i6fAdb95YSG22M7NWWHwxsz/f7ZpIY5/t7sLrasWt/GZvyStiUX8ymvBLW7jrAvM17iYkyxpzckVvP6KaD2JqRhq456NxKIs2Qc46VuUVMXbaDaSt2smt/GS1ioxnSrQ35B8rZnF/CwRpXs0uOj6F7RhJn9szghuFddL3sZkjlICL/R3W1Y+GWAj5YsYMFmwvo0KoF3dOT6Z6RRPeMJE7ISCYjJV6bjZq5cJpzEJEgiIoyhnZP0/EH0iDa3UBERGpROYiISC0qBxERqUXlICIitagcRESkFpWDiIjUonIQEZFaVA4iIlJL2B0hbWZ5wFYgFSg67On6PFbzfjqQH4CYR8vTVK851nJHel7jFrhxO9b9QI5bY8asIa/TuDXudUdbrqHPHe+4dXHOZRwr8L8558LyC5jYmMdq3gcWBTtjU73mWMsd6XmNW+DGrR73AzZujRkzjZu349bQ54I9buG8WemDRj5W1zKB0pjPqu9rjrXckZ7XuDXu+caMUaiPWUNep3Fr3OuOtlxDnwvquIXdZqWmZGaLXANORCU+GrfG0bg1jsatcY533MJ5zaEpTPQ6QJjSuDWOxq1xNG6Nc1zjFtFrDiIiUrdIX3MQEZE6qBxERKQWlYOIiNSicjgKM0sys8VmdpHXWcKFmZ1oZs+Y2dtm9iOv84QLM7vEzJ4zs/fN7Byv84QLM+tuZi+Y2dteZwl1/t9nL/p/zq471vLNshzMbJKZ7TGzVYc9fp6ZrTWzDWb263q81a+ANwOTMvQ0xbg559Y45yYAVwERsfthE43be865HwI3AWMDGDdkNNG4bXLOjQ9s0tDVwDG8DHjb/3N28bHeu1mWAzAZOK/mA2YWDTwJnA/0Ba4xs75m1t/Mph321dbMzga+AXYHO7yHJnOc4+Z/zcXALODz4Mb3zGSaYNz8fud/XSSYTNONW6SaTD3HEOgEbPcvVnWsN45p0pghwjk308y6HvbwqcAG59wmADN7HRjjnLsPqLXZyMxGA0n4Bvegmf3LOVcd0OAea4px87/PVGCqmX0IvBq4xKGhiX7eDLgf+Mg5tySwiUNDU/28RbKGjCGQg68gllGPFYNmWQ5H0JH/tCb4BmrokRZ2zv0WwMxuAvKbezEcRYPGzcxG4Vt9jQf+FdBkoa1B4wbcBZwNpJrZCc65ZwIZLoQ19OctDfgrcIqZ/cZfIpHuSGP4GPCEmV1IPU61EUnlYHU8dswjAJ1zk5s+Slhp0Lg552YAMwIVJow0dNwew/c/b6Rr6LjtBSYELk5YqnMMnXMlwM31fZPmOudQlxygc437nYAdHmUJJxq3xtG4NY7G7fg1yRhGUjksBHqaWTcziwOuBqZ6nCkcaNwaR+PWOBq349ckY9gsy8HMXgPmAr3NLMfMxjvnKoEfA9OBNcCbzrnVXuYMNRq3xtG4NY7G7fgFcgx14j0REamlWa45iIjI8VE5iIhILSoHERGpReUgIiK1qBxERKQWlYOIiNSicpCAM7PiIHzGxfU8DXtTfuYoMzutEa87xcye99++ycyeaPp0DWdmXQ8/9XMdy2SY2cfByiTeUTlI2PCfirhOzrmpzrn7A/CZRzv/2CigweUA/D/g8UYF8phzLg/YaWane51FAkvlIEFlZr8ws4VmtsLM/lTj8ffMd9W91WZ2W43Hi83sz2Y2HxhuZlvM7E9mtsTMVppZH/9y//4L3Mwmm9ljZjbHzDaZ2RX+x6PM7Cn/Z0wzs39999xhGWeY2f+Y2VfAT8zsB2Y238yWmtlnZpbpP03yBOCnZrbMzM7w/1U9xf/vW1jXL1AzSwEGOOeW1/FcFzP73D82n5tZlv/xHmY2z/+ef65rTcx8V/n60MyWm9kqMxvrf3yIfxyWm9kCM0vxryF87R/DJXWt/ZhZtJk9WOO/1e01nn4POOaVxCTMOef0pa+AfgHF/u/nABPxnTUyCpgGnOl/ro3/ewtgFZDmv++Aq2q81xbgLv/tO4Dn/bdvAp7w354MvOX/jL74zm0PcAW+04hHAe2AfcAVdeSdATxV435r/nM2gVuBv/tv/xH4eY3lXgVG+G9nAWvqeO/RwJQa92vm/gC40X/7FuA9/+1pwDX+2xO+G8/D3vdy4Lka91OBOGATMMT/WEt8Z2JOBBL8j/UEFvlvdwVW+W/fBvzOfzseWAR089/vCKz0+udKX4H9iqRTdov3zvF/LfXfT8b3y2kmcLeZXep/vLP/8b34rlg15bD3ecf/fTG+a0fU5T3nuwbHN2aW6X9sBPCW//FdZvblUbK+UeN2J+ANM2uP7xfu5iO85mygr9m/z5jc0sxSnHMHaizTHsg7wuuH1/j3vAw8UOPxS/y3XwUequO1K4GHzOxvwDTn3Ndm1h/Y6ZxbCOCc2w++tQx85/U/Gd/49qrj/c4BBtRYs0rF999kM7AH6HCEf4M0EyoHCSYD7nPOPft/HvRdIOhsYLhzrtTMZgAJ/qfLnHOHX9Kw3P+9iiP/DJfXuG2Hfa+Pkhq3Hwceds5N9Wf94xFeE4Xv33DwKO97kP/8246l3ic+c86tM7PBwAXAfWb2Cb7NP3W9x0/xXf52oD9zWR3LGL41tOl1PJeA798hzZjmHCSYpgO3mFkygJl1NN91gFOBff5i6AMMC9DnzwIu9889ZOKbUK6PVCDXf/vGGo8fAFJq3P8E39kwAfD/ZX64NcAJR/icOfhOrwy+bfqz/Lfn4dtsRI3n/w8z6wCUOuf+iW/NYhDwLdDBzIb4l0nxT7Cn4lujqAbGAXVN9E8HfmRmsf7X9vKvcYBvTeOoezVJ+FM5SNA45z7Bt1lkrpmtBN7G98v1YyDGzFYA9+L7ZRgIU/BdCGUV8CwwHyiqx+v+CLxlZl8D+TUe/wC49LsJaeBuINs/gfsNdVyhzDn3Lb5LgaYc/pz/9Tf7x2Ec8BP/4/8F3GNmC/Btlqorc39ggZktA34L/MU5VwGMBR43s+XAp/j+6n8KuNHM5uH7RV9Sx/s9D3wDLPHv3vos/1lLGw18WMdrpBnRKbslophZsnOu2HzXHl4AnO6c2xXkDD8FDjjnnq/n8onAQeecM7Or8U1OjwloyKPnmQmMcc7t8yqDBJ7mHCTSTDOzVvgmlu8NdjH4PQ1c2YDlB+ObQDagEN+eTJ4wswx88y8qhmZOaw4iIlKL5hxERKQWlYOIiNSichARkVpUDiIiUovKQUREalE5iIhILf8fGwBU35cm4c8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot learing rate\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above learning rate looks good, i remember Jeremy mentioning that you should set the other learning rates between 3 and 10 times smaller each time. So i picked the below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "lrs=np.array([lr/9,lr/3,lr])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bae3d949c6c946cb927d494be293c651",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=26), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                  \n",
      "    0      0.470533   0.013419   1.0       \n",
      "    1      0.28691    0.015132   1.0                       \n",
      "    2      0.235851   0.013688   1.0                       \n",
      "    3      0.198013   0.011895   1.0                       \n",
      "    4      0.163334   0.010592   1.0                       \n",
      "    5      0.138481   0.01724    1.0                       \n",
      "    6      0.134769   0.027559   1.0                       \n",
      "    7      0.130257   0.031887   1.0                       \n",
      "    8      0.125633   0.026149   1.0                       \n",
      "    9      0.115097   0.021358   1.0                       \n",
      "    10     0.103829   0.018229   1.0                       \n",
      "    11     0.091046   0.013599   1.0                        \n",
      "    12     0.077217   0.00525    1.0                        \n",
      "    13     0.075805   0.057111   1.0                        \n",
      "    14     0.065214   0.012825   1.0                        \n",
      "    15     0.056631   0.009379   1.0                        \n",
      "    16     0.05504    0.029291   1.0                        \n",
      "    17     0.051926   0.040629   1.0                        \n",
      "    18     0.045275   0.009856   1.0                        \n",
      "    19     0.041415   0.006447   1.0                        \n",
      "    20     0.040161   0.012116   1.0                        \n",
      "    21     0.040294   0.018272   1.0                        \n",
      "    22     0.035134   0.010635   1.0                        \n",
      "    23     0.038321   0.010204   1.0                        \n",
      "    24     0.040475   0.018785   1.0                        \n",
      "    25     0.038852   0.008794   1.0                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.00879]), 1.0]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, 3, cycle_len=2, cycle_mult=3) #, cycle_save_name=\"resnet50_299_all_data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.save('resnet50_224_final')\n",
    "learn.save('resnet50_224_final_all_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn.load('resnet50_224_final')\n",
    "learn.load('resnet50_224_final_all_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Time Augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             \r"
     ]
    }
   ],
   "source": [
    "log_preds,y = learn.TTA(True)\n",
    "probs = np.mean(np.exp(log_preds),0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9830339321357285"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# validation accuracy after TTA\n",
    "accuracy_np(probs, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Analyzing results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "preds = np.argmax(probs, axis=1)\n",
    "probs = probs[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(y, preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[102   0   0   0   0   0   0   0   0   0   0   0]\n",
      " [  0  72   0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0  39   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0 108   0   0   0   0   1   0   0   0]\n",
      " [  0   0   0   0  53   0   1   0   0   0   0   0]\n",
      " [  1   0   0   0   0  91   0   0   0   0   0   0]\n",
      " [  7   0   0   0   0   0 152   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  41   0   0   0   0]\n",
      " [  0   0   0   0   1   0   0   0 100   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   2  43   0   0]\n",
      " [  1   0   0   0   0   0   0   0   0   0 102   0]\n",
      " [  0   0   1   0   0   1   0   0   1   0   0  82]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWYAAAEmCAYAAABRfjp6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXecVEXWhp93BoakEgRFkuSck6KImDGsmFFx1xxWXVdXXdO3ijnnnLOCWcQEKlkykiWIYEBUsmSG4Xx/VPXQM9PdM92T7gz38Ve/6a57bt1zW7q6blWd98jMCAkJCQkJDmml7UBISEhISE7CjjkkJCQkYIQdc0hISEjACDvmkJCQkIARdswhISEhASPsmENCQkICRtgxh4QUAElVJH0iaZ2kdwvRzkBJw4vSt9JC0kGSFpS2H+URhfuYQ8oTks4E/gO0BtYDM4A7zWxcIdv9O/Av4AAz215oRwOOJANamNkPpe3Lrkg4Yg4pN0j6D/AIcBewN9AIeAroXwTN7wss3BU65YIgqUJp+1CuMbOwhKXMF6A6sAE4NYFNJVzH/ZsvjwCV/LG+wK/A1cCfwHLgXH/sVmAbkOmvcT4wCHgjqu3GgAEV/PtzgB9xo/YlwMCo+nFR5x0ATAHW+b8HRB0bBdwOjPftDAdqx7m3iP//jfL/BOAYYCGwGrgxyr4nMAFY622fADL8sTH+Xjb6+x0Q1f51wO/A65E6f04zf42u/n09YCXQt7T/bZTFEo6YQ8oLvYDKwIcJbG4C9gc6A51wndP/RR2vi+vg6+M63ycl1TSzW3Cj8CFmtpuZvZjIEUnVgMeAo81sd1znOyOGXS3gU2+7J/AQ8KmkPaPMzgTOBfYCMoBrEly6Lu4zqA/cDDwPnAV0Aw4CbpbU1NtmAVcBtXGf3WHApQBm1sfbdPL3OySq/Vq4p4eLoi9sZotxnfabkqoCLwOvmNmoBP6GxCHsmEPKC3sCKy3xVMNA4DYz+9PMVuBGwn+POp7pj2ea2We40WKrFP3ZAbSXVMXMlpvZ3Bg2xwKLzOx1M9tuZm8D84G/Rdm8bGYLzWwz8A7uRyUembj59ExgMK7TfdTM1vvrzwU6ApjZNDOb6K+7FHgWOLgA93SLmW31/uTAzJ4HFgGTgH1wP4QhKRB2zCHlhVVA7XzmPusBP0W9/8nXZbeRq2PfBOyWrCNmthH3+H8JsFzSp5JaF8CfiE/1o97/noQ/q8wsy7+OdJx/RB3fHDlfUktJwyT9Lukv3BNB7QRtA6wwsy352DwPtAceN7Ot+diGxCHsmEPKCxOALbh51Xj8hnsMj9DI16XCRqBq1Pu60QfN7EszOwI3cpyP67Dy8yfi07IUfUqGp3F+tTCzPYAbAeVzTsItXJJ2w83bvwgM8lM1ISkQdswh5QIzW4ebV31S0gmSqkqqKOloSfd5s7eB/5NUR1Jtb/9GipecAfSR1EhSdeCGyAFJe0s63s81b8VNiWTFaOMzoKWkMyVVkDQAaAsMS9GnZNgd+AvY4Efz/8x1/A+gaZ6zEvMoMM3MLsDNnT9TaC93UcKOOaTcYGYP4fYw/x+wAvgFuBz4yJvcAUwFZgGzgem+LpVrjQCG+LamkbMzTcPt7vgNt1PhYPzCWq42VgHHedtVuB0Vx5nZylR8SpJrcAuL63Gj+SG5jg8CXpW0VtJp+TUmqT/QDzd9A+7/Q1dJA4vM412IMMAkJCQkJGCEI+aQkJCQgBF2zCEhISEBI+yYQ0JCQgJG2DGHlCQv4cKF50TV1QJG4AITRgA1ff1A3MLaLOBbXKReNpL6SVog6QdJ1+d34dC+fNuXN8LFv5CSpA9u69hrFapVb1epRl3uuOla1qxdy4NPPs/Vl11IjerV+d9dD7Bf9y4sWLSYtev+4shD+nD79f/m7LPPAZy+y9y5c2jRoiUVK1ZkwYL5NG7chCpVqsS8aGhf/ux/+mkpK1euzG/fdVzS99jXbHue4EV3/c0rvjSzfqm2XSSUtlhHWHa50tjM5lSt19J63DnSlq7caP0eGW897hxp/R4Zb0tXbrQed47MUQ59cKzt2GG2OdOVkWO+tcOPODL7/W133GW33XFX9vvcJbQvf/Zdu3azwvw7VJW9rHKXf8UswNTS/p6EUxkhpUqtahms2rgNgFUbt1GzasU8Nsd32oesqAe7335bRoMGDbPf16/fgGXL4gfLhfbl2z4lBEixSwAIO+ZSQlKWpBmSZkqaLukAX99Y0pz8zo/T5ihJ3YvW09Kl2741OL5TXbbv2Flnlnf6TQm+UKF9+bZPmbT02CUAhB1z6bHZzDqbWSdcOO/dpe1QNJJK5F/o6o3b2LNaBgB7VstgzabM7GPN61TjpmNace17OX+n6tdvwK+//pL9ftmyX6lXrx7xCO3Lt31qKOyYQ/JlD2BN7ko/eh7rR9TZo2p/7L+SZvsR9z25zkuT9KqkPOHGXkPiHUmzJA2RNCkyypa0QdJtkiYBvSTdLGmKpDmSnpMftki6QtI838ZgX3ewfwKYIek7SbsX5MbHLFrJsR2d/s+xHesyZqGLRt57j0rce3J7bhn6PT+vzrlI071HD374YRFLlyxh27ZtvDtkMMced3zca4T25ds+JRTsjjlMD1N6VJE0Aydsvg9waAybP4EjzGyLpBY4EZ7uko7GqajtZ2ablFPFqwLwJjDHzO6M0ealwBoz6yipPTkF3Kv5824GkDTPzG7zr1/H6Tp8AlwPNDGzrZJq+HOvAS4zs/FyKmN55CFnz549uW7dup1r1qxZYcHkkbwy4y9em/Azd53YjuM71eWPv7ZywwdOtviC3o2pXqUC1/VrCUBGOmzzMkAVKlTg4Uef4G/HHkVWVhZnn3Mebdu1i/tBh/bl2z5lFNxxabhdrpSQtMHMItq4vYAXcDq2+wLDzKy9nGrZEzhx9CygpZlVlfQgMN+cMHl0m6Nw+4DfidMpI+kjnHj6SP9+OnCRmU2VtB2XainLHzsZJ6xTFbff+HEzu0fSF7htbx8BH5nZBr/X9ETcj8IHZvZrovuvVr+Vtbvs2QJ/XmP+27fAtiHlnwP36860aVNTnnhO262eVep8fsxjW8bfMc3MSnWtJrg/GbsQZjYBJ1JeJ9ehq3Dyi52A7rjUQuDWlOP9on4LHCKpMoCkE6OmGLqTWHN3S1SnXBmXyPQUM+uAUyCr7O2OBZ7EpSyaJqmCmd0DXABUASYqtjB8SEgwEIGeygg75gDgO7F0nPRjNNWB5Wa2A5cCKfKvZjhwnlxuNXJNZbyI0/l913eYH/pFxs5mNhUYB5zmz2sLdIjjVqQTXumnJk7x56QBDf2I+79ADWA3Sc3MbLaZ3YuT1gw75pAAI0hPj13yO1N6SdKfsXZPSbpGksnpfSPHYz6CcZakrgXxLuyYS48qkZEsTgv3bNuZFijCU8DZkiYCLXFZMzCzL4ChwFR/fo4EneZ0iacDr/uONHebdSTNwiXPnIXL0JwDM1uLGyXPxk1ZTPGH0oE3JM0GvgMe9rZX+kXCmbgURp8n/YmEhJQUhRsxv4LTns7ZpNQQOAL4Oar6aKCFLxfhMsfkS9gxlxJmlh41ku1kZp/6+qVm1t6/XmRmHc1sfzO7ITIn7Y/dY2Zt/fk3+rq+flSMmd1iZmf40XY0W4CzzKwjcBsuielP/pwc+eTM7P/MrLmZHW5m55rZIHOJSnubWQcza++nMDCzf/n3nfx1k8r3tm7hZGY//A9mPTiQ5aPfynN84YIF7Netc3bZq9YePP7oI9xw3bV0at+aHl06ctopJ7J27dqY7Q//8gs6tmtFu9bNuf++e2LahPZl1z555Bb/YpV8MLMxuAQIuXkY9xQZPc3YH3jNHBOBGpL2KchFwrILFVxKoanATNxo+ejS8KNr127Z4bUbtmy3Jk2b2rwFi23dxq3WoUNHmz5zbo4Q3Gs/mZ9drv5orlWtUdsuevFrO+XWF+zqj+bYtZ/Mt54nX2A9T77Arv1kfo5zC9J+aF+27Asdkr17fat8xH0xC7DUf0ci5aIY36PGuB1MkffH4xbV8efX9q+HAb2j7L4GuufnXzhi3sUwl8q+u7mRbUczK/UphymTJ9OsWXOaNG1KRkYGpw44nWGffBzX/qeZE6ixT0Oq71WfJl17k5budn3u06oT61f+nsc+2fZD+7JlnxKJ9zGv9N+RSHkucVOqCtyEyyGZ53CMuny3woUdc0ipk6w2wvyxn9Gmz7F56ueMeJ8m3foUuv3QvmzZp0aRBpg0A5oAMyUtBRoA0yXVBX4FGkbZNqAAmdnDjrkMIqmupMGSFvsIvM8kXSSpUNmVJQ2SdE3+lnnO61uYa/tHvNxtxrTNytzG4knf0OrAnGsvE4Y8g9Ir0Lbv3wrVfmhf9uxTpohEjMztRtrLnHJiY1xn3NXMfsct0v/D787YH1hnZsvzazPsmMsYPiz6Q2CUmTUzs7bAjcDehWy31KJAk9FG+HHaWPZq1pZqNWtn1835+kMWTxnJcVffH/MLHDRth9A+AFoZEqRViF3yPVVvAxOAVpJ+lRQ7UsXxGfAj8ANul1OebOmxCDvmsschQKaZPROpMLMZwFjcfuL3JM2X9GaUtkU8zYtRku6SNBr4d/RFJHWWNNHvvfxQUk1f31zSV9qpitcs13k9vFZG04LeUDLaCPPHfEqbg3dOYyyZNpbJ77/ASf97moqVYwuvB03bIbQPgFYGpDyVYW7X0T5mVtHMGpjZi7mONzazlf61mdllfhDVwfyuqfwItTLKHu2BaXGOdQHa4eawxgMH4gJKnrDYmhcANczsYH9sUFRbrwH/MrPRkm4DbgGuxIVc32NmH/rowDT8HJqcyNLjQH8zi97LiT9+EW4vJw0bNcquL6g2QuaWzSydMZ4jL7s1u+6rZ28nK3Mb7/zvPADqteqU43gy7Yf2ZdM+JSKLfwEl1MooY0i6AicgdFWu+r7ATWZ2hH//NDDezN5IoHkxCrjFzEb7cwbhNDCeB2abWSNf3wx4FzgY+N7MGsS49ou4wJIjzSzfxY1u3brb+EkFGjwAcPMXCwpsC3Bbv1ZJ2YeULQqtlVGzsVU+NNYmCtj8wfmhVkZI0szFaVTEIjqoIwuokI/mBfhowgKS6IuwHBe80iWJ9kJCSgWXwEQxSxAIO+ayxzdAJUkXRiok9cCNZmMRU/MiEWa2Dlgj6SBf9XdgtJn9Bfwq6QR/3UoRvQ5gLU7c6C4/gg4JCS4SSotdgkDYMZcxzM09nQgc4bfLzQUGEWdvpMXXvMiPs4H7vaZGZ1z4NrhO+gpf/y1QN+pafwB/A56UtF+StxYSUqKkpaXFLEEgGF6EJIWZ/WZmp/mV3nZmdqyZPW9mx0XZXG5mr/jXeTQvfH3f6FVic1oYD/jXM8xpdHQ0sxPMbI2vX2Rmh/r6bmb2o5mNilzbzH72Pk1K5p6S1UZYMm0sL1zSj+cvOpJJ7+YNzNqyZQu9e/WkZ9dOdO3UjttvvQWAUSO/oVePrnTr3J4Lzj2b7du3F4k/oX3p2ieNCPSIudS1G8Kya5ZktTKS1VIYu2CVDf/uZxu3cLWNmvuHtenY1Z4e/LntVbeevfXlZBu3cLWdc9k1dv2dj9q4hasDpwUR2hevVkZ6rSZW/czXYxZgaml/P8IRc0ipUxxaCpKoWs2J5W3fnknW9u2kpadTMaMSjZo0B6DHAYcw6stPUmo/tA+OfaqEUxkhIQkoLi2FrKwszjm+D3/r1YruB/albcdubN+eyfzZ3wEw8suP+fP3vOcFTQsitC8GrYyAT2WEHXMZJI5WRkvFyKhQFjArHi2F9PR0Xhk6hg/GzOH7WdNZsuh7bn34BR676yYuPPlwqlbbnfT0vDFWxeVPaF889qkgFI6YQ4qO4tLKKIw/ypslJSmKW0th9z2q06XngUwc+zXtu/Tkqbc/4/n3v6Jzj140aJw3cjxoWhChfTFoZZD6iFkxUktJut9LIUQkDGpEHbtBLrXUAklHFcS3sGMue8TTysj+lywp3f9DmeL/oVzs63eT9LXXuJgtqb+vv1fSpVHnD5J0tX99bVQ7t/q6xpK+l/QULoVVQ0mveC2O2ZJyRCXmR3FoKaxZvZL1f7mMWVu3bGbqt6PZt2lL1qxaAcC2bVt587nHOOH0c0vEn9A+YFoZKlSAySvkTS01AmhvLjPQQuAGyM6reTpOKqEf8JSkfGPBQ62MskcirYwI5+PkBXtIqgSMlzQc13mfaGZ/ySWLnChpKDAYeAQXIQguWWs/SUficpX1xAVLDZXUB5fTrBVwrpldKqkbUN98Sqzo0UI0hdXKSMZ+1Z9/cOd1l7JjRxY7duzg0KNP4MBDjuLJe2/m25FfssOME884l2698uo3B00LIrQvBq0MSHnawszGSGqcq2541NuJ7Azk6g8MNpdqbYmkH3DfpwmJrhFqZZQxEmhlNAaGmVl7Se8BHYFN/nB14GJgJC4vWR9gB65zbWJmv0v6HjgMqAM8ZWYHSnoA9w8skkhvN+BuXHqckWbWxF+7Ji4Fz2fAp8Bwy5trMAfJamUky7Qla5Ky79akZjF5ElIcFFYrI6NOc6t98n0xjy1/9uSfgJVRVc9Zriwm0d+33OdL+gQYYk6n5glgopm94Y+9CHxuZu8l8i8cMZc95pJ/WLVwynBf5qiUzsF1vN3MLFMu20IkZPs9325d3Ag60s7dZvZsrnYaE6WxYWZrJHUCjgIuw424z0vyvkJCSg4lHDGvtBRFjCTdBGzHqTD6K+UhTC1VDomnlbFvlM2XwD8lVfTHW0qqhhs5/+k75UNynTMYNxd2Cq6TjrRzntfYQFJ9SXvldshPi6SZ2fvA/4CuRXOrISHFR1GLGEk6GyepO9B2TkWklFoqHDGXMczMJJ0IPCLpepyi21KcVnKEF3BZfKf7XRwrgBNwv+KfSJoKzADmR7U7V9LuwDLzqW/MbLikNsAE/w92A3AWTrkumvrAy1G7M24oujsOCSkeinLPsqR+wHXAwWa2KerQUOAtSQ8B9XBrNpPzay8cMZdBLLZWxqLIfJeZ7TCzG81lTGhvZoeY2TozW2lmvcxl/r3AzNqY2dKodjuY2SG5rvWor+/gz11sZkuj59bMbKaZdTWzzr4knXm7uLUUJo75ijOO6smAw7vx+rOPANB8r6r0aFKdzo32yGFbMQ0y0t3fWPUZ6ZCe6zsdNO2IZO0vvuA8GtXbi26d80yZlgn/k0VKfR+zYqeWegLYHRghaYakZ8ANeIB3gHnAF8BlZpZ7YJOX0o4JD8uuWYpbK6Mg9lsyzbZkmmXtMGt303Brd9Nwe3HMEnvoy4XW7qbh9tCXC+2F0T9au5uG2yPDd77ufedI27EjWNoRqdhv2rYjuwz/epSNnzTV2rZtl6M+ugTJ/8JqZWTs1dwaXzksZiHUyggJKT3thVgrMIe0rsPH090U4MfTf+PQNm5K3QyqVXIzf1Urpec4N2jaEaloTfQ+qA+1atZKaBNk/5NGoVZGSEhCgqS9sOduGazcsA2AlRu2UWu3DADemvgLTetUY+R1ffjw8l5sj9oMGCT/U7FPlrLuP0QymMQuQSDsmENKHbPgay8c2GJP5i9fzyH3juHkJydSIeqbEzT/i+J+E1HW/fctkpYWuwSBctExK46oT2n7lQyS/uFDmuf6e7jG14+SlGdPpaTukh7Lp80NxeVvgmv2lTQsmXOCpL2wasM2avtRcu3dMljtR88ndq3HV/P+BOCX1Zsx27lBNUj+p2KfLGXdf8BPZYQdc7Hht4MFRtQnFSQdjdvudqSZtcPtA16X6Bwzm2pmV5SEf8VNkLQXRs1fQf+urhPo37UeI+c7bY3la7ewfzM3B7tntQzStDNKIEj+p2KfLGXdf3A/qunpilmCQJnvmIkv6jPOC/lEhHUGQPaIbrSkdyQtlHSPpIGSJnu7Zt7uFUlPSxop6UdJB8upSn0v6ZXItSSd4c+bI+neqPoNku6UNFPSREmJfihuAK4xs9+8/1vM7Pmo46d6/xbKJ0iNHpnKiRO97P2YJenk6MYl1ZY0QdKxkp6SdLyv/1DSS/71+ZLu8K/P8tebIelZedEVSUf6dqZLejcq8KSfnLLWOOCkeDcp6SJJUyVNXbFyRXZ9tDZC5w5tOPnU0wqspVAY+8j2NwFfXXsQJ3WrxwtjltKrWS0+vfJAejWrxQtjlgDwzKgldG5Ugw8u358XzutG5o782y9u/4vKHuDss86kb58DWLhwAc2bNOSVl18sU/6nQlEHmBQlZV4rQ/G1I04GLsEpOtXGJSHdD6cP8RHQBlgN/Ai8YGa3SPq3b+tK3/lWBs4AjgdeBw7EhURPwQkF/YkTLOkGrAGGA4+Z2UeSDDjezD6RdB/wl5ndEeceVvvr5hklSxoFTDOzqyUdA/zHzA6Xy0R9jZkd538QKpnZlf6cmubCpDcAzXCb3P/PzEZIOh0Xkn2tpMnADjPbX9LLuOi/n4H7gJPMRQg+5e/xM+AD4Ggz2yjpOqCSt10EHAr8AAwBqlpU/sFYFLdWRrJ0HzQiKfupg44oJk9KjmS/+0HptKDwWhlV67W05hc8FfPY7NuPmGYphmQXFeVhxByP3sDbZpZlLnvzaKCHPzbFzJabU3xajOtQwWWSbhzVxifm/vXOBv4ws9nmxHnmerseuCmUFWYWiY+PyJVtAyJzrdNytZssH+TTzuHAk5E35hOnAhVxgkP/NbNIzzMWOEhOjnAe8IekfYBeuKzXh+F+aKZImuHfNwX2B9rilOpm4LJo7wu0BpaYC3Ax4I1C3GdISAkR7MW/8hCSHU/UJ9EnvDXq9Y6o9zvI+ZlsjWETbRc7xbIj03YOSbJI/FnPxXWG3+Tjb7x2RGxhlO24zvwo3A8TZrZMTg2uHzAGqIUTHdpgZuv9nP2rZpYjrFrS34ARZnZGrvrOca4dEhJc/OJfUCkPI+Z4oj5rgAFyovF1cCPZfGPUk2QScLCfw03HTXuMTqGdu4H7JNUFkFTJT9EUlOHA5ZE3vuMF12GeB7SW09WIMAG32DgGN4K+xv8FN8I+RV6sSFItSfvipjMOlNTc11eV2/kyH2gSmZvHfQYhIYFGhLsyihU/Kj0ROEJuu9xcYBDwFjALmInrvP9rZr8X8bWX4xbuRvrrTDezpEOUzOwz3FTEV97/aST3NHMHUNMvQM7ELYhG2s7CqcYdop1ZSsYCFczsB1wGklq+DjObB/wfMFzSLFxmhn3MbAVwDvC2r58ItDazLTjx+0/94t9Pyd4/BE97Yf3iKSx6+jwWPXUOK74dHNOmVfPGdO/cgf26debA/dyU5K23/I8eXTqyX7fOHHf0kfz2W2whsSBqWXRq15r2bVrwQAA+/+LWyoDUF/8UO7VULUkjJC3yf2v6ekl6TC611CxJBVNeLO2Y8LDsmiUIWhmFsX9r2i9We58G9sxXM+2tab9klxdGz8t+/Y9rbrXDTj7L3pr2S9Lt59apKGtaFkHXyqhav6X1vGtUzEI+Whm4p++uwJyouvuA6/3r64F7/etjgM9xg/T9gUkF8S/uiFnSHolKgXr9kJACEDTthcJoNVTdbffs11s3b4pps6tpWQRRK0OFWPwzszG4HV3R9Ade9a9fxcnsRupfM8dEoIZfbE9IoqmMucAc/3durvdzEpwXEgdJN/m9wdHlptL2q7QJmvZCQe0lcc9lA7lx4DF8/cGb2fVDnryXy4/pyfgvPuTUf15TaH+SJSifT0nZp0qCjrl2ZL+9LxcVoLm9baeO+XIgklCiPlGJknHC+fXzayzuPKaZNYx3LCQ1zOxO4M7S9iNo+Ee+HCSa6wuK/aCXPqBmnbqsW72Suy89k3qNm9Gm6/4MuOw6Blx2HR+/9ATDh7zCKZdcXSh/kiUon09J2adEYsGilFNLxb5SHoomtZSk0yXd6F83kMuKvEugcqDDkQhJg+R1OYqgrSslVU32vKBpLxTUvmadugBUr1Wb7of0Y/GcGTmOH3D0CUz+5rNC+5MsQfl8Sso+FdyujCKV/YzEA+D//unrU0otla8XclleDwH+7qs2Ac/EP6P84Pf0lmkdjhLmSiDpjjlo2gsFsd+yeRObN27Ifj174hgaNm/F8p+XZNtMHz2Ceo2bF9qfZAnC51OS9qlSxNvlhuKCrvB/P46q/4ffnbE/sC4y5ZGIgmzJOsDMukr6DsDMVkvKSMHxskhMHQ7/Id8PHI17LLnDzIbIhUnfCvwBdMZF7M0G/g1UAU4ws8Vy4d6bcVFz+wLn4v5n9sKt2p4DTocD90Mg4FMzu87XbwAexSV+3Az0NxfdmAO/t3oRLiy7Om7Boq+ZjZE01l8XoK1c6Hcj4BEze8yffxZwBZCB27N9qZllSXoaF/VYBXjPXDj7FbicZiMlrbRcKap8exfhttbRsFGj7PpobYSsrCzOPue8AmsplJb9ulUrePgat3U+KyuLA/v1p9MBh/DwtRex/KfFSGnU3qcB5994V6H9AadlMWbMKFatXEnzJg35v5sHcc655wf28ylJ+1SQUu+E5VJL9cXNRf8K3ALcA7wjl2bqZ+BUb/4ZbmfGD7hB7bl5Gox1jVjzObmcmITrMKb6DnpP4Csz65L0HZUxVD50OL4Argaa4P4BfQQ8AMw3syaSBgFH4n6EdgcWAHWB5sTQzDCz1yTV8j/Q6biAlCvMbJakpUB3M1uZ32cbNK2MZPlw1q9J2Z/YsUFS9vl9L2NR5POwAaawWhl7NGpjPa59Keaxb644oExoZTwJvA/UkXQrMA64N/Ep5Z6ypMMx1p/XBxdh2Nu3PSXK5lMz2+o71D9xUzXxNDMATpM0HfgOaIfT0AgJKVOkpylmCQL5TmX4EdI0nFAOwKlmtqtslysPOhxjcaP7esDNwLW4x7AxcXyOtBdPM6MJLoS7hzkFu1dwo/+QkDKDRGA64VgUdAkyHcjEjdTKfBh3EpQHHY5JwAE4ec8twAzgYnZqY8QjnmbGHsBGYJ2cxvTRUeesx02HhIQEnjKtleEDIN7GjbgaAG9JuiHxWeUDPyot6zocW3Eb3Cf6qrG4znN2PufF08yYiZvCmAu8BIyPOu054HNJI5P1syS0IIrTfua3I7n6pIO5qn9vhr78ZEybdDlh/ox09xogzddVSk+56HwwAAAgAElEQVT8GFbWtSyCppUhIF2KWQJBfjHbwPc44fPI+6rA94WJUw9LWLp07ZaUFkTQtBqStf/n4Fm2ZNVGO/H5KXbcM5Psu1/W2gVvzbCL355pF74102YuW2dXvDfHjnl6kh3z9KTA+R80+8JqZVTft40d/9yUmIV8tDJKohRkWuIncs5hVsDtNggJKTLKuxZEw5qVWfDHBrZu38EOg9m/radXk5r8snYLy9ZtKXf3W5L2qSCCvfiXSMToYUkP4fbezZX0gqTncY/Aa0vKwZCCsSvpcARNq6Eg9j+t3kz7ffZg90oVqFQhje6NalCnWqW4bQbN/yDbp0qQ55gTreZHdl7MBT6Nqp8YwzaklLFdSIfDLFhaDQWx/2XtFt6b8Rt3HNeKLZlZLFm1iawY55WUP+XJPhWCvisjkYhR/DS5IYFAUhY5F/FOMLOlcWwb46I430rQXl98gteouleAYWb2XuE9LhqCptVQUPvh81cyfL6LvflHzwas2rgtbptB9D+o9qkS3G65YLsymnkRn1mSFkZKSTgXki+bzaxzVFmawLYxcGbJuFW8BE2roaD21Su7cVCd3TI4oElNRi9aVa7vt6TsU6Gwc8ySrpI0Vy5r0NuSKktqImmSXBaTIYWRriiIVsYruNRFD+D2rJ6LC4IICSB+ZPw6UM1XXW5m3+Ji+dv4KL5XzezhFNruBjwE7AasBM4xs+VeZ2MSLqy7BnC+meXZJx1PKwN2DS2IG49qwR6VKrB9h/H0uJ/YsC2LXo1rcknvfalepQKDjm7Jj6s2cfOnCwLpf1DtU6JwWhn1cRoybc1ss6R3cOnbjgEeNrPBkp7BSSs8ndI1Ys3n5HJimpl1kzTbzDr4urFmdlAqFwwpOnJNZSwxsxO97OYOM9siqQUudLx7rGmKGO31xaliLYmqboTrTD/GBbj0N7MVkgYAR5nZeb5jnmZmV0s6BviPmR1OArp2627jJ05JZJLbtwLbBpGTX0gu/uj9C3oWkyflg8JqZdRu2s6Ou/PtmMdePbNTQq0M3zFPBDoBf+H0Zx7HySbUNbPtknoBg8zsqFT8K8iIeavct2KxpEuAZexU5w8pXTabWedcdRWBJyR1xoVXJ6sdPTbGHDM4gab2wAjfSaYD0fKFH/i/+Wl3hIQEgrQUf+zNbJmkB3AqcptxejjTgLXmdG2ggJlK4lGQjvkq3KPrFbhV/+rAealeMKTYuQonO9oJt4aQeJNswREw18x6xTke0dvIT7sjJKTUkRJ2zLUlRUsfPmdmz+08VzVxufya4LYOv0tOaYIIyUsEegoiYjTJv1zPTrH8kOBSHfjVzHZIOhs3soXC61gswCkM9jKzCZIqAi3NbG4h/Q0JKRUSzDHnl1rqcNzU4QoASR/g9GhqSKrgR80FylQS17d4ByR9KOmDeCXVC4YUO08BZ0uaiJvG2OjrZwHbJc2UdFXcs+NgZttwSnv3SpqJE0M6oCgcXrhgAft175Jd9t6zOk889kjCc4Km1ZCs/Yq5Exh9y6mMuvlkFn/pkiv/u28T3jy7C0+etlMvpEKa09GIaGxE+pKIxkbu+qDeb/C0MmLvyCjgroyfgf0lVfXTvIcB83C6NhE1yugsJskTL1bbXyxuKe1Y8rCU7dK5SzdbvXF7nrLir622115728zvF+eoD5pWQ3HYb8k025JplrXDrHLny6xy58vs9qc/tesf/CD7faTsN+Bua3LEjVa582XW9eQ7bNkfa0rd/7KklVGnWTu79IN5MQsF0MrAZSqajwvEex2ohNMrn4zLVvIuUClV/+KOmM3s60Ql5V+CkJAEjB75NY2bNqVho33j2gRNq6Go7JOZkJy54FeWr1gHwLzFy6mUUbHU/S8t+1RJi1MKgpndYmatzay9mf3dXKKJH82sp5k1N7NTzSk7puxbyC6GpKNi6Gp8WNp+AXzw3jucfOrpCW2CptVQ3PaXnN6HyUNu4JlbBlJj9yp5jp94eGdmLtgZKRc0/4OolVFmRYxCyi9m9qXljBjsbGYnlrZf27Zt44vPPqH/ibGSxuzEP0rmoCxpOyRj//y7Y2n7t0Hsd/o9/L7yL+75z0k5jrdpWpc7rujP5XcMDqT/JWGfChGtjDLfMUsqmBRWGUcuA3Vp+5Am6TEf7jlb0hS5lE5I+kxSDf96g//bV9KwRG3mc70iuWdJjSWlHPb91fAv6NipC3vtvXdCu6BpNRSn/Z+r17Njh5t3fOmD8XRvv3OKp/5eNRjy0EVc8L/XWfLrzvy3QfK/JOxTJT0tdgkCBdHK6ClpNrDIv+8k6fFi92zXZgAuY0xHc9GWJ+KlVs3sGDMLquxqYwqhx/H+u4PzncaA4Gk1FKd93dp7ZL/uf2gn5i12MT3Vd6vCB49fws2PD2XCzJzy6EHyvyTsU0FABSlmCQIFCQR4DDgOF3aImc2UdEixehUw5HLdvQTUAVYA55rZzwnqTwVuwQVbrDOzPnJ5++7BJUKtBDxpZs/GueQ+wHJzWbMxs1+jfFkKdDeX0TqWrz1wKZ5OBr7EKcqtkJQGLAT2j3WupAdxWhdrgNP9Oc1wWdLr4HS5LzSz+T4a8C+gO1AXl1rrPfLR44jWymjQMKdWxqZNmxj1zVc8/Fj+0gJB02ooKvuKaTu3vf3wxe3c/sxn9OnWgo6tGmBm/LR8Nf+6w4URX3J6H5o1rMP1F/bj+gv7BcL/0rJPlYD0wTEpiFbGZDPrKek7M+vi62aaWacS8bCEkbTBzHbLVfcJ8J6ZvSrpPOB4MzshQf1soJ+50M0aZrbWd0p7mdkdflpoPC7j+JIYPjQAxuFGyV8Db5jZd/7YUnzHHPE1ooMB3IWL2T/R/0DcgvtheETSkcDFZnZyjOsZcJaZvSnpZu/n5ZK+Bi4xs0WS9gPuNrNDfcdcDTeybw0MNbPmBdHjiNCla3f7Ztyk/MyyqZKRnr9ROaJmj8uTPmfNlCeKwZNgUlitjHotO9gFj8cOx7i9X8uEWhklQUFmVH6R1BMwuazQV+JGXrsSvXAJWMHtWeydT/144BW57NqRHuVI4B9+NDkJ2BNoEetifoTcCpeMdQfwtaTD8vGxDW6k/Dcz+9nXvQT8w78+D3g5zrk7gCH+9RtAb0m74QJI3vU+P4sbyUf4yMx2mEvamnhSOCQkYAiokKaYJQgUZCrjn7jpjEY4DYavfN2uTLzHDAMws0v8CPNYYIYXFBLwLzP7skAXcHsgP8dlnf4DOAE3eo7HcqAy0AUfCmpmv0j6Q9KhwH7AQD+lMs2fM9TMbo5zH2k4UZbcIkkRovdoBuNfc0hIQVFwFvpika9rZvanmZ1uZrV9OT3e/GY55luc3irAQNw0Q9x6Sc3MbJLv9FYCDXHzvf/0GhNIaikpopmcA0ldJdXzr9OAjrikuIlYi/shuMtPKUR4ATcKfsfMsnyJbJGLdMpp7AwlPRMYZ2Z/AUv8fDly5Dd9VVg9jpCQEkNx/gsCBdmV8byk53KXknCulKgq6deo8h+cst65kmbhhJz+7W3j1d/vt7nNAcYAM3Ed5Dxguq9/lvhPLHsBn3i7WcB2IN8JRDP7A/gb8KQfsQMMxakDxpvGAKen0U7SNOBQ4DZfPxA432tjzMUpaiUiZT2Or4Z/Qc/ObenWoRWPPHBvvvZB02oobvusv35i6/dvsnXe62z/Y1p8u6ws9u/ehZP6u2n+p598gnatm1Ololi5Mv54Kmj3W/xaGU6HJFYJBAWICR8QVc7Gi0KnGgMelpItuJ0TY0vbj9wlWitjxV9brXGTpjZ9zkL7fc0ma9e+o307ddYup5URbROti1Gp0z9NGXtYRpuzrFLHS0yV97SM1mfk0c/YnGl2z30P2mmnn2FHH3Osbc40mzB5us1ftMQa7buv/bJ8RWDvt6S1Mhq0bG8PjFocs1AwrYwawHs4vYzvcetNtYARuK3FI4CaqfpXENnPIdHvJb3uLxoScCRdj1sPGFjavuQmTTt3WkycMJnmzZvTppVbCx1w+ul89cUwunTqEPPcaC0FIFtLoU3btuXGPnqHxcQJE7jz9sV88tnrANx/r9tqeO11N+Ro95PxsxnywUecdclVvPvy00xcvAr2aMTyLNiauYNpS1dTfd3OR/X9m+0ZmPstjH1KFD5L9qPAF2Z2ilxuv6rAjcDXZnaP/+5dD1yXSuOpDNybAPEVZkIKjKQOyqtZUfA9ZPlgZveY2b5mNi5/69IjaNoLZdX+ibtu4uJrbiFNyX2tg+J/qvap4LQyUov8k7QH0Ad4EZwkrrmgr/7Aq97sVdyCfUoUZI55jaTVvqzFjZZvTPWCITsxs9mWV7Niv/zPTA1J5p94Iu8rSFqRXzi3pO6SHisuv/yjYe5rhvZJ2H/26TBq7FmbVu3jbaKJTxD8L4x9aoi0OKUANMUFlL0s6TtJL/iF/L3NbDmA/5tyCr6EUxlyn0YnXJ4/cEk+U06XElLqbATaS6piZpuBI9j5/zYuZjYVmJqfXaoETXuhLNpP+HY8337zBZNGf8W2bVvZtGE9d157CTfd/0zcdoPkf2HsU0GJt8slTC2F6ze74ra/TpL0KG7aougowCT3tNJeKApL0RRgAy468BT//jXcHNgw/74nbgvgd/5vK1/fN8rmM1z2khnAOtyCcDpwPzAFtzPj4vx86dq1W/ZCzvrNmda4SRP7fuGP2Ys902bMibs4FNrHth85f6WNnL/SHn71I9v/4COy34+cv9L2rtfQPpqwIEdd0PxPxr6wi3/7tu5gz01cGrOQz+IfToZgadT7g4BPcenX9vF1+wALUvWvIAEmkyV1NbPpqXX9IQFjMHCzn77oiIsOPMgfmw/0MZd+/XBcJ54jhNvMjgGQ1A23Be8j4Hxc6HePSLi5pOGWK9w8WiujYaOdWhlB014o6/bRvP/acwx+8XFWr/yT84/vw34HH861dzwaaP9LSisj1cU/M/td0i+SWpnZAnamlpqHG6jcQyFTS8XVypBPKuh1H9oAi3GPwnK+WddULxpSOkRpa0zFiRO1wKVev8bMjpPUEBfl2QIX/VfRzFrn1sCQVBu3P/s0M5sj6T1cJ7/JX6o6btQ8PJ4v3bp1t/GTim12ZJdk4uJVSdlHdmWURQqrldGkTUe75bXYSyvn9tw3X60MH837ApAB/Aici1uzewcXJf0zTgtndSr+JRoxT8bNo6S8shgSWIYCD+CmKKK/nbcDI83sREmNgVG5T/Qh3YOB28xsTqSaJMLNQ0JKHUFaIRYUzWwGLkYgN/lp2hSIRB2zvAOLi+JCIYHiJdzUw+xc4dvV2bkYeE6cc+8BZpnZ4Ki6SLj5N2aWKaklsMzMNsZuIiSkdBGQHmDdz0Qdcx0fjhwTM3uoGPwJKQHMqdc9GuPQfcCr/v/7N3FOvwaY6xXnAG7GPdI1xoWbC7eVKHzSCgkwIi0gSnKxSLSPOR2nsbB7nBJSxrBcOtO+blRk7tjMJphZSzM70Mz+Z2aNY9jIXGbgyL7roebkP280sw7+2CFmti4Z34KmvZCs/cUXnEejenvRrXP7fG1Lwp/JY7/mH/32Y+CRPXjrOfcb3KpuNQ5oVpMejavnsK2YBhnp7m80FXx9Rnpe+cCgff7JEhkxxyqBIMGWkOlFtU0rLGHJXaK3ywVBe6Ew9pszzUZ8M9q+nTTN2rZrl9CuOP2JbIP7au4fVq9hY3tzxFQbPus3a9qqnb08bLxN/2mtTVmyxjZsyczeLpeZZbZtuzt/23b3fnOm2dbtZtv96y2ZZlk7gvX5F3a7XNM2He2d75bFLBRAK6O4S6IRc0B+OkLKO9HaCBkZGdnaCEG2z/1FOrD3QdSsWRMs9mCnJPzfv9me7N9sT9JXLaZt65ac1LcbB7XZh3P+PpBfZoymTb3qdGhQgyoZFdi/2Z4c+tAYlq3dxEnPTOTQh8Zw0jMTWbZ2E4c+NIbPZi/ntk+/59CHxnDIQ2P4Zc2mpP0pKftUkII9Yk7UMRfJ6mJISH4ETXuhuLUaguR/raoZrNq4DYBVG7dRs2pFAOrsnsEff+3MhbBi/dbskVqQ/C8MaVLMEgTiLv5ZivvvQkKSJfeIEsq3VkNZ8D+RdVnwPz+CvisjKLLQRYakmyTNlTTLq7UVmSiQpCslVY16v9QHWxQLks7xwkOHRdWd6OtOSXRuaZHKZxI07YXi1moIkv+rN21jz2oZAOxZLYM1mzIB+HP9Nvbeo1K2XZ3dK2XnUwuS/4VBil2CQLnqmCX1Ao4DuppZR+Bw4JfEZyXFlTjd1ZJkNnBG1PvTcRlRyg3de/Tghx8WsXTJErZt28a7QwZz7HHHlxn7ZAmS/+N+WMUx7V0u3WPa783YRauy6/u1c/Xt9tmdjVu3B9L/1Ik9jRGUqYxy1THjhENWmktkipmtNLPfJPWQ9K1PeTRZ0u5yGb/vlzTFj64vBpDUV9IoSe9Jmi/pTTmuAOoBIyWNzH1hSWf5tmdIeta3ny7pFUlz5FJNXeVtr5A0z193cO62cjEW6Cmpolzm6uY4AaHIdW/29zBHLu2XJDWTND3KpoWkaZJ6SvrA1/WXtFlShqTKkn709c0kfeHtx0pq7evrSHrfX2uKpAN9/Z6ShsvJHz5LgqdgSRdJmipp6oqVK7Lro7UROndow8mnnlZgLYUg2AOcfdaZ9O1zAAsXLqB5k4a88vKLgfM/si1OQKV0OK5DXV6f+As9GtdgyIU96NG4Bq9PcuOYb39czW9rt/DuhT24vl9LHhjxQ6n7X5QEffEvrlZGWcR3XONwo9qvgCHABJw4zwAzmyIncr0JOA/Yy8zuiAjvAKfikgB8DLTDZZseD1xrZuMkLQW6m09GG3kP1MEFZ5xkLvLtKWAiLk/ePWZ2hLevYWZrJf0GNDGzrZG6OPdzjm9/Gy7gozquY26CU3t7T1KtyHqAnNbyO2b2if/xuMrMZki6C5dF+2lgkZk1kfQAcDDuKaACcImZnSHpa/96kZ8GutvMDpX0FvCU/xwaAV+aWRs5neaVZnabpGOBYUAdyydhb1nXykj2e1PUc6RFwaEPjUnK/pv/9CkmT5KnsFoZLdt3tiff/SrmsSPb1slXKwOy5Qmm4qJcj5PUBCdXUAuYDvzdzLal4l+5GjGb2QagG07BbAWuY74YWG5mU7zNX2a2HTgS+IdcBNsknGZEC9/UZDP71cx24EanjfO59GH+ulN8e4fhxLR/BJpKelxSP+Avbz8LeFPSWbhEq/kxGDeFcTrwdq5jh0iaJCc2dSjuBwVcNN65/h/PAOAtf98/SGqDk/h8CJeJ4SBgrP9hOwB419/Hs7inEHDTQk/4+qHAHpJ29+e/AWBmnwJrCnA/ISGlShEFmPwbl+8vwr3Aw2bWAvc9OD9V/woi+1mmMLMsnPjOKN9ZXQbEGt7EFN6R047YGlWVRf6fk4BXzeyGPAekTsBR3o/TcCP1Y3Ed2vHA/yS1851mvHuaLKk9sNnMFkZGX5IqA0/hRvG/SBoEVPanvQ/cghtpTzOziPTYWOBoIBP3VPEKLsrzGtwP9Vozi5UGIw3oZU5gP/r+IPbnGxISaArzECOpAe57fCfwH7kvwqHAmd7kVWAQ7ik1acrViFlSK0ktoqo6437R6knq4W12l1SBncI7FX19S7n0MIlYT+xw9K+BUyTt5duqJWlfud0JaWb2PvA/oKukNKChmY0E/ovLtpsnVDoGN5A3pVekE17pR7vZOzXMbIu/x6dxuskRxuCmLyaY2Qrck0JrYK6Z/QUskXSqvw/5HxZw8qCXRxqRkz2MtDfQ1x0N1CzAvYSElCpFMGJ+BPf93eHf74kb1EQGWL8C9VP1r1x1zLgO7tXIwhrQFieyMwB4XNJMXM7CyrhH/Xk44Z05uMf2/EbGzwGfK9fin5nNA/4PGO6vOwI3BVAfN3KfgRuZ3oAbnb7hR/Pf4R59Ys4x57rG574zj65bCzyP27nxES6DSDRv4kaz0brIk4C9cR0quGmVWbZz0nQgcL7/rObiEkwCXAF09wuW84BLfP2tQB+/2HgkToc2aYKmvZCKfad2rWnfpgUPBMSfgtpvXfsnc5/9NzMe+DszHjyb5ePei2ublZXF/t27cFL/4wC45MLz6dm1Ez26dOSMAaewYcOGEvc/JeLsyPC7MmpHFql9uSjnqToO+NPMpkVXx7hK6k+SpR0THpbiK7jpidtL249YpaxrZWzatiO7rN+caU2aNrW583+wtRu2ZKdCirYJmv+bM8163Tvaet072rrd9IF1uOJ563XvaOt52+dWuXYD6/SfV7OPR8rmTLN77nvQTjv9DDv6mGNtc6bZH6vWZbf3r39fZbfdeXeZ0Mpo3b6zfbtoTcxC/qml7saNiJcCv+M2E7wJrAQqeJteuAXylPwrd3PMIQ5JHwLNcPNegSZaGwHI1kZo07ZtYO2jd1lMnTKFZs2a07RZs2z7T4cNjbvFKwj+A4y48qBYpzNgclcuPqwuhx6e8/iHY2cx+P2POPPiq3j/1acZt8hvvPljJWbGkt/XsCl9t+z63i1ql8j9pkqqU8zm1pJugOw1qWvMbKCkd3HTiYMpZGqp8jaVUSaRdK7c/ufo8mRh2jSzE82so+WzbS0IBE17YVezj+anpUuZOfM7uvfMGzD79D03ceE1t5CWlrPbuP/Gf3Fan3b8smQRJwy8oFT9TwZJMUshuA63EPgDbs45/mb2fAg75gBgZi/bTn3jSLmstP0qKfyjXw4SfUFC++LRmtiwYQMDTz+Fex94mD322CPHsc8/HUaNWnVo2a5TnvOuvetxBo+aTaOmLRn1+Uel5n+ypCl2SQbLqVX+o5n1NLPmZnaq+UC3lHxL9cTyhmJobKiItDAkNfYLjEVGKr7JRTQukIuAHC+pVVH6lCpB017Y1ewBMjMzGTjgFAacfib9Tzgpz/GJE8YzYeQXnHV4V+68+kJmTBrHPf/9Z/bx9PR0Dj66P+NG5E1wGkStDFEsI+aio7QXgYJQcBP1E4BK/n1tXPj1UqB2EbTfGJhTiPMrxKhL2jfc/u7u/vVFwNDC+FCYEr34t35zpjVu0sS+X/hj9mLPtBlz4i4OhfZFY79h6w7bsHWHrd+SZWcM/LtdevkV2XWxyoh5K2zEvBX2wCsf2X4HH2HD5/5pr3w+yUbMW2HD5/5pp5xzqZ1yzqXZdsXpf2EX/9p06GzTlqyLWQiAUH64+OfIo7EB2Y9P/5L0N6AiLh35fL/f+XGgA26L3SAz+1guhPpEoBIubPotM7vVXyNd0vO4yLplQH8z2yypGfAkLqx7E3Chv8YrwGqgC25L3124qL86uAzm8j5Ww6VMb4Dbine7mQ0pwD1H9jNnh5ab2UpJ3YEHzKyvD1iph/thWSnpTtye6Azc09bJ5kK3z8Jtp8vAbce71FygTw78tqOLABo2apRdH62NkJWVxdnnnFdgLYXQvvD2E74dz9tvvk679h3o1aMLAINuu5Ojjj4m7jngBnX33Xg5mzZsADOatmrHFbfcX+L+p0xABsexKFdaGamiGBobZjbad1gPmtnjki7FqdZd4DvJeWb2hqQauI6yC05r426gPa6TnYLLNr0S+AHX+c2Q9A5utPqG4mtTvIIbufc3syzF0aTA6V30M7ML/b1Utzj59iSNwq0gT5V0rfdnQD4d89+A3v5H5HFgopm9KSkD90PQmBg6IWb2WqLPvKxrZZQHsnYk992fsHhV/kZRRHZlFAeF1cpo27GrvTVsdMxjXfbdo0BaGcVJOGLGaWxI6obTjDgEGCLpen/4A/93GhCZfDsSOF7SNf59ZSAyBBxhPvxZTsmtNy74Y4mZzYhqq7FyalNE3NkpggvvRo08+0Sub2afSopoUswGHpB0L07YaGw+t/umpM24qZB/5WML7gckEoY9AbhJLhz1A/9jEq0TAlAF+LMA7YaElBpujrm0vYhP2DF7LK/Gxtn+UGRlNVozQ7jH+AXRbfgRb+5hSOR9bv2NKiTWpgDYGKetaL8X+h+VY4C7JQ03s9vitAcw0MxyD1W3s3MhuHKuY9k+mNlbkibhNAK+lHQBCXRCQkKCTFC0l2MR7sogrsbGTwlO+RI39xyZ5+0SdewIOa2MKsAJONnQmFhibYrcxNSkkFQP2GRmbwAPAF0T+B2PpbhRL8DJ8YwkNQV+NLPHcApzHYmjE5KCDyEhJYrilCAQdsyOWBobgxLY345bDJzlt8HdHnVsHPA6Ti70/Rij09zE06bITTxNig7AZDk9jpuAO/K5Xry2H5U0Fjeaj8cAYI6/VmvgNYuvE5IUF19wHo3q7UW3zu0LZB8krYnyYD/iyy/o0r41Hdu04MH787efMvZrzj1mf84+qgeDn38UgLb77EafFrXYv0mNHLYRgf6KUb1NmlxdpOTuEItfKyPcLrfLFNxC3xOl7UdZKNHb5TZnmo34ZrR9O2matW3XLu42qqBpTZR1+8g2uHWbMq1Jk6Y2+/sfbPX6Lda+Q0ebMmNO3O1yX8z+3fZp2Nhe+3KKfTZjmTVt1c5eGDrOpixZYxN/XG3rN2dmb5fLzDLbtt1db9t2935zptmWKD+2bjfL2lGyWhntOnaxecs2xCwEYLtcOGIOKRWMnIOCA3sfRM2aNcFiDxaiidZSyMjIyNZSiEdoH9s+PU2kp4npU6fQrHlzmjdvRpXKlThtwOl8Pmxo9vFI6d2iNr1b1KbSmh9p17olpx7anUPa1eOcvw/kt1ljaN+gBp0b1qRqpQr0blGbPveNYtmaTRz/xLf0uW8Uxz/xLcvWbKLPfaM46L5R9PHl6EfHsXLD1qT9LyxBHjGHHXMRYmavmNnl+VsWL5I+VF7tjaNK26+iImhaE6F9fPta1TJYtdFlV1q1cRs1q1bMY3N8p32YsHh1yv6kSqoh2ZIaShop6XsfLfxvX19L0ghJi/zflLXJw465HGJOwCi39saX+Z9ZNsg9goaypU2xq9knotu+NTi+U12eGK0dukoAACAASURBVLm4WNpPSOqrf9uBq82sDbA/cJmktsD1wNfmUkt97d+nRLF2zIqhP1EEbW7wf+PqT0hq7a/3nVzW59jq3aVEIt+jbHpKGiOnbTFf0guSqpaUj1F+DIrarx1dX0/Se/51X0nD/OtzJD1RnD4FTWsitI9vv3rjNvaslgHAntUyWLMpM/tY8zrVuOmYVlz73hzWbd6ZWa1EtDJEIqH8hJjZcjOb7l+vx2VJqo9buH/Vm72K25WVEsXWMUvqBRyHi5briEvm+Uvis4qME4CPzayLmS3O17qIkEt8WhTt7A28C1xnZq2ANsAX5EprJZciq1Qws9/M7JT8LYue7j168MMPi1i6ZAnbtm3j3SGDOfa440P7ANqPWbSSYzvWBeDYjnUZs9Cp0O69RyXuPbk9twz9np9X50gjmbQ/qSLFLsm1oca4qN9JwN5mthxc5w3slbJzxbWqiItS+yRG/VLgLlwU2VTcvtsvgcW40GRw29e+xqUAn40LS46cv8H/bUwMYSBcoMXvOD2KkbnOEXA/MMe3O8DXPwUc719/CLzkX58P3OFfn4ULvY5kj06PtA3c5v/H9MbtBx6Ni+77EtjH23UDZvr7vj+W71H3cBtwW5xjg3AproYDb/nPYaz/rKYDB3i7vriAmfeA+bgMC4ryJZaPV+DSbc0CBkdd73VcUtdFOC2PHJ+/v9Yw//ocCrAzpUvXbjkyfJx62um2d926VqFCBatXv7499ezzcTOAbM40+3Dop9a8RQtr0rSpDbrtjnx3coT2xW+/Pctsx46d5fZh39vhD421yUtW20+rNtrkJavtsAfHWo87R9pH3/1m6zZtswW/r7cFv6+3eb/9lZQ/hd2V0b5TV/txxeaYxfdRU6PKRXG+i7v579BJ/v3aXMfXpOpfsWllKLH+xL1m9rSkh4HDgANxEWdzzWwvPxKsamZ/yUlbTgRamJlJ2mBmu/lfqmFmlmfjq5zGwwYze8C/j5xzMi5XXT+cDsUUYD+c3kQ3M7tW0mRgh5ntL+llXDaCn4mjByHJcB38O3KJXUfjfkhWSBoAHGVm5/k9vv/yn8H9wNGxfPf+foCLpsuzFK28+hVVvb9bfJDM22bWXS6zwsdAO+A3XKDLtbgfkHg+/gY0MbOtkmqY2Vp/vRNxc2nVcHkK98MJFg0zs/bamcXhODkhp+4WYxFUOUWMui34YWms249JUFbLQwpOn/tGJWU/5r99C2xbWK2Mjp272dCvYsd+NalTJV+tDP9dH4ZLH/WQr1sA9DWz5ZL2AUaZe+JNmmJ7FLbE+hND/d/ZwG7m5mnWS9oiJwq0EbhLUh9cFtr6uASivxfSrd64jisL+EPSaKAHbsR5pZ/AnwfU9B9sL9wo8mzi60FkAe/7161wAkYjvF06sFxSdaCGmUVUU14Hji7EfUTrV1QEnpDLWp0FtIyym2xmvwL4oJDGwNpYPnr7WTgtjY9w+h4RPvbX2yyXiLYn7skhKczsOdxon67duofqWSGlSqo/9nInvgh8H+mUPUNxfcU9FDK1VLHOUVr++hM7yKkhscP7NBCnnNbNj1CXklfDIRs/su0C/GZmibQKY/6fMLNlfmtLP1zocy3gNNyoe73/HxFPD2KL7RQaEm7U3yuXfzVILmPuXNwPQbz/sdEaGlcBfwCdcGsGW6KO5dbnqBDPR8+xOLGk44H/SYpoLcbT/wgJKbMkm60kigOBvwOz/YAH4EZch/yOpPNxT9mnpuxbyq7lg5LXn4imOi49eKakQ4CE2gtmdq65LWGJBWRdpztAUrqkOrhOaLI/NgGnTzwGN4K+xv+FgutBLADq+IVPJFWU1M7M1gLrJPX2dgPz8fMJ4OzoXSySzpJUN4ZtdWC5me3A/WPJbwEypo+S0oCGZjYS+C9QAzeHxv+3d+ZxV87pH39fPa1SKrJrSpIlhPBDCGnFZF/HNPbsw1jGNvasYzAYWxhkGGTLkn1CVMg2iCFjydiyVlJdvz8+191zdzzVs5ync576fl6v+/Wcc5/7/i73c871vb7X8rmAX5tZczNbGtmTxy2gj4SE8kYdUrLd/Vl3N1dNzSwc9SF3/8rdt3X3LvH36wU2Ng/UZ7hcTfkn8rgN6GFm45EQe7tIYxqBtuuvImfWCe6emUdGoyod7yEnWrs4h1eTD8LdZ6AquReYuC8mIFpPgN8BV5rZGGBa4b0F7fwP2BPReb5jZm8hk9B3VVx+FRLiLyAzRiEjXXXHWAHcGjubV4BLY0EBLV4jka3/bHf/dH591AajHn2E9dZeg25rduHieuCCKDcujsVtPN9OHMvrl+7Ha5fsw+Rnhld5TUWOPyPj1ch4NppWQOMiSquM9rOuURn1hrp4NtORjtoe+aiM76f97J1WXdXffPs9/+aH6XNKCc0rKqOm3BHTfi4vLo7FZTwbnfuUb3TuU97j7Me9WbsVfZ3jbvMNzxzlLZZf1bsdfeOczzc69ykfcNlzc/FlzAyOjZ9m/vJcMaIy1u2+gX/6zU9VHpQBV0biY04oCbJimADjx42jc+fVWLVzZwB222NPRj54/zzLCeW5FLLrH3zgPtZca60517jPbQbfvOcWfDhp0hwujl+MJ6cqVaf9hj6ewjEtaDz5MVW3/SzK4oUxYzj3X9144Py9Abio4kBgMsefMHiu66dMncGpD01k2s+zOGarTox6+wve+KwyN6zvGsvQqllj7nr1MyZNme+ms1oo50iflJJdQphZX/slp8WIUo9rYaO+uR3SeBrGeB5+6wsuHbQml++8NlNnzJpLKFcYbN6pLa99+n1R5mDz4Mmog0OwqEgacwnh4q9YZDgsaosFaYx1vT6Np2GMZ4OVW3PcfW8xdcYsjtiiI5t1bMPzk+Tm+O3GK/PO5z8y8Yv5ulBqhKQxJ8wTVg98IrUYw6RI5KlrO93NbEGRMb9AfXM7pPGU/3gaGXzxwwy+/2kWsxzGf/QtXdq3BGDQOsvRqlljhr9UXJ9zqmCSUCVsIfOJWJG4POaD7iglvkaob26HNJ7yH487dF6mJU0rJBrXXn5JPv32J7bq3I51VmjFVc99WOTg+aoJjMqmDmCpvY+L88H8+USWidc9UGonKOnmMRTOdw2KC8+uuxfl7b9JLrefAi6PeYxjEnABCosbC6yW6+9uFLc8Dtg8zrcEhsW5VxCrVlMUVP8FCsHbY35zL6xgUmxuh3xERzlwcZTbeArHtKDxFI6pPsZzz6uT/ZNvpvlHU6b6s+9/5b8b/qrPnDXbP/tuuk/6aqpP+mqq3zVhsv/m1gnertNaXpffXvf1N/Svf5xZ5UEZRGXUG1dGwoJh8+cT6eHuX5pZD+Bid+9lotP8xN2Hmlk/4GGgfVzXzt2/NhWBHQds5e5fWY7LYz7jmARc5+7nmtl+wO4u3ovhwFXu/qyZdUC8AGua2XnAv9391shqHIsyL3djHjwZ0c9cXBkT/1PdfKOao6bf6/q2N5bbeKD8xnTwHa9W+9qRp+3NV++/WesBrb9BD3/6ubFVftZmiYoFcmXUN5Lzr4Tw+fOJVIWeiFAId3/EzKbkPjvKzHaK16sAXYCvmJvLY364Pff30njdG1gr94NsbWatUDHYHa2Sp7k50GFBHXiOK2PDxJWRUEqUUzJJFUiCucTwqvlEZlJp/89zhFT5VTKxu/UGNnX3qWb2dO6+PJfHfIdSxetG0eZcQaPBHbKLu79TcH6hOy4TEmqDLPOv1vdrx3oZypi93t2LWso7Of9KiPnwiUxCJEYAu+Q+fxaRK2FmfYCspthSiPt1qpmtgSg6a4o9cn/HxOtRwByzRDDYgUL8jgwBjZmtH+e/p4DMPyGhXFFb51840a9EDJFrAXuZmCmLN7ZiNpZQY8yLT+RM4DIzG41MERnOBPqY2cvoSzEZCcNHgMbRxtmI06KmaGZmLwJHI8Y6EOVpjwjl+zfisib6aAK8ZiqRdXacfwqZPiaYeJ4TEsoWdeDK2Bh4z93fd3HP/AM5wIuGZMooIdz9JSpJjvIYzdy8yhm+RaT2MyPUbmt3z6g9q+R3dvclqzpfcE3HeHlmwfkvqdSk8+enAYdUcf5rxG+dkFDWMKhLaNxKzB3W+jEqHlE0JMHcsNAB8b02AmYAB5V4PLXGyy+/9GWLJlZVWMYywJc1aCpdv3heP18q4AXh5ZdferRFk3kmVTUPZssM14bjOkNVEr2ozuwkmBsQ3P1dFJZWKwQPR6eC0ye6UsMXKty9fVXnzWx8TUKV0vXp+trA3fvV4faPUeRThpVR+baiIQnmxQjuvtOCr0pISFgAxgFdzKwTKvq8J7B3MTtIgjkhISGhBggfzxEoOqkCGObubxazjySYE8oN1y74knR9ur60cPeHgIfqq/2Ukp2QkJBQZkhxzAkJCQllhiSYExISEsoMSTAnJJQ5Im49YTFC+ocnLDLIuDuqOF+j73lhO/Nqtz5hZsub2WkA7j47P4a6jGdB95ZqEQi62pLDzFqWegyQBHPCIgIzM3d3M9vBzC42szkRR+4+O6757YLIZrJ24nVXM6vwhewhD+G5AtDdzDIekqY5odqmtu3m5tbTzHqbWescGVUzoF+8XjtHI1uvMLPWwGgz61uk9poHjzlm1sXMulXzvqWAc81s12KMoy5IgjlhkUAI5YGI72NUxJquZ2an5C7rD0xdUDsAZvZ74DRg+eyzhaU5u/AKqhKzoZmdjNLvDzGzrYG7zWypmo6nYG5DUWLEo8C6cUlTYDkzewG4B6g+c30d4O7fAX8BLjGzzWOMVvB3XVNNyY7VaHIFoKeZ3YK4yL9b0A2xKP2A2B23M7Pi1eKqBVIcc8KihIHAicArIaQPBdqaqrgMRWx+bc3sw0ItuECb3BPRqw5w9ylmtjTwvbvPyF9Xn8iN/1uUVbYS4th2oJe7f1ubhcLM1kPkV1uY2bEotfh1AHf/3sw+RgRarxBEPbFrqA6nd42Ra/tZ4DNgpJnt5u6P5XZBvwZORqyJK5nZVe7+5Hya/RBohhae6939v9FXo2z3VDCGNsDVwBVxHIQKQeDu9xdxutVG0pgTGixy2lTHOPU9MBi4C7HcTQO+BtYDTgDeAaYTPNZmtoqZNSoQyi0RWdSTQEczOx24A3jQzJovJKHcCjgJUcCejmopjqVSg70iuzR3z7zs6/lrWqEU4vFmdjWqRLM94Ga2q5k1dvfHgG1RqbPbzKyju88Km3fR5UW0vS0wAi2e1wC3mFnvEMorAIcD26D/30rABKuisHA21xC+wxCX+Ddm9sf4380Os0n+nsbu/g16viciLprrUP3MHUulOSfBnNBgET/c/sDf4wd8PiqNdYS7n4F4ctvHuY0Rz/T1wK1m9iAwHGiZE8r7o6oUoxAd68VIazwGCceVF9LUKhDf9TLA3+P9WKA1cAPivH49BM1GZrZeVQtGwYJzIHBefNQVWAPYz91/BvZDxXhvMrNLw4zyV+A/wEVmtjtwCbBiPc13XeAxd3/C3U9EAvIuM9sOLT4fAAegncNvgl52MzObQ4SV0677mZym+6H/73Mx7kNNlX7+ELZkzGwZ4HqTP+J6VOj4DOYWzgPNbOd6mve84WVQLTod6ajNgThw3wU2ivdNc59dgzTmfZEJoCfSoi7IXdM+93oP4EZglXjfBGgSr3cC3gCWq6d5ZBm4XZBjrxFwIEr5vRLZSN8CngE2BdYGfkSa/MfAdgto/xAk3D6NZzEAaahXA1fFM3wLmW/uBsbGfc2BU5CA2r7Y88293wUtOAZUxLkX0GLYDrgcLRLd47NtUSX21Qra2RZ4DQnWL5G/YQnkW7gSLbLbF9zTCegMdIj3hwMj0Y6rMSoccSOw7EL9bpf6x5WOdNT0iB+wIbPF2cByqNrKWOCB+IF+BOyVu6cpsDlydv0hayP3+Xmo1uI68b4CaAHsFUJr7Xqe00BU0msoqgF5C9L2pocA/hkV1z03hM1ywD7A/1XR1oZAx3h9MPAiMgFcB7SL8+uGgD4V7Sj+mLv/n8DzuffZPVaEeWaL0HaEQxPZg59G2moX4DhkWz8gd+2fkYnqcODfeQGb+z78GRU23irm3KGg745VzQM4K9rMFuXDgftQibbGhe0slO94qX9k6UhHdY/cjzrTqjoA7yN78BHAmsATwP7AxXHNEkCjeL1U/NhWyrW5YvyoG4VQHAusGJ81jzY71vO8VkNa6fLArUhD3h3RS76AzAyfIBvrW0jT7TyPtpoCv4u2KtCuomN89kAmyIENkKlkfRSBcTuwRq6dB4HX88+7iPPtg+zlW6JdzcEokmI4Mj9NB46Ka7sAPdCiuj9wGHJ+ViVgjwJuQo7EznHuQOCweYyjWe71H/PCHJm9HgGWKsV3PUVlJDQI5GyIvYFfR0jXSGAdJHi/N7POVFa82MvM7nL3F+L+/amsaJzZXY9BGtbXyEwwFGnNt5vZvu7+ERKE9TYn5IjsAjyMhORWyD76M9LYd0Y27r2RsB2AhNgvogsAXJEjN0ebtyB7+6Rwlv0MVITN9GzgWGTiOAi4FBhgZrPdfaK7b29RZNeLFJER861A0RL7xDxeReGNk83sSlSZ50ugtZldhhaQH4Hh7j4s31Z8H9ZHZqfP0CJ2LHC4u/8nIlCOQjukzKa8qruPNbMBwJ4RJjcULXazgeHxv7/UzO5x92+LMfeaIjn/EhoE4kfYF9kbn0Ya8iXIvvx9/IhfQfbST5CZ404zO8TM/oBshWNyQnkQMAhppusgTfI7ZJccj5xCFbUJSVsQcm22QlEXvZEWOQzZTm+I182RoFoW+BuKOukAnO/uH8yjTVxRCd8BzwPnmNmaIVwfR9v2M5Aw60SlmeMvQDdgNzPrGk1NKOa8XZhJ5f/nZOC3sXD8CfkFZgNvIqftI+6+BdJc18/PM74P/YDbkP/gPeBz5LAdYmZ3o2d2qruPMrMmaAHa28z2Rrbzm2IshwPbuPsFSNu+K6I4qip9tnBQCjU9Hemo7kGl+WJ5JJRXRzbkN4ELEV/v2UiInIIE9qvILtkTaYJ/AdYsaHc/pLUdhOzOzeJ8tpVtX8/z6ouEykVI878dRQV8AUxCW/wXYx7/iWsfpwonHHPbytdD0SNNkGA/DTn6OqJKzrORZtkRaeMvASvEvesiU8qq9fD/WxtYFTnzdkA7kx7x2SC0a/lbvG+Uu28jtOD2ybeJTFBPIDPQdsDbwDLx+YqoJuBqBWPohhbevwNX5No7EC32reJ9l5J/70s9gHSkY0FHCOJBSFvsgDTawfHjey8Eze656/sje2rzKtraEtkrewEvA0/lPjsOCf/G9TyfAWhh2TSE8e1Ic3sshNDIEMwjY4wHxX1Lxd8qnXDAkfEsLka22mbI5nwqCgFsHUL7fGSvfQ5YPe7dBZlVWtbDfPujJJaTUPJHK6S5j0cRD2+gyvAjmdvOvSGyf/+6ijaborC6w2LOXeL8boQTL3dt5mOoQJr4UGS62jp3zUjCdl0ORzJlJJQ1zGxdYFfgU1cGVxukPW+KNK/XkDDbP3fbWJRe27KgrRZIa9wACaUJwJNmNsDMfoM06Gtd2+36mk8LtIV/CWmvbZGgegKZH1ZEccarI6fYX4FNzKylh73TM+ls1jbX7q4o5K9PnNoYCeMmSFC/CLR294/R87se6OfuE81sY+T8WtndfyziXM1UF+809NwnIXNMY3c/Pc53QWam36BFdiczWy2aeAU5Ae/LEkrMrG2YJX5GgnsosJW7v2tmGyJhvXR+DK547/4oquNHZCoajWzq+5r4U9ZAUS/lgVKvDOlIx7wOJIR/AO6L91k0xu0oHO4rZC+tQOFNd6Dt+17Ivrpcrq3s3k3QtncdJACPQOFhNwHd6nk+a8TxJxQO9glaIB4Djkea7CtIkzwAmWImADtW0VafuK5PvN8AOdMOQaaZtaKt59Fu4yUkkM5Hqen3APcjgfwKVWilRZpzthDtgRaHTLMdiMwQ58Y4lkcRMJcgbbprXNcRWDdebx9zvjXuaxRjvxYJ6AlVzQPZ8CciAZ6dWybaeAdFg/Qr9fd9rjGXegDpSEf+oNIeuHT87YPCp3bLXbMT0gLPRxrl9igs7i60hR+TF7LIfHEY0giJ18fG62yb26ye59UUaYjHoGSH15E9eTqKnpiJ7J/9kDb3ArIr75h/Lrn2jox7RwE7ZHNBDrRewI6IV+O1aL8P0qJvQAtbY2AI0lS3qqqPOs53Y+CGeD0G7QayxXF3ZEpYHdmcT0WmhPZowbyCynC3w4DJyPwzLObRJdq8KuYxGO2YehbOIz4/HdnTW6KEo2dRSOHSKH69a6m/9794fqUeQDrSkR05obwx8BSwR7zvi7bAu6KY1nvjfCMUDnU5lZqjEU6cXLuDkSb2GtKefh/CqdVCmldnpKVujjT7E0PofhDzegc5MF9Fi84SSJA3zz+XgjaXQY7NY5H2mz2ra5FGfhLSLn8A7so9r+bAneScacX+/8XrpiFst0Vmp4/QonkUYvi7t2Auw9EisyywZEG7R6EswL8XnH+eBThD4/2OyLn4KNohHBDfhfZU4YcohyPFMSeUDdznhEAdgJxfl5rZdJeNcSf0w52E7MIVLgKcYcBvUdzyEsjs8T2AmW0KzHD3m+L9fkjjaou21u8hLbbeYCJYuhMJlsHR52Zo+74iEhAe8z0JaYGnufutWRueSRvZ23H315CgmYFMFlcDR5rZ50gDPBBFQRyATAEjzKyPu48CppvZZ9QD70X8//qgxfMSpKn3Q4vhi8g0swSKYz7NzM5y99Pd/UszGx+fd3D38THfRu4+290vN7PpwClmtpG7j4sun0b2+TnIxTf3ReaSycj0sRliCPzEzFZE0Tgt3P2LYj+HoqDUK0M60pEdyEb6IrBZvN8fbVl3i/f9UFrxMyiutXGcb4UcZXmb8lHIHHBTtJGFhC2NbMt3EBEJ9TifLATvIsRTcTUyZXyOhNMuSBAPBe5FccUDgC2qaGtp5Cz8L9o5ZFwOV6IdxZ4oEmVgXF+Rez57IGfZ8Uh7fZNcREKR53wCMlscjswq36C4a9Bi8RyiGu2AbMLno9DF0VRh4ydMTfH6WJTwcwSVduNtqrinH9ot9EUL4HAqTVZ7xfx3KvX3fb7PsdQDSEc6sgNttW9G2W+ZWeMsYEr8zRxHJ8UPLi+c89voHUMANEFZX19kAiF3TdHsqfOYy/oxl+2RBnkpcoLtjWzDHyHH2GBkijkKaXDzHB+ivpwdz+K0EDgXAfvG579DC04rKheFZZBZYYe4dziwfj3MtxtybLZE6dx7IKa+mUhrXT+uG04l6dRKSLMeBgya3/ci93oIMv9cQsRBV3H98ch81B8tzhkHRktkWum3ML4DdXqepR5AOtLhPsc23AI59U4i+Czi/RQUgfEEsHecPx5pzusW/sCQdrky2so/HOceRtpW1m69/ShRGNduVJowvkW23wlI090Q2Vu3Q5rtkciJucCxhWD5MOZ4SDyD20L4bgqMzF3bkVxSClqw6sO23Cnm+x5ahI5Cdu+myJzxNVqQDgL+xdysfnlGuWwxXpVf+gnywvkI5qPxo2iLMfFsMofvAOCQUn/Pq/1MSz2AdKTD3aGSYnP1+FFfhaIspsQP9cUQrNfnhPPRzK0Fr06OChJlBg6J10OQQ3HFehj7aii1OHufMZPtjrb176Ftt4ewuANty48NwdwMbfN/X83+BqCojiXjfafcZ/cDu8brm8ixxuWuKWb0RVtkNlgjxnUvlZl0g+KaE0JQvk5kFeYFbX5MKHPxWWD5KvpqNK+5xH0ZQdMKyDSSEVltFc+/d6m/59U9kvMvoWQIZ9YP7v6+u/9sZqsijfAUJIzXR3bZ36Ct8jpIuzwhHEOX5do6HtkdK8zsVcTINhHYMpdAsJ+7f1oPU2kF3BjJI9fEucaIT/lMFI3RDGnO9yFhvAfSnv+HohcuRzHZC4S7PxSUEeNMNfI+hzk8EiPRAgXSEH+Kz+aUVfKQVnVBjoT/W2S6+FWM61MkJDsgB9/D7n6hmX2ENOY2wUMxvWBObmbboPC4c939MysoBeVKFGnsqufYDGjj7v8zVUC5CvjWzEajuPRjgL+Y2b1o9/R7d3+8rvNeaCj1ypCOxedAHMK/QkJrKaTdrRWftUGa10lI2+yItrkbIxvxZXHdQcjhlXf0bQs8FK+vAUbE685Ia/0b9cSnTOU2fAO0Zd8ZCYkLkEa/M0poeRoleTyOYom3QMI4izb5hcNvAf0ayqZ7He0khiAbdHvErbF5/tp6mHfX3Ov9UKLHkrlzbWOOedv/IcjWXyWxP4o3nw2cHe8bFdyfPes2KCmnYzz3jAukNbK5n0uwDsbzyGzMZWtT/sWzKPUA0rH4HIhM6E4q+YGHIzPAWiikqT+KxHgLOW2ORNvS/iic7DpEaL4d0tKybexWSEM6EzGRZY6vdep5Pln/a8TfHsh04cimPQ3xRT+NnFVjkTNsFLKnL4FMG1vVtM94vV08m/7IhjsO2V/vBf4Q1/xi+1/XOcfYH0KaaR8Uo31SjMGYu5JMYUGCIcDGBc9vKSrNMr2Qc3Tv/P0FQnlUXNccxSV/R2V2YAe0KF5BFUUEGsqRuDIS6h05qsZjENfuWcGh8CNy3vVEAngXJGzWQdEUayIzxnNo2z8BCYBJKPzrjmh7MnJsbQLs7O4/mdnhwOVm1ro+qDtjPm5mhwD3mtkGSFMdF3OcgnYAs4CfUNLHILRwLIuiJZZBVTqeqU5/php3fw1eYYDuyEn2sLsPR3G72W/6CDNr7VVUha4Ncs9wCXefhrT1x5CW+3z83Qf9/+bwN3vAopCru1/t4kPO4o0HoeiV282sr7s/jYT95WY22Csxy1TN+m6kUT/tModcHvefYWarufhUrkTfjynFmHtJUOqVIR2L/kFOM4q/NyAzxvso9OlfyDn0LtKIs+u3RGaBPyEzyEAUg7obigS4Gbgzrj0G0TmeEMfr1IP5glw1v2l+cAAAH45JREFUDyo1+T9ln6EFZjekNR+FOCBmA7dnz4Icg1oN+26HnJ83Is3xRKI6B/wiMuWvwDFF/v9lREAnApvk5tMPZVL+jJJj8vfmoymy0MZmyMSwHVrIlkORJe+hYqsgs8xUlAiTactnIPPIQBQyeD4yi7WI93dQybFRryn29f6bKfUA0rF4HPGjfpDKRI/bQ6idjMK8eqFY25GEPTmu2xbFAO8QP9xNcp8tGT/o2+L9QMS7cF5thd8C5tAe2S+bIPPL22hRsTjXMV6vg+ye/ZAWPzTmUVHQXrVKNiHn1T3xemkUH3wdcA7KemyLzAGrUbn4nQZcWMS590b0nBujsMVHo29DCSPjkVnhMBQOuCe5yArmNkU8gkwOQ+J//2u0KzoC7ToOjWvbFoyhUXwPXkY7qPHx/JvGcQEy48wpJ9ZQj5IPIB2L/oEcNG8TGX1x7lgUFnUjlYTmFSHwrs0LFaQRHQscHe+b5D5rGcL55ty5otaoy7W7InIoroJMJ7NzQmZ8CIyLYk7DkdPxIyrJdSoKx1+Dvsehih6gBeJStFWfHP08hXYTa8YzOZ86sOWhyI7fxkJQgbTVbkjLfQX5AT5CpECG/AevoRjrzP58MVXbh/fJ9dMWLWJrx/u7kYM0L9QzO3OL6GcdZBZ6EmU7vk5lYYCSk9wX5btW6gGkY9E/kPZ0Vby20G6uQBrf1SgsznKf/wZpZefnfpRXAOdk1xS03xWFof0j3tebthQC4EIUa70jSnz5BoWs3YkiM0aFUNuqCqHcBi08v4jTraKvTsAJufePA4/H62Xi+d1FpbOzbe7apnWYY0W0+0UI3lbxP2uHnJoZ898PIRQ7IA32mZh30xCcV+T+r23ifzoY+BjV8Mv6uwntRHrGc92kijH1QYUM2qKdwTgqK5ZMAcaW+ntezCM5/xIWBj4EVjKzri7MQMxqA1Fx1FeQf6nC9UtbAWmdf4nrHWlV/2dmG7rLmZQ5lJCZ4Og48CI5vKqCu/+MtNX3kfA5HZkRnkfOzJOQOWYrYLa7P5sjXFoq5nGru39Wje6+Bp4ys1Wi794AZvaEu3+JTBpTgGFm1hT4PudonVGHOc5CWu/nyCzx65jjbLRYdDGzgTG+zmjBHOfuW7n7zSjc8XTg0fjfgezSj6DwwMuAnmb29/jsJrQb+RuiCn0xPx4zWxP5EB5x9ynIBPY60NLMNkELwHG1nW9ZotQrQzoWrYNKDWkzJDA3j/dXoUiLQchO+QYSvhcCG+bu3ws5AjsUtNsSbacLr98Dba1Xroe5tKKSerN/9H9UvG+HBPT1yC46ndCCkRngBGRiaRvn2qCwuZ7V6LcLsEu8zsjgr8t9/jiVZo1liFjwIsy3OZWafQtkVrgDcVkMjvO/RYvST0gYdkNmnFvi847IKZtl/WWVsTMOjX3ifBNk+hmW6/9X8bcT2jW1Rmajq4Enc9etHM/9OkQOtW3+u7coHCUfQDoWnYNKr/sAZIPcH231eyNb5fFIE7s3BPRKSLN6BtkjzyGqi8yj/XldX/TKIyGY7g8BsXHM57gY+93IfNInhMYdyL76U04Qd6EyXrsJipLYshr9rhltHUuQGiGb8gvMXUB0DLl6hUWYb0cUEXMmlex+2yEmu0GI22NwLASbU8nxnAneD1EsemOUkTdHUObmcRGRLh7vuyFzSN6fsBbShs+JzxvH9+j++Ju19SuUzblBqb/39fJbKvUA0tHwD3L8EyGQxsbfHREJ/AyiWGoIqUx4ZQ6dnkgbPYQFUHHW9PpazqddCJveyC56FZVhXE1DKH6NNNc7CD4KpNm+NI82F1h1G4XWvUiQ3hd81haZS/IRK7+wxdZhzlugzMTn0G5mT7QI3ofirrdB/BPPIzrNUflnj0wNHxf+P5Cp4xFkqtgt7s+cveuiOOS3UQz7ymhRyp51JtgbI/6Nv6AswxbFmne5HokrI6FOCJvmqWa2hrtv4yqKuQ8Sbme6e9codPoPM2vm7rcQgf+uX900FJ3xbHX6cyU3VPv6WswnI55/BsXGGloEppvZPUhI/ICE13S0bR9oZlu4+/pmNt7Mengl2bu5UB1C9uWBz9z9jrh3EBKIM5Gmvh0w1sz+6u5HeIEtto4Yg0INuyPnWmu0oG6PNOUR8f5ad3/UzDYDbjCzK9BiuyXQ390nxtjzXBoT0I5hPxSzfI2ZfYjs8ANQ7cPpyLfwpLvfEv4Dz3FjDEO0ppsDjczs5mh/kUQSzAl1gru7mf0B+JuZjXD3nUI474FsqhBeeIJsp1wRTqbbUfr0bS4n2GNm5khoZTzPbRER/M+IKKknEjCj3b1Hvs0aCo//AV+Z2VnIfPI9Wrg+iP6GoHjvrrWcYpUIITrTzJ5AQrYNMj1lRFCfo3jzdshcg7v/KSqhdEcmh2uBL82sk7t/gEwjH7iqk1yEkm1uQ9mBI5F2fE5ctw8ymawKbGNmbdz9mxheVrF8FWSjdmDcoiyUoXKrkJBQI5hZZ/SjnOXu95pZc+SMaeXug6Ks07HIUbQ9InN/JadJlRVCQxsK/Nfdr8ydzzS2wUjjuwXZfDsCl7v726ayV9sTwjoEem3HMQglpSyDtu7/jsVvBDDU3cfWtu1q9t8U2c6zLMsb3f3HSAc/Gy0ed7j7vwvuWRWFup2FFuSnEJnUyXFNe6Q1L4VY7z6MtPxbUPbia5FyfSGKgX7G3WeYWRMX8+BxwGRX6vkijxQul1BjmNnqyPa4GXCimR3s4i04BPjRzO5w9zGIs+Az4DhXSFxNNciFBleIXRtiF5kLO5tpZgNQKF5b5MB8Fy04/zCzk1B0xp3uPr22QjnX373u/kd3P8jd3wyhvC5aCKbVaZLVgCvM7lHkbNsBmU6uRcL6dKTp/trMuuVuWxHFPV/i7v9whQJuCwww0bESppxxKM26fZz7ABjgqmFIaMmfIg26p5m1DKG8MSp68HH9zr58kDTmhBohbLC3Aae7+wNmliUgPO/ur4bmfD3Q2t13zN1XlppyHmZ2FMrKuyTjAjaz1ii+9jtkh90OJVmMRhEEKwLvuvvoehjPUsgOex5wsrvfX6R2F/i/MLPeVIY4bo0iSjY0s66Iu+QdRET/o5n9Duju7kfHzmMD9FzWRmnWf0bETgegwgVvRh8V2UKWH5OZnY3szV0RUdLewInufl8x5t8QkARzQo1gZj2Bf7l7o3j/GnLerAC84e77hnC+Ef1wXyrdaGuGML/cg1K/7zRV7F4VcSovgbLgNkHa4/Mo0/D7IvRbpaAMTXlvZLseWYzFLcfq1gtFivyzqmtQIsjnyPRwKqLh/CAWi6UQJ8frcX22eJyF4spbIDPXPSja4604d0+YvZYPrbpQOM8hxjezX6Ed2XfAl+7+YkNY3IsGL4PQkHQ0rAMlW7yPwsVOj3NNUdjTyfG+Xvgq6nFOmZKyPRJI56N6fTcih9djuWt/g5yEK9WxrwUyoFHJUzwXr3Ed5zoIhfb1zZ1rxNxMcPujxJlnqYzH7ovC25oXtLcECpebgEwaWyCzz/rIzNMyN+cKlOl3f+7+PGNfgyYfKtaRNOaEWsFUzudRxMmQaTkHoOSCS0o6uFoip00ORlEEjrLTRqNKKJ+4+yFx7QruPrkOfQ1Egm4acoC97e4zC67JzClztMq6wszaIufaYGTP3QSZIa4OzXdVZJZ6x8xuQdEmWyAb98WIgP+hebTdzt2/zr3vhTTpXVAYYGaqaIEWts/d/eA494s5FmjQi4+2TDJlJNQB4RS73N1XM7PVUDjTUe4+qsRDqzViG78NSqn+L3Ceawu/PBImk9z9d3URFGa2BXKM7oqyCMcAR3iO38Iq+TXaIp7pvb2WZpPcgrMkCvEbHUd7ZJZpjgiYtkS7nm9iTHsi/o+fUFzzpfMSygX9NUG2+KFoBzWyYBybIq19V+CJqoRzbv6t0M7k7drMvcGi1Cp7Ohr2gTiHp6LQqn6lHk81xmv5vwXn1kU8Dc2R42o4cChyZDVCdvSN6tB3xkNxAiIG6oky/TrE+SzdOEttz2gyty7CfPsSqc+oBNaVyKm3L0rFnkSkNwMHI236rFw7S1WzvyYoCeRxYIcqPv+/6GtbJJjvIXg2smfE3Ex8TwE9Sv29Wejf01IPIB0N/4gf2U6lHkcNxjswhO6QnDDaGtFc3hyfrYocWLcg+2mt7MnR9hzqy/i7C9pdjKWSuGcfFO2QXdsWRSTUqEjrPPrvTSTCVPHZjojnYmo8j0ZU1k+8HhHfz2V/rkZ/TagkdCqkaN0GuCheZxVfXkDx0vnrlgrhXuf5N8QjxTEn1Bnu/oS7j8hiccsZEe51DCLKWQk40sz6IEa4HVEB2JdQcsds5NjaGgmR2vSXbd8HACNia/5RtDsMmGaqF3gikdFmZhUoGWOo1yEMz8wqIvljb1Qn71kz28PMrjWz30fSx5FxDI6/26MsvvNQdZCn3X2214BK1d1/9nnTmn4H7GNma7v7LJedfjTQLaJQiGc0CqX0Fz0MsSEg2ZgTFhuYWQ/k+DrJ3e+IRJljkUb5ExJMzyBt9SCUTDIEeMjrEBYXMcFXoBjep+PcNojUpwMynVzmEaccscCtvTItuab9zWX/NrOdUZTJJCqrjPwW8VM0RruCp5CD7xzgXHe/u64Ot9yi1BcJ/MnIlr05SvE+JC49GjkV/xOLe2/gW6/nLMdyRhLMCYsVzGw8gLv3MLONUKTB+8gJ9jDKbpuK+BwmoC32+Dr0Z0jwTEKZb9ui1O0rEOvaTFR5+tN6iFPujbiSJ6Fwux/dfaKJx+RvKLOvNYrMcERg3w89g61Q/HCdig5ELPg5SBAfBkxz973MbAgyKTUFrnb3Ebl7ihaF0lCRBHPCIouckOqKCHhecvEvPI2iExoDryJHWGOUSPEtlWaGMa6KJbXqN/d+V5R8MRkJ/KmoIMCB7v5uHaZYZb9m1h9xTlyEtNIn3f20uCYTyl+6e5c41wOZLs5192fqGgpYMKZjkD19NWS33s3d/5v7vLm7T8/MYHVdmBYVJHa5hEUWIaQGIYH7DTDZVJapl5k9iXgfVkbVQB4xswOR4+874MW6COUwX/RB4WbXIzt1Y3f/xMxWQhSWRYGZLevun0e/zVC0xyBEI9oC0ZhiZisjR95lQC8zO8HdL3T38Wb2BnJ2PlMMoRwhgTMQl/OtyFS0c8x/IGKLG0YlW10SyDkk51/CIotIHz4MscJth+KQ+5uIdY5CtJq3AIeHA3A6cpRd6u4/1abPEI7bIOrQD6gsDdU5hNKeqIrL+cXQlkPTvMXMboj+f0Ia+bAYw45hJjkVRX20Q9r0ZUBnM7sxzB590O6hzsjZ7j9HWvss5Nj8JJJYLkX8IjOSQJ4HSh0Wko501NeBEiheIuJgkRb5TZx7HTn41kMRCU+RS1GuY79/QIx62ft9kdBri+y628T5YqVYt0URFFfH+00Q9ebh8f5IpJkegUj+T0DEUzugRJIRwFZxba1T6dEi1CX6ODPONUMFXZ+LfsYDA0v93Sj3I5kyEhYZ5MwIHVBMcju0nT7DzC5F2vCfUb24O1Ga9ZdIa26G6hPWut/cqZ+RwM8+u9XMtkbcGA9kFxXcU9M+M/6JH1Dc8HbA02Z2CXAKYgDsH6ac7nHuVZRMcrsryuSB0Lh7IV6LZ7wOTjeXo/BdM7sS+L2ZXeGq5v1SmDaWQ/HQn9S2j8UFyfmXsEgh4oXPRcJxWcQbPQVpy9ORRrevuz9soiw9CEUvzK6NUCpIM14aaabPIM3wYeA0lGl3Dap8PbGOU8z63RJl6D2ChO6m8dFjwOPu/kczW4KouI3szWsD+7nC0g5GdQvvQxl4G6L086+pAXLz3wzRoL7m7mPN7DyUNLOxu//PcrwXCQtGsjEnNHhkHv3QIvdF4WgtUSLHWGRvPQnRT45Ezj1QTO/nyClXK00xhFJflJSxJYq+uATYCPESX4eiPk4qllCOfv+FQs2GIT6Kb1xxz72RY++faMfwDqIoHQCcGkJ5XWTe+M7l4LwXOKOmQjnG4Wa2I5pje+BSMzvQVblkOPBWOCeTUK4BkikjocEjhEPGndwExQu/hOysvZE99TbES3Es8AczOxzVqjvDVeC1VjAR9hyE6E9HxLkxKJOvL4oTbuXuHxUzTjne3oaiPo411d/b1N3/bGYXIFNNM+TgHILKQh0f814FOMXdR4UmW5f5t0R81dsiHoxd0U6B0Nobo+f8ZG37WByRTBkJDR6mlOabkDDaFwnm7khLboe0xZNcdeZWQfbYr4EPvRZ1CAsTIMzsauA+d38k3q+JHGwH1IemGGaTlZFN+HMzuxzNaWmUytwKRUNMRLuH7igypTHSat1F61nXzL5lo/9LkQMyMxN9ECFxk7yyWsliRdtZVyRTRkKDRmzLDwOudPdzULxwW2RX3gVFXxxLZb247igl+GGvYR1CM+tkZku56Cjzu81/oyrhK8X7lVB685J1mds8xrAVMl8cDFxp4sDui3ijJ6E5r46E8heI2P5lFBGxlLu/7e7vQJ2dj8sBp8XzH4+chxeFUN4ChcTNmX8SyjVDEswJDR3t0da8h5mt7EqOOBB9tzdEtJ1rArPNbHPgAuBur12ccmdgkpm1cRVpbQrg7lcgjX2Umf0ZpVtf7O7fzbupmiOE4B+BQe6+HbIfXwSMdPd9UNhfLxSJMtiFr5D9dzTSqOvSf2bLXxLtOGYih+L9SPAfYWbXIXv77939xbr0tzgjmTISGhRyUQDrIvvpZCQwT0GxvPe7EipWQPzKqyON+uW4bqi731fbrXXYsq9EsdFTLFKK47OBiMe4wpVNV7Ttu4nYaE9UIPVEd7/GVAR1MNKUR6BdQVYE9WjEp3xV3N/Ea5HJWMU4NkBcGhegmPD7gD3d/WUzWw9FwnzuKsybzBe1RBLMCQ0OERN8J8qga4IIctogk8XLwD/zsbKx7Z6N4og/LoJttT+i5dwoi2SI8LXdkS37h9q2XdBPtgi1Q2F401C9wd1RtMcUxHHxbypD9bIiqFsCawBXufufijGeGFM/tDv4Ei14fdHid4S7/69Y/SzuSKaMhAaB3Da6DXNzJ49HdtRvEIfyphRwJ7v7/9z9C3f/ON7XSRtx94dRFt34GNPaiE70iWIJ5egn4/q4FUVgHIxMErchro2Vo9+N0MJzFaosfRfaPfRCcc11hpmtHWGBjyJH4pvInt8E/S96FqOfBCFpzAkNBqGp7oUE88nu/qSZLYO0yIEobO1LrwN3ci3Gcw9ipDvU3e8tsvliA6SZ90f189ZCGmozFKK2J+JVnhm25Oy+XsjU0C/MLcUI0zsAmYbaorhnEK/zf5D2frW7p5C4IiEJ5oQGARN38tnIvnsosqueG/bk9gTfhdeBO7mW49oGlYy6p9g21dBQVwc+A36PQtHeN7HE/Q9YtsBkU2UR1Fr2nZlR1kLJOu8jTudtov0vUJbfwYX31LbPhEokwZxQ9ghH3nDgVXc/JpIarkU21gsieaMozq06jLGYJPfd3P0NE0/yn1A69R4hlAchk8ae+aiPEMobI87jyzzHyVGLcTRy99nhzLwMJet0inZvM7NNgANQQdktEVNcyuwrIpJgTih7mGrADUGmiiNd3MktkLD+HDiqluFvZYdwrv0VCb1JqCRUI5Q4MgOFx53k7g9WcW8TYGl3/6w2C4WZtcs5M9ujeOmzXdwXe6DqJje6+7/ChNTMEyFRvSAJ5oSyQ05zXB9VUf5vHINQnbwrXOnELYCu7j6hhMOtE8xsRWQznoTC+UYAh3kUITWz1ij64VcoYeN2d3+oHswmzYlEFHc/Jc7dDdzl7rfH+5ORc/HXvpiXfqpvJK6MhLJBJGz8HEK5H6q88RDaLl+OCIkaAX8MwfQoqsvXIGFmawB3A2cgm21jYLy7j4645abu/p2ZXRwJLS3d/Ucobiadma2DsiH3Be41s5kRYvcvoIuZdY/F73HER1KByO8T6glJMCeUBUx1+Y4DRpjZ28D+iGviyXCw7U0RuJPLBWbWEYW1/dnd/xnnvgB6mtku7n43MN1UWWUzJLxrTTY0n3EsiaIqbookkR7Aa2Y2BS2MF6KF8DuiurW7zyj2OBLmRhLMCSVHeP7vQALiJRcxztdANzN7JoTzikBW2PP6RWArvTWKe74htOP1gQ7Ibn6imS2LsgjPQA7AjIi+2JiGQv66mdl77r6aiY/jWbR7OTqE9brAdWFvTtEX9YwkmBNKirChXoHq7A3LEklQ8kYnRCX5HHNzJxddcywB3gcOjJC4PVDR1HWRk89Q0sZExKE8sr6EoYuQaRYKNxxmZo1dzHObo4ooy7j7mUQyTdyThHI9Izn/EkqKiCS4AUVbfJuFvZlZW5Ri3Q1pdRl38j0lHG7RYKoucjASiO+hsLQ3ECvdPojfeWpcWy9COedk7YlSuZdD5ER3RQhiV0Sy/3/A+4vALqXBIAnmhJIiUqxHoxCwkXEuE8515k4ud+RD1OJ9L8R/sSswudhzzTsQC5+lqRJJX+Bt4N4QzktkC0TCwkPiykgoKVzlkK4AdjGz7nE6s6XWmju5oSAXN9zEVK/wMiKjsciRFxbx4I+Y2c65803ibytUdus+xE63m5k1Q8RICQsZSTAnlANGIPrOQyMCoxjcyQ0Guay9Ywmbcj1009jFITIMEdwPdCHbmTwN9Hb3UajA6yPu/lMyX5QGyZSRUBYwUXPuThG5kxsS6pq1t4C22yFb8U7u/paZ7YWoUk929wfM7GYUP31FsfpMqBuSYE4oK1iRuZMTBDM7G8WGb+3uE3PCeU6WYVzXKPFelB4pXC6hrOAFZOtJKNcNmaB199PMbDrwrJn1dPfbI376cjM7y91HxCKYhHIZIAnmhIRFFJmgzaJc3P3cEMaZcL7NzCqA88zsWZRZmVAGSKaMhIRFELkY5R0Q18iSwDHu/pOZ/RFVYOkdNucVXEVsE8oEKSojIWERRAjlAcBZwI3AFqiK9yruPhSlvz8fiS6pVl+ZIQnmhIRFEBHlsSMitF8VJel8CjxoZh3c/QxU6XtqsiuXH5IpIyFhEUHOfJFVIGkJLIuKtw501f/7GFGl7grMSEK5PJGcfwkJiwhCKPcCNjCzie7+oJl9D3wCdDWznxC/9c3uPr2UY02YP5IpIyFhEYGp4sutiK/672Y2BPgGeBM4GlW3/qe7P1e6USZUB0ljTkhowMiZL36FKosc5arY/RQS0t+hqtbLAu3c/dUSDjehmkiCOSGhASOEcm/EgfExMNXMXnT3F8xsH+B+oK27/xX4qJRjTag+kvMvIaGBwcxWAJq7+wfByLc78ADgwE4opf1yd59sZv8X1z5dsgEn1BhJMCckNCCYCrjeg+KTn0IVT2YA27n7N2a2BTAAWAK4wN0/jfsS50gDQnL+JSQ0EFhlAddL3P0fwSuyHSpFdQhAEBKNAn4GWmf3JqHcsJA05oSEBgIz+x3Q3VUgtRGwAbAiIrY/Gjjb3a+Ma9u6+5TSjTahLkjOv4SEhoOqCrh2R6aNd4EzzGw5dz89CeWGjSSYExIaDsYB/0SVXQoLuAIcCrQtycgSiopkykhIaGCYRwHXC4B+kXadHH0NHEljTkhoYMgXcEXOv6GoTNSU+DwJ5QaOFJWRkNAAsZAKuCaUCMmUkZDQQFGfBVwTSoskmBMSEhLKDMmUkZCQkFBmSII5ISEhocyQBHNCQkJCmSEJ5oSEhIQyQxLMCYs9zGyWmU0wszfM7J9RObq2bfUyswfj9Y5mdtJ8rm1jZofVoo8zzOwP1T1fcM1NZrZrDfrqaGZv1HSMCXVDEswJCTDN3bu7ezdEoXlo/kMTavxbcff73f38+VzSBqixYE5Y9JEEc0LC3BgNrBaa4ltmdhXwMrCKmfUxszFm9nJo1ksCmFk/M3vbzJ4Fds4aMrPBZvbXeL2cmY0ws1fj2Aw4H+gc2vpFcd3xZjbOzF4zszNzbZ1iZu+Y2eNA1wVNwswOinZeNbO7C3YBvc1stJlNNLPt4/oKM7so1/chdX2QCbVHEswJCQEzawz0B16PU12Bv7v7+sCPwKlAb3ffABgPHGtmzYHrgB2ALYDl59H85cAz7r4eout8EzgJ+E9o68ebWR+gC8ro6w5saGZbmtmGwJ7A+kjwb1SN6dzj7htFf28BB+Q+6whsBQwE/hZzOAD41t03ivYPMrNO1egnoR6QuDISEqCFmU2I16OBGxDP8Yfu/kKc/z9gLeA5MwNoCowB1gA+cPd3AczsVuDgKvrYBtgPwN1nAd+aWSETXJ84Xon3SyJB3QoY4e5To4/7qzGnbmZ2DjKXLAk8mvvsTnefDbxrZu/HHPoA6+bsz0tF3xOr0VdCkZEEc0JC2JjzJ0L4/pg/BTzm7nsVXNcd1dorBgwY6u7XFPRxTC36uAkY5O6vmtlgoFfus8K2PPo+0t3zAjyrmpKwkJFMGQkJ1cMLwOZmthqAmS1hZqsDbwOdzKxzXLfXPO5/AhgS91aYWWvge6QNZ3gU2D9nu17JzJYF/gXsZGYtzKwVMpssCK2AycGnsU/BZ7uZWaMY86rAO9H3kLgeM1vdzFpWo5+EekDSmBMSqgF3/yI0z9vNrFmcPtXdJ5rZwcBIM/sSeBboVkUTRwPXmtkBwCxgiLuPMbPnIhzt4bAzrwmMCY39B2Bfd3/ZzO4AJgAfInPLgnAa8GJc/zpzLwDvAM8AywGHuvt0M7se2Z5fNnX+BTCoek8nodhIJEYJCQkJZYZkykhISEgoMyTBnJCQkFBmSII5ISEhocyQBHNCQkJCmSEJ5oSEhIQyQxLMCQkJCWWGJJgTEhISygz/DwFeUSAnd4eKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(cm, data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(794, 12)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_preds_test = learn.predict(is_test=True)\n",
    "log_preds_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_preds_test = np.argmax(log_preds_test, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_preds_test = np.argmax(log_preds_test, axis=1)\n",
    "preds_classes = [data.classes[i].replace(\"_\", \" \") for i in log_preds_test]\n",
    "probs = np.exp(log_preds_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.DataFrame({'file': os.listdir(f'{PATH}test'), 'species': preds_classes})\n",
    "submission.to_csv(f'{PATH}plantsubmission_no_tta.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/plant-seedlings-classification/plantsubmission_no_tta.csv' target='_blank'>data/plant-seedlings-classification/plantsubmission_no_tta.csv</a><br>"
      ],
      "text/plain": [
       "/home/ubuntu/github/fastai/courses/dl1/data/plant-seedlings-classification/plantsubmission_no_tta.csv"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import FileLink\n",
    "FileLink(f'{PATH}plantsubmission_no_tta.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             \r"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'numpy.ndarray' object has no attribute 'exp'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-34-2d24e4892a26>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mlog_preds_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTTA\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_test\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlog_preds_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mpreds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'exp'"
     ]
    }
   ],
   "source": [
    "log_preds_test = learn.TTA(is_test=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.argmax(preds, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "preds_classes = [data.classes[i].replace(\"_\", \" \") for i in preds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.DataFrame({'file': os.listdir(f'{PATH}test'), 'species': preds_classes})\n",
    "submission.to_csv(f'{PATH}plantsubmission_tta.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/plant-seedlings-classification/plantsubmission_tta.csv' target='_blank'>data/plant-seedlings-classification/plantsubmission_tta.csv</a><br>"
      ],
      "text/plain": [
       "/home/ubuntu/github/fastai/courses/dl1/data/plant-seedlings-classification/plantsubmission_tta.csv"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FileLink(f'{PATH}plantsubmission_tta.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to have a look at the sample submission file to see what layout the file needs to be in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samp_sub_df = pd.read_csv(f'{PATH}sample_submission.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From above we can see that we need a filename andthe predicted class `species`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samp_sub_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will run TTA with is_test=True to make sure it use the test set images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                               \r"
     ]
    }
   ],
   "source": [
    "log_preds_test, y = learn.TTA(n_aug=10, is_test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then get a 3 dimensional array out of TTA, this is 1 dimension for the original images and n for the augmented set of test images (default n_aug = 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_test = np.mean( np.exp( log_preds_test ),0)\n",
    "preds_test = np.argmax(preds_test, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to replace the \"\\_\" with spaces in the classes to resemble the original class names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_classes = [data.classes[i].replace(\"_\", \" \") for i in preds_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.DataFrame({'file': os.listdir(f'{PATH}test'), 'species': preds_classes})\n",
    "# submission.to_csv(f'{PATH}plantsubmission_tta.csv', index=False)\n",
    "# submission.to_csv(f'{PATH}plantsubmission_tta4_all_data.csv', index=False)\n",
    "# submission.to_csv(f'{PATH}plantsubmission_tta6_all_data.csv', index=False)\n",
    "# submission.to_csv(f'{PATH}plantsubmission_tta3_all_data.csv', index=False)\n",
    "submission.to_csv(f'{PATH}plantsubmission_tta10_all_data.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/plant-seedlings-classification/plantsubmission_tta10_all_data.csv' target='_blank'>data/plant-seedlings-classification/plantsubmission_tta10_all_data.csv</a><br>"
      ],
      "text/plain": [
       "/home/ubuntu/github/fastai/courses/dl1/data/plant-seedlings-classification/plantsubmission_tta10_all_data.csv"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import FileLink\n",
    "# FileLink(f'{PATH}plantsubmission_tta.csv')\n",
    "# FileLink(f'{PATH}plantsubmission_tta_all_data.csv')\n",
    "# FileLink(f'{PATH}plantsubmission_tta6_all_data.csv')\n",
    "# FileLink(f'{PATH}plantsubmission_tta3_all_data.csv')\n",
    "FileLink(f'{PATH}plantsubmission_tta10_all_data.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:fastai]",
   "language": "python",
   "name": "conda-env-fastai-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "nbpresent": {
   "slides": {},
   "themes": {
    "default": "f37a1638-141e-4181-bc65-b8e0438d25db",
    "theme": {
     "09307d3b-d5a1-4486-bfcb-0db5d42fa7d7": {
      "backgrounds": {
       "dc7afa04-bf90-40b1-82a5-726e3cff5267": {
        "background-color": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "id": "dc7afa04-bf90-40b1-82a5-726e3cff5267"
       }
      },
      "id": "09307d3b-d5a1-4486-bfcb-0db5d42fa7d7",
      "palette": {
       "19cc588f-0593-49c9-9f4b-e4d7cc113b1c": {
        "id": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "rgb": [
         252,
         252,
         252
        ]
       },
       "31af15d2-7e15-44c5-ab5e-e04b16a89eff": {
        "id": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "rgb": [
         68,
         68,
         68
        ]
       },
       "50f92c45-a630-455b-aec3-788680ec7410": {
        "id": "50f92c45-a630-455b-aec3-788680ec7410",
        "rgb": [
         197,
         226,
         245
        ]
       },
       "c5cc3653-2ee1-402a-aba2-7caae1da4f6c": {
        "id": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "rgb": [
         43,
         126,
         184
        ]
       },
       "efa7f048-9acb-414c-8b04-a26811511a21": {
        "id": "efa7f048-9acb-414c-8b04-a26811511a21",
        "rgb": [
         25.118061674008803,
         73.60176211453744,
         107.4819383259912
        ]
       }
      },
      "rules": {
       "a": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c"
       },
       "blockquote": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3
       },
       "code": {
        "font-family": "Anonymous Pro"
       },
       "h1": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 8
       },
       "h2": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 6
       },
       "h3": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-family": "Lato",
        "font-size": 5.5
       },
       "h4": {
        "color": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "font-family": "Lato",
        "font-size": 5
       },
       "h5": {
        "font-family": "Lato"
       },
       "h6": {
        "font-family": "Lato"
       },
       "h7": {
        "font-family": "Lato"
       },
       "li": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3.25
       },
       "pre": {
        "font-family": "Anonymous Pro",
        "font-size": 4
       }
      },
      "text-base": {
       "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
       "font-family": "Lato",
       "font-size": 4
      }
     },
     "25df69a3-8a9e-40c3-bec1-7d78b14fe19a": {
      "backgrounds": {
       "dc7afa04-bf90-40b1-82a5-726e3cff5267": {
        "background-color": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "id": "dc7afa04-bf90-40b1-82a5-726e3cff5267"
       }
      },
      "id": "25df69a3-8a9e-40c3-bec1-7d78b14fe19a",
      "palette": {
       "19cc588f-0593-49c9-9f4b-e4d7cc113b1c": {
        "id": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "rgb": [
         252,
         252,
         252
        ]
       },
       "31af15d2-7e15-44c5-ab5e-e04b16a89eff": {
        "id": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "rgb": [
         68,
         68,
         68
        ]
       },
       "50f92c45-a630-455b-aec3-788680ec7410": {
        "id": "50f92c45-a630-455b-aec3-788680ec7410",
        "rgb": [
         197,
         226,
         245
        ]
       },
       "c5cc3653-2ee1-402a-aba2-7caae1da4f6c": {
        "id": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "rgb": [
         43,
         126,
         184
        ]
       },
       "efa7f048-9acb-414c-8b04-a26811511a21": {
        "id": "efa7f048-9acb-414c-8b04-a26811511a21",
        "rgb": [
         25.118061674008803,
         73.60176211453744,
         107.4819383259912
        ]
       }
      },
      "rules": {
       "a": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c"
       },
       "blockquote": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3
       },
       "code": {
        "font-family": "Anonymous Pro"
       },
       "h1": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 8
       },
       "h2": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 6
       },
       "h3": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-family": "Lato",
        "font-size": 5.5
       },
       "h4": {
        "color": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "font-family": "Lato",
        "font-size": 5
       },
       "h5": {
        "font-family": "Lato"
       },
       "h6": {
        "font-family": "Lato"
       },
       "h7": {
        "font-family": "Lato"
       },
       "li": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3.25
       },
       "pre": {
        "font-family": "Anonymous Pro",
        "font-size": 4
       }
      },
      "text-base": {
       "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
       "font-family": "Lato",
       "font-size": 4
      }
     },
     "f37a1638-141e-4181-bc65-b8e0438d25db": {
      "backgrounds": {
       "dc7afa04-bf90-40b1-82a5-726e3cff5267": {
        "background-color": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "id": "dc7afa04-bf90-40b1-82a5-726e3cff5267"
       }
      },
      "id": "f37a1638-141e-4181-bc65-b8e0438d25db",
      "palette": {
       "19cc588f-0593-49c9-9f4b-e4d7cc113b1c": {
        "id": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "rgb": [
         252,
         252,
         252
        ]
       },
       "31af15d2-7e15-44c5-ab5e-e04b16a89eff": {
        "id": "31af15d2-7e15-44c5-ab5e-e04b16a89eff",
        "rgb": [
         68,
         68,
         68
        ]
       },
       "50f92c45-a630-455b-aec3-788680ec7410": {
        "id": "50f92c45-a630-455b-aec3-788680ec7410",
        "rgb": [
         197,
         226,
         245
        ]
       },
       "c5cc3653-2ee1-402a-aba2-7caae1da4f6c": {
        "id": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "rgb": [
         43,
         126,
         184
        ]
       },
       "efa7f048-9acb-414c-8b04-a26811511a21": {
        "id": "efa7f048-9acb-414c-8b04-a26811511a21",
        "rgb": [
         25.118061674008803,
         73.60176211453744,
         107.4819383259912
        ]
       }
      },
      "rules": {
       "a": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c"
       },
       "blockquote": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3
       },
       "code": {
        "font-family": "Anonymous Pro"
       },
       "h1": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 8
       },
       "h2": {
        "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
        "font-family": "Merriweather",
        "font-size": 6
       },
       "h3": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-family": "Lato",
        "font-size": 5.5
       },
       "h4": {
        "color": "c5cc3653-2ee1-402a-aba2-7caae1da4f6c",
        "font-family": "Lato",
        "font-size": 5
       },
       "h5": {
        "font-family": "Lato"
       },
       "h6": {
        "font-family": "Lato"
       },
       "h7": {
        "font-family": "Lato"
       },
       "li": {
        "color": "50f92c45-a630-455b-aec3-788680ec7410",
        "font-size": 3.25
       },
       "pre": {
        "font-family": "Anonymous Pro",
        "font-size": 4
       }
      },
      "text-base": {
       "color": "19cc588f-0593-49c9-9f4b-e4d7cc113b1c",
       "font-family": "Lato",
       "font-size": 4
      }
     }
    }
   }
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "266px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}