Q-learning-cart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-learning\n",
    "\n",
    "In this notebook, we'll build a neural network that can learn to play games through reinforcement learning. More specifically, we'll use Q-learning to train an agent to play a game called [Cart-Pole](https://gym.openai.com/envs/CartPole-v0). In this game, a freely swinging pole is attached to a cart. The cart can move to the left and right, and the goal is to keep the pole upright as long as possible.\n",
    "\n",
    "![Cart-Pole](assets/cart-pole.jpg)\n",
    "\n",
    "We can simulate this game using [OpenAI Gym](https://gym.openai.com/). First, let's check out how OpenAI Gym works. Then, we'll get into training an agent to play the Cart-Pole game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Note:** Make sure you have OpenAI Gym cloned into the same directory with this notebook. I've included `gym` as a submodule, so you can run `git submodule --init --recursive` to pull the contents into the `gym` repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-09-12 14:31:42,653] Making new env: CartPole-v0\n"
     ]
    }
   ],
   "source": [
    "# Create the Cart-Pole game environment\n",
    "env = gym.make('CartPole-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We interact with the simulation through `env`. To show the simulation running, you can use `env.render()` to render one frame. Passing in an action as an integer to `env.step` will generate the next step in the simulation.  You can see how many actions are possible from `env.action_space` and to get a random action you can use `env.action_space.sample()`. This is general to all Gym games. In the Cart-Pole game, there are two possible actions, moving the cart left or right. So there are two actions we can take, encoded as 0 and 1.\n",
    "\n",
    "Run the code below to watch the simulation run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset()\n",
    "rewards = []\n",
    "for _ in range(100):\n",
    "    env.render()\n",
    "    state, reward, done, info = env.step(env.action_space.sample()) # take a random action\n",
    "    rewards.append(reward)\n",
    "    if done:\n",
    "        rewards = []\n",
    "        env.reset()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To shut the window showing the simulation, use `env.close()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you ran the simulation above, we can look at the rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'rewards' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-f6344eaeebcf>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'rewards' is not defined"
     ]
    }
   ],
   "source": [
    "print(rewards[-20:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The game resets after the pole has fallen past a certain angle. For each frame while the simulation is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Network\n",
    "\n",
    "We train our Q-learning agent using the Bellman Equation:\n",
    "\n",
    "$$\n",
    "Q(s, a) = r + \\gamma \\max{Q(s', a')}\n",
    "$$\n",
    "\n",
    "where $s$ is a state, $a$ is an action, and $s'$ is the next state from state $s$ and action $a$.\n",
    "\n",
    "Before we used this equation to learn values for a Q-_table_. However, for this game there are a huge number of states available. The state has four values: the position and velocity of the cart, and the position and velocity of the pole. These are all real-valued numbers, so ignoring floating point precisions, you practically have infinite states. Instead of using a table then, we'll replace it with a neural network that will approximate the Q-table lookup function.\n",
    "\n",
    "<img src=\"assets/deep-q-learning.png\" width=450px>\n",
    "\n",
    "Now, our Q value, $Q(s, a)$ is calculated by passing in a state to the network. The output will be Q-values for each available action, with fully connected hidden layers.\n",
    "\n",
    "<img src=\"assets/q-network.png\" width=550px>\n",
    "\n",
    "\n",
    "As I showed before, we can define our targets for training as $\\hat{Q}(s,a) = r + \\gamma \\max{Q(s', a')}$. Then we update the weights by minimizing $(\\hat{Q}(s,a) - Q(s,a))^2$. \n",
    "\n",
    "For this Cart-Pole game, we have four inputs, one for each value in the state, and two outputs, one for each action. To get $\\hat{Q}$, we'll first choose an action, then simulate the game using that action. This will get us the next state, $s'$, and the reward. With that, we can calculate $\\hat{Q}$ then pass it back into the $Q$ network to run the optimizer and update the weights.\n",
    "\n",
    "Below is my implementation of the Q-network. I used two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class QNetwork:\n",
    "    def __init__(self, learning_rate=0.01, state_size=4, \n",
    "                 action_size=2, hidden_size=10, \n",
    "                 name='QNetwork'):\n",
    "        # state inputs to the Q-network\n",
    "        with tf.variable_scope(name):\n",
    "            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')\n",
    "            \n",
    "            # One hot encode the actions to later choose the Q-value for the action\n",
    "            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')\n",
    "            one_hot_actions = tf.one_hot(self.actions_, action_size)\n",
    "            \n",
    "            # Target Q values for training\n",
    "            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')\n",
    "            \n",
    "            # ReLU hidden layers\n",
    "            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)\n",
    "            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)\n",
    "\n",
    "            # Linear output layer\n",
    "            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, \n",
    "                                                            activation_fn=None)\n",
    "            \n",
    "            ### Train with loss (targetQ - Q)^2\n",
    "            # output has length 2, for two actions. This next line chooses\n",
    "            # one value from output (per row) according to the one-hot encoded actions.\n",
    "            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)\n",
    "            \n",
    "            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))\n",
    "            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience replay\n",
    "\n",
    "Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on. \n",
    "\n",
    "Here, we'll create a `Memory` object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maxmium capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.\n",
    "\n",
    "Below, I've implemented a `Memory` object. If you're unfamiliar with `deque`, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "class Memory():\n",
    "    def __init__(self, max_size = 1000):\n",
    "        self.buffer = deque(maxlen=max_size)\n",
    "    \n",
    "    def add(self, experience):\n",
    "        self.buffer.append(experience)\n",
    "            \n",
    "    def sample(self, batch_size):\n",
    "        idx = np.random.choice(np.arange(len(self.buffer)), \n",
    "                               size=batch_size, \n",
    "                               replace=False)\n",
    "        return [self.buffer[ii] for ii in idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploration - Exploitation\n",
    "\n",
    "To learn about the environment and rules of the game, the agent needs to explore by taking random actions. We'll do this by choosing a random action with some probability $\\epsilon$ (epsilon).  That is, with some probability $\\epsilon$ the agent will make a random action and with probability $1 - \\epsilon$, the agent will choose an action from $Q(s,a)$. This is called an **$\\epsilon$-greedy policy**.\n",
    "\n",
    "\n",
    "At first, the agent needs to do a lot of exploring. Later when it has learned more, the agent can favor choosing actions based on what it has learned. This is called _exploitation_. We'll set it up so the agent is more likely to explore early in training, then more likely to exploit later in training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Learning training algorithm\n",
    "\n",
    "Putting all this together, we can list out the algorithm we'll use to train the network. We'll train the network in _episodes_. One *episode* is one simulation of the game. For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:\n",
    "\n",
    "* Initialize the memory $D$\n",
    "* Initialize the action-value network $Q$ with random weights\n",
    "* **For** episode = 1, $M$ **do**\n",
    "  * **For** $t$, $T$ **do**\n",
    "     * With probability $\\epsilon$ select a random action $a_t$, otherwise select $a_t = \\mathrm{argmax}_a Q(s,a)$\n",
    "     * Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$\n",
    "     * Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$\n",
    "     * Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$\n",
    "     * Set $\\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\\hat{Q}_j = r_j + \\gamma \\max_{a'}{Q(s'_j, a')}$\n",
    "     * Make a gradient descent step with loss $(\\hat{Q}_j - Q(s_j, a_j))^2$\n",
    "  * **endfor**\n",
    "* **endfor**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "One of the more difficult aspects of reinforcememt learning are the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_episodes = 1000          # max number of episodes to learn from\n",
    "max_steps = 200                # max steps in an episode\n",
    "gamma = 0.99                   # future reward discount\n",
    "\n",
    "# Exploration parameters\n",
    "explore_start = 1.0            # exploration probability at start\n",
    "explore_stop = 0.01            # minimum exploration probability \n",
    "decay_rate = 0.0001            # exponential decay rate for exploration prob\n",
    "\n",
    "# Network parameters\n",
    "hidden_size = 64               # number of units in each Q-network hidden layer\n",
    "learning_rate = 0.0001         # Q-network learning rate\n",
    "\n",
    "# Memory parameters\n",
    "memory_size = 10000            # memory capacity\n",
    "batch_size = 20                # experience mini-batch size\n",
    "pretrain_length = batch_size   # number experiences to pretrain the memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate the experience memory\n",
    "\n",
    "Here I'm re-initializing the simulation and pre-populating the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize the simulation\n",
    "env.reset()\n",
    "# Take one random step to get the pole and cart moving\n",
    "state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "memory = Memory(max_size=memory_size)\n",
    "\n",
    "# Make a bunch of random actions and store the experiences\n",
    "for ii in range(pretrain_length):\n",
    "    # Uncomment the line below to watch the simulation\n",
    "    # env.render()\n",
    "\n",
    "    # Make a random action\n",
    "    action = env.action_space.sample()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        # The simulation fails so no next state\n",
    "        next_state = np.zeros(state.shape)\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        \n",
    "        # Start new episode\n",
    "        env.reset()\n",
    "        # Take one random step to get the pole and cart moving\n",
    "        state, reward, done, _ = env.step(env.action_space.sample())\n",
    "    else:\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Below we'll train our agent. If you want to watch it train, uncomment the `env.render()` line. This is slow because it's rendering the frames slower than the network can train. But, it's cool to watch the agent get better at the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Total reward: 5.0 Training loss: 1.1135 Explore P: 0.9995\n",
      "Episode: 2 Total reward: 19.0 Training loss: 1.0469 Explore P: 0.9976\n",
      "Episode: 3 Total reward: 16.0 Training loss: 0.9867 Explore P: 0.9960\n",
      "Episode: 4 Total reward: 17.0 Training loss: 0.9942 Explore P: 0.9944\n",
      "Episode: 5 Total reward: 11.0 Training loss: 1.1122 Explore P: 0.9933\n",
      "Episode: 6 Total reward: 20.0 Training loss: 1.0318 Explore P: 0.9913\n",
      "Episode: 7 Total reward: 14.0 Training loss: 1.1660 Explore P: 0.9900\n",
      "Episode: 8 Total reward: 40.0 Training loss: 1.0401 Explore P: 0.9860\n",
      "Episode: 9 Total reward: 21.0 Training loss: 1.0544 Explore P: 0.9840\n",
      "Episode: 10 Total reward: 18.0 Training loss: 1.1086 Explore P: 0.9822\n",
      "Episode: 11 Total reward: 18.0 Training loss: 1.0341 Explore P: 0.9805\n",
      "Episode: 12 Total reward: 18.0 Training loss: 1.2568 Explore P: 0.9787\n",
      "Episode: 13 Total reward: 30.0 Training loss: 1.2591 Explore P: 0.9758\n",
      "Episode: 14 Total reward: 31.0 Training loss: 1.2429 Explore P: 0.9729\n",
      "Episode: 15 Total reward: 11.0 Training loss: 1.2713 Explore P: 0.9718\n",
      "Episode: 16 Total reward: 20.0 Training loss: 1.2872 Explore P: 0.9699\n",
      "Episode: 17 Total reward: 25.0 Training loss: 1.7295 Explore P: 0.9675\n",
      "Episode: 18 Total reward: 28.0 Training loss: 1.3458 Explore P: 0.9648\n",
      "Episode: 19 Total reward: 15.0 Training loss: 2.1950 Explore P: 0.9634\n",
      "Episode: 20 Total reward: 10.0 Training loss: 1.7199 Explore P: 0.9624\n",
      "Episode: 21 Total reward: 15.0 Training loss: 2.6349 Explore P: 0.9610\n",
      "Episode: 22 Total reward: 24.0 Training loss: 1.9690 Explore P: 0.9587\n",
      "Episode: 23 Total reward: 9.0 Training loss: 2.7537 Explore P: 0.9579\n",
      "Episode: 24 Total reward: 12.0 Training loss: 2.3463 Explore P: 0.9567\n",
      "Episode: 25 Total reward: 17.0 Training loss: 5.2483 Explore P: 0.9551\n",
      "Episode: 26 Total reward: 27.0 Training loss: 3.2285 Explore P: 0.9526\n",
      "Episode: 27 Total reward: 50.0 Training loss: 3.1694 Explore P: 0.9479\n",
      "Episode: 28 Total reward: 16.0 Training loss: 5.9571 Explore P: 0.9464\n",
      "Episode: 29 Total reward: 12.0 Training loss: 5.1244 Explore P: 0.9452\n",
      "Episode: 30 Total reward: 26.0 Training loss: 3.5526 Explore P: 0.9428\n",
      "Episode: 31 Total reward: 14.0 Training loss: 9.0665 Explore P: 0.9415\n",
      "Episode: 32 Total reward: 11.0 Training loss: 9.8520 Explore P: 0.9405\n",
      "Episode: 33 Total reward: 16.0 Training loss: 2.5396 Explore P: 0.9390\n",
      "Episode: 34 Total reward: 12.0 Training loss: 2.8262 Explore P: 0.9379\n",
      "Episode: 35 Total reward: 25.0 Training loss: 6.0055 Explore P: 0.9356\n",
      "Episode: 36 Total reward: 11.0 Training loss: 9.9105 Explore P: 0.9345\n",
      "Episode: 37 Total reward: 16.0 Training loss: 7.2444 Explore P: 0.9331\n",
      "Episode: 38 Total reward: 24.0 Training loss: 3.6672 Explore P: 0.9309\n",
      "Episode: 39 Total reward: 14.0 Training loss: 4.9742 Explore P: 0.9296\n",
      "Episode: 40 Total reward: 22.0 Training loss: 3.3407 Explore P: 0.9275\n",
      "Episode: 41 Total reward: 13.0 Training loss: 3.8486 Explore P: 0.9264\n",
      "Episode: 42 Total reward: 27.0 Training loss: 4.8694 Explore P: 0.9239\n",
      "Episode: 43 Total reward: 9.0 Training loss: 7.4307 Explore P: 0.9231\n",
      "Episode: 44 Total reward: 11.0 Training loss: 32.7631 Explore P: 0.9221\n",
      "Episode: 45 Total reward: 36.0 Training loss: 32.4630 Explore P: 0.9188\n",
      "Episode: 46 Total reward: 31.0 Training loss: 20.8429 Explore P: 0.9160\n",
      "Episode: 47 Total reward: 10.0 Training loss: 4.1655 Explore P: 0.9151\n",
      "Episode: 48 Total reward: 15.0 Training loss: 4.4446 Explore P: 0.9137\n",
      "Episode: 49 Total reward: 31.0 Training loss: 26.4540 Explore P: 0.9109\n",
      "Episode: 50 Total reward: 26.0 Training loss: 9.0251 Explore P: 0.9086\n",
      "Episode: 51 Total reward: 10.0 Training loss: 28.3238 Explore P: 0.9077\n",
      "Episode: 52 Total reward: 14.0 Training loss: 5.0416 Explore P: 0.9064\n",
      "Episode: 53 Total reward: 13.0 Training loss: 3.5019 Explore P: 0.9053\n",
      "Episode: 54 Total reward: 20.0 Training loss: 10.5312 Explore P: 0.9035\n",
      "Episode: 55 Total reward: 15.0 Training loss: 10.9353 Explore P: 0.9021\n",
      "Episode: 56 Total reward: 13.0 Training loss: 5.8705 Explore P: 0.9010\n",
      "Episode: 57 Total reward: 19.0 Training loss: 53.5708 Explore P: 0.8993\n",
      "Episode: 58 Total reward: 34.0 Training loss: 49.0695 Explore P: 0.8963\n",
      "Episode: 59 Total reward: 14.0 Training loss: 17.4837 Explore P: 0.8950\n",
      "Episode: 60 Total reward: 16.0 Training loss: 16.3644 Explore P: 0.8936\n",
      "Episode: 61 Total reward: 33.0 Training loss: 5.5948 Explore P: 0.8907\n",
      "Episode: 62 Total reward: 17.0 Training loss: 4.1340 Explore P: 0.8892\n",
      "Episode: 63 Total reward: 15.0 Training loss: 5.4491 Explore P: 0.8879\n",
      "Episode: 64 Total reward: 32.0 Training loss: 36.8293 Explore P: 0.8851\n",
      "Episode: 65 Total reward: 14.0 Training loss: 19.7579 Explore P: 0.8838\n",
      "Episode: 66 Total reward: 8.0 Training loss: 38.8983 Explore P: 0.8831\n",
      "Episode: 67 Total reward: 12.0 Training loss: 37.9880 Explore P: 0.8821\n",
      "Episode: 68 Total reward: 46.0 Training loss: 15.7353 Explore P: 0.8781\n",
      "Episode: 69 Total reward: 41.0 Training loss: 60.4021 Explore P: 0.8745\n",
      "Episode: 70 Total reward: 16.0 Training loss: 33.3378 Explore P: 0.8732\n",
      "Episode: 71 Total reward: 12.0 Training loss: 65.3904 Explore P: 0.8721\n",
      "Episode: 72 Total reward: 11.0 Training loss: 67.4169 Explore P: 0.8712\n",
      "Episode: 73 Total reward: 15.0 Training loss: 34.7701 Explore P: 0.8699\n",
      "Episode: 74 Total reward: 12.0 Training loss: 14.2784 Explore P: 0.8689\n",
      "Episode: 75 Total reward: 17.0 Training loss: 40.2716 Explore P: 0.8674\n",
      "Episode: 76 Total reward: 37.0 Training loss: 52.9214 Explore P: 0.8642\n",
      "Episode: 77 Total reward: 19.0 Training loss: 4.9850 Explore P: 0.8626\n",
      "Episode: 78 Total reward: 35.0 Training loss: 21.9255 Explore P: 0.8596\n",
      "Episode: 79 Total reward: 38.0 Training loss: 100.1122 Explore P: 0.8564\n",
      "Episode: 80 Total reward: 11.0 Training loss: 52.6482 Explore P: 0.8555\n",
      "Episode: 81 Total reward: 27.0 Training loss: 22.5384 Explore P: 0.8532\n",
      "Episode: 82 Total reward: 10.0 Training loss: 90.1052 Explore P: 0.8524\n",
      "Episode: 83 Total reward: 12.0 Training loss: 49.3395 Explore P: 0.8513\n",
      "Episode: 84 Total reward: 30.0 Training loss: 57.6037 Explore P: 0.8488\n",
      "Episode: 85 Total reward: 26.0 Training loss: 37.1015 Explore P: 0.8466\n",
      "Episode: 86 Total reward: 41.0 Training loss: 37.5723 Explore P: 0.8432\n",
      "Episode: 87 Total reward: 40.0 Training loss: 24.6072 Explore P: 0.8399\n",
      "Episode: 88 Total reward: 32.0 Training loss: 4.1799 Explore P: 0.8372\n",
      "Episode: 89 Total reward: 42.0 Training loss: 3.8772 Explore P: 0.8338\n",
      "Episode: 90 Total reward: 9.0 Training loss: 28.6997 Explore P: 0.8330\n",
      "Episode: 91 Total reward: 43.0 Training loss: 4.2603 Explore P: 0.8295\n",
      "Episode: 92 Total reward: 12.0 Training loss: 128.7484 Explore P: 0.8285\n",
      "Episode: 93 Total reward: 11.0 Training loss: 44.2926 Explore P: 0.8276\n",
      "Episode: 94 Total reward: 12.0 Training loss: 46.4630 Explore P: 0.8266\n",
      "Episode: 95 Total reward: 12.0 Training loss: 34.9408 Explore P: 0.8257\n",
      "Episode: 96 Total reward: 28.0 Training loss: 3.3755 Explore P: 0.8234\n",
      "Episode: 97 Total reward: 86.0 Training loss: 3.0469 Explore P: 0.8164\n",
      "Episode: 98 Total reward: 18.0 Training loss: 79.0495 Explore P: 0.8150\n",
      "Episode: 99 Total reward: 12.0 Training loss: 133.3604 Explore P: 0.8140\n",
      "Episode: 100 Total reward: 21.0 Training loss: 3.0632 Explore P: 0.8123\n",
      "Episode: 101 Total reward: 22.0 Training loss: 65.0454 Explore P: 0.8106\n",
      "Episode: 102 Total reward: 38.0 Training loss: 114.5933 Explore P: 0.8075\n",
      "Episode: 103 Total reward: 11.0 Training loss: 1.4899 Explore P: 0.8066\n",
      "Episode: 104 Total reward: 24.0 Training loss: 2.5832 Explore P: 0.8047\n",
      "Episode: 105 Total reward: 15.0 Training loss: 1.6163 Explore P: 0.8035\n",
      "Episode: 106 Total reward: 11.0 Training loss: 39.9080 Explore P: 0.8027\n",
      "Episode: 107 Total reward: 22.0 Training loss: 62.3060 Explore P: 0.8009\n",
      "Episode: 108 Total reward: 17.0 Training loss: 1.5050 Explore P: 0.7996\n",
      "Episode: 109 Total reward: 10.0 Training loss: 80.6076 Explore P: 0.7988\n",
      "Episode: 110 Total reward: 22.0 Training loss: 29.4337 Explore P: 0.7971\n",
      "Episode: 111 Total reward: 15.0 Training loss: 94.2126 Explore P: 0.7959\n",
      "Episode: 112 Total reward: 22.0 Training loss: 25.4378 Explore P: 0.7942\n",
      "Episode: 113 Total reward: 18.0 Training loss: 87.9994 Explore P: 0.7927\n",
      "Episode: 114 Total reward: 14.0 Training loss: 56.0689 Explore P: 0.7916\n",
      "Episode: 115 Total reward: 19.0 Training loss: 93.4653 Explore P: 0.7902\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 116 Total reward: 19.0 Training loss: 53.0275 Explore P: 0.7887\n",
      "Episode: 117 Total reward: 17.0 Training loss: 0.9808 Explore P: 0.7874\n",
      "Episode: 118 Total reward: 9.0 Training loss: 28.3738 Explore P: 0.7867\n",
      "Episode: 119 Total reward: 8.0 Training loss: 1.4699 Explore P: 0.7860\n",
      "Episode: 120 Total reward: 11.0 Training loss: 1.1479 Explore P: 0.7852\n",
      "Episode: 121 Total reward: 12.0 Training loss: 78.0813 Explore P: 0.7843\n",
      "Episode: 122 Total reward: 16.0 Training loss: 85.3888 Explore P: 0.7830\n",
      "Episode: 123 Total reward: 11.0 Training loss: 23.4610 Explore P: 0.7822\n",
      "Episode: 124 Total reward: 12.0 Training loss: 1.7128 Explore P: 0.7812\n",
      "Episode: 125 Total reward: 10.0 Training loss: 53.8748 Explore P: 0.7805\n",
      "Episode: 126 Total reward: 9.0 Training loss: 0.7206 Explore P: 0.7798\n",
      "Episode: 127 Total reward: 13.0 Training loss: 42.5871 Explore P: 0.7788\n",
      "Episode: 128 Total reward: 13.0 Training loss: 87.4049 Explore P: 0.7778\n",
      "Episode: 129 Total reward: 8.0 Training loss: 1.3326 Explore P: 0.7772\n",
      "Episode: 130 Total reward: 9.0 Training loss: 1.1823 Explore P: 0.7765\n",
      "Episode: 131 Total reward: 13.0 Training loss: 23.0877 Explore P: 0.7755\n",
      "Episode: 132 Total reward: 14.0 Training loss: 61.6805 Explore P: 0.7744\n",
      "Episode: 133 Total reward: 14.0 Training loss: 24.0416 Explore P: 0.7733\n",
      "Episode: 134 Total reward: 12.0 Training loss: 1.2429 Explore P: 0.7724\n",
      "Episode: 135 Total reward: 21.0 Training loss: 19.1593 Explore P: 0.7708\n",
      "Episode: 136 Total reward: 18.0 Training loss: 75.3962 Explore P: 0.7695\n",
      "Episode: 137 Total reward: 14.0 Training loss: 1.2113 Explore P: 0.7684\n",
      "Episode: 138 Total reward: 57.0 Training loss: 1.2131 Explore P: 0.7641\n",
      "Episode: 139 Total reward: 11.0 Training loss: 1.5932 Explore P: 0.7633\n",
      "Episode: 140 Total reward: 15.0 Training loss: 1.5978 Explore P: 0.7621\n",
      "Episode: 141 Total reward: 11.0 Training loss: 40.5025 Explore P: 0.7613\n",
      "Episode: 142 Total reward: 15.0 Training loss: 72.5052 Explore P: 0.7602\n",
      "Episode: 143 Total reward: 8.0 Training loss: 23.2750 Explore P: 0.7596\n",
      "Episode: 144 Total reward: 21.0 Training loss: 20.7752 Explore P: 0.7580\n",
      "Episode: 145 Total reward: 15.0 Training loss: 36.7589 Explore P: 0.7569\n",
      "Episode: 146 Total reward: 19.0 Training loss: 1.6520 Explore P: 0.7555\n",
      "Episode: 147 Total reward: 14.0 Training loss: 29.4250 Explore P: 0.7544\n",
      "Episode: 148 Total reward: 23.0 Training loss: 75.3529 Explore P: 0.7527\n",
      "Episode: 149 Total reward: 16.0 Training loss: 58.8462 Explore P: 0.7515\n",
      "Episode: 150 Total reward: 18.0 Training loss: 1.9167 Explore P: 0.7502\n",
      "Episode: 151 Total reward: 16.0 Training loss: 17.6803 Explore P: 0.7490\n",
      "Episode: 152 Total reward: 17.0 Training loss: 21.5109 Explore P: 0.7477\n",
      "Episode: 153 Total reward: 13.0 Training loss: 19.5638 Explore P: 0.7468\n",
      "Episode: 154 Total reward: 22.0 Training loss: 20.1236 Explore P: 0.7452\n",
      "Episode: 155 Total reward: 11.0 Training loss: 77.8483 Explore P: 0.7444\n",
      "Episode: 156 Total reward: 21.0 Training loss: 44.8093 Explore P: 0.7428\n",
      "Episode: 157 Total reward: 15.0 Training loss: 38.7777 Explore P: 0.7417\n",
      "Episode: 158 Total reward: 21.0 Training loss: 21.9319 Explore P: 0.7402\n",
      "Episode: 159 Total reward: 15.0 Training loss: 2.4389 Explore P: 0.7391\n",
      "Episode: 160 Total reward: 19.0 Training loss: 39.7182 Explore P: 0.7377\n",
      "Episode: 161 Total reward: 23.0 Training loss: 1.9047 Explore P: 0.7360\n",
      "Episode: 162 Total reward: 11.0 Training loss: 20.8252 Explore P: 0.7352\n",
      "Episode: 163 Total reward: 15.0 Training loss: 59.0132 Explore P: 0.7342\n",
      "Episode: 164 Total reward: 12.0 Training loss: 2.0644 Explore P: 0.7333\n",
      "Episode: 165 Total reward: 15.0 Training loss: 17.6074 Explore P: 0.7322\n",
      "Episode: 166 Total reward: 74.0 Training loss: 20.8284 Explore P: 0.7269\n",
      "Episode: 167 Total reward: 14.0 Training loss: 1.5734 Explore P: 0.7259\n",
      "Episode: 168 Total reward: 20.0 Training loss: 1.5810 Explore P: 0.7244\n",
      "Episode: 169 Total reward: 39.0 Training loss: 70.5313 Explore P: 0.7217\n",
      "Episode: 170 Total reward: 10.0 Training loss: 1.6514 Explore P: 0.7210\n",
      "Episode: 171 Total reward: 10.0 Training loss: 1.7757 Explore P: 0.7202\n",
      "Episode: 172 Total reward: 18.0 Training loss: 16.6155 Explore P: 0.7190\n",
      "Episode: 173 Total reward: 11.0 Training loss: 16.6054 Explore P: 0.7182\n",
      "Episode: 174 Total reward: 14.0 Training loss: 72.1413 Explore P: 0.7172\n",
      "Episode: 175 Total reward: 11.0 Training loss: 40.4051 Explore P: 0.7164\n",
      "Episode: 176 Total reward: 12.0 Training loss: 57.5355 Explore P: 0.7156\n",
      "Episode: 177 Total reward: 15.0 Training loss: 1.7269 Explore P: 0.7145\n",
      "Episode: 178 Total reward: 16.0 Training loss: 17.4474 Explore P: 0.7134\n",
      "Episode: 179 Total reward: 10.0 Training loss: 60.1354 Explore P: 0.7127\n",
      "Episode: 180 Total reward: 21.0 Training loss: 1.5862 Explore P: 0.7112\n",
      "Episode: 181 Total reward: 31.0 Training loss: 56.2704 Explore P: 0.7090\n",
      "Episode: 182 Total reward: 28.0 Training loss: 32.8521 Explore P: 0.7071\n",
      "Episode: 183 Total reward: 12.0 Training loss: 1.7133 Explore P: 0.7062\n",
      "Episode: 184 Total reward: 37.0 Training loss: 1.7937 Explore P: 0.7037\n",
      "Episode: 185 Total reward: 14.0 Training loss: 52.2032 Explore P: 0.7027\n",
      "Episode: 186 Total reward: 10.0 Training loss: 30.0956 Explore P: 0.7020\n",
      "Episode: 187 Total reward: 10.0 Training loss: 45.1228 Explore P: 0.7013\n",
      "Episode: 188 Total reward: 29.0 Training loss: 36.3887 Explore P: 0.6993\n",
      "Episode: 189 Total reward: 11.0 Training loss: 32.2396 Explore P: 0.6986\n",
      "Episode: 190 Total reward: 12.0 Training loss: 1.4191 Explore P: 0.6977\n",
      "Episode: 191 Total reward: 9.0 Training loss: 19.2088 Explore P: 0.6971\n",
      "Episode: 192 Total reward: 15.0 Training loss: 37.3820 Explore P: 0.6961\n",
      "Episode: 193 Total reward: 14.0 Training loss: 42.6056 Explore P: 0.6951\n",
      "Episode: 194 Total reward: 15.0 Training loss: 18.6653 Explore P: 0.6941\n",
      "Episode: 195 Total reward: 21.0 Training loss: 1.4376 Explore P: 0.6927\n",
      "Episode: 196 Total reward: 11.0 Training loss: 1.6606 Explore P: 0.6919\n",
      "Episode: 197 Total reward: 10.0 Training loss: 32.2514 Explore P: 0.6912\n",
      "Episode: 198 Total reward: 14.0 Training loss: 50.1935 Explore P: 0.6903\n",
      "Episode: 199 Total reward: 17.0 Training loss: 18.6795 Explore P: 0.6891\n",
      "Episode: 200 Total reward: 13.0 Training loss: 1.6508 Explore P: 0.6882\n",
      "Episode: 201 Total reward: 10.0 Training loss: 1.2041 Explore P: 0.6876\n",
      "Episode: 202 Total reward: 9.0 Training loss: 1.4620 Explore P: 0.6870\n",
      "Episode: 203 Total reward: 18.0 Training loss: 29.1202 Explore P: 0.6857\n",
      "Episode: 204 Total reward: 15.0 Training loss: 28.4505 Explore P: 0.6847\n",
      "Episode: 205 Total reward: 52.0 Training loss: 78.8927 Explore P: 0.6812\n",
      "Episode: 206 Total reward: 13.0 Training loss: 68.9032 Explore P: 0.6804\n",
      "Episode: 207 Total reward: 14.0 Training loss: 36.5976 Explore P: 0.6794\n",
      "Episode: 208 Total reward: 14.0 Training loss: 0.8633 Explore P: 0.6785\n",
      "Episode: 209 Total reward: 32.0 Training loss: 0.7632 Explore P: 0.6763\n",
      "Episode: 210 Total reward: 10.0 Training loss: 59.6192 Explore P: 0.6757\n",
      "Episode: 211 Total reward: 15.0 Training loss: 1.0048 Explore P: 0.6747\n",
      "Episode: 212 Total reward: 34.0 Training loss: 15.5524 Explore P: 0.6724\n",
      "Episode: 213 Total reward: 19.0 Training loss: 89.9568 Explore P: 0.6712\n",
      "Episode: 214 Total reward: 18.0 Training loss: 18.0470 Explore P: 0.6700\n",
      "Episode: 215 Total reward: 15.0 Training loss: 0.6492 Explore P: 0.6690\n",
      "Episode: 216 Total reward: 28.0 Training loss: 40.2396 Explore P: 0.6671\n",
      "Episode: 217 Total reward: 9.0 Training loss: 15.8342 Explore P: 0.6666\n",
      "Episode: 218 Total reward: 19.0 Training loss: 0.6549 Explore P: 0.6653\n",
      "Episode: 219 Total reward: 36.0 Training loss: 0.7956 Explore P: 0.6630\n",
      "Episode: 220 Total reward: 50.0 Training loss: 0.6843 Explore P: 0.6597\n",
      "Episode: 221 Total reward: 17.0 Training loss: 19.3502 Explore P: 0.6586\n",
      "Episode: 222 Total reward: 37.0 Training loss: 18.4212 Explore P: 0.6562\n",
      "Episode: 223 Total reward: 25.0 Training loss: 31.1406 Explore P: 0.6546\n",
      "Episode: 224 Total reward: 24.0 Training loss: 0.7038 Explore P: 0.6530\n",
      "Episode: 225 Total reward: 48.0 Training loss: 24.5643 Explore P: 0.6500\n",
      "Episode: 226 Total reward: 29.0 Training loss: 47.1199 Explore P: 0.6481\n",
      "Episode: 227 Total reward: 71.0 Training loss: 1.0754 Explore P: 0.6436\n",
      "Episode: 228 Total reward: 11.0 Training loss: 13.6922 Explore P: 0.6429\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 229 Total reward: 29.0 Training loss: 0.9098 Explore P: 0.6411\n",
      "Episode: 230 Total reward: 35.0 Training loss: 0.9095 Explore P: 0.6389\n",
      "Episode: 231 Total reward: 50.0 Training loss: 16.0957 Explore P: 0.6357\n",
      "Episode: 232 Total reward: 99.0 Training loss: 26.9346 Explore P: 0.6296\n",
      "Episode: 233 Total reward: 57.0 Training loss: 0.8806 Explore P: 0.6260\n",
      "Episode: 234 Total reward: 18.0 Training loss: 0.8269 Explore P: 0.6249\n",
      "Episode: 235 Total reward: 74.0 Training loss: 0.9112 Explore P: 0.6204\n",
      "Episode: 236 Total reward: 68.0 Training loss: 0.9322 Explore P: 0.6163\n",
      "Episode: 237 Total reward: 43.0 Training loss: 12.0461 Explore P: 0.6137\n",
      "Episode: 238 Total reward: 81.0 Training loss: 31.4040 Explore P: 0.6088\n",
      "Episode: 239 Total reward: 24.0 Training loss: 23.3348 Explore P: 0.6074\n",
      "Episode: 240 Total reward: 24.0 Training loss: 1.0206 Explore P: 0.6059\n",
      "Episode: 241 Total reward: 15.0 Training loss: 15.0853 Explore P: 0.6050\n",
      "Episode: 242 Total reward: 23.0 Training loss: 38.9416 Explore P: 0.6037\n",
      "Episode: 243 Total reward: 23.0 Training loss: 14.0702 Explore P: 0.6023\n",
      "Episode: 244 Total reward: 20.0 Training loss: 1.0021 Explore P: 0.6011\n",
      "Episode: 245 Total reward: 16.0 Training loss: 12.2022 Explore P: 0.6002\n",
      "Episode: 246 Total reward: 19.0 Training loss: 1.0435 Explore P: 0.5990\n",
      "Episode: 247 Total reward: 13.0 Training loss: 40.4388 Explore P: 0.5983\n",
      "Episode: 248 Total reward: 23.0 Training loss: 1.0815 Explore P: 0.5969\n",
      "Episode: 249 Total reward: 31.0 Training loss: 1.1641 Explore P: 0.5951\n",
      "Episode: 250 Total reward: 23.0 Training loss: 0.9003 Explore P: 0.5938\n",
      "Episode: 251 Total reward: 17.0 Training loss: 27.7752 Explore P: 0.5928\n",
      "Episode: 252 Total reward: 16.0 Training loss: 0.8877 Explore P: 0.5918\n",
      "Episode: 253 Total reward: 14.0 Training loss: 14.8302 Explore P: 0.5910\n",
      "Episode: 254 Total reward: 28.0 Training loss: 19.4606 Explore P: 0.5894\n",
      "Episode: 255 Total reward: 20.0 Training loss: 13.2488 Explore P: 0.5882\n",
      "Episode: 256 Total reward: 18.0 Training loss: 24.4358 Explore P: 0.5872\n",
      "Episode: 257 Total reward: 16.0 Training loss: 1.2232 Explore P: 0.5863\n",
      "Episode: 258 Total reward: 22.0 Training loss: 13.5509 Explore P: 0.5850\n",
      "Episode: 259 Total reward: 21.0 Training loss: 28.0917 Explore P: 0.5838\n",
      "Episode: 260 Total reward: 22.0 Training loss: 28.9177 Explore P: 0.5826\n",
      "Episode: 261 Total reward: 44.0 Training loss: 0.9377 Explore P: 0.5800\n",
      "Episode: 262 Total reward: 83.0 Training loss: 1.0211 Explore P: 0.5753\n",
      "Episode: 263 Total reward: 42.0 Training loss: 13.8436 Explore P: 0.5730\n",
      "Episode: 264 Total reward: 18.0 Training loss: 17.2339 Explore P: 0.5719\n",
      "Episode: 265 Total reward: 19.0 Training loss: 16.6741 Explore P: 0.5709\n",
      "Episode: 266 Total reward: 34.0 Training loss: 42.9108 Explore P: 0.5690\n",
      "Episode: 267 Total reward: 32.0 Training loss: 1.0328 Explore P: 0.5672\n",
      "Episode: 268 Total reward: 62.0 Training loss: 0.6322 Explore P: 0.5637\n",
      "Episode: 269 Total reward: 57.0 Training loss: 1.0425 Explore P: 0.5606\n",
      "Episode: 270 Total reward: 58.0 Training loss: 34.8877 Explore P: 0.5574\n",
      "Episode: 271 Total reward: 50.0 Training loss: 1.2829 Explore P: 0.5547\n",
      "Episode: 272 Total reward: 22.0 Training loss: 44.1177 Explore P: 0.5535\n",
      "Episode: 273 Total reward: 66.0 Training loss: 0.9675 Explore P: 0.5499\n",
      "Episode: 274 Total reward: 32.0 Training loss: 1.1746 Explore P: 0.5482\n",
      "Episode: 275 Total reward: 18.0 Training loss: 1.1780 Explore P: 0.5472\n",
      "Episode: 276 Total reward: 120.0 Training loss: 19.9839 Explore P: 0.5408\n",
      "Episode: 277 Total reward: 63.0 Training loss: 25.9887 Explore P: 0.5375\n",
      "Episode: 278 Total reward: 87.0 Training loss: 13.7030 Explore P: 0.5329\n",
      "Episode: 279 Total reward: 59.0 Training loss: 16.7966 Explore P: 0.5298\n",
      "Episode: 280 Total reward: 74.0 Training loss: 1.0362 Explore P: 0.5260\n",
      "Episode: 281 Total reward: 31.0 Training loss: 33.3587 Explore P: 0.5244\n",
      "Episode: 282 Total reward: 24.0 Training loss: 16.3720 Explore P: 0.5232\n",
      "Episode: 283 Total reward: 30.0 Training loss: 25.6119 Explore P: 0.5216\n",
      "Episode: 284 Total reward: 21.0 Training loss: 12.7965 Explore P: 0.5206\n",
      "Episode: 285 Total reward: 20.0 Training loss: 67.9830 Explore P: 0.5195\n",
      "Episode: 286 Total reward: 60.0 Training loss: 1.6650 Explore P: 0.5165\n",
      "Episode: 287 Total reward: 92.0 Training loss: 20.6553 Explore P: 0.5119\n",
      "Episode: 288 Total reward: 54.0 Training loss: 1.2518 Explore P: 0.5091\n",
      "Episode: 289 Total reward: 17.0 Training loss: 1.3696 Explore P: 0.5083\n",
      "Episode: 290 Total reward: 75.0 Training loss: 12.8616 Explore P: 0.5046\n",
      "Episode: 291 Total reward: 44.0 Training loss: 1.7323 Explore P: 0.5024\n",
      "Episode: 292 Total reward: 45.0 Training loss: 0.9569 Explore P: 0.5002\n",
      "Episode: 293 Total reward: 37.0 Training loss: 13.4833 Explore P: 0.4984\n",
      "Episode: 294 Total reward: 14.0 Training loss: 14.8676 Explore P: 0.4977\n",
      "Episode: 295 Total reward: 23.0 Training loss: 1.1391 Explore P: 0.4966\n",
      "Episode: 296 Total reward: 55.0 Training loss: 26.1906 Explore P: 0.4939\n",
      "Episode: 297 Total reward: 38.0 Training loss: 72.4363 Explore P: 0.4921\n",
      "Episode: 298 Total reward: 35.0 Training loss: 49.0798 Explore P: 0.4904\n",
      "Episode: 299 Total reward: 100.0 Training loss: 1.5414 Explore P: 0.4856\n",
      "Episode: 300 Total reward: 56.0 Training loss: 66.8442 Explore P: 0.4830\n",
      "Episode: 301 Total reward: 109.0 Training loss: 49.6279 Explore P: 0.4778\n",
      "Episode: 302 Total reward: 68.0 Training loss: 34.9066 Explore P: 0.4747\n",
      "Episode: 303 Total reward: 65.0 Training loss: 56.9847 Explore P: 0.4716\n",
      "Episode: 304 Total reward: 75.0 Training loss: 26.7449 Explore P: 0.4682\n",
      "Episode: 305 Total reward: 31.0 Training loss: 50.8275 Explore P: 0.4668\n",
      "Episode: 306 Total reward: 40.0 Training loss: 24.8033 Explore P: 0.4650\n",
      "Episode: 307 Total reward: 66.0 Training loss: 1.7454 Explore P: 0.4620\n",
      "Episode: 308 Total reward: 109.0 Training loss: 2.1917 Explore P: 0.4571\n",
      "Episode: 309 Total reward: 68.0 Training loss: 0.9941 Explore P: 0.4540\n",
      "Episode: 310 Total reward: 55.0 Training loss: 1.6395 Explore P: 0.4516\n",
      "Episode: 311 Total reward: 135.0 Training loss: 1.1570 Explore P: 0.4457\n",
      "Episode: 312 Total reward: 62.0 Training loss: 0.9965 Explore P: 0.4430\n",
      "Episode: 313 Total reward: 56.0 Training loss: 38.8352 Explore P: 0.4406\n",
      "Episode: 314 Total reward: 36.0 Training loss: 34.5867 Explore P: 0.4390\n",
      "Episode: 315 Total reward: 70.0 Training loss: 50.9113 Explore P: 0.4360\n",
      "Episode: 316 Total reward: 145.0 Training loss: 85.6069 Explore P: 0.4299\n",
      "Episode: 317 Total reward: 50.0 Training loss: 2.2769 Explore P: 0.4278\n",
      "Episode: 318 Total reward: 27.0 Training loss: 1.7176 Explore P: 0.4267\n",
      "Episode: 319 Total reward: 83.0 Training loss: 1.7466 Explore P: 0.4232\n",
      "Episode: 320 Total reward: 51.0 Training loss: 46.4260 Explore P: 0.4211\n",
      "Episode: 321 Total reward: 31.0 Training loss: 1.5277 Explore P: 0.4199\n",
      "Episode: 322 Total reward: 99.0 Training loss: 20.5415 Explore P: 0.4158\n",
      "Episode: 323 Total reward: 21.0 Training loss: 49.8342 Explore P: 0.4150\n",
      "Episode: 324 Total reward: 190.0 Training loss: 1.5660 Explore P: 0.4073\n",
      "Episode: 325 Total reward: 93.0 Training loss: 51.1021 Explore P: 0.4037\n",
      "Episode: 326 Total reward: 37.0 Training loss: 2.0783 Explore P: 0.4022\n",
      "Episode: 327 Total reward: 44.0 Training loss: 27.4150 Explore P: 0.4005\n",
      "Episode: 328 Total reward: 31.0 Training loss: 1.2908 Explore P: 0.3993\n",
      "Episode: 329 Total reward: 33.0 Training loss: 1.3743 Explore P: 0.3980\n",
      "Episode: 330 Total reward: 32.0 Training loss: 90.0787 Explore P: 0.3968\n",
      "Episode: 331 Total reward: 68.0 Training loss: 2.5739 Explore P: 0.3941\n",
      "Episode: 332 Total reward: 169.0 Training loss: 41.0620 Explore P: 0.3877\n",
      "Episode: 333 Total reward: 101.0 Training loss: 51.9687 Explore P: 0.3839\n",
      "Episode: 334 Total reward: 80.0 Training loss: 2.0961 Explore P: 0.3809\n",
      "Episode: 335 Total reward: 65.0 Training loss: 1.3380 Explore P: 0.3785\n",
      "Episode: 336 Total reward: 114.0 Training loss: 1.9196 Explore P: 0.3743\n",
      "Episode: 337 Total reward: 57.0 Training loss: 2.0645 Explore P: 0.3723\n",
      "Episode: 338 Total reward: 55.0 Training loss: 1.6788 Explore P: 0.3703\n",
      "Episode: 339 Total reward: 79.0 Training loss: 1.3417 Explore P: 0.3675\n",
      "Episode: 340 Total reward: 64.0 Training loss: 70.1281 Explore P: 0.3652\n",
      "Episode: 341 Total reward: 185.0 Training loss: 1.0922 Explore P: 0.3587\n",
      "Episode: 342 Total reward: 69.0 Training loss: 67.1897 Explore P: 0.3563\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 343 Total reward: 95.0 Training loss: 54.1397 Explore P: 0.3530\n",
      "Episode: 344 Total reward: 121.0 Training loss: 47.3858 Explore P: 0.3489\n",
      "Episode: 345 Total reward: 121.0 Training loss: 64.3302 Explore P: 0.3448\n",
      "Episode: 346 Total reward: 49.0 Training loss: 34.4985 Explore P: 0.3432\n",
      "Episode: 347 Total reward: 74.0 Training loss: 35.0932 Explore P: 0.3407\n",
      "Episode: 348 Total reward: 130.0 Training loss: 2.5263 Explore P: 0.3364\n",
      "Episode: 349 Total reward: 76.0 Training loss: 60.5445 Explore P: 0.3340\n",
      "Episode: 350 Total reward: 44.0 Training loss: 33.1582 Explore P: 0.3325\n",
      "Episode: 351 Total reward: 46.0 Training loss: 1.2880 Explore P: 0.3311\n",
      "Episode: 352 Total reward: 63.0 Training loss: 1.4582 Explore P: 0.3290\n",
      "Episode: 353 Total reward: 104.0 Training loss: 77.5228 Explore P: 0.3257\n",
      "Episode: 354 Total reward: 198.0 Training loss: 1.5249 Explore P: 0.3195\n",
      "Episode: 355 Total reward: 54.0 Training loss: 1.2237 Explore P: 0.3179\n",
      "Episode: 356 Total reward: 82.0 Training loss: 1.7854 Explore P: 0.3154\n",
      "Episode: 357 Total reward: 112.0 Training loss: 2.0025 Explore P: 0.3120\n",
      "Episode: 358 Total reward: 116.0 Training loss: 1.4570 Explore P: 0.3085\n",
      "Episode: 359 Total reward: 69.0 Training loss: 0.9450 Explore P: 0.3064\n",
      "Episode: 360 Total reward: 84.0 Training loss: 1.8019 Explore P: 0.3039\n",
      "Episode: 361 Total reward: 140.0 Training loss: 2.1334 Explore P: 0.2999\n",
      "Episode: 362 Total reward: 75.0 Training loss: 2.9094 Explore P: 0.2977\n",
      "Episode: 363 Total reward: 57.0 Training loss: 18.1824 Explore P: 0.2961\n",
      "Episode: 364 Total reward: 133.0 Training loss: 1.6073 Explore P: 0.2923\n",
      "Episode: 365 Total reward: 122.0 Training loss: 56.1993 Explore P: 0.2889\n",
      "Episode: 366 Total reward: 148.0 Training loss: 2.0079 Explore P: 0.2848\n",
      "Episode: 367 Total reward: 46.0 Training loss: 3.6002 Explore P: 0.2835\n",
      "Episode: 368 Total reward: 34.0 Training loss: 2.5416 Explore P: 0.2826\n",
      "Episode: 369 Total reward: 119.0 Training loss: 0.9749 Explore P: 0.2793\n",
      "Episode: 370 Total reward: 117.0 Training loss: 1.6382 Explore P: 0.2762\n",
      "Episode: 371 Total reward: 62.0 Training loss: 1.4070 Explore P: 0.2746\n",
      "Episode: 372 Total reward: 82.0 Training loss: 2.7600 Explore P: 0.2724\n",
      "Episode: 373 Total reward: 92.0 Training loss: 2.5500 Explore P: 0.2700\n",
      "Episode: 374 Total reward: 86.0 Training loss: 1.5967 Explore P: 0.2678\n",
      "Episode: 375 Total reward: 130.0 Training loss: 96.1389 Explore P: 0.2645\n",
      "Episode: 376 Total reward: 101.0 Training loss: 87.3475 Explore P: 0.2619\n",
      "Episode: 377 Total reward: 70.0 Training loss: 2.3725 Explore P: 0.2601\n",
      "Episode: 378 Total reward: 148.0 Training loss: 1.5031 Explore P: 0.2565\n",
      "Episode: 379 Total reward: 118.0 Training loss: 112.1792 Explore P: 0.2536\n",
      "Episode: 380 Total reward: 153.0 Training loss: 85.8152 Explore P: 0.2499\n",
      "Episode: 381 Total reward: 137.0 Training loss: 1.7995 Explore P: 0.2466\n",
      "Episode: 382 Total reward: 82.0 Training loss: 70.3034 Explore P: 0.2447\n",
      "Episode: 383 Total reward: 91.0 Training loss: 1.8676 Explore P: 0.2425\n",
      "Episode: 384 Total reward: 103.0 Training loss: 72.0954 Explore P: 0.2402\n",
      "Episode: 385 Total reward: 124.0 Training loss: 90.3143 Explore P: 0.2373\n",
      "Episode: 386 Total reward: 59.0 Training loss: 1.2550 Explore P: 0.2360\n",
      "Episode: 387 Total reward: 133.0 Training loss: 1.2349 Explore P: 0.2330\n",
      "Episode: 388 Total reward: 161.0 Training loss: 80.4714 Explore P: 0.2294\n",
      "Episode: 389 Total reward: 103.0 Training loss: 1.6743 Explore P: 0.2272\n",
      "Episode: 390 Total reward: 97.0 Training loss: 1.5255 Explore P: 0.2251\n",
      "Episode: 391 Total reward: 102.0 Training loss: 71.3923 Explore P: 0.2229\n",
      "Episode: 392 Total reward: 82.0 Training loss: 2.0358 Explore P: 0.2212\n",
      "Episode: 393 Total reward: 88.0 Training loss: 78.5940 Explore P: 0.2193\n",
      "Episode: 394 Total reward: 167.0 Training loss: 1.1797 Explore P: 0.2159\n",
      "Episode: 395 Total reward: 121.0 Training loss: 1.0248 Explore P: 0.2134\n",
      "Episode: 396 Total reward: 130.0 Training loss: 201.6953 Explore P: 0.2108\n",
      "Episode: 397 Total reward: 155.0 Training loss: 1.0410 Explore P: 0.2077\n",
      "Episode: 398 Total reward: 148.0 Training loss: 1.0987 Explore P: 0.2048\n",
      "Episode: 399 Total reward: 199.0 Training loss: 134.5375 Explore P: 0.2009\n",
      "Episode: 400 Total reward: 140.0 Training loss: 0.5533 Explore P: 0.1983\n",
      "Episode: 401 Total reward: 181.0 Training loss: 0.8945 Explore P: 0.1949\n",
      "Episode: 402 Total reward: 137.0 Training loss: 1.7010 Explore P: 0.1924\n",
      "Episode: 403 Total reward: 140.0 Training loss: 1.1660 Explore P: 0.1898\n",
      "Episode: 404 Total reward: 163.0 Training loss: 43.5393 Explore P: 0.1869\n"
     ]
    }
   ],
   "source": [
    "# Now train with experiences\n",
    "saver = tf.train.Saver()\n",
    "rewards_list = []\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    step = 0\n",
    "    for ep in range(1, train_episodes):\n",
    "        total_reward = 0\n",
    "        t = 0\n",
    "        while t < max_steps:\n",
    "            step += 1\n",
    "            # Uncomment this next line to watch the training\n",
    "            env.render() \n",
    "            \n",
    "            # Explore or Exploit\n",
    "            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*step) \n",
    "            if explore_p > np.random.rand():\n",
    "                # Make a random action\n",
    "                action = env.action_space.sample()\n",
    "            else:\n",
    "                # Get action from Q-network\n",
    "                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "                Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "                action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "    \n",
    "            total_reward += reward\n",
    "            \n",
    "            if done:\n",
    "                # the episode ends so no next state\n",
    "                next_state = np.zeros(state.shape)\n",
    "                t = max_steps\n",
    "                \n",
    "                print('Episode: {}'.format(ep),\n",
    "                      'Total reward: {}'.format(total_reward),\n",
    "                      'Training loss: {:.4f}'.format(loss),\n",
    "                      'Explore P: {:.4f}'.format(explore_p))\n",
    "                rewards_list.append((ep, total_reward))\n",
    "                \n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                \n",
    "                # Start new episode\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                state = next_state\n",
    "                t += 1\n",
    "            \n",
    "            # Sample mini-batch from memory\n",
    "            batch = memory.sample(batch_size)\n",
    "            states = np.array([each[0] for each in batch])\n",
    "            actions = np.array([each[1] for each in batch])\n",
    "            rewards = np.array([each[2] for each in batch])\n",
    "            next_states = np.array([each[3] for each in batch])\n",
    "            \n",
    "            # Train network\n",
    "            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})\n",
    "            \n",
    "            # Set target_Qs to 0 for states where episode ends\n",
    "            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)\n",
    "            target_Qs[episode_ends] = (0, 0)\n",
    "            \n",
    "            targets = rewards + gamma * np.max(target_Qs, axis=1)\n",
    "\n",
    "            loss, _ = sess.run([mainQN.loss, mainQN.opt],\n",
    "                                feed_dict={mainQN.inputs_: states,\n",
    "                                           mainQN.targetQs_: targets,\n",
    "                                           mainQN.actions_: actions})\n",
    "        \n",
    "    saver.save(sess, \"checkpoints/cartpole.ckpt\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing training\n",
    "\n",
    "Below I'll plot the total rewards for each episode. I'm plotting the rolling average too, in blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def running_mean(x, N):\n",
    "    cumsum = np.cumsum(np.insert(x, 0, 0)) \n",
    "    return (cumsum[N:] - cumsum[:-N]) / N "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1240cdb38>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXl4ZFd54P17771VpX1Xr+q92228YBs3xpiYzQn2JBDi\nQAgkw5gkg8M3Dl8YwmSZkISPJ84KyZeZEBizhGXYHIwDQyAYHIMhYONu721bvbda3Wq11Nql2u69\nZ/64i26VqkpVpSotrfN7Hj2quutR6dZ5z7uLUgqNRqPRaPIxVnoAGo1Go1mdaAGh0Wg0moJoAaHR\naDSagmgBodFoNJqCaAGh0Wg0moJoAaHRaDSagmgBodFoNJqCaAGh0Wg0moJoAaHRaDSaglgrPYCl\n0NPTo3bu3LnSw9BoNJo1xaFDh0aVUr2LHbemBcTOnTs5ePDgSg9Do9Fo1hQicrqc47SJSaPRaDQF\n0QJCo9FoNAXRAkKj0Wg0BdECQqPRaDQFqZuAEJFtIvKQiDwnIodF5Lf97V0i8h0ROer/7oyc8wci\nckxE+kXk1nqNTaPRaDSLU08NwgZ+Ryl1BXAjcJeIXAH8PvCgUmof8KD/Hn/fW4ErgduAfxARs47j\n02g0Gk0J6iYglFJDSqnH/dfTwPPAVuCNwGf8wz4D/IL/+o3Al5RSaaXUSeAYcEO9xqfRaDSa0ixL\nHoSI7ASuAx4FNiqlhvxd54GN/uutwCOR0wb9bfnXuhO4E2D79u31GbBGU4RkMomIMD09jVKKhoYG\nANra2up630wmw/T0NJZl0d7ezuzsLKZpMjc3h2VZzM7OEo/HicVitLS0YBgGExMTJBIJGhsbAZid\nnSWZTALwjaeGGJ1J1XXMUZpi8KYDO7FMA6UUT52Z4ImB8SVdM24o3vTSHSQsE9u2vfs0NTE3N4fj\nOCQSCZRS2LZNLBbjwnSKbz1znsXaLF+5vYf/cN2uBdtnZmbIZrN0dHQgIhWPN5lMMjk5SWdnJ4lE\nouLzV4K6CwgRaQHuA96jlJqKfrBKKSUiFTXFVkrdA9wDcODAAd1QW7OsDAwMFNxebwFx/vz5cHJP\nJBIMDg4WPba9vZ3e3l6Gh4cxTZO9e/fiOE54zlQqyycffNo7uPJ5rnL8b2mDyvCKvT2MzqT5s68d\nJmO71d/fv+Y//fgou3ub+b3bLsc0hMnJyfCQmZmZ8PXEXJYPfuMwU0m79D0VfPvJU9x27c4FQuDs\n2bMANDY2hguDSjhz5gxKKSYnJ9m/f3/F568EdRUQIhLDEw6fV0p91d88LCKblVJDIrIZuOBvPwts\ni5ze52/TaFY9SqmqVpXlEqyQAVzXXfTYYJXsOA4AQ0Oe0t7Y2IjZuYUz7mn+x9uu4+ev2VKnEc9z\nbnSct/7tN/nHfz/Frp5m/ss3zpPMtvGlO2/kxt3dVV0zmUzy8W/9hMPnpnju3BTDUym2dDSG+x1X\n8XcPHuW2KzdxxZY2nhw3OTzbwjtu2skHfv7Kote9518P8emH+xmZTrOhrbAQWEwDKUa1560k9Yxi\nEuCTwPNKqb+J7Po6cIf/+g7ga5HtbxWRhIjsAvYBP6nX+DSatUo1E002mw3PnUx6wqatYXkq7bQ3\nxvntn74MgD//1gsksw7vuGln1cIBwDAMbr1yE296SR8AZyeSOfvPTiR57twUf/OdIwCcGZ+jvTFW\nUjgA9LZ6QmF4Kl312C4l6vmEvAJ4O/CMiDzpb/vvwF8A94rIbwCngbcAKKUOi8i9wHN4EVB3KaWc\nOo5Po6kZ9dYgKmGxcUwlPWHR3hhbjuEgIly1pY1f/6ldPD80xcu6+/hvty7NxGIY3tp2a0cjDTGT\n//X9E2ztaGRLRyNKKb7z3HB47FODE3z5sWGu2dGz6HUb4iYCpOziU89a1ASqpW4CQin1Q4pb+24p\ncs7dwN31GpNGUy/W0qTxUL9n1W1bRgEBcNOebm7a010T+3sgICxT+OUDfXzmx6f53COn+d1b9/OX\n/9rPsQvz/of/+eAxFK28oQxzWoPlRdanslpAgM6k1mjWHYPjnjlmR1fTstwvqtHUSssKBATAzZf1\nsm9jC0eHZ/jo905w+MLC6Ky/+eVrueOmnYteNxEzERTJjDZewBov963RrBaWc1VZzr1KHZPKOhzY\n0YllLs/6MCoUohN7La5pWRa2bfO+1+3n7n95nscHxnGVxRuv28JrL99AzDQ4Mz7H664uzxmfiPka\nhF08EEBrEBqNpiLW0qSRtl0SseX76kcFhGnWrjjC/v372bBhg3ddQ7hxTxcAt7+kj9e/eDPNCYu4\nZbCnt6VsweRpEJAqoUGspf/1UtEahEZzibGYGSdtO3Qsk/8BcsfT2dlZ4sil8Zr9G9jc3sjLL9/G\nzPRU0TGUIvBBJLN20WPWk4DQGoRGUwNWm4mpGCJCKrsyGoSI0NHRUZdrA8RMg6u3thc0nZUtIHwT\nk/ZBeGgBodHUgLW0qkzbDgnr0qiDWWjiz99WSUme0Aeho5gALSA0mpqQTqcXzXCuFUHSWzEWmzRT\nWZeGFfJBLAf5/4dK7m+ZBpYhCzSIqFDQAkKj0VTE0NBQyfpItWRkZGRJ56ezy6tB1FNAFLr2Uifw\nuGWQLqFBrCe0gNBoakRQTG81UDLMdYWimNrb25flfkop9u3bt+D+5SAixEyDlHZSAzqKSaNZVyil\nyNjusmsQ+/btq4smUeiarutiGAYiUvFkLiLELWOBD0KbmDQazSVP1vEmt+X0QQDhhL0cBBN4NHqq\nEuLmQgFR6PrrAS0gNJpLjFITYtbPEL5UopgKTdZLDQMupEEslbUqVLSA0GiWyEP9IznF4VYzaccT\nEMutQSwnTU1ejalqNAjPByF1NTE1Nzcv6fzlRPsgNJolMJXM8vlHTgPwiTsOrPBoFicwMV2qGsSe\nPXsWlPOo2MRkGYzV2MSkNQiNZh3yoQf6V3oIi+K6bjhJZi4xDSIWmy8Z0tDQgGVZVfseAuKmWbIf\nxHpCaxAazRI4N7GwtPRKkx+9c/r06bBY3aXmg4jH4+zevRsRKVqQr5LV+7wPIjfZbqkmJq1B5CEi\nnxKRCyLybGTbl0XkSf/nVNBpTkR2ikgysu9j9RqXRlNLXra7K3y9WieBTCYz/9oJBMSloUGAp0VY\nlrVAQAS2/kpLjMethT6IKKv1/1wP6qlBfBr4e+CzwQal1C8Hr0Xkw8Bk5PjjSqlr6zgejabmxMx5\nM0ayRA+B1ULGDkxMl4YGUYqNGzfS1dVVcYnxuGmQsjNF92sBUQOUUg+LyM5C+8QzDr4FeG297q/R\nLAe2Mz9ZzKWLZ9+uFtL2peWDKIWIEI/HKz4vpktthKzUU3IzMKyUOhrZtss3L31fRG5eoXFpNBWR\njQiIqWTpInqrgdmMJ8Tal7EfxFojbhrYjsJ25jVCnUm9vLwN+GLk/RCw3TcxvRf4goi0FTpRRO4U\nkYMicnCpRcs0mqViOy4tCc+EcXJ0doVHU5wgomcmpQXEYsR9/0yxtqNaQNQREbGAXwS+HGxTSqWV\nUhf914eA48Blhc5XSt2jlDqglDrQ29u7HEPWaIqSdRQb2xrobo5zdJmS5ZZSsmI2bSMCrQ1aQBRC\nRIj7DYeijupaCoW1JGBWQoP4aeAFpVRYG1lEekXE9F/vBvYBJ1ZgbBpNRWRdF8sUtnc3cW5ieaq5\nLiYgSk1AM2mb1oSFaSxvj4a1RKBBFOsqt5Ym+KVSzzDXLwI/BvaLyKCI/Ia/663kmpcAXgk87Ye9\nfgV4l1JqrF5j02hqhe24xEyDxrhZ8/o9xShngip2TDLj0JLQ6U+liPkaRLpIstx6EhD1jGJ6W5Ht\n7yiw7T7gvnqNRaOpF1lHYZkGjTFzQXLVaiTtuDTEtYAoRSLUIFb//7PeXPqxbhpNHUnbLgnLIBEz\nSNnuql1dBuPK2i4Nl0gWdb0INIhouQ0dxaTRaCom7TffabBMXFeFeQbj4+P09/fjOLU3Oy1lgkrb\nLo1xLSBKEXTb0z4ILSA0miWRzjo0WJ6JCTwnMMDExAQAtl2b5LlaTUoZx10XSXJLwTIWRjGtV/ST\notFUiVKexpCImWHpiiDPIGC5uqiVSzrrhsJMsxARIREzECBZJMxVaxAajWZRMvZ84btgVT5Tp3Ib\ntRI0GcchoQVEScIophoGHaxVAaMFhEZTJUGmbVSDmF0l9ZiKTUKZrHZSL0bMEECFlW+j5JdSv9TR\nAkKjqZKwMqplzAsIv9bRSkwiWcfl7x86xsDF4iU/0o5LY1x/7Uth+EmEjrs2V/21RD8pGk2VBE7M\nRERATNfJB1HOBHX8wgxPDkzwkYeOFT0/o8NcFyUQENkCGsR6QwsIjaZKvvP8MOD1Vpg3Ma1c5EvM\nWhi/HyVwqusw1+KICEGLj6gGEd2/ntACQqOpgrmsw4+OXQS8uPnASZ3vg6iVaaKSienYcOGigbY/\n4a2HZkFLwRTvf2lrE5MWEBpNNbiRySNhGTRY9Y1iKodskf4F4X5bC4hy8IOYtAZBfVuOajSXLI4T\nFRCmFz9vGcykbUZHR8lma9s8KDrhf+knZ4hZwn+4ajNNEXNR0LxIyG1kFBBE5ehEudKICCLkNAxa\nr2gBodFUQdadnzyC0gwNMZPZtM3Fixfrd1/H5bu+7+OJ0+P86e1Xz+8LG9woMgVKfAQahk6UWxzL\nkIImpvWmQeilhEZTBdHJo9Uvn90YMxaYmGptux6eSoWvz0+lc/YFQksonOQVlK/WJqbFsQyjoIlp\nvaEFhEbjo5Ti5MmTTE9PF9x/8uTJ8HVgYupqjoWryoSvQdSD4B5TyeLXD3wMIgtDNNPpdGh20iam\n4gSfs2XkLgJqyVpyeOsnRaPxUUqRyWQYGhoqui8gmDx+5WU7wm0NBTSIWo4NYCpV3LcxLxRUmMQX\nJag0qzWIxTENyfFBaBNTjRGRT4nIBRF5NrLtAyJyVkSe9H9+NrLvD0TkmIj0i8it9RqXRlOMQl9+\n13U5efIkqVQqZ7vtm3NMQzCMwAdhMZOXB1Hr1eJksrCAUEqFAkKgoIDIOi6u0gKiHMw8H8R6pZ5O\n6k8Dfw98Nm/73yqlPhTdICJX4LUivRLYAnxXRC5TSul6u5plJzqpJ5NJMpkMo6OjOcfYvrkmZght\nbW1MTEzQGDOYnbYxDAPXrU8EzIXpdMHtd33hiRyhkHFc4nnHZGyXMdWkndRlYBqiw1ypowahlHoY\nKLev9BuBLyml0kqpk8Ax4IZ6jU2jCRgbG+PChQsVnxdMHpZpLPBB1GMSCa55YSq1YJ9SuSYlQZEp\nkE2ddVxcDK1BlIFlGDpRjpXxQbxbRJ72TVCd/ratwJnIMYP+No2mroyMjDA+Pg5UNglETUw9PT0A\nNDY0MJO26zKZBNcs1MQmP1xfmPc3RAmEiNYgFscypWgeRDX/37UqYJZbQHwU2A1cCwwBH670AiJy\np4gcFJGDIyMjtR6fRlMWgYlpw8bNGIZBc3MzjTGTtO0WdG7Wioy98Hp2njlLUCUFhI5iKk6gqRlS\n2Afx3LkpDp0eX+5hrRjL+qQopYaVUo5SygU+zrwZ6SywLXJon7+t0DXuUUodUEod6O3tre+ANZoi\nBJNHW1tLuC1YmSez9Su3kR++6riq4ESWX3bj7HgykkmtNYjFsIzC5b4/+C/P86EH+ldqWMvOsgoI\nEdkceXs7EEQ4fR14q4gkRGQXsA/4yXKOTaMpRGB+yieYlOPm/GQbVElNFWl2XwuyruvZkMJxuAuE\nRr6J6ZvPDPEnXz/M8RGvT0TC0hrEYph5Poj1St2imETki8CrgR4RGQT+BHi1iFwLKOAU8JsASqnD\nInIv8BxgA3fpCCbNamB21ptU801Fjj8pxyxvthYRf2WuSGXrZ2LK2i6vvqyXsdkMTw9OknVUaO4K\nEFSk7Ab0+9VdL86kaYg1rrtInGoo5YNYKmvJH1E3AaGUeluBzZ8scfzdwN31Go9GU0uC1WXQvxjm\nbfsp20EpRcp2eezUGDdf3hg2oVkqWUcRNw2u29bhC4iFGgTk5kEE+12ltIO6TEwpXItpvaGL9Wk0\nVRDUPYoKiKCyajLr8r3+ET7/6AAX3GZ+62eSvPuWfTW6ryJmGWFzoKytilRudQBvPIGASNuu9j+U\nSb4PIkCxvspPrKe/VaOpGUGx1Kg9vzFmIkAyY/P0sJev0GUkeexkbaLtHFfhuoqYIaFgyjpugSim\nPA3Cj3yaSztag1iEwPymM6k9tIDQaHyqyYMIJmoRoaslDiguzmRIxDzl3MIlkZmqyfiyod/DwPL7\nYqZsJ0eD2N7VBMwLiEdOXOTsRBKAZNYhoQVEWVimoftBoAWERlOQ4eFhnAI9FQJsVyHirTQDmmIm\n3U1xLkynGJ1Js6k9wd4NLUzNLsx+roYgMiluGsR9wfTn33yBbz/rFRe87apNvO91+wH42PdPAPCJ\nH5zMuUajzoEoC1OkYJhrEEK2XoSH9kFoNAWYmJgoud9xFVYBx3NPS4Lp1Cwj0w4v3thCzBQOnSlc\nPrxS5vzw2aaEmeP7eOasp6G8dGcXTQlPQyjmEtc+iPLIbxgUEGxJ2y6WeekL20v/L9RoirCYECiF\n7eROEIHtujlhkMw4jM5m6G6O094QYy6VZnYuueTxzmW8BLymuJUjIAJiZlQsFDaXaQFRHkYRJ3Xw\nsRaqlnspogWEZt0yPDxccn80XyCZzJ3g7SIaRHPC4vxUCsdVdDXHaWuMoRQcPnJ8yeMNNYi4mScM\nPCy/7PiunmbAC2vNp7VBGw3KwTKMBdnokKtBrAe0gNBoqsB23QUCQilFU8xiYi6LQuhqSdDWGANg\nYq54o59ySeYIiIVf3cBxfWBnJwI5CXsBvS2JJY9jPVCs3HdApRrEWs2j0MsJjaYIpTKObQdMc6GA\naGVe0+hsjKEava/Y0OTSTUxpv5JrwjIxCowtEBAJyzMjBe1PRSCYn3pbtYAoxeJhrt7+TIkAhsVY\nS8JCaxAaTRFKCog8DUJEsG2bJvEa+ig8Z/Lm9kZilnBqdG7J4wmK7cUto7CJydcq4paBCJwc9cqE\n/PJL5+tgagFRHsWK9QUU0s4uRbQGodEUoZSA8KKYFq6v2hpi4evGmIlpCNs7mzg9Nrvk8WQjXewK\njS3mC6z2xhhtkuLB5z0fS9QcpQVEeRhSuFhfsCWzTsJctQah0RShlIDIOm5o0onS2+ZNwAoJs6x7\nWxsYn8sseTyB09Qyi2gQvoC4YnMrFi6ZQKBoAVExllG6WF96nWgQWkBoND6V2IYdV2EW0CC6mucn\n4EDANMWNMAJpKWQcF9MQzCIaRLBNRLhyaxtnxjyzVjwqILSTuiwsc2GxvujzsV40iKImJhF5gmLB\n1IBS6iV1GZFGs0oo7YNQoUknemwQRhr94jTHLeYyDq6rllTVNWurgtFLhWjy8x0sQ7hic1u4vbMp\nXvX91wOhk9rJLIhiir5dL3kQpXwQb/Z/vwuvLOTn/Pe/CuheDZpLjpmZmZz3paOYFPFEcR+EiuQy\nNyZMUDCTsXN8FJWSdV3iVnkCJkiI29rZGGZXAzUrO36pI3ZqgQ/CVSr8v6bt9TEFFl2OKKWOK6WO\nA7copd6rlHrC/3kf8DPLN0SNZnnIT5xbTIMolChXqNZRS8ITCgdPjS1pfFnbJVbArFWIINS1R5uU\nqsLM80EopXI0iqVoEBNzGd73T0+Ryq5+IVPO02aKyI3BGxF5GUGheY1mneK4hUttiAi/euMOPnnH\nS8N9L9neAcDBU0trdp9x3LCDXT4v39Od8z5oXtSc8IwE/+8te7nrNXuWdP/1hGkIrgLXFwqTk5M5\nAmIpmdSfe+Q0Xzk0yLcPn1/yOOtNOWGuvwF8WkQa/PdJ4NcXO0lEPgW8HriglLrK3/bXwBuADHAc\n+DWl1ISI7ASeB4Ju4I8opd5Vwd+h0SwrozMZtvmltfN5zf5eNm1q4/x5z0ncEDPpbU0wMLa0XIis\n7RIzF67NPvxL19CSV0Ij7kdQBQ7qF/d1LOne642gSm/GcVHpDLZt5wiIJwbGedsN26u6dirMiF/9\nWQYlNQgRMYEd/gT/MuBlSqmrlVKPlXHtTwO35W37DnCVUurFwBHgDyL7jiulrvV/tHDQrFqeOTtJ\nxnZ5fmi+z0N+aXAjzxTU25LgzHh12dRTU1O4rkvKdkLNIEp7Uyyn7DgQZlrHY9rnUA0JX7Cmsk74\nv4027gt6bFSKiISmpeb46jfElBRhSilHRP47cJ9S6mIlF1ZKPexrBtFtD0TePsK8I1yjWTOc8yeH\nqaRd9Bgzb6Xf05rgsTOVaxCpVIqhIa/fQ9p2aUnMf2X/8k1XU8zScf2OTs6Mz3HTnp6K77meCUyF\nQUvXVNalxf9XOn6TKEGFXfqqIWk7rJUc5XJ8EA+IyHtEZLOItAU/Nbj3rwPfirzfJSJPisj3ReTm\nYieJyJ0iclBEDo6M1KaVo0ZTCcEkPaWKO4AtK3cC6GmJMzGbCkt2l4sbaSeayjo55bq7WxJsbCs8\nht7WBO+8eTeb2hoK7teUJkhyTGadMP/BcWuTSR2U6ciugZam5Yix/+j//p3INgVUZ4ADROQPARv4\nvL9pCNiulLooItcD/ywiVyqlFvRqVErdA9wDcODAgdX/CWsuOQL7vkq0FD1GRGhubmZ21iux0dYY\no1vmuDiToamr/NVjNDkrnXULmpg0tScWMTGR8LQKV0X6fC9BQMykbCC+JrrSLfqkKqW2LXZMJYjI\nO/Cc17co/+lXSqWBtP/6kIgcBy4DDtby3hpNMSrNogb4/H8Og/sWnC8i9PX1oZRiaGiI1sQElriM\nzRZ3bi9GynbD8FVNffGc+4pU1kEp7zOPOqkrFRDR52PGr7Ib7SW+WilrKSMilwNXAKG+qpT6QqU3\nE5HbgN8FXqWUmots7wXGfJ/HbmAfcKLS62s0taSY0LD9L3Znc/Gs5GjYq+u6tDRYuAhjs9XVZHKV\nN1k16o5wy0LMMkjgMJNMQ4cn0AOZkLDMmkzuS9FClotFBYSIvB94HXA58G3gVuCHQEkBISJfBF4N\n9IjIIPAneFFLCeA7/hcoCGd9JfBBEckCLvAupdTSsoo0mhLkT/6VaBBBhm2hgnkB0SS79vZ22hrG\nyCizYgERjGsqaaMUtOtSGctC3BQ2GDMMnx1AbdoPzDupE5bBdJV5ENHnwnYvAQEB/DJwLfC4Uurt\nIrIZL4S1JEqptxXY/Mkix94H3FfGWDSaFSdY+SUK5CQERCeC1tZWmhssHIyqNYigGmxXc/WlOjTl\nE/iZohnTgdIQt4yqV/9RobAWTEzleLySSikHsEWkFTgP7KjvsDSa5aWQBpFKpQoe++XHzgAQjziM\nC/kgojTFTGIGjFVZ9nsq5bUsXUotJ83iBP+3uC/8M7YbiWLyFwax6gVEtEy4vQYERDkaxBMi0gF8\nCs9pPAX8pK6j0miWmUICYnJysuQ5hSqrmqa5IGkOvImnvTHG2Ex1JqYg7j5eZjVXzdII8iAyzryA\nCBb/CdOoevU/l3EQP1j2kjAxKaV+03/5ERH5NtCmlHq8vsPSaFY/hYr1bdq0iZaWheGvIkJnU5zR\nmXRV98pG2o1q6k/c9y/lmJh831MiZlSdBxGt4bQWTEzlOKn/EXgY+IFS6lj9h6TR1J9qndTRUMdS\n1V7zERG6m2OcnSpstipGqEFEuslp6k+8gAZhh05qk6y/vZJnALzEu4C1kAdRztP2BWAX8HEROS4i\nXxaRu+o8Lo1mVbKUKp49rXGGp8rXIJRSoR8kEBClIqc0tcMyBATsyP87WBskLAOlWNBQqByiAuKS\nCHNVSn1HRL4LXA/cAtzlv/5Incem0VRFJpPBsqwFBfNKUa4GERRa29lTWbKbp0HEuTg7RdZxy+oM\nNzY2xvi4VyI86yweWqtZOtH8lbhpkHaiTmrfD2SZgEPWUVSat5jKERCr38S06FPq+x1+DNwBnARu\nVErtrffANJpqOXnyJGfOnKnonEoFxOuu2FTx+d3NCZSCC9PlaRHRKKp5DcL7yjY1VZeNrSmfuClk\nC/kgCoTAlksqE4liWgNO6nKWWEfw6ibtwyt/sVdEdLaOZlVTLEQ1oJLEuChzQS3/RGHlu5hNOtAg\nAM5PVuaHAE9AmIaEZbwr0Y401RGzPGd08Cw5Kk9AVGEiyvVBrH4NohwT07sBRKQd+E94vak3AI31\nHZpGs3yUKzCCOjoticpLXgRJbsMVOqrBM0dEzVJaQNSfhGmQsd2wou68icn77KvxIQQCojFWm3Id\n9aYcE9O7ROTzwGPAm4DPAm+s98A0mlpj23ZO+ewolQuIyur5iwjdfn/oc2PTZZ0THVPGcYn77Ub3\n7t1btoDQpqjKEBF27PDygC3LJOOoSB6E97thCQIiY3utahMxY02YmMp5yjuAfwAeU0pVlwaq0VRJ\nkHSW34CnGo4fP45lWezZU31v5tklCIjWuEmnlWX47BmSyW00NpavhGcdl5gvFEzTLDu8Mr8vhWZx\ngs8s8EGEHeUCDcIvmFiNgLBdRcwQYmb12djLyaLLEKXUXwAO8FYAEekSkap7QWg0lXDs2DGOHatd\n+o1tF27YU76Ter5gWzVsblRMpbJks9mKzrMdVVYEUyyWW4qj0jh9zTxx3weRyXjr4tAHYQZO6vJN\nRGEklFKYhhAz5JIxMb0frxLr+/1NjSxSyVWjWe1U66S2Xc9ZnD/xLnY9EWFmZobOBMymyusql29i\niibJFZv4N2/eXNa1NYsTN3Mzpu1IJjVUp0E4jicgLNNYE4ly5eifbwauAx4HUEqdrVHLUY1m1VCu\nwLAdVbDERsBiK/aWBovp9OICwnGcnDHl506UqxkEx/X09NDR0VHWORqPmGWQseejjtwaOKkdpTBF\nsExZEy1Hy9GT037nNwUgItrrpVnTBG1Ao5QtIFx3SclqLQmL6TI0iGPHjpFMJsP3Wbu8++YLDomE\nxdbCj7OeiJtGWCQRPAGhkLBgYjVhrq4bmJjWhgZRjoD4qoh8BGgXkV8DHgD+sb7D0miKo5QqGo1U\nDovlSKSyDh/93nHG5xb6CWxHVVUPKfB9bO1oYGQ6zUSFZb+zjgorjMJCQbBp0yb27NmjfQ41JG4Z\nZCPPmaN5iSt0AAAgAElEQVS8zOlAg6zGh2C73vmWKWsiD6IcJ/VfAt8Avg5cA9ytlPrbxc4TkU+J\nyAUReTayrUtEviMiR/3fnZF9fyAix0SkX0Rure7P0awHBgcHOXr0aMF95WgChY6Jbnv05BiHTo/z\nga8/m7N9dCZNMutgFVjJl6uBbO9uBmBgbG6RI3NZrDyHYRg6YqnGxAwjp3+Do/DNQ76JqYpMasd1\nMQyv6OKlYmJCKfUtpdR/VUq9B/hXEfnlMk77NHBb3rbfBx5USu0DHvTfIyJX4EVJXemf8w8iovXh\nS5xqNYG5ucom12L3LvY+UP1n0w6nLs6F+3//vmc4eGq8oA8iCFldbJIOGv5U2lnOdtycXhDFTEnF\nNIhqnfLrkbBpUMzzE4QRSI6LZRjh/78aE5PtgmkYC8p4rFaKCggRaRGR/yYi/7+IvFY83gUcx8uo\nLolS6mEgv6/0G4HP+K8/A/xCZPuXlFJppdRJ4BhwQ4V/i2aNcebMmaKawEoSrdKZ9EtrRM0JVgFb\nfk9PDzt37iQeL12Fpq3REyCFzFel8ArDVW4+0ian6okbBq6rCORAEKIayGm7Cg3AUa5nYjLWfqLc\n/wZm8Ar13QX8IZAA3qKUOljl/TYqpYb81+eBjf7rrcAjkeMG/W2aS5ioE3alyY8Yyn8d3RYroEGI\nCIlEYtH7tCQsRODkyEzZYzs5OstUKktTbF4wlar5pKkNQUvZjOtimSaO6zv7xdvuVDHBuy4YBogp\nJLOrX6srJSD2KKWuBhCRj+FN6NuVUjX5ViullIhU/AmJyJ3AnQDbt+t8PU11DA8P57w/f/48AONz\nGe5/4ly4PRMKiKgGUf0kHDMNrtzSxqMnL5Z1/Nhshrv/5XkAulrmtZNiJiZN7Yj5mmLWdiFmehFs\nBgQfdXVOahdLDDDXhgZRygcR6sBKKQc4UwPhMCwimwH83xf87WeBbZHj+vxtC1BK3aOUOqCUOtDb\n27vE4WjWK4VCXQG++njuY3dkeJqTo7M5GkSpPIhiNDQ0hK/7OptITo6V1XBmJpIz0d2cyLlOlMV8\nEJrKyW87ajuKeMzE9P//1TQMclwwDe8ZWutRTNeIyJj/Mw68OHgtIvm+hXL5Ol5fCfzfX4tsf6uI\nJERkF15p8Z9UeQ+Npmryez4/9MIId//L80sWEJs3b6avrw+ADa0JHFdxvoyqrkF58b6uJm694UXh\nNSrVILSTunLyE+KyfqBAICCq8UHYrsI0DC/MdQ1EMZUyMS2p54OIfBF4NdAjIoN45Tr+ArhXRH4D\nOA28BUApdVhE7gWew+s9cZevtWg0FVFtmGtAsbk/18RUeR6EYRg0N3shrp1N3ldreCrF1o7SBfuS\nGU+D+PVX7KS9pXnRZLd8QRHUZsqv0aRZnPyEOK+iroXpf8bVJLo5rotpGJhrJFGuqIBY6gStlHpb\nkV23FDn+buDupdxTo6mG54emMETYv6k1p+NXlA99uz98LY1tYUnoamhv8ibrC5H+1BcuXGB8fJz9\n+/fnHBs2KIrnVnAtV4Po6OggFouFwklTPkHeScbPhcjaLomIBlGJiWm+bSkYQamNNWBi0pk1mlVJ\nf38/bW3LU/Lrww8cAeATdxxgLlu4DEa0E1jMsor6Asqho9EXENPzJqag93Q+gQ+iOa+8eCV5EFo4\nVEbwGSbCKCZvIs86iljDvICoZoJ3XC/MNWYYVfkwlhvdlkqzapmamiq5vxZ29eg1nhgY5+Solxj3\nrlftIWYVXpWXymguh9YGC8OQsjrLjc9mSVgGjbHSPSC0c7r2WEaQMR3kwng+iOCjribM1VF4uRSm\nrPkoJo3mkiEqCKKvpyKF8z7y0HGmklmu297BgZ2dNMUKK9gzZVRjLYWI0N5o5ZiYomOLjm88maHT\nb1UaFQL5GoyOYqo98bze01nHJWGZ8z6IqqKY1lY/iKImJj9yqdBfIHhpDF11G5VGs0yMTC+cpJv9\nftPNCZPJ5HzGswgoBens0uMnuprinL64eMmQidlM6NSOTv6xWIy+vj4GBweXPBZNYUIBYUeimCzB\nMARDqCpM1XbXVj+IUhpED9Bb4CfYrtGsKIsV3SuH0ZmFAiLQHBpiuRFDXf5EnamB7XhPbwtPnpko\n2FMg+BvSWYfjI7O0N8bC8NYoUd+C1iBqT34eRNZRoXnRMo0laRBrJcy1qIBQSjnRH6AdrzRG8KPR\nrHmCKKEojXFPMMTzfA2v2u+ti2qx8tvY3kDGcRdoMEeOHAkd1t8/OgLA0GRqUUezFgy1Zz4PIohA\nUmFfcC/RrZowV79hkLHGBUSAiPyciBzBq4/0qP/73+o9MI2mlhTTLFIFzEXBxHDT3h5etLk13L6p\nzbP7V9KLOJ9du3YB0OmHuhZKlhsdHfUH7f16y4FtC47JRwuI2pOfBxGs/sFzNC9Jg/CjmFZ7AmM5\nTuq7gVcA/UqpbcCtwA/qOiqNpgxq8eVK2QsFRGBGuGlPN7/zuv30dXrJbEH+wlLq+MfjcTo7O+lq\n8YTN+cnikUwzaRvDEC7b2FL1/TSVEzXXxUwJFxGuUpim99zFzOrCVG0/DyLoDrjaHdXl5EHYSqkR\nETFERJRS3xGRD9V9ZBrNIhw7doy9e/eW1UqzmDBJZ90F6n5+tdb3vu4yLkyl2dDqVWu99YpNSxi1\nN/F0NFpsNSYZODvEYGdhp/dM2varvxbXDmKxGNlsVmsQdaKzKR727nCV18sBAg2ichNT1lVYBkuq\n57SclCMgJkWkBfgh8FkRuQCsnjrNmnVNKrW4fR7g3Ln5Cq1KKZqampibmyOZdWhrtBibnY9WiuXV\nY2priIWNfj76H1/Cjm0LHcaVICI0x03iJoyOjTM7W3j80ymblkRp4bd9+3aSyWRBAbFt2+KmKU1p\neloTjM6kUUrhuArD/5xjVRTbc5VC+XkQQYpN1nVpZPX2RivHxPQLeALhPcD38Kqsvr6OY9JoqiZf\nUzh9+jRjY2NFu9ClbZdEXrRSkCBViJhpYJTYXy4iwqa2Bo5dKN4X4sz4HK0NpddwlmXR2tpacF9T\nU9OSxqjxkhrn0g7BQj8o1GhWEYXkuAqFFyYbrEFWe0XXcp70P/AjmbJKqU8qpf4GeG+9B6bR1IJU\nKsXIyMiC7YEgse2FvZ6X0u+hHILV/lVb2zk1OlvQ/HX43BSj0xmOjxQuS16KLVu2sHPnzqUOUwPE\nTCHrurj+/ygwDXkd4SrUIFyFwutrHVaEXeW5EOUIiPy+0gA/V+uBaDTVsFRHtWcTzqtrtMg5tbL3\ndzfHcVyVk80dcHbc03iqWWG2traW1d1OsziWYZB1FK6bLyCk4lIbjgsK8TOpvW2rPdS1VCb1bwLv\nAi4Tkccju1qBQ/UemEZTT+ara7oLBES9HYeBgOnwo6KGJ1O0JCzSWZcm3+cQ5Gf87q37tQN6BYmZ\nBlnbJZDT0TDXSiOQbOWi8ISLEWoQa1RAAPcCDwJ/Dvx+ZPu0UupC4VM0mtVPtN6R7SgSMZOrtrbx\n7FmvOOBi8qFWE/a+ja0kLIOfnBrj4aMjPHJijGv62nn1/g2MzqTpbo5z2abC/gXN8hDzfQ2BthBE\nMVUa5uo5uQl9EEGkXHaVF+wrlUk9rpQ6ppT6JaAB+Bn/R5fZ0Kwalmpisn0T012v2csrL/Me7eZF\nIoeWSiBgWhIWeza0cGR4mkdOeE0anxqc5O8fOsboTIbuliX17NJUSU7NK18QBNpC4J6qJlHOcT0N\nwjQkvM5qD3MtJ5P6LuCfgO3+z70i8l+qvaGI7BeRJyM/UyLyHhH5gIicjWz/2WrvodGUS1hd0zT4\n1Zdt57deu5cXbS7dh6KWJp89vS2cm8hNllPAhek0Pa3aj7DSBAEMab8eUxDAUE2pjUCDsAwhiIso\nVItrNVFOHsRvAjcopWYAROTPgB8B/1DNDZVS/cC1/rVMvLDZ+4FfA/5WKaWT8DRVU6lGYTvzUUym\nIVy7raMew8ohKmB+am83Pzo2SmPcZGwu44VUuoqpZJaeFi0gVppAIKT9jPvAxFRNsT1XecLAEIm0\nLV3dGkQ5AkKATOR9lsUDPcrlFuC4Uuq0dsRpqqFaE1NwXjZSX6dclvqsRvMoulsS/OWbXwzAVCrL\ndNLmT75+GEALiFVAsHj46uNnAcKJ3TIM5jKV9QVxgjBXQ8LAiNXeNKioiUlEAuHxOeBREXm/iLwf\nT3v4TI3u/1bgi5H37xaRp0XkUyLSWaN7aNYYlUz6juMwODiIbVfXxMcpEOZabwol2okIbQ0xtnY2\n0uInx+3oagr3aVaGQEA8d84LYAjDXE2p2H8QDXO9FBLlfgKglPorPDPTnP/zrlqYgUQkDvw8nn8D\n4KPAbjzz0xDw4SLn3SkiB0XkYKEEKM36YnJyktnZ2YI9nYsJmuh221kY5roYtdQgAqJ5C7/1mr3c\ntKebq3ZvXdJ9NEsnlpc0Gc2DqNTEZIeRUJFEuVXupC5lYgo/GaXUT/AFRg35D8DjSqlh/x7D4Y1F\nPg58o9BJSql7gHsADhw4sLo/XU3dqcbEFO11bbte85flZDEBs3dDC3s3tNDT0834+NgyjUpTiPws\ne9MQlPLMkpWu/j0NwjNTBTWd1rKTuldEipbU8EtuLIW3ETEvichmpdSQ//Z24NklXl+jKclTgxOk\nsk7FPoilUq4Gstp7BawHFpRhCU1MRsX+A89JvbZqMZUSECbQQu0c0iEi0oyXU/Gbkc1/JSLX4gnZ\nU3n7NJcwSqma2dmjk+piE+x9h7x+zls6Giu6x1LHmkgk6OzsRCnFzMzMov4T7YNYOfLNj14zqYUl\n4svBcb2e5laOk3rtCoghpdQH63FTpdQs0J237e31uJdm7VHJyrnSVfYHvn6Y67Z18PprtrClo5GR\nmTQv391V6RCXhIiwYcMGwBv/5ORkQSFgmiZdXV20tZXOy9DUj3wfREvCAjJVmpgWJsqt9iimsnwQ\nGs1qQkTKEgz5xySzDoPjSQbHkySzDqmsw5b2xopX6LVc0S/2d/T26sIFK0m+iam1IQZ2hphReUc5\n73jP/7BWajGV8s7dsmyj0GiqJJvNLn6Qz4WpdPj6uaEprxeEVbmDWpt81g/5zaOCEGSvH0Q1mdSB\nicnbttqd1KVqMenwCc2aoRyNYjo9b+vvbo6TWcUCYtOmpbU11dSG/PazTX5zqViFPgilFI5fzdU0\nZc20HF3e+D6NZhnJFxqZ7Hzv55ZEjLTtLOgmVw7LYWJqb2+v2T001bMgzDUsy2JUFeYKfsOgIMxV\nCwiNpjT5k+Ri2kCpCbrUuZmIOp+23VVhYgrGq81Wq5N45Pm4fsd8cQfTqNzB7DmpPf9D6KRe5Sam\ncmoxaTRrAqUUo6OjRfcHFTm7m+Oksk7VAqIetLW1oZQimUyu9FA0ERpiJne+cjfdLXH29LaE272O\nctVpENFqrtrEpNEsE9PT08zMzITv87WJoCJnV0ucqbRNOusSt+rb+6FcTNNk+/btKz0MTQFu2NWV\nIxzAm+SzjqoozDradCg0Ma3hKCaNZk2x2Jf13se8xLitHY2cn0ziuGrFNYienh4SiQRNTU0rOg5N\nZQTlWSpRABz/+bRMI9QgVruJSQsIzZqjmlIV0deb2htCB2MitrJfgUQiwc6dOwsW8NOsXoIopErC\nVB0/RsIyjPlaTNrEpNFURtRMtBQmJibC19EQ17aGWPg6ETExaUexplysKsJUg4ZBZqQrnbPKM6m1\ngNAUJZlMMjs7W5drl6qZdP78+bKvc+zCDEOTqYL7oiXAR6a9JLl3v3YvbQ3zsRnJjLPgPI1mMQIT\nUyW5EHbESa2U8rrSaR+EZq0yMDDA4OBgXa5dK8HzF996gT/652exbbtkBNO5cS86aEvHfEOefKIa\nREuL55Ts7JwPbWxra6O1tbUWw9asccJie5WYmJSLaQiGYaCUImYYq95JrcNcNStCLWzuUfV+enq6\n5LHnJlPELYOeljhTqXlz0y0v2hC+DgTEtm3bSKfTC0xdGzZswDRXR9STZmUJelVXZGJy5/tJTE1N\nVZVLsdxoDUKzIuSblSYmJnDL/LIEE/nYbGaRI+c5N5lkc3sDIuJX5PSIZsoG11VKrVgCW0dHB83N\nzct6T03lBBpEJU5m21HETCMs795sOGu63LdGsyzMzMwwPDxMJpMJy2CXw3NDXme4xkXKZUwmsxw5\nP8012zoAijYIWg1O6o0bN670EDRlYPoasFOBiSgwMQXEzNUf5qoFhGZFiGoQjh//57pu+LocJua8\nSq5dzfGSx/3OvU8B0NEYK3ncYgJiNQgQzeog6BNRiYnIVbkNiNaCk1oLCM2KExUW586dK/u8ZMZT\n1cv9knY0zQuS3/7pfTTFC2seK2li0qxugufBrKIjnOO6WBHfmyWGNjEVQkROAdOAA9hKqQMi0gV8\nGdiJ13L0LUqp8WLX0KxtpqamwtfRybiS/g5zfnXWjJ0rII6PzLCxrSHH1wDQ2TyvQVy9dWG1VK1B\naMrFqrDhj1IKx5l3bkN1Bf+Wm5V0Ur9GKXWtUuqA//73gQeVUvuAB/33mkuQZDJZMOqo3Ak4OC7I\nYRify/LB//NcuP/Pv/lCaFaK0r6IiUmjKZdqEuUcpXKCIixz9Ye5rqYopjcCn/Fffwb4hRUcy4oz\nNze3aDP7tUp+tFLUxFTJKv3izHwU08DYnHdt/1rBF/f81HwS3b4NpXMYovfWJiZNKcJSGxVoAI6r\niLaXsAxZ9U7qlRIQCviuiBwSkTv9bRuVUkP+6/NAwXAOEblTRA6KyMGRkZHlGOuKcObMGU6dOrXS\nw1gWqpmMZ9I2A2NzvOGaLfzUvm7AEw7ZiLnp4Klx3n//swC8+fq+otFLAdEw11L7NZpYEMVUoQYR\nfQYb4yazqzyTf6Wc1D+llDorIhuA74jIC9GdSiklIgU/eaXUPcA9AAcOHFjd+tkSqSSiZy1TjYCY\nTnm+is3tDTTHTeAicxknZ3L/3pEL4esDOzvzL1HWmDSaQlRXrM8NBQtAS8Li1FT5PreVYEU0CKXU\nWf/3BeB+4AZgWEQ2A/i/LxS/guZSIp1OL3pMfiOd6ZQnPFsSVlg6Yzbt5DisXxia93P0tCSKXjvI\nji6UJa21Bk0hYlVkUjtqvmUpQHPCZDq1us3Iyy4gRKRZRFqD18DrgGeBrwN3+IfdAXxtuce2nlFK\n1a0wXymGJpN84qEXcJUqKSgGBgbC1yLCjK9BtDRYNMc9AXF2fI7fu++ZBedetbWt5Bi6urrYv39/\nURNTtB6TRgNgVBnmGouYmFrjFlOp1a1BrISJaSNwv/9ltIAvKKX+VUQeA+4Vkd8ATgNvWYGxrVsm\nJia4cOECW7ZsWdaCdB/8xnNkbcX2riZuvkyIx0snvQXM+jkQzQkrVPP/4XvHCx77W6/ZV9Y129ra\nmJ6epqGhAcuyGBsbo6mpiYaGhooyvDWXPkE+QyWJbo6bG+bakjCYSdu4rgoFzmpj2QWEUuoEcE2B\n7ReBW5Z7PBqPIP+gkjyEmtzX9r5gn/nxaTa2NXDV9u6yzktlPaHQaJnYiYXhq790oI9/OjhIW6OV\n86UsRKA5tLS0sH//fgDi8Xj4WqPJZ75YX2VRTJZp0NTUxNzcHC0JC6VgJmPn9ChZTaymMFfNCrIa\nbO3DUykymfIK8KX9JLlEzKA5sdB3sG9DC6/a38vv3nZ5Tceo0UCkWF9FtZi8KKa+vj6AMJN/Nfsh\ndKmNdYTjODiOU9CMs1iIZz3Iv1e069tipGwXyxRMQ3JKZuzpbWb/plZ29TSzO6/RvEZTK4KGQZWV\n+1ZYhiDi/TSHAiILNNZjmEtGC4glkkwmaWys7T+3XpP0yZMncRyn5qaTaj+DIOKoIWaSyjrMRATE\nZ398msPnJnnpzi4eOznGn95+FUqp8MuVyjo0+FVcDRH+19uvZ2Bsjl09ulS2pn4ECymrCie17bph\nJrVhGKGAmEquXg1Cm5iWwMzMDAMDAzm9j1czpfIqqtUgpqamGBgYyKmtVPa5vmr9Ky/bTldzLEdA\nPHxkhIszGf712fNcnM1wfCS3eU8669IQm398TUO0cNAsG4Fbq6KOcu58/oSIEMMGFFPJ1RvJpAXE\nEggcuuXazS9Fgr+9Gud2EOLX2mDRGLNK9ocem83mCK+ZdJammFaANSuDaVauQXgmpqCPhENCHDol\nyfjc6p0/tIDQAMvrgwjuNTTh1UnqaIzREPfMTMXI3zc6m6G7pbyQ2ICtW7eSSBRPmNNoymU+zLV8\nDcJ2VZhgp5SirdEiLg6jM1pAaCpgvZR5OHJhmrbGGH2djTTGjFAIFHL8feHRAR7qHwa8Yn+j02l6\nWyub7EWEHTt2LH3gmnVPpf0glFK4kVpMhmGQsEwaLIPRmcUrCawUWkBogJWJYsrYLs0JExGhIWaS\nDPo7+KuyuJX7eL7rc4cAr4Vo1lH0liifUYzVEM6rWftUVWrDVTlOaoCeljgn8vxrqwktINYoSqll\nT2orNo5qydrzUR2NMZO5TG4DoJ+9ahO3v2RrePzWDi9SangqTVJZ9FSoQRRDm500lVJdRzmV46QG\neNGmFg6dulj7AdYILSCqoNYTs1Iqp/dDoUk3/54jIyOcOHGi5j0jlkODCO6RdlwSvpawub2BqaTN\nvzwzxP/nN//paknwc1dvDs87N5Fk4OIc06ksClly9mlrayt79uyhqalpSdfRrD9iVZXaUGH+REBP\nU4z27BizFeQALSdaQFTI9PQ0J06cqGlhu7Nnz3L8eOE6QuCFkp44cYK5ublwW/C63JLg0XPzsW17\nyaaXas6PahBXbvFagN7/+Nkw7C/u7/uz26/mj15/BY6r+OazQ6R9DWPX7t1LGrNSCsvSkVCayjEM\nQaSylqG2UjnF+gDam7xFzoXp1emH0AKiDFzXDSfiVMqLvCmnRHW5BMLGtu2CGkEt7lns3Gw2y/Hj\nx7l48WI4huUi7ajQz9BZICIp4ec5bGhLsL2rERHFXMYhbfulvhsXj2LatGlT+Fr7HzS1xDKkbBOT\n6ypQLDAxdfhtcC9EOh+uJrSAKIOBgQGOHTsGzK/Y6zHZHD9+nOPHjy8wJwUOrfxWndUSNSMFAiH4\nPTMzUzCvw3Gcpfkbstlw/MF1bNsJtYSm2MJ6Stu75k0/IkLcNEjbTlioL5oHsWvXroL3bW9vL6ol\nrJdoMU19sAyjbCd10Jo038TU1ug9mxemU7iuW9Z3XCm1bM3EtIAog+jqe3Jysu73y1/Fl4owKneS\nq0SgFdIijh07xrlz58q+RhTHcThx4gRnz57N2Z52XOJW8XG1N+b6GBKWQTpjk85642uM1GCKxRb6\nI7Zt2wZQtIS4FhCapWAZUnZHOcf3VVh5JqaOJu/ZvDCZ4ujRozlthl3XLfhdHBkZ4dixYzVbMJZC\nG2CL4LpuWPcnoFYTiuM4BbuXFaOcEFSlFEqpUNsoRVDTqFJmZqoLxwse5Hw/SNp2Qw0iyk+/aCOv\nu3JhS/KEZZAdHyKdMDEMyQmDzf97+vr6tPNZU1csU8rWIAJTVL4G0Rw3MQ1heDoJG2M51oOjR48C\n3rPc3DxfRiYoa7McCxytQRTh6NGjDA0N5Wy7cGG+C2o1E6xSitHRUY4dO1aRrb+QgMi//+joKEeP\nHq3LqqKaB9FxnAUmpfxrzmWcsBscwH++eRdvePFm3nrDNrqaC/gkLIOs45K23bBQX5Q9e/bMHxsJ\nXS02/oaGhvL/II0mD9Mwyi73HTizg/yJ4PsrIvS2xDlzsXgQSeCDDAie5+UIc1+JlqPbROQhEXlO\nRA6LyG/72z8gImdF5En/52eXe2z5TE9P57yPmpeqERBnzpwJncGV2BCDe5U6J1hVlGvDrIRKjx8f\nH+fYsWNFBZZSimTWBQUtrfMluW/c3c0br9u64PiABstT6TN2bqG+gMUikvL/Zz09PYv9KRpNDtFn\nyDKk7IZBdmhiyvNBtLWxoa2Bw2cny9ZGgu9jdMFaL1ZCg7CB31FKXQHcCNwlIlf4+/5WKXWt//PN\nFRgbUDtncD7JZLKs4/In5OChLFQxNf/Y4P1i5qhS5E+khe4R/cknKlijmkSUIO67vaV8M1A3M2Qd\n5WkQSwhP3bNnD7t379ZRTZqqUUphmeVHMYUmpjwfRCwW47rtHZyfnGNgrLAWsZLP6Uq0HB0ChvzX\n0yLyPFB82bhMhMlb6TSnT58ueIyIVLSaVkrhum5F/gZggTM4uGfgX1hsDLZtc/z4cXp7e+nq6lqw\n//jx4+zZs6fsHID8+x05coTGxsayBN7AwEDBJMAga7q1gmQ3EXjqzAT7NrbQYlYm5KLo3AdNLbAM\nKTtRLijqZ+a1vxUR9vS2YDDK8FSKXT3NZfkal4sV9UGIyE7gOuBRf9O7ReRpEfmUiHTW+/7RD3pw\ncJAjR44sEA5jY2MFzz1//jyjo6Mlrx9EG9SyHHgymeTIkSOhXTL4G6J+iiDpbmRkZIGZLKCU/bKY\nVpI/jijFJuR8X0s265XtDjSItsbyBcSAb6c9OjxDokT0Uz6BP6JSQa3RlMIyyw9zdYqEuYoIG1ob\naIoR1mQ6ffp06KAOjgk4cuRIwe31YsUEhIi0APcB71FKTQEfBXYD1+JpGB8uct6dInJQRA6OjIxU\nff/Z2VmOHDkShrAWyzSO+h3yJ8rFTFHBNQsJiGKrgEIPXNRslD/OBW078wRCtZFHpe6xFALBNJvx\nBERrQ/mr+dddMR/ZlLBKP7rRL8+GDRvYvn170XBXjaYavES58szRfupOmEkdBEiYpollCjfs6OCp\nM5MopUin04uGtD968iKHz9U/5H5FBISIxPCEw+eVUl8FUEoNK6UcpZQLfBy4odC5Sql7lFIHlFIH\nent7qx5DMJEODg7S399f8BilFBenZnPeB2RsN8zoLUawYi0UsRSYn6L3npzL8jv3Psn/eap4vsFi\nk3W+gJiamqK/v7+s1YbtKF44P73gHlEtaqnCIhCqs2nvs6ukntJbXrqNAzs9xTIQEOVM+iJS87aw\nGhujsDUAABc3SURBVI1ZgYkpzIPwNYhg0RJotzfu6uTibIaHjxa3SkSjAr/82CDfeW54KcMvi2U3\nxoo3U30SeF4p9TeR7Zt9/wTA7cCzyzGeUuGmn/zhKR45cZGYJfzx669kc/t8WOR/v/8ZJuay3POf\nri96fuAzKCYgomokwI9PXGQm7fD1p85hGMLlm1qB/pxzCl2n1PvFiB5//5Nn+faz59myZQuX9c1P\nvFEtqhoHfnCPZ89NYcUbyabnQg2ipcFiqgzfvWVZ2LZNm69xnE55/4vt27cXNJdpB7Sm3limUbaT\nOsykjvSDaGxsDC0YN+zs4JPAD4+O8Mp9PYh4EXtf/MkZXnEl7Gm7EEbdPT00y1QyyxVb2mr/R+Wx\nEhrEK4C3A6/NC2n9KxF5RkSeBl4D/NcVGFvIA8+d55ETXkhq1lb80T8/i1KKqVSWjzx0jIk5b1L6\nfv8IMzMz9Pf350xUo6OjoXmnnIn93ESSrxwa9Pd5Rev+/JsvLDrOagREKuswWCBi4oy/7b3/+BC3\nfPCrOdf8vfue5qEXLlSczf3ZH5/inZ89xH2Pn+XvvnuU/+fTP+Kdnz3E2fEkCcugocy2oYE29oq9\nntb4ppfuDLfrfAbNSlCJicn1BYlpLHRSA7TEhFtetIGTo3O887OHOHR6nCcGJnj4yAh//NUnmUnb\njI+PA/DkOe97esvlC5NJa82yCwil1A+VUqKUenE0pFUp9Xal1NX+9p+PaBMrwmMnx+luifPTL5r/\nJ/z1t/v53z8e4ImBiXDb5x8d4GP/dgSlVI55J8h3gMITaHQlPpXK8sdfOwzAL74kN6DriYHx8PVM\n2mZyLhv2SyhEsck62D6VzPJbX3iCOz/6beYyds7x0We3UezQhDadsrk4k+Hzjw4s0CDOTSR552cP\n8b8fepr+/n76+/tDB/ZM2ubhI57K/K/PngcImwIdOj1eUcvQQEDs6G7iE3cc4Fdv3Fn2uRpNLdid\nVz24kiimYrWYAgGRTCZ57eUb2djmmZw++r3jfONpz9RsCHzse8fD72P/8DT7N7UsuFY9WLeZ1PkT\n6VQqy3m/oqKrFIMTc1y3rZO33rCNP3q9l6ZxZHiGxyMTdsCPT1zk5OhsUfNLoUk7Gsr6wOF5W+LP\nXr2Z266ar0D6kYeO8+jJizzUP8J7vvQkb/rov/OH9z8TPpilNIhnzk7y1ccHc/b/yzOe3LVdxanR\nuZzz+oenecn2+eCx6ZTNU2cmODLsaUIiuYJtJm3zo+OeILz34OCCsTx6IrcRyvU756/tuKqilqH5\nJUQWMyFpE5Om1uTX+6qk1Ebgg8gv9x19Ti/fsYm7b7+a//JqryLAuYkU1+/oZFtHAy+cn+bB587z\n5JkJjo4kuW5757IEXaxbARHl0Olx3vvlp3j//c/ywtAUd372EFlbsWuTN6Ht6G7id2+7POect798\nB+94xc7w/eceGWA65ZmYgoxnpRT3HjzDPz8xmNPvYTZt82fffJ5/e8HLhBzxe9K+79b9ALz28g30\nRSqZfvzhk3z+ES/8tlXSjM9lOXphOrxHlLBSqqP4u+8e5ZvPnA9NQydHZ3nw+fnsy7/76vfDbMyp\nlE3WVrxocyvvvmWv/zed5n/+2zE+9v3j/rXhxPC8P+LDDxwJNQOAd372EEcvzEdNPTU4yab2BO/9\nmcvY0JrgP924g4/+x5eE+1965b7w9WJd3SoVEBpNPTl+/DimCNmyE+VKaxDgZfbHYjFesqOTG3Z5\n+UtXbmnjfT/jfR9Pjc7yyImLNDc18SuvvoalBOmUy7oXEGfHk3z0e/OT94cemI8zvmr7fCmGyzbO\nl4T4o9dfwasu6+Wn9vbwiTsOAJ79/tc+9hDPPd8fOqX7h6d54PAw93z/OD884mkJozNp/uu9T3Fi\nZJYvPDrAxdkMj58e55pt7b5TGrqa43zgDVfkTKYAB3Z2ctdrvNXF4XNTfO6R0/zd/3ksp/ie67o8\n+Pww7/vKU+F5X/jJAOenUnyv/wINMZM/vf0qAJIZJwzBHZv1fnc2x+nw+ywcPrswc/stH/t3zk0k\nmUplQ59FlI8/7H2Wk3NZ+s9Pc+22Tq7Y0saf/eLVNCessEEQwGVb5jWKUoX1+vr6cgREX1+fFhCa\nFSH63MUMFi21MTAwQH9/P5MXvZD8/Ezq6HNtGEZoSv21V+zk3bfs5ZX7N9LVZHHzvh6eGpzk8Nkp\nXr6vl86O9mX5DqzblNJgpR3Y+N/y0j4eODzMxFyWN1/fR0vCYnt3MxcvzhfK+vAvXYNhLMz+vePl\nO/jMj0/juIqJZCYs2dt/3ltNb2lv4BM/OElLQ4xTIzOhwwrg977yNAAJa2ESV8w0uPv2q/nD+58B\n4LarNrGzu5nu5njOyv2Hf/1tXrpB6GiK8/Unc0Nk3/e6y/jQA0foH5ri3ESKXT1NbGpr4LrtHTwx\nMMGjJy/ysl3dXJzxBURTnE1t86v5tsYY73vdZTTETL5yaJDvnpjlH//9JL9wXR8Ab76+j6u2tjE+\nl+UTPzjB2GyWmbTNF34ygOMqbtrTveDvSlgGadtlU1sDhuF9FoHpyjRNDMMgm83S19dHIpHAsqwc\n/060sqVGUw8Mw1g0Ys+SxVuOBv644CtvFcikjhIEusRMg2v6Ouhob2NiYoIX97XzAz8E9mV76q85\nBKx7AXHzvl42dzRy/Y5Ort/eySMnx7j1yo1eg5o8G1/QHjCfmy/rpb0pxv948BiTcxm6m+M4ruL7\nR0bo62zkF166nb9/4DCf+9EpNnU00NMa589vv5p/+N7x0OHdU8RhuzEyWfd1eKvsHd3NXJydT76b\nnJjg4QkHI2918t6fuYzdvS3ELYOvHDzDufE5XnXZBgBaEt6//uMPn+Rbz5xncNx7kLtb4iRiJls6\nGjg3keLKLW1s6WjENE3ufOVutm+a4is/PsInf3gSBG7e10NzwqKvE95x004+8tBx3vOlJ8MxbOlY\nmH/wrlfv4etPnmNTewNO1vs7bNumr6+PeDyOiJBKpXIEge7doFlOdu3aFU7u09PTYcmaaJmWmEHZ\nYa4XZz0zcn6xvnyiBTm3bNkSCqmrtrazb2MLw1NpXnv5pmKn15x1KyBc1yWRSLCrqyuc+LtbEvzc\n1ZvZsmXLgl4Qi9Hum2X+6tv9vHy3t2qeSmZ52a4ubtrTzeN7u/nRsYtcnM3ws1dvQkS4/bqtvDA0\nzU17u3nDNVuKXvs1+3txmV993HbVRh4fGKe3NcE7b97F80NT3P/EOVxX8de/dA0x04uu2NLTztzc\nHD/34s3c/7jXrOfFfV7v55/a1xOuSALhAF59eoBrtnVwbuJ82HMhFovhOA6/eO1mDh09y8nRWS7f\n1EpzYv4R2rNh3gwH8I5X7KSvr4/BwVxH+dVb27l6aztxy8TG++wNw8gRCC0tudfSaJYTy7JobfVM\nvsFvyHVUN9vjpEYv8tVHj/CLL7tswTWCRc0L56e49zHvO9DauNBSsHnz5tC0FNR72717N7FYLGxH\nHDMNfvfW/XRv2FhRcMdSWbc+CNd1sSxrQclnEaG1tZWWlpaCAiJqK48+OH2djWzvbsJ2FD84OsoP\njo7yyst6eMuBPubm5rh803xSy837PBVxS0cj//NXruNtN2zPsc3n86s37uDtN+4I3+/ubeGP33AF\n737tXnb3toS5AZdvbqWzKUZLwqKjKRb+bddt6wjP3eqv6Pf0tvDxSJJfW2OMN167Jfybd3Z7k3WT\nLzACbWpubpZfe8VOrulr546bduaMs60hxt+85Rq2dDRw21WbeNX+jTQ3N5cUtIlEgs2bN7NxY+mY\nbl1HSbMaiMfjtLd7i6zbrtzE5vYEf/u1x/jEQ88vOHZmZgbbUdx3aJDu5jgf+Pkr2NC6UKNua2sL\nF0c7d3qLqkAQRTWWLVu20NtV9xJ1OaxbDSLoviYibN26lbm5OcbHx3Mms6amJjZu3MjwcOGU9s2b\nN2NZFuPj45iG8Mevv4Jk1uEDXzvMxdkMb3755cQsG9u22eFHJc2qeNEVQKDWTk9PhyuHYkT7NXc0\nxfgfb70OIyJkRCR0gG3paOT1L97M8+enwx64wTF3vnI333xmiD/8uRcRMw3i8TiZTIaX/N/27j02\nsrKM4/j315l2Or3MTC/b7k63XVZFdA2EZTcEARVEo3hbIyaIEjHRKEZRJF5ATAT/UmKUEI1RAYOK\nEOMFjVe8kHhBQVRArrIssLCwFly7O7R77+Mf553Z0/Zst+12O7Nznk8y6Zl3embOM52+z5z3fc/7\njpT44BkvrHXOt7e316YbL5fyXLZhLePj4zOafgr5Vj63IeoEryaV4eFhdu3aRTabZceOHTPmhyoU\nDn1FaH9/f+1CodmsXr16USdHdG66wcHBMOpulEteexzf/OMmrv/tPeyczPK+049h69NPUSqVGB0d\n5YbbH+eeZ/dz8SuGWNnTkbg0blxbW9uUpu14gpjL/8liS22CmJycrFWgXV1dtba+6d92S6XSlAQR\n/yYbr4Sr8q0ZPn/O8Yzt3MvxLxyqzQ67PEzTcUx/cgdrV1dX7cNxsGm0q5V3ksH+EgMDA1OG08aP\n7a1rh/hAQrI7eXVvbUgdwLJly9iyZQuSWLfqwLeV6R3D5XKZzZs3T1ntKt6xVygUasPw8vl8bS6k\njo4ONm7cCMxvqOpcllKFmf9gzi02SfT09NDS0kJp1y4uOjPDdX/exPd/dycPPPYkF6wb4JFnxrj9\n0ef4y6b/8q5XreOcU8sLuuI/k8kwMDBQt890qhNEvIKqbif9EcvlMpVKhUqlMuMbQG9v75SrpqvP\n9eJVQ2Sz2QOjc1rEFW9Zw/DyZdju2c8OkirOcrlMPp9nfHycrVu3Tnmsu7ubcnlmH0a8Us3lcpRK\nJXK5HJOTk1QqFfbt2zfjTKWzs3PGWg/FYnHGB/Rgx1jtb+jv709cdyGTydDX1zfj+OZiZGTEO6td\nwygWixSLRXbt2sVFrz6W2x4a5cY7NvOJTQcmgTjlRQNc/JoXT1k/fb56epa2WSku1QkiXkF1dHTQ\n29tba1+M6+7urk2zPX1W0KRKbmhoqNbJmsvl2LlzJ52dnawESh1tVPbvZt++fQwMDLBnzx7GxsYS\n15tetmwZ1SnNq/0dxWKxNsqnVCrx/PPPUyqVSBI/tuqZT/X4Ozs7awsLxUlieHiYbdu2UalU2L17\nd+35BwYGGB0drR1fuVxmYmKilrDi/TOzLcqz0KU+fUZW18jOfMkAhXwrT26bYKS3g+XFdk4/6WW0\nHkZyqLdUJojqqkzTK9DZrkzs6+ujtbV11jH45XKZbDY7pSIbGhpi+/btdHZ20traSqFQoKenh4mJ\nCYrFIuPj44yNjU15nviIhqGhoRkdtIVCodYeOX3FuJGRETZv3kwmk5nyLX/58plD4+Lxd3V11ZKQ\nJPr6+mpJsfo8pVIJM6sl0dbW1lrCyuVySGLVqlXs2bPHL2RzqbFixQomJiZoa2ujUCiQy+Xo6Ohg\n7969h+xzaHSpTRAwvyaObDZbq4xHRkamTOE9PDzM5ORk4tDMTCZT2y8+UqdayVYr0vixVH+/VCrN\nu6LN5/MMDg7Wvs0vX76c9vb2xA9qS0sLfX197N+/P3EU0YoVK6hUKrVpMCQlLmEa7zxrb2/32VVd\nqsT7veJn0c3wf5DKBHGwDum5mt7UMds0EXN5rv7+/inNRNVv8AsVf66kJrO42Zp7stlsXds/nXP1\nleoEMd9O0iPhcJOBc84dKfWvIeugkRKEc841qoarISW9XtLDkjZKuvRIvEZLSwvd3d2zjrRxzrm0\na6gEISkDfBU4G1gDnCdpzWK/TltbG+Xywi5ccc65tGioBAGcDGw0s01mtge4GdhQ52NyzrlUarQE\nMQQ8Gbv/VChzzjm3xBotQRySpPdLukvSXdWrjJ1zzi2+RksQW4Dh2P2VoazGzL5hZuvNbP1SrMnq\nnHNp1WgJ4m/AsZJWS2oD3gH8tM7H5JxzqdRQ4zzNbJ+kDwO/BjLA9WZ2f50PyznnUqmhEgSAmf0C\n+EW9j8M559Ku0ZqYnHPONQgdzQuwSHoWeGKBu/cDzy3i4RwNPOZ08JjT4XBiXmVmhxzlc1QniMMh\n6S4zW1/v41hKHnM6eMzpsBQxexOTc865RJ4gnHPOJUpzgvhGvQ+gDjzmdPCY0+GIx5zaPgjnnHOz\nS/MZhHPOuVmkMkEsxaJES03SsKTbJD0g6X5JHw3lvZJ+I+mR8LMnts9l4T14WNLr6nf0h0dSRtI/\nJf0s3G/qmCWVJP1A0kOSHpT08hTE/LHwub5P0k2S2psxZknXSxqVdF+sbN5xSlon6V/hsWskaUEH\nZGapuhFN4fEo8AKgDbgHWFPv41qEuFYAJ4XtbuDfRIsuXQVcGsovBb4QtteE2HPA6vCeZOodxwJj\nvwT4HvCzcL+pYwZuAN4XttuAUjPHTDTl/2NAPtz/PvCeZowZeCVwEnBfrGzecQJ3AqcAAn4JnL2Q\n40njGURTLkpkZs+Y2T/CdgV4kOgfawNRhUL4+dawvQG42cx2m9ljwEai9+aoImkl8Ebg2lhx08Ys\nqUhUiVwHYGZ7zGyMJo45yAJ5SVmgA3iaJozZzP4AbJtWPK84Ja0ACmb2V4uyxbdj+8xLGhNE0y9K\nJOkYYC1wBzBoZs+Eh7YCg2G7Wd6Hq4FPApOxsmaOeTXwLPCt0Kx2raROmjhmM9sCfBHYDDwDbDez\nW2nimKeZb5xDYXt6+bylMUE0NUldwA+Bi81sR/yx8G2iaYatSXoTMGpmfz/Y7zRbzETfpE8CvmZm\na4FxomaHmmaLObS5byBKjmWgU9L58d9ptpgPZqnjTGOCOOSiREcrSa1EyeFGM/tRKP5POOUk/BwN\n5c3wPpwGvEXS40RNha+W9F2aO+angKfM7I5w/wdECaOZY34N8JiZPWtme4EfAafS3DHHzTfOLWF7\nevm8pTFBNOWiRGGUwnXAg2b2pdhDPwUuCNsXAD+Jlb9DUk7SauBYoo6to4aZXWZmK83sGKK/4+/N\n7HyaO+atwJOSjgtFZwEP0MQxEzUtnSKpI3zOzyLqY2vmmOPmFWdojtoh6ZTwfr07ts/81LvXvh43\n4A1Eo3weBS6v9/EsUkynE5163gvcHW5vAPqA3wGPAL8FemP7XB7eg4dZ4CiHRrkBZ3BgFFNTxwyc\nCNwV/ta3AD0piPlK4CHgPuA7RCN3mi5m4Caifpa9RGeL711InMD68F49CnyFcFH0fG9+JbVzzrlE\naWxics45NweeIJxzziXyBOGccy6RJwjnnHOJPEE455xL5AnCuRhJ+yXdHbvNOtuvpAslvXsRXvdx\nSf2H+zzOLSYf5upcjKTnzayrDq/7OLDezJ5b6td27mD8DMK5OQjf8K8Kc+zfKelFofwKSR8P2x9R\ntB7HvZJuDmW9km4JZX+VdEIo75N0a1jj4FqiaZmrr3V+eI27JX1dUqYOITvnCcK5afLTmpjOjT22\n3cyOJ7oy9eqEfS8F1prZCcCFoexK4J+h7NNEUy8DfBb4k5m9DPgxMAIg6aXAucBpZnYisB941+KG\n6NzcZOt9AM41mJ2hYk5yU+znlxMevxe4UdItRFNgQDQFyjkAZvb7cOZQIFrT4W2h/OeS/hd+/yxg\nHfC3sAhYngOTszm3pDxBODd3dpDtqjcSVfxvBi6XdPwCXkPADWZ22QL2dW5ReROTc3N3buznX+IP\nSGoBhs3sNuBTQBHoAv5IaCKSdAbwnEXrdPwBeGcoP5towj2IJmV7u6SB8FivpFVHMCbnDsrPIJyb\nKi/p7tj9X5lZdahrj6R7gd3AedP2ywDfDUuCCrjGzMYkXQFcH/ab4MC0zVcCN0m6H7idaEprzOwB\nSZ8Bbg1JZy/wIeCJxQ7UuUPxYa7OzYEPQ3Vp5E1MzjnnEvkZhHPOuUR+BuGccy6RJwjnnHOJPEE4\n55xL5AnCOedcIk8QzjnnEnmCcM45l+j/pjqo1l9aaboAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x124468e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps, rews = np.array(rewards_list).T\n",
    "smoothed_rews = running_mean(rews, 10)\n",
    "plt.plot(eps[-len(smoothed_rews):], smoothed_rews)\n",
    "plt.plot(eps, rews, color='grey', alpha=0.3)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Total Reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "Let's checkout how our trained agent plays the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/cartpole.ckpt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-09-12 00:19:12,056] Restoring parameters from checkpoints/cartpole.ckpt\n"
     ]
    }
   ],
   "source": [
    "test_episodes = 10\n",
    "test_max_steps = 400\n",
    "env.reset()\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
    "    \n",
    "    for ep in range(1, test_episodes):\n",
    "        t = 0\n",
    "        while t < test_max_steps:\n",
    "            env.render() \n",
    "            \n",
    "            # Get action from Q-network\n",
    "            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "            Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "            action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            if done:\n",
    "                t = test_max_steps\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                state = next_state\n",
    "                t += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extending this\n",
    "\n",
    "So, Cart-Pole is a pretty simple game. However, the same model can be used to train an agent to play something much more complicated like Pong or Space Invaders. Instead of a state like we're using here though, you'd want to use convolutional layers to get the state from the screen images.\n",
    "\n",
    "![Deep Q-Learning Atari](assets/atari-network.png)\n",
    "\n",
    "I'll leave it as a challenge for you to use deep Q-learning to train an agent to play Atari games. Here's the original paper which will get you started: http://www.davidqiu.com:8888/research/nature14236.pdf."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}