ch17_Model_Based_Policy_Optimize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1b4d195-dbf9-4b5b-8dc2-4df7b58437a3",
   "metadata": {},
   "source": [
    "# 第 17 章 基于模型的策略优化"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "439a369e-4b81-4851-a618-c231d034298d",
   "metadata": {},
   "source": [
    "## 17.1 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052e457c-63c0-401d-86c5-67903f7078ad",
   "metadata": {},
   "source": [
    "第 16 章介绍的 PETS 算法是基于模型的强化学习算法中的一种，它没有显式构建一个策略（即一个从状态到动作的映射函数）。回顾一下之前介绍过的 Dyna-Q 算法，它也是一种基于模型的强化学习算法。但是 Dyna-Q 算法中的模型只存储之前遇到的数据，只适用于表格型环境。而在连续型状态和动作的环境中，我们需要像 PETS 算法一样学习一个用神经网络表示的环境模型，此时若继续利用 Dyna 的思想，可以在任意状态和动作下用环境模型来生成一些虚拟数据，这些虚拟数据可以帮助进行策略的学习。如此，通过和模型进行交互产生额外的虚拟数据，对真实环境中样本的需求量就会减少，因此通常会比无模型的强化学习方法具有更高的采样效率。本章将介绍这样一种算法——MBPO 算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45ca0d88-e60f-4b83-8ae6-1daea8726710",
   "metadata": {},
   "source": [
    "## 17.2 MBPO 算法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c87294-49fb-422b-8fa0-878a33d2de9e",
   "metadata": {},
   "source": [
    "基于模型的策略优化 (model-based policy optimization，MBPO）算法是加州大学伯克利分校的研究员在 2019 年的 NeurIPS 会议中提出的。随即 MBPO 成为深度强化学习中最重要的基于模型的强化学习算法之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c57edef-3453-47f1-89b3-50f93f324ac7",
   "metadata": {},
   "source": [
    "MBPO 算法基于以下两个关键的观察： (1) 随着环境模型的推演步数变长，模型累积的复合误差会快速增加，使得环境模型得出的结果变得很不可靠； (2) 必须要权衡推演步数增加后模型增加的误差带来的负面作用与步数增加后使得训练的策略更优的正面作用，二者的权衡决定了推演的步数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca32799-0fea-4001-b8f9-29dbead78332",
   "metadata": {},
   "source": [
    "MBPO 算法在这两个观察的基础之上，提出只使用模型来从之前访问过的真实状态开始进行较短步数的推演，而非从初始状态开始进行完整的推演。这就是 MBPO 中的分支推演（branched rollout）的概念，即在原来真实环境中采样的轨迹上面推演出新的“短分支”，"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11b48325-d8d2-4ddb-8d47-c38984935559",
   "metadata": {},
   "source": [
    "## 17.3 MBPO 代码实践"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b9147f-cc3a-4138-8f72-d08313b302d6",
   "metadata": {},
   "source": [
    "首先，我们先导入一些必要的包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1470ae3e-ca21-45b5-8b85-390ebee16ec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "from collections import namedtuple\n",
    "import itertools\n",
    "from itertools import count\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.distributions.normal import Normal\n",
    "import numpy as np\n",
    "import collections\n",
    "import random\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e544b86-98e0-47cd-87f9-b2a537e599c0",
   "metadata": {},
   "source": [
    "MBPO 算法使用 SAC 算法来训练策略。和 SAC 算法相比，MBPO 多用了一些模型推演得到的数据来训练策略。要想了解 SAC 方法的详细过程，读者可以阅读第 14 章对应的内容。我们将 SAC 代码直接复制到此处。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a4b4e47f-e761-4c86-b491-8daa2d5f5122",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PolicyNet(torch.nn.Module):\n",
    "    def __init__(self, state_dim, hidden_dim, action_dim, action_bound):\n",
    "        super(PolicyNet, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(state_dim, hidden_dim)\n",
    "        self.fc_mu = torch.nn.Linear(hidden_dim, action_dim)\n",
    "        self.fc_std = torch.nn.Linear(hidden_dim, action_dim)\n",
    "        self.action_bound = action_bound\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        mu = self.fc_mu(x)\n",
    "        std = F.softplus(self.fc_std(x))\n",
    "        dist = Normal(mu, std)\n",
    "        normal_sample = dist.rsample()  # rsample()是重参数化采样函数\n",
    "        log_prob = dist.log_prob(normal_sample)\n",
    "        action = torch.tanh(normal_sample)  # 计算tanh_normal分布的对数概率密度\n",
    "        log_prob = log_prob - torch.log(1 - torch.tanh(action).pow(2) + 1e-7)\n",
    "        action = action * self.action_bound\n",
    "        return action, log_prob\n",
    "\n",
    "\n",
    "class QValueNet(torch.nn.Module):\n",
    "    def __init__(self, state_dim, hidden_dim, action_dim):\n",
    "        super(QValueNet, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(state_dim + action_dim, hidden_dim)\n",
    "        self.fc2 = torch.nn.Linear(hidden_dim, 1)\n",
    "\n",
    "    def forward(self, x, a):\n",
    "        cat = torch.cat([x, a], dim=1)  # 拼接状态和动作\n",
    "        x = F.relu(self.fc1(cat))\n",
    "        return self.fc2(x)\n",
    "\n",
    "\n",
    "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\n",
    "    \"cpu\")\n",
    "\n",
    "\n",
    "class SAC:\n",
    "    ''' 处理连续动作的SAC算法 '''\n",
    "    def __init__(self, state_dim, hidden_dim, action_dim, action_bound,\n",
    "                 actor_lr, critic_lr, alpha_lr, target_entropy, tau, gamma):\n",
    "        self.actor = PolicyNet(state_dim, hidden_dim, action_dim,\n",
    "                               action_bound).to(device)  # 策略网络\n",
    "        # 第一个Q网络\n",
    "        self.critic_1 = QValueNet(state_dim, hidden_dim, action_dim).to(device)\n",
    "        # 第二个Q网络\n",
    "        self.critic_2 = QValueNet(state_dim, hidden_dim, action_dim).to(device)\n",
    "        self.target_critic_1 = QValueNet(state_dim, hidden_dim,\n",
    "                                         action_dim).to(device)  # 第一个目标Q网络\n",
    "        self.target_critic_2 = QValueNet(state_dim, hidden_dim,\n",
    "                                         action_dim).to(device)  # 第二个目标Q网络\n",
    "        # 令目标Q网络的初始参数和Q网络一样\n",
    "        self.target_critic_1.load_state_dict(self.critic_1.state_dict())\n",
    "        self.target_critic_2.load_state_dict(self.critic_2.state_dict())\n",
    "        self.actor_optimizer = torch.optim.Adam(self.actor.parameters(),\n",
    "                                                lr=actor_lr)\n",
    "        self.critic_1_optimizer = torch.optim.Adam(self.critic_1.parameters(),\n",
    "                                                   lr=critic_lr)\n",
    "        self.critic_2_optimizer = torch.optim.Adam(self.critic_2.parameters(),\n",
    "                                                   lr=critic_lr)\n",
    "        # 使用alpha的log值,可以使训练结果比较稳定\n",
    "        self.log_alpha = torch.tensor(np.log(0.01), dtype=torch.float)\n",
    "        self.log_alpha.requires_grad = True  # 可以对alpha求梯度\n",
    "        self.log_alpha_optimizer = torch.optim.Adam([self.log_alpha],\n",
    "                                                    lr=alpha_lr)\n",
    "        self.target_entropy = target_entropy  # 目标熵的大小\n",
    "        self.gamma = gamma\n",
    "        self.tau = tau\n",
    "\n",
    "    def take_action(self, state):\n",
    "        state = torch.tensor([state], dtype=torch.float).to(device)\n",
    "        action = self.actor(state)[0]\n",
    "        return [action.item()]\n",
    "\n",
    "    def calc_target(self, rewards, next_states, dones):  # 计算目标Q值\n",
    "        next_actions, log_prob = self.actor(next_states)\n",
    "        entropy = -log_prob\n",
    "        q1_value = self.target_critic_1(next_states, next_actions)\n",
    "        q2_value = self.target_critic_2(next_states, next_actions)\n",
    "        next_value = torch.min(q1_value,\n",
    "                               q2_value) + self.log_alpha.exp() * entropy\n",
    "        td_target = rewards + self.gamma * next_value * (1 - dones)\n",
    "        return td_target\n",
    "\n",
    "    def soft_update(self, net, target_net):\n",
    "        for param_target, param in zip(target_net.parameters(),\n",
    "                                       net.parameters()):\n",
    "            param_target.data.copy_(param_target.data * (1.0 - self.tau) +\n",
    "                                    param.data * self.tau)\n",
    "\n",
    "    def update(self, transition_dict):\n",
    "        states = torch.tensor(transition_dict['states'],\n",
    "                              dtype=torch.float).to(device)\n",
    "        actions = torch.tensor(transition_dict['actions'],\n",
    "                               dtype=torch.float).view(-1, 1).to(device)\n",
    "        rewards = torch.tensor(transition_dict['rewards'],\n",
    "                               dtype=torch.float).view(-1, 1).to(device)\n",
    "        next_states = torch.tensor(transition_dict['next_states'],\n",
    "                                   dtype=torch.float).to(device)\n",
    "        dones = torch.tensor(transition_dict['dones'],\n",
    "                             dtype=torch.float).view(-1, 1).to(device)\n",
    "        rewards = (rewards + 8.0) / 8.0  # 对倒立摆环境的奖励进行重塑\n",
    "\n",
    "        # 更新两个Q网络\n",
    "        td_target = self.calc_target(rewards, next_states, dones)\n",
    "        critic_1_loss = torch.mean(\n",
    "            F.mse_loss(self.critic_1(states, actions), td_target.detach()))\n",
    "        critic_2_loss = torch.mean(\n",
    "            F.mse_loss(self.critic_2(states, actions), td_target.detach()))\n",
    "        self.critic_1_optimizer.zero_grad()\n",
    "        critic_1_loss.backward()\n",
    "        self.critic_1_optimizer.step()\n",
    "        self.critic_2_optimizer.zero_grad()\n",
    "        critic_2_loss.backward()\n",
    "        self.critic_2_optimizer.step()\n",
    "\n",
    "        # 更新策略网络\n",
    "        new_actions, log_prob = self.actor(states)\n",
    "        entropy = -log_prob\n",
    "        q1_value = self.critic_1(states, new_actions)\n",
    "        q2_value = self.critic_2(states, new_actions)\n",
    "        actor_loss = torch.mean(-self.log_alpha.exp() * entropy -\n",
    "                                torch.min(q1_value, q2_value))\n",
    "        self.actor_optimizer.zero_grad()\n",
    "        actor_loss.backward()\n",
    "        self.actor_optimizer.step()\n",
    "\n",
    "        # 更新alpha值\n",
    "        alpha_loss = torch.mean(\n",
    "            (entropy - self.target_entropy).detach() * self.log_alpha.exp())\n",
    "        self.log_alpha_optimizer.zero_grad()\n",
    "        alpha_loss.backward()\n",
    "        self.log_alpha_optimizer.step()\n",
    "\n",
    "        self.soft_update(self.critic_1, self.target_critic_1)\n",
    "        self.soft_update(self.critic_2, self.target_critic_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a203a35d-ee77-4e20-9ed8-2fd2a8877192",
   "metadata": {},
   "source": [
    "接下来定义环境模型，注意这里的环境模型和 PETS 算法中的环境模型是一样的，由多个高斯分布策略的集成来构建。我们也沿用 PETS 算法中的模型构建代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6c95cabb-aeae-4f2f-b277-45ce5a94864c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Swish(nn.Module):\n",
    "    ''' Swish激活函数 '''\n",
    "    def __init__(self):\n",
    "        super(Swish, self).__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return x * torch.sigmoid(x)\n",
    "\n",
    "\n",
    "def init_weights(m):\n",
    "    ''' 初始化模型权重 '''\n",
    "    def truncated_normal_init(t, mean=0.0, std=0.01):\n",
    "        torch.nn.init.normal_(t, mean=mean, std=std)\n",
    "        while True:\n",
    "            cond = (t < mean - 2 * std) | (t > mean + 2 * std)\n",
    "            if not torch.sum(cond):\n",
    "                break\n",
    "            t = torch.where(\n",
    "                cond,\n",
    "                torch.nn.init.normal_(torch.ones(t.shape, device=device),\n",
    "                                      mean=mean,\n",
    "                                      std=std), t)\n",
    "        return t\n",
    "\n",
    "    if type(m) == nn.Linear or isinstance(m, FCLayer):\n",
    "        truncated_normal_init(m.weight, std=1 / (2 * np.sqrt(m._input_dim)))\n",
    "        m.bias.data.fill_(0.0)\n",
    "\n",
    "\n",
    "class FCLayer(nn.Module):\n",
    "    ''' 集成之后的全连接层 '''\n",
    "    def __init__(self, input_dim, output_dim, ensemble_size, activation):\n",
    "        super(FCLayer, self).__init__()\n",
    "        self._input_dim, self._output_dim = input_dim, output_dim\n",
    "        self.weight = nn.Parameter(\n",
    "            torch.Tensor(ensemble_size, input_dim, output_dim).to(device))\n",
    "        self._activation = activation\n",
    "        self.bias = nn.Parameter(\n",
    "            torch.Tensor(ensemble_size, output_dim).to(device))\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self._activation(\n",
    "            torch.add(torch.bmm(x, self.weight), self.bias[:, None, :]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d47b0b5-9df4-40fc-88cf-95dd1cc92181",
   "metadata": {},
   "source": [
    "接着，我们就可以定义集成模型了，其中就会用到刚刚定义的全连接层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "413bc610-4784-440a-a3a6-f4ba971a8e71",
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnsembleModel(nn.Module):\n",
    "    ''' 环境模型集成 '''\n",
    "    def __init__(self,\n",
    "                 state_dim,\n",
    "                 action_dim,\n",
    "                 model_alpha,\n",
    "                 ensemble_size=5,\n",
    "                 learning_rate=1e-3):\n",
    "        super(EnsembleModel, self).__init__()\n",
    "        # 输出包括均值和方差,因此是状态与奖励维度之和的两倍\n",
    "        self._output_dim = (state_dim + 1) * 2\n",
    "        self._model_alpha = model_alpha  # 模型损失函数中加权时的权重\n",
    "        self._max_logvar = nn.Parameter((torch.ones(\n",
    "            (1, self._output_dim // 2)).float() / 2).to(device),\n",
    "                                        requires_grad=False)\n",
    "        self._min_logvar = nn.Parameter((-torch.ones(\n",
    "            (1, self._output_dim // 2)).float() * 10).to(device),\n",
    "                                        requires_grad=False)\n",
    "\n",
    "        self.layer1 = FCLayer(state_dim + action_dim, 200, ensemble_size,\n",
    "                              Swish())\n",
    "        self.layer2 = FCLayer(200, 200, ensemble_size, Swish())\n",
    "        self.layer3 = FCLayer(200, 200, ensemble_size, Swish())\n",
    "        self.layer4 = FCLayer(200, 200, ensemble_size, Swish())\n",
    "        self.layer5 = FCLayer(200, self._output_dim, ensemble_size,\n",
    "                              nn.Identity())\n",
    "        self.apply(init_weights)  # 初始化环境模型中的参数\n",
    "        self.optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)\n",
    "\n",
    "    def forward(self, x, return_log_var=False):\n",
    "        ret = self.layer5(self.layer4(self.layer3(self.layer2(\n",
    "            self.layer1(x)))))\n",
    "        mean = ret[:, :, :self._output_dim // 2]\n",
    "        # 在PETS算法中,将方差控制在最小值和最大值之间\n",
    "        logvar = self._max_logvar - F.softplus(\n",
    "            self._max_logvar - ret[:, :, self._output_dim // 2:])\n",
    "        logvar = self._min_logvar + F.softplus(logvar - self._min_logvar)\n",
    "        return mean, logvar if return_log_var else torch.exp(logvar)\n",
    "\n",
    "    def loss(self, mean, logvar, labels, use_var_loss=True):\n",
    "        inverse_var = torch.exp(-logvar)\n",
    "        if use_var_loss:\n",
    "            mse_loss = torch.mean(torch.mean(torch.pow(mean - labels, 2) *\n",
    "                                             inverse_var,\n",
    "                                             dim=-1),\n",
    "                                  dim=-1)\n",
    "            var_loss = torch.mean(torch.mean(logvar, dim=-1), dim=-1)\n",
    "            total_loss = torch.sum(mse_loss) + torch.sum(var_loss)\n",
    "        else:\n",
    "            mse_loss = torch.mean(torch.pow(mean - labels, 2), dim=(1, 2))\n",
    "            total_loss = torch.sum(mse_loss)\n",
    "        return total_loss, mse_loss\n",
    "\n",
    "    def train(self, loss):\n",
    "        self.optimizer.zero_grad()\n",
    "        loss += self._model_alpha * torch.sum(\n",
    "            self._max_logvar) - self._model_alpha * torch.sum(self._min_logvar)\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "\n",
    "\n",
    "class EnsembleDynamicsModel:\n",
    "    ''' 环境模型集成,加入精细化的训练 '''\n",
    "    def __init__(self, state_dim, action_dim, model_alpha=0.01, num_network=5):\n",
    "        self._num_network = num_network\n",
    "        self._state_dim, self._action_dim = state_dim, action_dim\n",
    "        self.model = EnsembleModel(state_dim,\n",
    "                                   action_dim,\n",
    "                                   model_alpha,\n",
    "                                   ensemble_size=num_network)\n",
    "        self._epoch_since_last_update = 0\n",
    "\n",
    "    def train(self,\n",
    "              inputs,\n",
    "              labels,\n",
    "              batch_size=64,\n",
    "              holdout_ratio=0.1,\n",
    "              max_iter=20):\n",
    "        # 设置训练集与验证集\n",
    "        permutation = np.random.permutation(inputs.shape[0])\n",
    "        inputs, labels = inputs[permutation], labels[permutation]\n",
    "        num_holdout = int(inputs.shape[0] * holdout_ratio)\n",
    "        train_inputs, train_labels = inputs[num_holdout:], labels[num_holdout:]\n",
    "        holdout_inputs, holdout_labels = inputs[:\n",
    "                                                num_holdout], labels[:\n",
    "                                                                     num_holdout]\n",
    "        holdout_inputs = torch.from_numpy(holdout_inputs).float().to(device)\n",
    "        holdout_labels = torch.from_numpy(holdout_labels).float().to(device)\n",
    "        holdout_inputs = holdout_inputs[None, :, :].repeat(\n",
    "            [self._num_network, 1, 1])\n",
    "        holdout_labels = holdout_labels[None, :, :].repeat(\n",
    "            [self._num_network, 1, 1])\n",
    "\n",
    "        # 保留最好的结果\n",
    "        self._snapshots = {i: (None, 1e10) for i in range(self._num_network)}\n",
    "\n",
    "        for epoch in itertools.count():\n",
    "            # 定义每一个网络的训练数据\n",
    "            train_index = np.vstack([\n",
    "                np.random.permutation(train_inputs.shape[0])\n",
    "                for _ in range(self._num_network)\n",
    "            ])\n",
    "            # 所有真实数据都用来训练\n",
    "            for batch_start_pos in range(0, train_inputs.shape[0], batch_size):\n",
    "                batch_index = train_index[:, batch_start_pos:batch_start_pos +\n",
    "                                          batch_size]\n",
    "                train_input = torch.from_numpy(\n",
    "                    train_inputs[batch_index]).float().to(device)\n",
    "                train_label = torch.from_numpy(\n",
    "                    train_labels[batch_index]).float().to(device)\n",
    "\n",
    "                mean, logvar = self.model(train_input, return_log_var=True)\n",
    "                loss, _ = self.model.loss(mean, logvar, train_label)\n",
    "                self.model.train(loss)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                mean, logvar = self.model(holdout_inputs, return_log_var=True)\n",
    "                _, holdout_losses = self.model.loss(mean,\n",
    "                                                    logvar,\n",
    "                                                    holdout_labels,\n",
    "                                                    use_var_loss=False)\n",
    "                holdout_losses = holdout_losses.cpu()\n",
    "                break_condition = self._save_best(epoch, holdout_losses)\n",
    "                if break_condition or epoch > max_iter:  # 结束训练\n",
    "                    break\n",
    "\n",
    "    def _save_best(self, epoch, losses, threshold=0.1):\n",
    "        updated = False\n",
    "        for i in range(len(losses)):\n",
    "            current = losses[i]\n",
    "            _, best = self._snapshots[i]\n",
    "            improvement = (best - current) / best\n",
    "            if improvement > threshold:\n",
    "                self._snapshots[i] = (epoch, current)\n",
    "                updated = True\n",
    "        self._epoch_since_last_update = 0 if updated else self._epoch_since_last_update + 1\n",
    "        return self._epoch_since_last_update > 5\n",
    "\n",
    "    def predict(self, inputs, batch_size=64):\n",
    "        inputs = np.tile(inputs, (self._num_network, 1, 1))\n",
    "        inputs = torch.tensor(inputs, dtype=torch.float).to(device)\n",
    "        mean, var = self.model(inputs, return_log_var=False)\n",
    "        return mean.detach().cpu().numpy(), var.detach().cpu().numpy()\n",
    "\n",
    "\n",
    "class FakeEnv:\n",
    "    def __init__(self, model):\n",
    "        self.model = model\n",
    "\n",
    "    def step(self, obs, act):\n",
    "        inputs = np.concatenate((obs, act), axis=-1)\n",
    "        ensemble_model_means, ensemble_model_vars = self.model.predict(inputs)\n",
    "        ensemble_model_means[:, :, 1:] += obs\n",
    "        ensemble_model_stds = np.sqrt(ensemble_model_vars)\n",
    "        ensemble_samples = ensemble_model_means + np.random.normal(\n",
    "            size=ensemble_model_means.shape) * ensemble_model_stds\n",
    "\n",
    "        num_models, batch_size, _ = ensemble_model_means.shape\n",
    "        models_to_use = np.random.choice(\n",
    "            [i for i in range(self.model._num_network)], size=batch_size)\n",
    "        batch_inds = np.arange(0, batch_size)\n",
    "        samples = ensemble_samples[models_to_use, batch_inds]\n",
    "        rewards, next_obs = samples[:, :1][0][0], samples[:, 1:][0]\n",
    "        return rewards, next_obs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add72c5e-e140-4eb4-9d3f-b7210daf1c73",
   "metadata": {},
   "source": [
    "最后，我们来实现 MBPO 算法的具体流程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ada9e3c2-791e-477e-8301-a3502fda3d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MBPO:\n",
    "    def __init__(self, env, agent, fake_env, env_pool, model_pool,\n",
    "                 rollout_length, rollout_batch_size, real_ratio, num_episode):\n",
    "\n",
    "        self.env = env\n",
    "        self.agent = agent\n",
    "        self.fake_env = fake_env\n",
    "        self.env_pool = env_pool\n",
    "        self.model_pool = model_pool\n",
    "        self.rollout_length = rollout_length\n",
    "        self.rollout_batch_size = rollout_batch_size\n",
    "        self.real_ratio = real_ratio\n",
    "        self.num_episode = num_episode\n",
    "\n",
    "    def rollout_model(self):\n",
    "        observations, _, _, _, _ = self.env_pool.sample(\n",
    "            self.rollout_batch_size)\n",
    "        for obs in observations:\n",
    "            for i in range(self.rollout_length):\n",
    "                action = self.agent.take_action(obs)\n",
    "                reward, next_obs = self.fake_env.step(obs, action)\n",
    "                self.model_pool.add(obs, action, reward, next_obs, False)\n",
    "                obs = next_obs\n",
    "\n",
    "    def update_agent(self, policy_train_batch_size=64):\n",
    "        env_batch_size = int(policy_train_batch_size * self.real_ratio)\n",
    "        model_batch_size = policy_train_batch_size - env_batch_size\n",
    "        for epoch in range(10):\n",
    "            env_obs, env_action, env_reward, env_next_obs, env_done = self.env_pool.sample(\n",
    "                env_batch_size)\n",
    "            if self.model_pool.size() > 0:\n",
    "                model_obs, model_action, model_reward, model_next_obs, model_done = self.model_pool.sample(\n",
    "                    model_batch_size)\n",
    "                obs = np.concatenate((env_obs, model_obs), axis=0)\n",
    "                action = np.concatenate((env_action, model_action), axis=0)\n",
    "                next_obs = np.concatenate((env_next_obs, model_next_obs),\n",
    "                                          axis=0)\n",
    "                reward = np.concatenate((env_reward, model_reward), axis=0)\n",
    "                done = np.concatenate((env_done, model_done), axis=0)\n",
    "            else:\n",
    "                obs, action, next_obs, reward, done = env_obs, env_action, env_next_obs, env_reward, env_done\n",
    "            transition_dict = {\n",
    "                'states': obs,\n",
    "                'actions': action,\n",
    "                'next_states': next_obs,\n",
    "                'rewards': reward,\n",
    "                'dones': done\n",
    "            }\n",
    "            self.agent.update(transition_dict)\n",
    "\n",
    "    def train_model(self):\n",
    "        obs, action, reward, next_obs, done = self.env_pool.return_all_samples(\n",
    "        )\n",
    "        inputs = np.concatenate((obs, action), axis=-1)\n",
    "        reward = np.array(reward)\n",
    "        labels = np.concatenate(\n",
    "            (np.reshape(reward, (reward.shape[0], -1)), next_obs - obs),\n",
    "            axis=-1)\n",
    "        self.fake_env.model.train(inputs, labels)\n",
    "\n",
    "    def explore(self):\n",
    "        obs, done, episode_return = self.env.reset(), False, 0\n",
    "        while not done:\n",
    "            action = self.agent.take_action(obs)\n",
    "            next_obs, reward, done, _ = self.env.step(action)\n",
    "            self.env_pool.add(obs, action, reward, next_obs, done)\n",
    "            obs = next_obs\n",
    "            episode_return += reward\n",
    "        return episode_return\n",
    "\n",
    "    def train(self):\n",
    "        return_list = []\n",
    "        explore_return = self.explore()  # 随机探索采取数据\n",
    "        print('episode: 1, return: %d' % explore_return)\n",
    "        return_list.append(explore_return)\n",
    "\n",
    "        for i_episode in range(self.num_episode - 1):\n",
    "            obs, done, episode_return = self.env.reset(), False, 0\n",
    "            step = 0\n",
    "            while not done:\n",
    "                if step % 50 == 0:\n",
    "                    self.train_model()\n",
    "                    self.rollout_model()\n",
    "                action = self.agent.take_action(obs)\n",
    "                next_obs, reward, done, _ = self.env.step(action)\n",
    "                self.env_pool.add(obs, action, reward, next_obs, done)\n",
    "                obs = next_obs\n",
    "                episode_return += reward\n",
    "\n",
    "                self.update_agent()\n",
    "                step += 1\n",
    "            return_list.append(episode_return)\n",
    "            print('episode: %d, return: %d' % (i_episode + 2, episode_return))\n",
    "        return return_list\n",
    "\n",
    "\n",
    "class ReplayBuffer:\n",
    "    def __init__(self, capacity):\n",
    "        self.buffer = collections.deque(maxlen=capacity)\n",
    "\n",
    "    def add(self, state, action, reward, next_state, done):\n",
    "        self.buffer.append((state, action, reward, next_state, done))\n",
    "\n",
    "    def size(self):\n",
    "        return len(self.buffer)\n",
    "\n",
    "    def sample(self, batch_size):\n",
    "        if batch_size > len(self.buffer):\n",
    "            return self.return_all_samples()\n",
    "        else:\n",
    "            transitions = random.sample(self.buffer, batch_size)\n",
    "            state, action, reward, next_state, done = zip(*transitions)\n",
    "            return np.array(state), action, reward, np.array(next_state), done\n",
    "\n",
    "    def return_all_samples(self):\n",
    "        all_transitions = list(self.buffer)\n",
    "        state, action, reward, next_state, done = zip(*all_transitions)\n",
    "        return np.array(state), action, reward, np.array(next_state), done"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b84400a6-d8f4-472a-a79d-0cb860a4ac8e",
   "metadata": {},
   "source": [
    "对于不同的环境，我们需要设置不同的参数。这里以 OpenAI Gym 中的 Pendulum-v0 环境为例，给出一组效果较为不错的参数。读者可以试着自己调节参数，观察调节后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "76698941-c35a-4a4f-9faa-898368ea9079",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n",
      "/opt/anaconda3/lib/python3.11/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
      "  deprecation(\n",
      "/var/folders/89/_dbj4_0x3cn29bsmx2cpn8800000gn/T/ipykernel_3385/438640068.py:71: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/utils/tensor_new.cpp:278.)\n",
      "  state = torch.tensor([state], dtype=torch.float).to(device)\n",
      "/opt/anaconda3/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:241: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)\n",
      "  if not isinstance(terminated, (bool, np.bool8)):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 1, return: -1770\n",
      "episode: 2, return: -1550\n",
      "episode: 3, return: -1486\n",
      "episode: 4, return: -1444\n",
      "episode: 5, return: -1356\n",
      "episode: 6, return: -1478\n",
      "episode: 7, return: -1671\n",
      "episode: 8, return: -1636\n",
      "episode: 9, return: -1535\n",
      "episode: 10, return: -1630\n",
      "episode: 11, return: -1523\n",
      "episode: 12, return: -1510\n",
      "episode: 13, return: -1517\n",
      "episode: 14, return: -1540\n",
      "episode: 15, return: -1292\n",
      "episode: 16, return: -1313\n",
      "episode: 17, return: -1261\n",
      "episode: 18, return: -1231\n",
      "episode: 19, return: -1232\n",
      "episode: 20, return: -1358\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAHFCAYAAAA9occoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABvQElEQVR4nO3deVxU9foH8M/MMAz7IjuIAq4gpiimaIb71qKllVom2e1eLX+VtNyyrpndlltmXttsUbGbebvlUi6VmmupqQluoLiwqGwCsguznd8fMCMjiwwOnDkzn/frNa9k5pwzz2lEHr7f5/t8ZYIgCCAiIiKiFpOLHQARERGR1DCBIiIiIjITEygiIiIiMzGBIiIiIjITEygiIiIiMzGBIiIiIjITEygiIiIiMzGBIiIiIjITEygiIiIiMzGBIpK4pKQkyGQyyGQy7N69u8HrgiCga9eukMlkGDZsmMlrhvMMD1dXV0RGRuL1119HZWWlybEJCQkmx6pUKvTo0QOvvfYaqqurG7zvzz//jLvuugt+fn5QqVQIDQ3FzJkzkZqaasnbv2X1///JZDI4ODigY8eOeOyxx3D58uV2j2f37t1NfpY3k5mZCZlMhqSkJIvH1R6WLl2K+++/H+Hh4Y3+fSWyJkygiGyEu7s7VqxY0eD5PXv24Pz583B3d2/0vClTpuDAgQM4cOAAfvjhB0yZMgWLFi3Co48+2uBYZ2dn47EbN27EwIEDsWjRIsycOdPkuBdffBHjx4+HXq/HJ598gu3bt+O1117D4cOH0a9fP6xfv94yN21Bq1atwoEDB7B9+3Y88cQTWLt2LYYOHdogkaS2s3z5cmRlZWHEiBHw8/MTOxyiZjmIHQARWcZDDz2ENWvW4OOPP4aHh4fx+RUrViAuLg5lZWWNnhcQEIBBgwYZvx41ahSysrKwZs0aVFdXw8nJyfiaXC43OXb8+PHIzMzE//73PyxZsgQhISFYu3Yt3nvvPcyZMweffPKJ8dg777wT06ZNQ3x8PGbMmIG+ffsiIiLCkv8Lbkl0dDRiY2MBAMOHD4dOp8Mbb7yBjRs34uGHHxY5OvuQmpoKubz29/ro6GiRoyFqHkegiGzEtGnTAABr1641PldaWop169Zh1qxZZl3L09MTMpkMCoXipscaEqqsrCwAwJtvvglvb28sXry4wbGurq748MMPUVVVhQ8++OCm1z558iQmTpwIb29vODk5oW/fvli9erXJMYYpr7Vr1+KVV15BcHAwPDw8MGrUKJw5c6Ylt9ui+xIEAZ988gn69u0LZ2dneHt7Y8qUKbhw4YLJecOGDUN0dDQOHz6MoUOHwsXFBREREXjnnXeg1+tNjj19+jTGjRsHFxcX+Pr6Yvbs2SgvL28QS1hYGBISEho8P2zYsJtOcyUkJCAsLKzB8wsXLoRMJjN5TiaTYe7cuVi1ahV69OgBZ2dnxMbG4uDBgxAEAe+99x7Cw8Ph5uaGESNG4Ny5c82+t0ajgb+/P2bMmNHgtZKSEjg7OyMxMdH4nCF5IpIC/m0lshEeHh6YMmUKVq5caXxu7dq1kMvleOihh5o8TxAEaLVaaLValJSU4IcffsDq1asxdepUKJXKm76v4Yeon58fcnNzcerUKYwZMwYuLi6NHh8XFwd/f39s37692eueOXMGgwcPxqlTp7Bs2TKsX78eUVFRSEhIwLvvvtvg+Pnz5yMrKwtffvklPv/8c5w9exb33HMPdDrdTe/hZvcFAH/729/w7LPPYtSoUdi4cSM++eQTnDp1CoMHD0Z+fr7JuXl5eXj44YfxyCOP4Mcff8T48ePx8ssv4+uvvzYek5+fj/j4eJw8eRKffPIJ/vOf/6CiogJz585tVbyWsnnzZnz55Zd45513sHbtWpSXl+Ouu+7Cc889h99//x0fffQRPv/8c6SmpmLy5MkQBKHJaymVSjzyyCNYt25dgxHQtWvXorq6Go899lhb3xJR2xCISNJWrVolABAOHz4s7Nq1SwAgnDx5UhAEQRgwYICQkJAgCIIg9OrVS4iPjzc5F0Cjj/HjxwsVFRUmx86cOVNwdXUVNBqNoNFohCtXrgj//ve/BZlMJgwYMEAQBEE4ePCgAEB46aWXmo154MCBgrOzc7PHTJ06VVCpVEJ2drbJ8+PHjxdcXFyEkpISQRAE4z1PmDDB5Lj//e9/AgDhwIEDzb6P4f/fwYMHBY1GI5SXlwubN28W/Pz8BHd3dyEvL084cOCAAEB4//33Tc69ePGi4OzsLLz44ovG5+Lj4wUAwh9//GFybFRUlDB27Fjj13//+98FmUwmpKSkmBw3evRoAYCwa9cu43OdO3cWZs6c2SD2+Ph4k880IyNDACCsWrXK+NzMmTOFzp07Nzj3tddeE278EQBACAwMNPnsN27cKAAQ+vbtK+j1euPzS5cuFQAIx48fb3Dt+o4fPy4AED7//HOT52+//Xahf//+TZ7X2N9XImvCESgiGxIfH48uXbpg5cqVOHHiBA4fPnzT6bsHH3wQhw8fxuHDh7F3714sW7YMR44cwbhx41BTU2NybGVlJZRKJZRKJfz8/PDss89i/Pjx2LBhg1lxCoLQYProRjt37sTIkSMRGhpq8nxCQgKqqqpw4MABk+fvvfdek69vu+02ANen4G5m0KBBUCqVcHd3x913343AwED89NNPCAgIwObNmyGTyfDII48YR+u0Wi0CAwPRp0+fBivmAgMDcfvttzeIp34su3btQq9evdCnTx+T46ZPn96ieNvK8OHD4erqavw6MjISQG29W/3PzPB8/SnO+v9vtFotAKB3797o378/Vq1aZTw3LS0Nhw4dMntqmciasIicyIbIZDI89thjWLZsGaqrq9G9e3cMHTq02XP8/PyMxdMAMHToUPj5+WHatGlISkrC3/72N+Nrzs7O2Lt3LwBApVKhc+fOJgXrnTp1AgBkZGQ0+55ZWVkNEqMbFRUVISgoqMHzwcHBxtfr8/HxMflapVIBAK5du9bs+xh89dVXiIyMhIODAwICAkzeOz8/H4IgICAgoNFzbyyGvzEWQzz1YykqKkJ4eHiD4wIDA1sUb1vp0KGDydeOjo7NPm9oYbF69eoG03FC3fTerFmz8NRTT+H06dPo2bMnVq1aBZVKZazbI5IiJlBENiYhIQELFizA8uXL8eabb7bqGobRm2PHjpk8L5fLTZKtGwUFBaFXr17Ytm0bqqqqGq2DOnDgAPLz8/HAAw80G4OPjw9yc3MbPJ+TkwMA8PX1vel9mCMyMrLJe/P19YVMJsO+ffuMiVl9jT13Mz4+PsjLy2vwfGPPOTk5NRgNBIDCwsKb/n9o7lxLuueee3D48OFGX5s2bRoSExORlJSEN998E//5z38wadIkeHt7WzQGovbEKTwiGxMSEoIXXngB99xzT4P+TC2VkpICAPD39zf73FdeeQVXr17F888/3+C1yspKPP3003BxccG8efOavc7IkSOxc+dOY8Jk8NVXX8HFxcWknUJbu/vuuyEIAi5fvozY2NgGj969e5t9zeHDh+PUqVMNktRvvvmmwbFhYWE4fvy4yXPp6ektWmUYFhaGgoICk0J3tVqNX375xeyYm+Pj49Pg/4uBt7c3Jk2ahK+++gqbN29GXl4ep+9I8jgCRWSD3nnnnRYfm5+fj4MHDwKonY5JSUnBP//5T3h5ebVqhdS0adNw9OhRLF68GJmZmZg1axYCAgJw5swZfPDBBzh//jy++eabm/aAeu2117B582YMHz4cCxYsQIcOHbBmzRps2bIF7777Ljw9Pc2OrbWGDBmCv/71r3jsscdw5MgR3HnnnXB1dUVubi5+++039O7dG3PmzDHrms8++yxWrlyJu+66C//85z8REBCANWvW4PTp0w2OnTFjBh555BE8+eSTmDx5MrKysvDuu++2qNnkQw89hAULFmDq1Kl44YUXUF1djWXLlrV6dWJrzZo1C99++y3mzp2Ljh07YtSoUQ2OOXLkCDIzMwEAZWVlEAQB33//PQBgwIAB6Ny5c3uGTNQsJlBEdu777783/pBSKpUIDQ3Fvffei1deeaXVP7Dee+89jBgxAh999BFmz56NsrIy+Pv7Y8SIEfjuu+8QFRV102v06NED+/fvx/z58/HUU0/h2rVriIyMxKpVqxrtidTWPvvsMwwaNAifffYZPvnkE+j1egQHB2PIkCENCsZbIjAwEHv27MEzzzyDOXPmwMXFBffddx8++ugjTJw40eTY6dOnIycnB8uXL8eqVasQHR2NTz/9FK+//vpN3yc8PBw//PAD5s+fjylTpiAoKAiJiYm4cuVKi863lFGjRiE0NBQXL17EK6+80mjPp48++qhBny/DVK9YnztRU2SC0EwTDyIiIiJqgDVQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJvaBagN6vR45OTlwd3e/6YapREREZB0EQUB5eTmCg4Mb7VVWHxOoNpCTk3PTjVKJiIjIOl28eBEdO3Zs9hgmUG3A3d0dQO0HUH+neiIiIrJeZWVlCA0NNf4cbw4TqDZgmLbz8PBgAkVERCQxLSm/YRE5ERERkZmYQBERERGZiQkUERERkZmYQBERERGZiQkUERERkZmYQBERERGZiQkUERERkZmYQBERERGZiQkUERERkZmYQBERERGZiQkUERERkZmYQBERERGZiZsJExER2aEr5TXQCwIc5DIoHeRQyuVQKmRQyGUt2kzX3jGBIiIisiOl1zR4/rtj2J6a3+QxSoUMSoUcDnIZHB3kcJDL4aCQwVFR+1+lQg4HhRxKueHP119zUMjhqJCje4A7ZsdH2GwyxgSKiIjITpzJK8ff/nMEmUVVAAC5DNALDY/T6ARodLpbfr/YMG8MCOtwy9exRkygiIiI7MCmYzl48fvjuKbRIcTLGcsf6Y/eHT2h0wvQ6PTQ6gVodXqodXpodQK0OqH2z3r99T/rTI/R6PTQ1J2n0emhqXt9y4lcHM68ih2p+UygiIiISHq0Oj3+9fNpfLEvAwBwR1dfLJsWgw6ujgAAhVwGhVxh0ff0cVPVJlBp+Xh5QqRFr20tmEARERHZqMKKGsz95igOXigGAMyO74IXxvaAQt62dUnxPfzgIJfh/JVKZBZWIszXtU3fTwxsY0BERGSDUi6W4J4Pf8PBC8VwdVTg04f74aXxPds8eQIADyclBkbUTt3tSGu6WF3KmEARERHZmLWHsvHg8gPILa1GhJ8rfpg7BON7B7VrDCN7BgAAfk0raNf3bS9MoIiIiGxEtUaHl9Ydx8vrT0Ct02NMVAB+eGoIuvq7t3ssoyJrE6hDmcUordK0+/u3NSZQRERENiCn5Boe+uwA/nv4ImQy4IWxPbD8kf5wd1KKEk8nHxd083eDTi9gd7rtjUIxgSIiIpK4/ecLcc+Hv+HYpVJ4uSix+rHb8dTwrpC3Q71Tc0ZG2u40HhMoIiIiiRIEAZ/vPY9HvvwDRZVqRAV5YNPcO3Bndz+xQwMAjI7yBwDsPlMAjU4vcjSWxTYGREREElRZo8WL645jy/FcAMD9MSF46/7ecFJatqfTregb6o0Oro4orlTjSOZVxHXxETski+EIFBERkcRcuFKB+z75HVuO58JBLsOiib3w/oN9rCp5AmqbdA7vUTsK9auNtTNgAkVERCQh21PzMfGj35GeXwE/dxX++9dBeDQuzGo37R0VWZtA7UjLhyA0svGeRHEKj4iISAJ0egH/3pGOZTvPAQBiO3vjk4f7wd/DSeTImje0ux8cFXJkFlXh/JVKdPV3Ezski+AIFBERkZUrqVLj8dWHjclTwuAwfPPEIKtPngDATeVg7EpuS9N4TKCIiIisWGpOGe756DfsPnMFKgc5ljzYBwvv7QVHB+n8CB9lg+0MpPN/n4iIyM5sTL6M+z/9HReLr6GjtzPWzRmM+/t1FDsss42sq4M6klWMq5VqkaOxDCZQREREVkar0+P1Tafw7LcpqNbocWd3P2z+vzsQHeIpdmit0tHbBT0D3aEXYDNdyZlAERERWZnF29Kx6vdMAMDc4V2xKmEAvFwcxQ3qFhmm8XbYyDQeEygiIiIrsif9CpbvOQ8AWPJgHzw/tgcUIm/JYgmGabw9Z65ArZV+V3ImUERERFaioKwaid+mAAAeGdRJkvVOTenT0Qu+bipU1GhxKKNY7HBuGRMoIiIiK6DTC3j22xQUVarRM9Adr94VJXZIFiWXyzCiZ+0efTtsoJ0BEygiIiIr8Onuc9h/vgjOSgU+mt7P6rZlsQRjO4PT0u9KzgSKiIhIZIczi7FkezoAYNHEXjbTrftGd3TzhaODHBeLr+FsQYXY4dwSJlBEREQiKqlS4+m1ydALwH0xIZjS33bqnm7k4uiAIV18ANTu6SdlTKCIiIhEIggCnv/uOHJLqxHu64o3JkVb7abAljLS2JWcCRQRERG1QtL+TOxIy4ejQo4Pp8XATeUgdkhtztDOIPliCQorakSOpvWYQBEREYng5OVSvL31NABg/oSeku0ybq4gT2f0CvaAIAC7Tku3qSYTKCIionZWUaPF3G+OQq3TY3RUAGYODhM7pHZlC5sLM4EiIiJqR4Ig4NUNJ5BZVIVgTye8N+U2m697upEhgdp79gqqNTqRo2kdJlBERETt6Ps/L2FjSg4UchmWTYuR/B53rREd4oEADxWq1DocvFAkdjitwgSKiIionZwrKMeCH04BAOaN6obYsA4iRyQOmUyGET2lPY3HBIqIiKgdVGt0mPtNMq5pdBjS1QdzhnUVOyRRjapbjfdrmjS7kjOBIiIiagf/3JKK03nl8HVzxAcP9YVCbl91Tzca0tUXTko5ckqrkZZbLnY4ZmMCRURE1MZ+OpGLrw9mAwDef7Av/N2dRI5IfE5KBe7o6gtAmk01mUARERG1oYvFVXhx3XEAwOz4Lojv7idyRNbDsBpvhwT7QTGBIiIimyQIAvR6cWtrNDo9nv5vMsqrtYjp5IXnxnQXNR5rM6JnbR3UsYslKCirFjka8zCBIiIim/T3dcfRZ9E2/HvHWVTWaEWJYfG2M0jOLoGHkwOWTY2BUsEfu/X5ezihT8faDuw7JTYKJZlP8s0338TgwYPh4uICLy+vBq8XFRVh3LhxCA4OhkqlQmhoKObOnYuysjKT406cOIH4+Hg4OzsjJCQEixYtalD9v2fPHvTv3x9OTk6IiIjA8uXL2/LWiIjIwgRBwE8n81BercUHO9IR/95ufH0wCxqdvt1i2JN+BZ/tuQAAeHfKbQjt4NJu7y0lhs2Fd0isnYFkEii1Wo0HHngAc+bMafR1uVyOiRMn4scff0R6ejqSkpKwY8cOzJ4923hMWVkZRo8ejeDgYBw+fBgffvghFi9ejCVLlhiPycjIwIQJEzB06FAkJydj/vz5ePrpp7Fu3bo2v0ciIrKMwgo1yqu1kMmAzj4uKKyowasbT2LsB3vx04ncNl82X1BWjcRvUwAAMwZ1xrjooDZ9PykzbC782zlpdSWXzLbPr7/+OgAgKSmp0de9vb1NkqvOnTvjySefxHvvvWd8bs2aNaiurkZSUhJUKhWio6ORnp6OJUuWIDExETKZDMuXL0enTp2wdOlSAEBkZCSOHDmCxYsXY/LkyW12f0REZDnnr1QAADp6O2P7vHj893A2/r3jLC4UVmLOmqPoG+qFl8f3xMAIH4u/t04v4NlvU1BUqUbPQHe8clekxd/DlkQFeSDY0wk5pdXYf77Q2GDT2klmBMpcOTk5WL9+PeLj443PHThwAPHx8VCpVMbnxo4di5ycHGRmZhqPGTNmjMm1xo4diyNHjkCj0bRL7EREdGsuXKkEAHTxc4OjgxyPxoVhz4vD8fTIbnBxVCDlYgke+vwgHk86jPR8y/Yg+nT3Oew/XwRnpQIfTe8HJ6XCote3NTKZTJLTeDaXQE2bNg0uLi4ICQmBh4cHvvzyS+NreXl5CAgwzWwNX+fl5TV7jFarRWFhYaPvWVNTg7KyMpMHERGJ50LdCFSEr5vxOTeVAxJHd8fuF4bhkUGdoJDL8OvpAoxbuhcvfHcMuaXXbvl9D2cWY8n2dADAG5Oi0dXf7SZnEHB9Gk9KXclFTaAWLlwImUzW7OPIkSNmXfODDz7A0aNHsXHjRpw/fx6JiYkmr9+447Xhg6r/fEuOqe/tt9+Gp6en8REaGmpWzEREZFmGKbwIP9cGr/m7O+Gfk3pj+7w7MaF3IPQC8N2flzDsvd14+6c0lF5r3WzD1Uo1nl6bDL0A3BcTgsn9Qm7pHuzJoAgfuDgqkF9Wg5OXpTEIIWoN1Ny5czF16tRmjwkLCzPrmoGBgQgMDETPnj3h4+ODoUOH4h//+AeCgoIQGBhoHGkyKCioHS40jDo1dYyDgwN8fBqfK3/55ZdNErWysjImUUREIrpQWDuF11gCZRDh54ZPHu6P5OyrePun0ziUUYzP9lzAfw9dxNzhXTEjrnOLp98EQcAL3x9Hbmk1wn1d8cak6CZ/6aaGnJQKDO3mi19O5WNHWj5617U2sGaiJlC+vr7w9fVts+sbRo5qamoAAHFxcZg/fz7UajUcHR0BANu2bUNwcLAxUYuLi8OmTZtMrrNt2zbExsZCqVQ2+j4qlcqkroqIiMRTo9XhYnEVAKCr382n0GI6eePbvw7CztMF+NfPp5GeX4E3t6YhaX8mEkd3x6SYkJvuW5e0PxM70vLhqJDjw2kxcFNJZo2W1RgZGYBfTuXj19P5mDfa+huOSqYGKjs7GykpKcjOzoZOp0NKSgpSUlJQUVE7TLt161asWrUKJ0+eRGZmJrZu3Yo5c+ZgyJAhxuRo+vTpUKlUSEhIwMmTJ7Fhwwa89dZbxhV4ADB79mxkZWUhMTERaWlpWLlyJVasWIHnn39erFsnIiIzZBVVQS/U1jz5ubfsl1tDIfNPz9yJd6fchiBPJ1wuuYbnvjuGu5btw64zBU3W5py8XIq3t54GAMyf0BPRIdY/emKNRvT0h0wGnLxchrxS6+9KLpkEasGCBYiJicFrr72GiooKxMTEICYmxlgj5ezsjC+++AJ33HEHIiMj8eyzz+Luu+/G5s2bjdfw9PTE9u3bcenSJcTGxuLJJ59EYmKiyfRbeHg4tm7dit27d6Nv37544403sGzZMrYwICKSiAv16p/MnUZTyGV4MDYUu54fhpfG94S7kwNO55XjsVWHMf2LP3D8UonJ8RU1Wsz95ijUOj3GRAVg5uAwC92F/fF1UyEm1AsA8Otp699cWCZIpdxdQsrKyuDp6YnS0lJ4eHiIHQ4RkV35eNc5vPfLGdwXE4IPHup7S9cqqVLjk93nkbQ/E2ptbRfzu24LwgtjeqCzjwvmfZuCjSk5CPZ0wtZnhsLLxdECd2C/DJ/d8B5+WPXY7e3+/ub8/JbMCBQREVFLGFfg+TZdQN5SXi6OmD8hEjufi8f9/UIgkwFbjudi1JI9mJV0GBtTcqCQy7BsWgyTJwsYVdcP6vfzRahSi7N/YUsxgSIiIptiaKIZ0YIC8pbq6O2CJQ/2xdanh2JYDz9o9QJ2nbkCAEgc3R2xYR0s9l72rHuAGzp6O0Ot1eO3s433XrQWTKCIiMhmCIJgrIHq4n/rI1A3igzyQNJjt+ObJwbijq6+eCg2FHPiu1j8feyVTCYzjkL9auVdybnOkoiIbEZhhRpldZsIh/lYPoEyGNzFF4O7tF0bHns2MtIfSfsz8evpAuj1AuQ3aSEhFo5AERGRzTCMPoV4OXMPOokaGO4DN5UDCitqcPxyqdjhNIkJFBER2QxDB/IuFqx/ovbl6CBHfHc/AMCOVOttZ8AEioiIbMb5gqb3wCPpMGwuvCONCRQREVGbu74HHkegpGx4D3/IZcDpvHJculoldjiNYgJFREQ2w7gCjyNQkubt6oj+nb0BADtPW+dqPCZQRERkE2q0OmTXbSLMGijpG1nXzmCHlbYzYAJFREQ2IbtuE2FXRwX8W7iJMFmvUXV1UAfPF6Gixvq6kjOBIiIim3C+rgN5F383szcRJuvTxc8NYT4uUOv02Jd+RexwGmACRURENsGSe+CR+GQymVVP4zGBIiIim9AWe+CRuAztDHadKYBOL4gcjSkmUEREZBMuFBpW4DGBshUDwjrA3ckBxZVqpFy8KnY4JphAERGR5AmCwCaaNkipkGNYD0NTTeuaxmMCRUREkldUeX0T4XDWQNkUw2q8X62sKzkTKCIikjxD/RM3EbY9w7r7QyGXIT2/AtlF1tOVnAkUERFJnnEFHuufbI6nixIDwmq7klvT3nhMoIiISPIusIWBTRtV187g19NMoIiIiCzmQr0mmmR7DP2g/rhQjLJqjcjR1GICRUREkmeYwuvCESibFO7rigg/V2j1AvZaSVdyJlBERCRpaq0eF69eA8AaKFtmnMazknYGTKCIiEjSsosrodMLcHVUIMCDmwjbqpE9a9sZ7DxdAK1OL3I0TKCIiEjizhVc38KFmwjbrv6dveHlokTpNQ3+zBK/KzkTKCIikjTDFi7sQG7bHBRyDK/rSv7rafGn8ZhAERGRpBlX4LH+yeYZNhe2hn5QTKCIWkhvZTuBE1Gt6000OQJl6+7s7gcHuQwXrlQio7BS1FiYQBG1QGFFDYb8aycSVh0SOxQiqkcQBOMIVIQvR6BsnYeTEgMjOgAQf288JlBELfD1wSzkllZj95kryBT5tx4iuq64Uo3SaxpuImxHRvasbWewPZUJFJFVU2v1+PpgtvHrbal5IkZDRPWdrxt9CvZ0hrMjNxG2B4Z+UMkXS1BZoxUtDiZQRDex5UQOCitqjF9vOyV+8SIR1brA+ie708nHBZ/P6I/D80fBVeUgWhxMoIiaIQgCVv2eCQB4eGAnAMCf2VdxpbymmbOIqL1cKOQKPHs0plcgPF2UosbABIqoGUezS3D8UikcHeRIHN0dt3X0hCCIX7xIRLXOF9TtgccRKGpnTKCImpG0PxMAMLFPMHzcVBjbKxAA8Msp1kERWQPDCBT3wKP2xgSKqAl5pdX46UQuACBhSBgAYExUbfHi7+eKUCFi8SIR1S7wyC6uAsApPGp/TKCImvD1wSxo9QJuD++AXsGeAICu/m4I93WFWqfHnjNXRI6QyL5xE2ESExMookZUa3T45lBt64JZdaNPACCTyTCmV+0oFKfxiMRlaGEQ7ufKTYSp3TGBImrEjyk5KK5UI8TL2dhzxGBMVG0d1K7TBVBr9WKER0TgHngkLiZQRDcQBAGr6orHH43rDAeF6bdJTKgX/NxVKK/R4uCFIhEiJCKgXg8obuFCImACRXSDPzKKkZZbBielHA8NCG3wulwuw+i6YnJ2JScSDzcRJjExgSK6QVJd48z7+3WEl4tjo8cYVuNtO5UPvV5or9CIqB420SQxMYEiqudicZVxVClhcFiTx8V18YGbygEF5TU4dqmkfYIjIqPiSjVKqjQAuIkwiYMJFFE9Xx/Mgl4A7ujqi+4B7k0ep3JQYFgPPwDANpF3BCeyR4bpuxAvbiJM4mACRVSnSq3F2rrWBc2NPhmwKzmReLiJMImNCRRRnQ3Jl1FWrUVnHxeM6Ol/0+OH9fCDUiHDhSuVOFe3HxcRtQ+2MCCxMYEiQm3rAkPx+My4MMjlN2/K5+6kxOAuvgC4Go+ovXEFHomNCRQRgN/OFeJsQQVcHRWYEtuxxecZpvG2nWIdFFF74ggUiY0JFBGuty54IDYUHk7KFp83KsofMhmQcrEEeaXVbRQdEdWn0V3fRJgjUCQWJlBk9zILK7HzTAGA2s7j5vB3d0JMqBcAYHsaR6GI2kNWURW0egEujgoEejiJHQ7ZKSZQZPdWH8iEIADDe/ghohXTAWOM03isgyJqD/VX4HETYRILEyiya+XVGnx35BIAIGFIeKuuYaiDOnC+CKXXNBaLjYgaZ+hAzj3wSEySSaDefPNNDB48GC4uLvDy8mr22KKiInTs2BEymQwlJSUmr504cQLx8fFwdnZGSEgIFi1aBEEw3Ypjz5496N+/P5ycnBAREYHly5db+G7IWqz78xIqarTo4ueKO7v5tuoa4b6u6ObvBq1ewO66qUAiajvnC7gCj8QnmQRKrVbjgQcewJw5c2567OOPP47bbrutwfNlZWUYPXo0goODcfjwYXz44YdYvHgxlixZYjwmIyMDEyZMwNChQ5GcnIz58+fj6aefxrp16yx6PyQ+vV7A6gNZAGobZ97KVMCYXtf3xiOitmUcgeIKPBKRg9gBtNTrr78OAEhKSmr2uE8//RQlJSVYsGABfvrpJ5PX1qxZg+rqaiQlJUGlUiE6Ohrp6elYsmQJEhMTIZPJsHz5cnTq1AlLly4FAERGRuLIkSNYvHgxJk+e3Ba3RiLZk34FGYWVcHdywP39Wt66oDFjewXi413nsftMAao1OjgpubUEUVsx1EB14QgUiUgyI1AtkZqaikWLFuGrr76CXN7w1g4cOID4+HioVCrjc2PHjkVOTg4yMzONx4wZM8bkvLFjx+LIkSPQaBqvb6mpqUFZWZnJg6zfyt8zAAAPxYbCVXVrv0v0DvFEoIcTKtU67D9faInwiKgRxZVqXOUmwmQFbCaBqqmpwbRp0/Dee++hU6dOjR6Tl5eHgIAAk+cMX+fl5TV7jFarRWFh4z8Y3377bXh6ehofoaGht3o71MbOFZRj39lCyGXAzBbse3czMpmM03hE7cAw+hTs6QQXR8lMopANEjWBWrhwIWQyWbOPI0eOtOhaL7/8MiIjI/HII480e9yNdS6GAvL6z7fkmBvfu7S01Pi4ePFii2Im8STtzwQAjIoMQGgHF4tcc0xU7Wq8HWn50OmFmxxNRK1h7EDuz/onEpeo6fvcuXMxderUZo8JCwtr0bV27tyJEydO4PvvvwdwPenx9fXFK6+8gtdffx2BgYHGkSaDgoLaVVOGUaemjnFwcICPj0+j761SqUymBcm6lVZpsO7PywCAhCFhFrvuwIgO8HByQGGFGkezr2JAWAeLXZuIahn3wOP0HYlM1ATK19cXvr6tWzp+o3Xr1uHatWvGrw8fPoxZs2Zh37596NKlCwAgLi4O8+fPh1qthqOjIwBg27ZtCA4ONiZqcXFx2LRpk8m1t23bhtjYWCiVLd/ig6zX/45cxDWNDj0D3REX0XhS3BpKhRwjIwOwIfkytp3KYwJF1AbOX+EKPLIOkqmBys7ORkpKCrKzs6HT6ZCSkoKUlBRUVNStxujSBdHR0cZHeHhtU8TIyEj4+/sDAKZPnw6VSoWEhAScPHkSGzZswFtvvWVcgQcAs2fPRlZWFhITE5GWloaVK1dixYoVeP7558W5cbIonV7A6gOZAG69dUFjxkTV1UGl5jfoL0ZEt+5CoWEFHhMoEpdkKvAWLFiA1atXG7+OiYkBAOzatQvDhg1r0TU8PT2xfft2PPXUU4iNjYW3tzcSExORmJhoPCY8PBxbt27FvHnz8PHHHyM4OBjLli1jCwMbsSMtH5euXoOXixKTYkIsfv34Hn5QOciRVVSF9PwK9Ah0t/h7ENkrjU6P7CJuIkzWQTIJVFJS0k17QNU3bNiwRkcAevfujb179zZ7bnx8PI4ePWpuiCQBq+paF0y7vVOb9GpycXTA0G6+2JFWgF9O5TGBIrKg7OLaTYSdldxEmMQnmSk8oluVlluGgxeKoZDLMGNQ5zZ7H8NqvG2p3FyYyJIuGOufXCGXcxNhEhcTKLIbSb9nAgDG9QpEsJdzm73PyEh/yGXAyctluFxy7eYnEFGLGFfgsf6JrAATKLILxZVqbEypbV3wmAVbFzTGx02F2LoVeNtOcRSKyFIusIUBWREmUGQX1h7KRo1Wj+gQD/Tv7N3m72dcjceu5EQWwyaaZE2YQJHN0+j0+M+BLADAY4PDLd66oDFje9XWQR3KLMbVSnWbvx+RPWATTbImTKDI5v1yKg95ZdXwdXPE3X2C2uU9Qzu4IDLIAzq9gJ2nC9rlPYls2dV6mwizhQFZAyZQZPNW1RWPTx/YGSoHy7cuaIphGu8X1kER3TJDA01uIkzWggkU2bTjl0rwZ9ZVKBUyPDKwU7u+95hetQnU3rNXcE2ta9f3JrI15wu4hQtZFyZQZNMMrQvu6h0E/3ZuvBcV5IEQL2dUa/TYd/ZKu743ka05X2hoYcDpO7IOTKDIZhWUV2PT8RwAwGNDwtv9/WUymbGY/BeuxiO6JcYVeByBIivBBIps1jd/ZEOjExDTyQt9Qr1EicEwjffr6XxodXpRYiCyBdebaHIEiqwDEyiySWqtHl8fzAYgzuiTQWxnb3i7KFFSpcHhzKuixUEkZaabCHMEiqwDEyiySVtO5KCwogYBHiqMjw4ULQ4HhRyjIuuaanJvPKJWuVhvE+EgbiJMVoIJFNkcQRCMrQtmDOoMpULcv+Zj6uqgtp3KhyAIosZCJEXn6+qfwn25iTBZDyZQZHOOZpfg+KVSODrIMe329m1d0Jih3XzhrFTgcsk1nMopEzscIsm5wPonskJMoMjmrPo9AwAwsU8wfNxUIkcDOCkVuLO7LwBgWypX4xGZiyvwyBoxgSKbklt6DT+drK01ShgSJm4w9Yw1TuOxDorIXFyBR9aICRTZlK8PZkGnF3B7eAf0CvYUOxyjET39oZDLcDqvHFlFlWKHQyQpFwo5AkXWhwkU2YxqjQ7f/FHbumCWFY0+AYCXiyMGhncAAGznNB5Ri12tVKO4Ug2gtoicyFowgSKb8WNKDq5WaRDi5WxsHWBNxtZbjUdELWPYRDjI0wmuKm4iTNaDCRTZhJIqNT7ZfQ4A8GhcZziI3LqgMaOjapO6w1nFKKyoETkaImkwtDBg/RNZG+v7KUNkpiq1FrOSDiOzqAoBHipMHSB+64LGBHs5o3eIJwQB+DWNo1BELcEVeGStmECRpKm1esz++iiOZpfA01mJr2YNhKeLUuywmjSmbhSK03hELWNcgcf6J7IyTKBIsnR6AYn/S8He9CtwViqwMmEAegS6ix1Ws8bWbSuz71whKmq0IkdDZP2uN9HkCBRZFyZQJEmCIGDhj6ew+XgulAoZls/oj/6dvcUO66a6+bshzMcFaq0ee9OviB0OkVXT6PTILq7dRLiLPxMosi5MoEiSPtiejv8czIJMBix5sC/iu/uJHVKLyGSyenvjsakmUXMuFldBoxPgpJRzE2GyOkygSHJW/paBZTtrV9wtmhiNe/oEixyRecb2qq2D+vV0ATQ6vcjREFmvC8ZNhN24iTBZHSZQJCkbki9h0eZUAMBzo7tjxqDOIkdkvr6h3vB1U6G8WouDF4rEDofIahl6QHVhCwOyQkygSDJ2ns7H898dBwDMGhKOuSO6ihxR6yjkMoyO8gfA1XhEzTlfYOgBxfonsj5MoEgSDmUUY87XR6HTC7g/JgSv3hUJmUy6Q/pjomrroLan5kOvF0SOhsg6cQSKrBkTKLJ6qTlleHz1YdRo9RjZ0x//mnKb5OshBnf1gaujAnll1Th+uVTscIisEptokjVjAkVWLbOwEo+uPITyai1uD+uAjx/uB6UVbtNiLpWDAsN6GqbxuBqP6EYlVWoUcRNhsmLS/0lENiu/rBqPrPgDhRU1iAzywJcJsXBSKsQOy2KMXclTWQdFdCPDHniBHtxEmKxTqxKoixcv4tKlS8avDx06hGeffRaff/65xQIj+1ZapcGjKw7h0tVrCPNxwVezboeHk/Vu0dIaw3v6Q6mQ4VxBhXG7CiKqZehA3sWfo09knVqVQE2fPh27du0CAOTl5WH06NE4dOgQ5s+fj0WLFlk0QLI/VWotHks6hDP55fB3V+E/jw+En7tK7LAszsNJibguvgC4Go/oRoYRqAhf1j+RdWpVAnXy5EncfvvtAID//e9/iI6Oxv79+/HNN98gKSnJkvGRnVFr9ZhTb3Pg/zw+EKEdXMQOq81cn8ZjHRRRfdf3wOMIFFmnViVQGo0GKlXtiMCOHTtw7733AgB69uyJ3Nxcy0VHdkWvF/Dcd8ewR0KbA9+q0XUJVHJ2CQrKqkWOhsh6XCjkCjyybq1KoHr16oXly5dj37592L59O8aNGwcAyMnJgY+Pj0UDJPsgCAIWbjqFTcdyJLU58K0K8HBCTCcvACwmJzLQ6vTIKjI00eQIFFmnViVQ//rXv/DZZ59h2LBhmDZtGvr06QMA+PHHH41Te0TmWLrjLL46ULs58PsS2hzYEgxNNZlAEdW6ePWacRPhYE9nscMhalSr1oYOGzYMhYWFKCsrg7f39VGCv/71r3Bxsd16FWobq37PwL9/PQugdnPgeyW2OfCtGtMrAP/6+TQOnC9EWbXG5lYbEpnLUP/ETYTJmrW6D5RCoTBJngAgLCwM/v7+txwU2Y+NyZfx+iZpbw58q7r4uaGrvxs0OgG7z1wROxwi0Rk6kHP6jqxZqxKo/Px8zJgxA8HBwXBwcIBCoTB5ELVE7ebAxwAAjw0Jk+zmwJZgWI2363SByJEQic/QF60LO5CTFWvVFF5CQgKys7Pxj3/8A0FBQZLe1JXEcTizdnNgrV7AfTEh+MddUXb99+j28A74ZPd5HL9UInYoRKIz7oHnzxV4ZL1alUD99ttv2LdvH/r27WvhcMgepOaUYVbS9c2B37WBzYFvVXSIJ4DapdsVNVq4cesKsmMXCut6QLGJJlmxVk3hhYaGQhAES8dCdiCryDY3B75Vvm4qBHk6QRCAtNwyscMhEk1plQaFFXWbCLMGiqxYq35yLV26FC+99BIyMzMtHA7ZsoIbNgf+YqZtbQ58q3oF145CnbhUKnIkROI5Xzf6FOCh4kgsWbVW/e186KGHUFVVhS5dusDFxQVKpemy6+LiYosER7ZDEGq7jF8svobOPi5YPWsAPJ25XL++3iGe2JGWj5M5TKDIfhnrn9iBnKxcqxKopUuXWjgMsnXbUvOx72whHBVyrEoYAH93J7FDsjrRIR4AgJOXmUCR/TrPPfBIIsxOoDQaDXbv3o1//OMfiIiIaIuYyMZUa3R4Y3Ntr6cn7gxHBH+zbFTvukLycwUVuKbWwdmR05tkf4ybCLOAnKyc2TVQSqUSGzZsaItYyEZ9vvcCLl29hkAPJzw13H57Pd2Mv4cT/NxV0AtAKgvJyU6xhQFJRauKyO+77z5s3LjRwqGQLbp0tQqf7D4HAJh/VyRcHFkU2hzDKNQp1kGRHdLq9Mg0bCLMJppk5VqVQHXt2hVvvPEGpkyZgrfffhvLli0zebSFN998E4MHD4aLiwu8vLwaPUYmkzV4LF++3OSYEydOID4+Hs7OzggJCcGiRYsatGTYs2cP+vfvDycnJ0RERDS4BrXc21tPo1qjx8DwDrjntiCxw7F60cG1dVBciUf26FLdJsIqBzlCvLiJMFm3Vg0HfPnll/Dy8sKff/6JP//80+Q1mUyGp59+2iLB1adWq/HAAw8gLi4OK1asaPK4VatWYdy4ccavPT09jX8uKyvD6NGjMXz4cBw+fBjp6elISEiAq6srnnvuOQBARkYGJkyYgCeeeAJff/01fv/9dzz55JPw8/PD5MmTLX5ftmz/uUJsOZELuQxYeG8vu+403lKGhponcziFR/bH0EAz3NfV7pvrkvVrVQKVkZFh6Thu6vXXXwcAJCUlNXucl5cXAgMDG31tzZo1qK6uRlJSElQqFaKjo5Geno4lS5YgMTHROGLVqVMn40rDyMhIHDlyBIsXL2YCZQaNTo+Fm04BAB4Z1BmRQR4iRyQNhgTqbH45qjU69skiu3K+gC0MSDpsrgX03Llz4evriwEDBmD58uXQ6/XG1w4cOID4+HioVCrjc2PHjkVOTo6xKeiBAwcwZswYk2uOHTsWR44cgUajafQ9a2pqUFZWZvKwd18fzEJ6fgW8XZRIHN1d7HAkI8jTCT6ujtDqBZzJKxc7HKJ2ZdzChS0MSAJaNQI1a9asZl9fuXJlq4K5VW+88QZGjhwJZ2dn/Prrr3juuedQWFiIV199FQCQl5eHsLAwk3MCAgKMr4WHhyMvL8/4XP1jtFotCgsLERTUsI7n7bffNo6QEVBYUYMl29MBAC+M7QkvF0eRI5IOmUyGXiGe2Jt+BScul6JPqJfYIRG1m/NsokkS0qoRqKtXr5o8CgoKsHPnTqxfvx4lJSUtvs7ChQsbLfyu/zhy5EiLr/fqq68iLi4Offv2xXPPPYdFixbhvffeMznmxjocQwF5/edbckx9L7/8MkpLS42PixcvtjhmW/Tez2dQXq1FdIgHHhoQKnY4ktO7rqEmV+KRvbnAJpokIa0agWqsD5Rer8eTTz5pVnPNuXPnYurUqc0ec+OIkTkGDRqEsrIy5OfnIyAgAIGBgcjLyzM5pqCgAMD1kaimjnFwcICPj0+j76NSqUymBe3ZsYsl+N+ftQnk6/f2goKFoGaLNuyJx47kZEdKr9XbRJgtDEgCLNaURy6XY968eRg2bBhefPHFFp3j6+sLX19fS4XQQHJyMpycnIxtD+Li4jB//nyo1Wo4OtZOK23btg3BwcHGRC0uLg6bNm0yuc62bdsQGxvbYM8/MqXXC3jtx1MQBOD+mBD079xB7JAkyVBIfiavHGqtHo4ONleqSNSAYfQpwEMFdyf+W0vWz6L/Mp8/fx5ardaSlzTKzs5GSkoKsrOzodPpkJKSgpSUFFRU1H7Tbdq0CV988QVOnjyJ8+fP48svv8Qrr7yCv/71r8bRoenTp0OlUiEhIQEnT57Ehg0b8NZbbxlX4AHA7NmzkZWVhcTERKSlpWHlypVYsWIFnn/++Ta5L1uy7uglpFwsgaujAi+N7yl2OJLV0dsZns5KaHQC0vNZSE72wVD/xC1cSCpaNQKVmJho8rUgCMjNzcWWLVswc+ZMiwR2owULFmD16tXGr2NiYgAAu3btwrBhw6BUKvHJJ58gMTERer0eERERWLRoEZ566injOZ6enti+fTueeuopxMbGwtvbG4mJiSb3Ex4ejq1bt2LevHn4+OOPERwcjGXLlrGFwU2UVWvwr59PAwCeHtkN/h7cLLi1ZDIZeod44rdzhTh5udQ4IkXUmIKyalSpdejUwUXSvZNY/0RS06oEKjk52eRruVwOPz8/vP/++zddoddaSUlJzfaAGjdunEkDzab07t0be/fubfaY+Ph4HD161NwQ7dqyHWdRWKFGhK8rHhsSLnY4ktcrxAO/nSvEiculaL5KkOxRaZUGW0/mYmPyZRzKLIYgAC6OCvQIdEfPQA9EBbkjMsgDPYM84KaSxvZJF7gCjySmVd9Zu3btsnQcJGHnCsqRtD8TALDgnijW7FhAb3YkpxtUa3TYdboAG1MuY9fpK1Drrve4c3SQo0qtQ3J2CZKzS0zO69TBBZFBtYlVZJAHooI80NHb2epGq85zBIokplUJ1IgRI7B+/foGe9KVlZVh0qRJ2LlzpyViIwkQBAELf0yFVi9gVGQAhvXwFzskm2BYiZeWWwaNTg+lgkmpPdLrBRzMKMLG5Mv46WQeyquv15j2CHDHpJgQ3Ns3GAHuKmQWVSI1txxpuWXGR35ZDbKLq5BdXIVfTuUbz3VTOaBnYO0oVe1IlTt6BrqLttm3Ti8gq6gKAEegSDpa9d2ye/duqNXqBs9XV1dj3759txwUSccvp/Lx27lCODrIseDuKLHDsRmdfVzg7uSA8motzhVUcCscOyIIAlJzy/BDSg5+TMlBXlm18bUgTyfc2zcYk/qGNPg70dXfHV393XFvn2Djc8WVapzOLUNqbhnS6pKrcwUVqKjR4kjWVRzJumo8ViYDwnxcERnkjsi60arIYA8Eezq1+T6Wl65WQa3TQ+UgRzA3ESaJMCuBOn78uPHPqampJv2SdDodfv75Z4SEhFguOrJq1Rod/rklFQDw16ER6OTjInJEtkMmk6FXsAcOXijGiculTKDswKWrVfghJQcbky/jbEGF8XkPJwdM6B2ESTEhuD2sg1lTbx1cHTG4qy8Gd73eLkaj0+PClUrjKJUhuSqsqEFGYSUyCiux9cT1f9s9nBwwKSYEz43uAU+XtmkvYJi+C/d1Ze84kgyzEqi+ffsaO4SPGDGiwevOzs748MMPLRYcWbfle87j0tVrCPZ0wpPDu4gdjs3pHeKJgxeKcepyKRDLju626GqlGltO5OKHlMs4nHl9NMjRQY6RPf0xsW8Ihvf0g8rBcptKKxVy9Ah0R4/A2ilAgyvlNTidZ5j+uz5aVVatxVcHsrD5eC5eGt8TU/p1tHj9lKGAnPVPJCVmJVAZGRkQBAERERE4dOgQ/Pz8jK85OjrC398fCgV3j7cHl65W4dPd5wEA8++KFK12wpYZ2hewI7ltqdbosCMtHxuTc7AnvQAanWGrKGBQuA8mxQRjXHQQPJ3bt5mkn7sKfu5+GNrt+r/rNVod/rhQjDc2p+JsQQVe/P44/nsoG29MikavYMu11+AeeCRFZv3U69y5M4DabVvIvr25JQ01Wj0GRXTAXb0bbrBMt86QQKXmlkGnFzi1IWE6vYAD54uwIfkyfjmVh4qa68XgUUEemBQTjHv6BCPI07rqf1QOCtzZ3Q9bnxmKpN8zsXRHOo5ml+CeD3/Do3FhmDe6u0USPa7AIylq9bDBf/7zHyxfvhwZGRk4cOAAOnfujA8++AARERGYOHGiJWMkK/P7uUL8dDIPCrkMC+/t1eYFpvYq3McVro4KVKp1OH+lAt0D3MUOiVphe2o+XtlwAgXlNcbnQrycMbFvMCbFhEjic1Uq5Hjizgjc0ycY/9ySis3Hc5G0PxObj+fg5fGRuL9fyC39O3CBXchJglq1NvrTTz9FYmIiJkyYgJKSEuh0OgCAt7c3li5dasn4yMpodHq89uMpAMCMQZ3RM5DFzW1FLpcZp0lOchpPkrQ6PV5eX5s8ebko8fDATvhudhz2vTgcL47rKYnkqb5ATyd8NL0f1vxlILr4uaKwQo3nvjuGBz87gLTc1vUsq91EuDa55AgUSUmrEqgPP/wQX3zxBV555RWTmqfY2FicOHHCYsGR9fnqQBbOFVSgg6sj5o3qLnY4Nq9XSG2Cyjooadp/vgiFFTXwdlHi4Msj8eZ9vTHAzJV01mhIV1/89Myd+Pu4nnBWKnA48yru/vA3LNqUivJqjVnXMmzh4u/OTYRJWlqVQGVkZBj3oqtPpVKhsrLyloMi63SlvAZLt6cDAF4Y23ZLmuk6Q0fyU5fZkVyKNiZfBgDc0ycYTkrbWmDj6CDHnGFd8Otz8RgfHQidXsDK3zMw4v09+CHlMgRBaNF1uAKPpKpVCVR4eDhSUlIaPP/TTz8hMjLyVmMiK/Xuz6dRXqNF7xBPPMhl9e3CUEh+KqcUen3LfiCRdahSa/HLqdp+ShP72m5/vGAvZ3z6SH+snnU7wn1dcaW8Bs/8NwVTPz+I9Pzym55/obB2BIor8EhqWpVAvfDCC3jqqafw7bffQhAEHDp0CG+++SZefvllvPjii5aOkaxAysUSfPfnJQDAwnt7cUVYO+ni5wYnpRyVah0yiji6KyXbU/NRqdahUwcX9OvkJXY4bS6+ux9+fnYoXhjbA05KOf7IKMaEf+/Dm1tSTVYd3uh8gWEEigkUSUurVuE99thj0Gq1ePHFF1FVVYXp06cjJCQEH374IYYOHWrpGElker2A1344CQC4v18I+nf2Fjki+6GQyxAV5IGj2SU4ebmUv6VLyA8pOQCASX2D7WalqspBgaeGd8W9fYLxxuZUbEvNxxf7MvDjsRy8elcU7r4tqMH/C8MIFKfwSGpavUPpE088gaysLBQUFCAvLw+HDh1CcnIyunbtasn4yAp8/+clHLtUCjeVA14a31PscOyOoQ7KXlfiVam1SPw2BWsPZYsdSosVVdRgT/oVAMDEGNudvmtKaAcXfP5oLFYmxKJTBxfkl9Xg/9Ym4+Ev/8C5guvTejq9gMzC2k2Eu/KXA5IYsxKokpISPPzww/Dz80NwcDCWLVuGDh064OOPP0bXrl1x8OBBrFy5sq1iJRGUXtPgXz+fBgA8M7Ib/N2dRI7I/vSy847k645exvrky1j44ymUVpm3wkssW07kQqcXcFtHT7seNRzRMwDb5t2JeaO6Q+Ugx/7zRRj/731456fTqKzRGjcRduQmwiRBZk3hzZ8/H3v37sXMmTPx888/Y968efj5559RXV2NrVu3Ij4+vq3iJJH8e8dZFFWqEeHnipmDw8QOxy7VX4mn1wuSXwJvru/rau9qtHpsSL6EhCHhIkd0cxvqVt/ZcvF4SzkpFXhmVDfcFxOChZtOYefpAizfcx4/pFzG2F6BAGqbxrKukqTGrBGoLVu2YNWqVVi8eDF+/PFHCIKA7t27Y+fOnUyebFB6fjlWH8gEACy8pxccHVo940u3oKu/Gxwd5Civ0SK7uErscNrV2fxyHLtYYvz6m0PZLV4eL5asokokZ5dALgPu6cNtjgw6+bhgZcIAfPloLDp6OyO3tBpJ+zMBAF38Wf9E0mPWT8ScnBxERUUBACIiIuDk5IS//OUvbRIYiUsQBLy+6RR0egFjogJwZ3e/m59EbUKpkCMyqLah5skc+5rGM4w+DYroACelHOn5FTiafVXkqJq3Mbm2eHxIV19OeTdiVFQAts+Lx9MjusJRUfsjqEcAdzQg6TErgdLr9VAqrzdPVCgUcHXlbw626OeTefj9XBEcHeR49a4oscOxe9HB9teRXKvTY33dVFjC4HDcfVswAOCbPy6KGVazBEHADym1Md9nh8XjLeXsqEDimB7YNu9OLLg7CgksDyAJMqsGShAEJCQkQKVSAQCqq6sxe/bsBknU+vXrLRchtbtrah3+uSUNADD7zgh08nEROSKyx47k+84W4kp5DTq4OmJET3/4uavw/Z+XsPl4DhbcHWWVnfCPXyrFhcJKOCnlGFNX30NNC/N1xaw7rL+mjagxZiVQM2fONPn6kUcesWgwZB0+3XMel0uuIdjTCXOGsS2FNYiutxJPEAS76Cv03Z+1I00T+wbD0UGOfp280CPAHWfyy7Ex5bJVLmrYWDf6NCYqEG6qVrXZIyKJMOs7fNWqVW0VB1mJi8VVWL7nPADglbui4OxoW/t3SVX3AHcoFTKUXtPg0tVrCO1g26OCVyvV2JFaAAB4oH/ttkEymQzTbg/Fwk2p+OaPbDwa19mqEkmtTo9Nx+qaZ8YEixwNEbU1LqsiE//ckgq1Vo+4CB9M6M0pCGvh6CBHj0B3APbRUHPT8RyodXpEBXkgKvh6gfF9MR2hcpDjTH45jmaXiBdgI34/X4TCCjU6uDpiaDcuuiCydUygyOjPrGL8ciofCrkMC+/tZVW/3VO9juR2sBLvuyO1q++m9O9o8ryni9JYTG5tnck31hW8331bEJQK/tNKZOv4XU5GBy8UAwDG9Qo0jnaQ9egVbKiDsu1C8tN5ZThxuRRKhQyTGlnJNn1g7ZTe5uM5KL1mHZ3Jq9Ra/HIqDwAajZmIbA8TKDI6k1e7R1WvEPZksUbXV+KVWn0zyVvxfd3o04ie/ujg6tjg9X6dvNE9wA3VGr2xZYDYtqfmo0qtQ2cfF8SEeokdDhG1AyZQZGRIoHoEcPTJGvUIdIeDXIaiSjVyS6vFDqdNaHR640o2Q/H4jWqLyTsBAL75wzo6k2+st3ULp76J7AMTKAIAqLV6nL9SAQCcvrNSTkoFugXYdiH5njNXUFihhq+bI+J7NF2IfX9dMfnpvHIk19vqRQyFFTXYe7YQADCpL1ffEdkLJlAEAMgsqoRWL8DVUYEQ7oputQwdyW01gTL0fprUN6TZQmxPFyXuuq12n7m1f4hbTL7leC50egF9Onoiws9N1FiIqP0wgSIAwOm66bvuge6cgrBivTsaVuLZXiF5UUUNfk2r7f00JbbjTY4GptdN4206noOyavGKyTfUm74jIvvBBIoAAOl1CVRPTt9Ztesr8WxvBOrHYznQ6gX0DvFEz8CbL2To39kb3fxri8kNNUjtLbOwEikXS6CQy3BPH07fEdkTJlAEoN4IFAvIrVpUkAfkMuBKeQ0KymyrkLyp3k9NkclkmD5Q3GJyQ8H7kK6+8HNXtfv7E5F4mEARACA9nyvwpMDZUYGu/rV1NrY0CnUqpxSpuWVwVMhxrxkjOffFhBiLyVPauZhcEAT8kJJTFwdHn4jsDRMoQpVai+ziKgBcgScFho2FT9pQQ811f9aO5IyK8od3I72fmuLl4oi7etcVk7dzZ/Jjl0qRUVgJZ6UCY6K47RGRvWECRUjPr21f4OvmCB83TkNYu2gbq4NSa2/e+6k50+qm8TYdy23XYnJD3dWYXgFwVZm1LzsR2QAmUGQsIOfokzQYVuKdspE98XadKUBxpRp+7ioM7eZr9vmxnb3R1d8N1zQ6/NBOxeRanR6bj9dO303i6jsiu8QEinAmnwXkUhIV5AGZDMgtrUZhRY3Y4dwyQ/H4/TEhcGjFJrwymczY0mBNOxWT/3auEIUVavi4OuKOViR9RCR9TKCIW7hIjKvKARG+rgCk31DzSnkNdp2p6/3UwtV3jbm/Xwgc64rJj11q+/8nhum7u28LarbhJxHZLn7nk3EEilN40nG9kFzaCdQPKZdru3iHehm3qWkNk2LyNu5MXlmjxS+n8gEAE2M4fUdkr5hA2bniSjWulNdOA93KDzBqX71tYCWeIAj4/k/zej81x7DB8I/H2rYz+fbUfFzT6NDZxwUxoV5t9j5EZN2YQNk5w/RdR29nuHElkWTYQkfyUzllOJ1XDkcHOe697db7KA0Iq1dMXtefqS0YVgxO7BvCbY+I7BgTKDtnaKDJLVykpVdI7VYnl0uu4WqlWuRoWscw+jQmKgCeLspbvp5MJjOOQrVVZ/LCihrsO1sIAJjUl80ziewZEyg7xxV40uThpESYjwsA4KQE2xnUaHXXez/Fmt/7qSn3x9QWk6flluF4GxSTbz6WU1uz1dETEX5uFr8+EUkHEyg7d4Y9oCSrl4TroHamFaCkSoMADxXu6Gq5NgDero6YEF3bFbwtOpNvqJsanMTicSK7xwTKjgmCwCaaEtZbwivxDNN39/frCIXcsnVE9YvJyy1YTJ5RWIljF0ugkMtwtwVqtohI2phA2bGc0mqU12jhIJchwpfTEVJj2NJFalN4BeXV2J1+BYBlVt/d6PbwDuji54oqtWWLyQ29n+7o6gs/d255RGTvmEDZMcPoU4SfKxwd+FdBaqLrCsmziqpQeq399oC7VRuTa3s/9evkhS5tUEfUFsXkgiDgh7qarUkxHH0iIiZQdo0F5NLm5eKIjt7OAKSzL1793k+WLB6/0eR+HeGokCM1t8wirR5SLpYgs6gKzkoFxkQFWiBCIpI6JlB2jFu4SJ/U6qCOXypFen4FVA5y3HVbUJu9j7erI8b3tlwxuWEqcEyvALiyXxoRgQmUXeMKPOmLlthKPMPo07joQHg43Xrvp+YYNhj+IeXWisk1Oj02HePqOyIyJZkE6s0338TgwYPh4uICLy+vJo9LSkrCbbfdBicnJwQGBmLu3Lkmr584cQLx8fFwdnZGSEgIFi1a1KBGYs+ePejfvz+cnJwQERGB5cuXt8UtiUqr0+PclQoATKCkTEp74lVrdMY6ogf6t930ncHt4R0QUVdM/uOx1heT/3auEEWVavi4OmKoBVsuEJG0SSaBUqvVeOCBBzBnzpwmj1myZAleeeUVvPTSSzh16hR+/fVXjB071vh6WVkZRo8ejeDgYBw+fBgffvghFi9ejCVLlhiPycjIwIQJEzB06FAkJydj/vz5ePrpp7Fu3bo2vb/2lllUBbVWD2elAqHeLmKHQ60UHVxbSH6hsNKiS/bbwo60fJRVaxHs6YS4Lj5t/n4ymcw4CnUr03iG1Xf39AmGg0Iy/2QSURuTzGT+66+/DqB2hKkxV69exauvvopNmzZh5MiRxud79epl/POaNWtQXV2NpKQkqFQqREdHIz09HUuWLEFiYiJkMhmWL1+OTp06YenSpQCAyMhIHDlyBIsXL8bkyZPb7P7aW7qxgNwNcgv34aH24+OmQrCnE3JKq5GaU4aBEW2fmLRWW/Z+asr9/Tri3Z/P4OTlMpy4VIreHT3NOr+yRottp/IBABO5dQsR1WMzv05t374der0ely9fRmRkJDp27IgHH3wQFy9eNB5z4MABxMfHQ6W63sNl7NixyMnJQWZmpvGYMWPGmFx77NixOHLkCDQa6/4N3xyG+ieuwJM+Y0fyHOutg8orrcbeNuz91JQO9YrJvzmUZfb521LzcE2jQ5iPC/qGelk4OiKSMptJoC5cuAC9Xo+33noLS5cuxffff4/i4mKMHj0aanXtZqt5eXkICAgwOc/wdV5eXrPHaLVaFBYWNvreNTU1KCsrM3lYOxaQ2w4prMTbkHwZegEYEOaNMF/Xdn3vafWKyStqtGaduzG5tnZqYt8QyGQcqSWi60RNoBYuXAiZTNbs48iRIy26ll6vh0ajwbJlyzB27FgMGjQIa9euxdmzZ7Fr1y7jcTf+I2goIK//fEuOqe/tt9+Gp6en8REa2vYFsrfKMIXHBEr6DA01rTWBqu39VDsS3B7F4zcaGN4BEb51xeRmdCa/Ul6DfWdrR824+o6IbiRqDdTcuXMxderUZo8JCwtr0bWCgmp7ykRFRRmf8/Pzg6+vL7KzawtIAwMDjSNNBgUFBQCuj0Q1dYyDgwN8fBqvL3n55ZeRmJho/LqsrMyqk6hqjQ6ZRZUA2APKFhhW4p2/UoEqtRYujtZV2ph8sQTnr1TCWanAhDbs/dQUQ2fyN7emYe2hbEwf2KlF520+ngO9APQJ9UJ4O4+aEZH1E/VfWl9fX/j6WmZZ8JAhQwAAZ86cQceOtTUWxcXFKCwsROfOnQEAcXFxmD9/PtRqNRwdHQEA27ZtQ3BwsDFRi4uLw6ZNm0yuvW3bNsTGxkKpbLxvjUqlMqmrsnbnCiqgFwAvFyX39LIB/u5O8HdXoaC8Bmm5ZejfuYPYIZkwFI+Pjw6Em0hNKCf374j3fjmDE5dLW1xMblh9dx+Lx4moEZKpgcrOzkZKSgqys7Oh0+mQkpKClJQUVFTU9jLq3r07Jk6ciGeeeQb79+/HyZMnMXPmTPTs2RPDhw8HAEyfPh0qlQoJCQk4efIkNmzYgLfeesu4Ag8AZs+ejaysLCQmJiItLQ0rV67EihUr8Pzzz4t275ZWvwM56zpsg6EO6sQl65rGq9bojE0op8S2X/H4jTq4OmJctKGY/OYtDS5cqcCxS6VQyGW4uw8TKCJqSDIJ1IIFCxATE4PXXnsNFRUViImJQUxMjEmN1FdffYWBAwfirrvuQnx8PJRKJX7++WfjyJGnpye2b9+OS5cuITY2Fk8++SQSExNNpt/Cw8OxdetW7N69G3379sUbb7yBZcuW2VQLgzOsf7I51roS75dTeSiv1qKjtzMGhYvbYsFQTP5jyuWbFpNvrKuVGtrNF75uHKUlooasq1iiGUlJSU32gDLw8PDAihUrsGLFiiaP6d27N/bu3dvsdeLj43H06NHWhCkJXIFne6x1JZ5h+m5yv46i9xsbFNEB4b6uyCisxKZjOcaE6kaCIBg7pk/qy+JxImqcZEagyHKMK/BYQG4zDCvxzhZUoFqjEzmaWjkl1/DbudrWH5P7iTd9Z1BbTF67uKO5zuTJF0uQVVQFF0cFxvQKaPI4IrJvTKDsTGmVBrml1QCAbkygbEaghxN83Ryh0wtIy7WOabwNyZchCLVtBDr5WMd2QVP6h8JRIcfxS6VNjtb9UFc8PiYqwOpWNBKR9WACZWfSC2pHn4I9neDp3PiqQpIemUyGXsHWUwdV2/updvrugVjraenRwdURY5spJtfo9Nh0PBcAMJG9n4ioGUyg7IxxCxfWP9kcYx2UFazE+zPrKjIKK+HiqMD4uoTFWhim8X5IvozKG4rJfztbiOJKNXxcHTG0q2VarBCRbWICZWfqtzAg22LsSJ4jfgJlGH2a0DsIriL1fmpKXIQPwnxcUKm+3mLBYGNd8fg9fYLhoOA/j0TUNP4LYWfYwsB2GTqSp+eXo0YrXiH5NbUOm+umwR5ox42DW8rQmRwwLSavrNFi26l8ANy6hYhujgmUHREEwbgCrztHoGxOiJczvFyU0OgEpOdViBbHz6dyUVGjRacOLrg93Lq6ohtM6d8RSoUMx+oVk29LzcM1jQ5hPi7o04JO5URk35hA2ZGC8hqUVGkglwFd/d3EDocsTCaTXe9ILmI/KMP03ZT+Ha22072Pmwpje9XWZhlGoTYk107nTYoJsdq4ich6MIGyI4b6pzBfVzgpFSJHQ23h+ko8cRKoS1ersP98EQDg/n7WPQ02vW4a74eUHGQVVeK3s1cAsHkmEbUMEyg7wgaatk/sjuTrj9b2fhrcxQcdva2j91NTBtUVk1fUaDH3m2ToBaBvqBfCfF3FDo2IJIAJlB05ncf6J1tnWIl3OrccGp2+Xd/btPeT9RWP30guv15MbpjynNSXGwcTUcswgbIjhhGonlyBZ7M6dXCBu5MD1Do9zua3byH5oYxiZBdXwU3lYKwvsnaT64rJAUAhl+HuPkygiKhlmEDZCb2+3go8JlA2SyaTITpYnGk8w+jTXb2DJLMFiq+bCmPqkr2h3Xzh66YSOSIikgomUHYiu7gK1Ro9HB3kCPNhjYct692x/QvJK2u02HKirveTBKbv6ntpXE/cHxOCVyZEih0KEUmINH5NpFtmaKDZzd8NCjmXaNuyXsG1dVDt2crgp5N5qFLrEO7riv6dvdvtfS0htIMLljzUV+wwiEhiOAJlJ7iFi/0wrMRLyy2Dth0KybU6vbGXkjX3fiIisiQmUHaCW7jYjzAfV7ipHFCt0eP8lco2fa9qjQ5PrjmKP7OuQqmQ4T5ugUJEdoIJlJ1Iz2MBub2Qy2WIqpvGa8tC8rJqDWauPIRtqflwdJDjw2n9EOzl3GbvR0RkTZhA2YEarQ4XCmtHIjiFZx8MK/Haqg6qoLwaD312EH9kFMNd5YCvZt2OcdHSaF1ARGQJLCK3AxeuVEKnF+Du5IAgTyexw6F20Ltj7QjUqTZYiZdVVIkZKw4hu7gKvm4qrJ41wLiFDBGRvWACZQfqb+HCAl/7YBiBOpVTBp1esNjKy5OXS5Gw6jAKK2rQqYML/vP47ejMthhEZIc4hWcHTrP+ye5E+LnBWalAlVqHjELLFJIfOF+EqZ8fRGFFDaKCPPD9nDgmT0Rkt5hA2QFDATm3cLEfCgsXkv98MhczVx5CRY0WgyI64L9/GwR/d04HE5H9YgJlBwwtDLiJsH0x9IO61QTqmz+y8eSao1Dr9BjbKwBJj90ODyelJUIkIpIs1kDZuIoaLS5dvQaAK/Dsza12JBcEAR/tPIf3t6cDAKbdHop/TurNTvZERGACZfMMBeT+7ip4uzqKHA21J8OeeKk5ZdDrBcjNSHz0egGLNqciaX8mAOD/RnRF4ujuXIRARFSHCZSNM9Q/sQO5/enq5waVgxzlNVpkFVch3LdlBd9qrR7PfXcMm47lAAAW3hOFhCHhbRkqEZHksAbKxp3mHnh2y0EhR2SQeYXklTVaPL76MDYdy4GDXIZ/T+3L5ImIqBFMoGycYQqPLQzsU3RIyxOo4ko1pn9xEPvOFsLFUYEVCQMwsS/3tiMiagyn8GzcGY5A2TXjSrybdCS/dLUKj648hAtXKuHtosSqx25H31CvdoiQiEiamEDZsMKKGhRVqiGTAd0C3MQOh0Rg2GLl5OUyCILQaBF4en45Hl1xCHll1Qj2dMJXjw9EV3/+fSEiag6n8GyYoYC8UwcXuDgyV7ZH3QPc4aiQo/SaxtjOor4/s4rxwPIDyCurRjd/N6x7cjCTJyKiFmACZcOMW7hw+s5uOTrIjSswb+wHtfN0Ph7+8g+UXtOgXycvfDc7DkGezmKESUQkOUygbJihgJxbuNi36EY6kq/78xKe+OpPVGv0GNbDD1//ZSC8XNgnjIiopTivY8O4hQsB11fiGUagvth7AW9uTQMA3BcTgnen3Aalgr9LERGZgwmUjdLrBTbRJACme+K9/VMaPttzAQDwlzvCMX9CpFkdyomIqBYTKBt1ueQaKtU6KBWyFnegJtvUPcAdDnIZrlZpjMnTS+N74m93RnBrFiKiVuK4vY0y1D918XPj9Iydc1IqjNO4chnw7pTbMDu+C5MnIqJbwJ+sNoor8Ki+qbeHIsTLGZ/NiMWDsaFih0NEJHmcwrNRhhEo1j8RADwaF4ZH48LEDoOIyGZwBMpGcQsXIiKitsMEygZpdHqcv1IBgCNQREREbYEJlA3KLKyERifA1VGBEC92liYiIrI0JlA2yNBAs1uAO3v8EBERtQEmUDbIUP/ELVyIiIjaBhMoG3SGLQyIiIjaFBMoG3SGLQyIiIjaFBMoG1Ol1iK7uAoAEygiIqK2wgTKxpwrqIAgAD6ujvB1U4kdDhERkU1iAmVjuIULERFR22MCZWPS81j/RERE1NaYQNkYFpATERG1PckkUG+++SYGDx4MFxcXeHl5NXg9KSkJMpms0UdBQYHxuBMnTiA+Ph7Ozs4ICQnBokWLIAiCybX27NmD/v37w8nJCREREVi+fHlb357FsIUBERFR23MQO4CWUqvVeOCBBxAXF4cVK1Y0eP2hhx7CuHHjTJ5LSEhAdXU1/P39AQBlZWUYPXo0hg8fjsOHDyM9PR0JCQlwdXXFc889BwDIyMjAhAkT8MQTT+Drr7/G77//jieffBJ+fn6YPHly29/oLbhaqUZBeQ0AoHuAm8jREBER2S7JJFCvv/46gNqRpsY4OzvD2fn6vm9XrlzBzp07TZKtNWvWoLq6GklJSVCpVIiOjkZ6ejqWLFmCxMREyGQyLF++HJ06dcLSpUsBAJGRkThy5AgWL15s9QlUet30XYiXM9ydlCJHQ0REZLskM4Vnrq+++gouLi6YMmWK8bkDBw4gPj4eKtX15f1jx45FTk4OMjMzjceMGTPG5Fpjx47FkSNHoNFoGn2vmpoalJWVmTzEYKh/4hYuREREbctmE6iVK1di+vTpJqNSeXl5CAgIMDnO8HVeXl6zx2i1WhQWFjb6Xm+//TY8PT2Nj9DQUEveSosZ65+YQBEREbUpUROohQsXNln4bXgcOXLE7OseOHAAqampePzxxxu8JpPJTL42FJDXf74lx9T38ssvo7S01Pi4ePGi2TFbgmEKrwcLyImIiNqUqDVQc+fOxdSpU5s9JiwszOzrfvnll+jbty/69+9v8nxgYKBxpMnAsELPMOrU1DEODg7w8fFp9P1UKpXJtKAYBEEwNtFkCwMiIqK2JWoC5evrC19fX4tes6KiAv/73//w9ttvN3gtLi4O8+fPh1qthqOjIwBg27ZtCA4ONiZqcXFx2LRpk8l527ZtQ2xsLJRK6y3MziurRnm1Fgq5DBF+rmKHQ0REZNMkUwOVnZ2NlJQUZGdnQ6fTISUlBSkpKaioqDA57ttvv4VWq8XDDz/c4BrTp0+HSqVCQkICTp48iQ0bNuCtt94yrsADgNmzZyMrKwuJiYlIS0vDypUrsWLFCjz//PPtcp+tZRh9Cvd1hcpBIXI0REREtk0ybQwWLFiA1atXG7+OiYkBAOzatQvDhg0zPr9ixQrcf//98Pb2bnANT09PbN++HU899RRiY2Ph7e2NxMREJCYmGo8JDw/H1q1bMW/ePHz88ccIDg7GsmXLrL+FAafviIiI2o1MuLENN92ysrIyeHp6orS0FB4eHu3ynon/S8H6o5eROLo7nh7ZrV3ek4iIyJaY8/NbMlN41Dxu4UJERNR+mEDZAJ1ewNmC2lowNtEkIiJqe0ygbEBWUSXUWj2clHKEdnAROxwiIiKbxwTKBhim77r5u0Mhb7zZJxEREVkOEygbYNgDjyvwiIiI2gcTKBvALVyIiIjaFxMoG8AtXIiIiNoXEyiJq9bokFlYCYAJFBERUXthAiVx569UQC8Ans5K+LuLu6ExERGRvWACJXFn6k3fGfbzIyIiorbFBErizrCAnIiIqN0xgZI4wybC3Vn/RERE1G6YQEmcYQqPW7gQERG1HyZQElZWrUFOaTUAoLs/EygiIqL2wgRKwgzTd4EeTvB0UYocDRERkf1gAiVh3MKFiIhIHEygJCydHciJiIhEwQRKwgxbuHRnCwMiIqJ2xQRKogRBMG4izBV4RERE7YsJlERdqajB1SoN5DKgq7+b2OEQERHZFSZQEmXo/xTm4wonpULkaIiIiOwLEyiJOsP6JyIiItEwgZIoQ/0Tt3AhIiJqf0ygJIpbuBAREYmHCZQE6fUC0vMrAHAKj4iISAxMoCTo0tVruKbRwdFBjjAfF7HDISIisjtMoCTodF4ZAKCrnxscFPwIiYiI2ht/+kpQOvfAIyIiEhUTKAniFi5ERETiYgIlQdzChYiISFxMoCRGrdXjwpVKAOwBRUREJBYmUBJzobACWr0Ad5UDgj2dxA6HiIjILjGBkhjjFi6B7pDJZCJHQ0REZJ+YQEmMcQsXFpATERGJhgmUxHALFyIiIvExgZKYMxyBIiIiEh0TKAmprNHiYvE1AGyiSUREJCYHsQOglsstrYa/uwoCgA6ujmKHQ0REZLeYQElIV383HHplFCprtGKHQkREZNc4hSdBrirmvURERGJiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJiZQRERERGZiAkVERERkJgexA7BFgiAAAMrKykSOhIiIiFrK8HPb8HO8OUyg2kB5eTkAIDQ0VORIiIiIyFzl5eXw9PRs9hiZ0JI0i8yi1+uRk5MDd3d3yGQyi167rKwMoaGhuHjxIjw8PCx6bWtjT/cK2Nf98l5tlz3dL+/V9giCgPLycgQHB0Mub77KiSNQbUAul6Njx45t+h4eHh42/Ze4Pnu6V8C+7pf3arvs6X55r7blZiNPBiwiJyIiIjITEygiIiIiMzGBkhiVSoXXXnsNKpVK7FDanD3dK2Bf98t7tV32dL+8V/vGInIiIiIiM3EEioiIiMhMTKCIiIiIzMQEioiIiMhMTKCIiIiIzMQEygp98sknCA8Ph5OTE/r37499+/Y1e/yePXvQv39/ODk5ISIiAsuXL2+nSFvv7bffxoABA+Du7g5/f39MmjQJZ86cafac3bt3QyaTNXicPn26naJuvYULFzaIOzAwsNlzpPi5AkBYWFijn9NTTz3V6PFS+lz37t2Le+65B8HBwZDJZNi4caPJ64IgYOHChQgODoazszOGDRuGU6dO3fS669atQ1RUFFQqFaKiorBhw4Y2ugPzNHe/Go0Gf//739G7d2+4uroiODgYjz76KHJycpq9ZlJSUqOfd3V1dRvfTfNu9tkmJCQ0iHnQoEE3va41frY3u9fGPh+ZTIb33nuvyWta6+falphAWZlvv/0Wzz77LF555RUkJydj6NChGD9+PLKzsxs9PiMjAxMmTMDQoUORnJyM+fPn4+mnn8a6devaOXLz7NmzB0899RQOHjyI7du3Q6vVYsyYMaisrLzpuWfOnEFubq7x0a1bt3aI+Nb16tXLJO4TJ040eaxUP1cAOHz4sMl9bt++HQDwwAMPNHueFD7XyspK9OnTBx999FGjr7/77rtYsmQJPvroIxw+fBiBgYEYPXq0cX/Mxhw4cAAPPfQQZsyYgWPHjmHGjBl48MEH8ccff7TVbbRYc/dbVVWFo0eP4h//+AeOHj2K9evXIz09Hffee+9Nr+vh4WHyWefm5sLJyaktbqHFbvbZAsC4ceNMYt66dWuz17TWz/Zm93rjZ7Ny5UrIZDJMnjy52eta4+fapgSyKrfffrswe/Zsk+d69uwpvPTSS40e/+KLLwo9e/Y0ee5vf/ubMGjQoDaLsS0UFBQIAIQ9e/Y0ecyuXbsEAMLVq1fbLzALee2114Q+ffq0+Hhb+VwFQRCeeeYZoUuXLoJer2/0dal+rgCEDRs2GL/W6/VCYGCg8M477xifq66uFjw9PYXly5c3eZ0HH3xQGDdunMlzY8eOFaZOnWrxmG/FjffbmEOHDgkAhKysrCaPWbVqleDp6WnZ4CyssXudOXOmMHHiRLOuI4XPtiWf68SJE4URI0Y0e4wUPldL4wiUFVGr1fjzzz8xZswYk+fHjBmD/fv3N3rOgQMHGhw/duxYHDlyBBqNps1itbTS0lIAQIcOHW56bExMDIKCgjBy5Ejs2rWrrUOzmLNnzyI4OBjh4eGYOnUqLly40OSxtvK5qtVqfP3115g1a9ZNN9aW6udqkJGRgby8PJPPTaVSIT4+vsnvX6Dpz7q5c6xVaWkpZDIZvLy8mj2uoqICnTt3RseOHXH33XcjOTm5fQK8Rbt374a/vz+6d++OJ554AgUFBc0ebwufbX5+PrZs2YLHH3/8psdK9XNtLSZQVqSwsBA6nQ4BAQEmzwcEBCAvL6/Rc/Ly8ho9XqvVorCwsM1itSRBEJCYmIg77rgD0dHRTR4XFBSEzz//HOvWrcP69evRo0cPjBw5Env37m3HaFtn4MCB+Oqrr/DLL7/giy++QF5eHgYPHoyioqJGj7eFzxUANm7ciJKSEiQkJDR5jJQ/1/oM36PmfP8azjP3HGtUXV2Nl156CdOnT292s9mePXsiKSkJP/74I9auXQsnJycMGTIEZ8+ebcdozTd+/HisWbMGO3fuxPvvv4/Dhw9jxIgRqKmpafIcW/hsV69eDXd3d9x///3NHifVz/VWOIgdADV042/qgiA0+9t7Y8c39ry1mjt3Lo4fP47ffvut2eN69OiBHj16GL+Oi4vDxYsXsXjxYtx5551tHeYtGT9+vPHPvXv3RlxcHLp06YLVq1cjMTGx0XOk/rkCwIoVKzB+/HgEBwc3eYyUP9fGmPv929pzrIlGo8HUqVOh1+vxySefNHvsoEGDTIqvhwwZgn79+uHDDz/EsmXL2jrUVnvooYeMf46OjkZsbCw6d+6MLVu2NJtcSP2zXblyJR5++OGb1jJJ9XO9FRyBsiK+vr5QKBQNfjspKCho8FuMQWBgYKPHOzg4wMfHp81itZT/+7//w48//ohdu3ahY8eOZp8/aNAgSf6G4+rqit69ezcZu9Q/VwDIysrCjh078Je//MXsc6X4uRpWVZrz/Ws4z9xzrIlGo8GDDz6IjIwMbN++vdnRp8bI5XIMGDBAcp93UFAQOnfu3GzcUv9s9+3bhzNnzrTqe1iqn6s5mEBZEUdHR/Tv39+4aslg+/btGDx4cKPnxMXFNTh+27ZtiI2NhVKpbLNYb5UgCJg7dy7Wr1+PnTt3Ijw8vFXXSU5ORlBQkIWja3s1NTVIS0trMnapfq71rVq1Cv7+/rjrrrvMPleKn2t4eDgCAwNNPje1Wo09e/Y0+f0LNP1ZN3eOtTAkT2fPnsWOHTtaldwLgoCUlBTJfd5FRUW4ePFis3FL+bMFakeQ+/fvjz59+ph9rlQ/V7OIVb1Ojfvvf/8rKJVKYcWKFUJqaqrw7LPPCq6urkJmZqYgCILw0ksvCTNmzDAef+HCBcHFxUWYN2+ekJqaKqxYsUJQKpXC999/L9YttMicOXMET09PYffu3UJubq7xUVVVZTzmxnv94IMPhA0bNgjp6enCyZMnhZdeekkAIKxbt06MWzDLc889J+zevVu4cOGCcPDgQeHuu+8W3N3dbe5zNdDpdEKnTp2Ev//97w1ek/LnWl5eLiQnJwvJyckCAGHJkiVCcnKycdXZO++8I3h6egrr168XTpw4IUybNk0ICgoSysrKjNeYMWOGyara33//XVAoFMI777wjpKWlCe+8847g4OAgHDx4sN3v70bN3a9GoxHuvfdeoWPHjkJKSorJ93FNTY3xGjfe78KFC4Wff/5ZOH/+vJCcnCw89thjgoODg/DHH3+IcYtGzd1reXm58Nxzzwn79+8XMjIyhF27dglxcXFCSEiIJD/bm/09FgRBKC0tFVxcXIRPP/200WtI5XNtS0ygrNDHH38sdO7cWXB0dBT69etnsrR/5syZQnx8vMnxu3fvFmJiYgRHR0chLCysyb/w1gRAo49Vq1YZj7nxXv/1r38JXbp0EZycnARvb2/hjjvuELZs2dL+wbfCQw89JAQFBQlKpVIIDg4W7r//fuHUqVPG123lczX45ZdfBADCmTNnGrwm5c/V0HLhxsfMmTMFQahtZfDaa68JgYGBgkqlEu68807hxIkTJteIj483Hm/w3XffCT169BCUSqXQs2dPq0kem7vfjIyMJr+Pd+3aZbzGjff77LPPCp06dRIcHR0FPz8/YcyYMcL+/fvb/+Zu0Ny9VlVVCWPGjBH8/PwEpVIpdOrUSZg5c6aQnZ1tcg2pfLY3+3ssCILw2WefCc7OzkJJSUmj15DK59qWZIJQV5lKRERERC3CGigiIiIiMzGBIiIiIjITEygiIiIiMzGBIiIiIjITEygiIiIiMzGBIiIiIjITEygiIiIiMzGBIiK7lpmZCZlMhpSUlDZ7j4SEBEyaNKnNrk9E7Y8JFBFJWkJCAmQyWYPHuHHjWnR+aGgocnNzER0d3caREpEtcRA7ACKiWzVu3DisWrXK5DmVStWicxUKBQIDA9siLCKyYRyBIiLJU6lUCAwMNHl4e3sDAGQyGT799FOMHz8ezs7OCA8Px3fffWc898YpvKtXr+Lhhx+Gn58fnJ2d0a1bN5Pk7MSJExgxYgScnZ3h4+ODv/71r6ioqDC+rtPpkJiYCC8vL/j4+ODFF1/EjTtmCYKAd999FxEREXB2dkafPn3w/fffG1+/WQxEJD4mUERk8/7xj39g8uTJOHbsGB555BFMmzYNaWlpTR6bmpqKn376CWlpafj000/h6+sLAKiqqsK4cePg7e2Nw4cP47vvvsOOHTswd+5c4/nvv/8+Vq5ciRUrVuC3335DcXExNmzYYPIer776KlatWoVPP/0Up06dwrx58/DII49gz549N42BiKyEuHsZExHdmpkzZwoKhUJwdXU1eSxatEgQBEEAIMyePdvknIEDBwpz5swRBEEQMjIyBABCcnKyIAiCcM899wiPPfZYo+/1+eefC97e3kJFRYXxuS1btghyuVzIy8sTBEEQgoKChHfeecf4ukajETp27ChMnDhREARBqKioEJycnBrsVP/4448L06ZNu2kMRGQdWANFRJI3fPhwfPrppybPdejQwfjnuLg4k9fi4uKaXHU3Z84cTJ48GUePHsWYMWMwadIkDB48GACQlpaGPn36wNXV1Xj8kCFDoNfrcebMGTg5OSE3N9fk/RwcHBAbG2ucxktNTUV1dTVGjx5t8r5qtRoxMTE3jYGIrAMTKCKSPFdXV3Tt2tWsc2QyWaPPjx8/HllZWdiyZQt27NiBkSNH4qmnnsLixYshCEKT5zX1/I30ej0AYMuWLQgJCTF5zVD43lwMRGQdWANFRDbv4MGDDb7u2bNnk8f7+fkhISEBX3/9NZYuXYrPP/8cABAVFYWUlBRUVlYaj/39998hl8vRvXt3eHp6IigoyOT9tFot/vzzT+PXUVFRUKlUyM7ORteuXU0eoaGhN42BiKwDR6CISPJqamqQl5dn8pyDg4Ox8Pq7775DbGws7rjjDqxZswaHDh3CihUrGr3WggUL0L9/f/Tq1Qs1NTXYvHkzIiMjAQAPP/wwXnvtNcycORMLFy7ElStX8H//93+YMWMGAgICAADPPPMM3nnnHXTr1g2RkZFYsmQJSkpKjNd3d3fH888/j3nz5kGv1+OOO+5AWVkZ9u/fDzc3N8ycObPZGIjIOjCBIiLJ+/nnnxEUFGTyXI8ePXD69GkAwOuvv47//ve/ePLJJxEYGIg1a9YgKiqq0Ws5Ojri5ZdfRmZmJpydnTF06FD897//BQC4uLjgl19+wTPPPIMBAwbAxcUFkydPxpIlS4znP/fcc8jNzUVCQgLkcjlmzZqF++67D6WlpcZj3njjDfj7++Ptt9/GhQsX4OXlhX79+mH+/Pk3jYGIrINMEG5oUEJEZENkMhk2bNjArVSIyKJYA0VERERkJiZQRERERGZiDRQR2TRWKRBRW+AIFBEREZGZmEARERERmYkJFBEREZGZmEARERERmYkJFBEREZGZmEARERERmYkJFBEREZGZmEARERERmYkJFBEREZGZ/h8v8hV+NNItYQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "real_ratio = 0.5\n",
    "env_name = 'Pendulum-v1'\n",
    "env = gym.make(env_name)\n",
    "num_episodes = 20\n",
    "actor_lr = 5e-4\n",
    "critic_lr = 5e-3\n",
    "alpha_lr = 1e-3\n",
    "hidden_dim = 128\n",
    "gamma = 0.98\n",
    "tau = 0.005  # 软更新参数\n",
    "buffer_size = 10000\n",
    "target_entropy = -1\n",
    "model_alpha = 0.01  # 模型损失函数中的加权权重\n",
    "state_dim = env.observation_space.shape[0]\n",
    "action_dim = env.action_space.shape[0]\n",
    "action_bound = env.action_space.high[0]  # 动作最大值\n",
    "\n",
    "rollout_batch_size = 1000\n",
    "rollout_length = 1  # 推演长度k,推荐更多尝试\n",
    "model_pool_size = rollout_batch_size * rollout_length\n",
    "\n",
    "agent = SAC(state_dim, hidden_dim, action_dim, action_bound, actor_lr,\n",
    "            critic_lr, alpha_lr, target_entropy, tau, gamma)\n",
    "model = EnsembleDynamicsModel(state_dim, action_dim, model_alpha)\n",
    "fake_env = FakeEnv(model)\n",
    "env_pool = ReplayBuffer(buffer_size)\n",
    "model_pool = ReplayBuffer(model_pool_size)\n",
    "mbpo = MBPO(env, agent, fake_env, env_pool, model_pool, rollout_length,\n",
    "            rollout_batch_size, real_ratio, num_episodes)\n",
    "\n",
    "return_list = mbpo.train()\n",
    "\n",
    "episodes_list = list(range(len(return_list)))\n",
    "plt.plot(episodes_list, return_list)\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Returns')\n",
    "plt.title('MBPO on {}'.format(env_name))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "332d2220-00e6-4df8-a78c-62476631e865",
   "metadata": {},
   "source": [
    "可以看到，相比无模型的强化学习算法，基于模型的方法 MBPO 在样本效率上要高很多。虽然这里的效果可能不如 16.3 节提到的 PETS 算法优秀，但是在许多更加复杂的环境中（如 Hopper 和 HalfCheetah），MBPO 的表现远远好于 PETS 算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886e7ab6-8958-4ddc-8005-db65838d58a8",
   "metadata": {},
   "source": [
    "## 17.4 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51b5fe78-e4ac-49a3-8153-3c220b1151a0",
   "metadata": {},
   "source": [
    "MBPO 算法是一种前沿的基于模型的强化学习算法，它提出了一个重要的概念——分支推演。在各种复杂的环境中，作者验证了 MBPO 的效果超过了之前基于模型的方法。MBPO 对于基于模型的强化学习的发展起着重要的作用，不少之后的工作都是在此基础上进行的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29808111-085a-433c-b783-d094ac375b8c",
   "metadata": {},
   "source": [
    "除了算法的有效性，MBPO 的重要贡献还包括它给出了关于分支推演步数与模型误差、策略偏移程度之间的定量关系，进而阐明了什么时候我们可以相信并使用环境模型，什么样的环境导出的最优分支推演步数为 0，进而建议不使用环境模型。相应的理论分析在 17.5 节给出。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7bc2d71-6ec3-465d-8c9b-b3d0a3b0f45d",
   "metadata": {},
   "source": [
    "## 17.5 拓展阅读：MBPO 理论分析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76476f9b-050c-499e-925c-4d4e118e5b70",
   "metadata": {},
   "source": [
    "### 17.5.1 性能提升的单调性保障"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23cb28fc-8379-4196-bee0-93cc3e1745dd",
   "metadata": {},
   "source": [
    "基于模型的方法往往是在环境模型中提升策略的性能，但这并不能保证在真实环境中策略性能也有所提升。因此，我们希望模型环境和真实环境中的结果的差距有一定的限制，具体可形式化为："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "802eb45f-afa2-47ee-bc90-9f29524854ac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}