examples/how_to_scale_op.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) 2024 Graphcore Ltd. All rights reserved.\n",
    "\n",
    "# How to unit-scale an op\n",
    "\n",
    "The unit-scaled maximal update parametrisation, [u-μP](https://arxiv.org/abs/2407.17465), enables hyperparameter transfer and low-precision training by paying careful attention to the _scale_ (standard deviation, or 'std') of tensors in the forward and backward passes.\n",
    "\n",
    "In order to construct u-μP models, we need _scaled_ ops. A scaled op produces approximately unit-std outputs when given unit-std inputs. Likewise, in the backward pass, it produces unit-std input gradients when given unit-std output gradients. The [unit-scaling](https://github.com/graphcore-research/unit-scaling) library provides implementations of many common ops, but it can never be exhaustive, so in this notebook we walk through how to unit-scale an op for ourselves.\n",
    "\n",
    "Structure:\n",
    " - **Introduction** - what is the task?\n",
    " - **Empirical scaling** - scaling our op via simulation and empirical measurement.\n",
    " - **Statistical scaling** - scaling our op via statistical analysis.\n",
    " - **Scaling constraints** - a mechanism for supporting the cut-edge rule.\n",
    " - **Summing up** - phew!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Imports/preamble/helpers (nothing much to see here - feel free to skip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import e, erf, pi, sqrt, exp\n",
    "from typing import *\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torch import Tensor\n",
    "\n",
    "from unit_scaling.constraints import apply_constraint\n",
    "from unit_scaling.scale import scale_fwd, scale_bwd\n",
    "\n",
    "matplotlib.rc(\"axes\", **{\"spines.top\": False, \"spines.right\": False})\n",
    "matplotlib.rc(\"legend\", frameon=False)\n",
    "\n",
    "def jointplot(df: pd.DataFrame, *, x: str, y: str,\n",
    "              xlabel: Optional[str] = None, ylabel: Optional[str] = None) -> sns.JointGrid:\n",
    "    g = sns.JointGrid(data=df, x=x, y=y, height=4, ratio=2)\n",
    "    g.plot_joint(sns.scatterplot, s=8, lw=0.2)\n",
    "    g.plot_marginals(sns.histplot, bins=20)\n",
    "\n",
    "    g.ax_joint.set_xticks([-2, -1, 0, 1, 2])\n",
    "    g.ax_joint.set_xlim(-2.5, 2.5)\n",
    "    g.ax_joint.set_xlabel(f\"${xlabel or x}$\")\n",
    "    g.ax_joint.set_yticks([-2, -1, 0, 1, 2])\n",
    "    g.ax_joint.set_ylim(-2.5, 2.5)\n",
    "    g.ax_joint.set_ylabel(f\"${ylabel or y}$\")\n",
    "\n",
    "    x_rms = sqrt((df[x]**2).mean())\n",
    "    y_rms = sqrt((df[y]**2).mean())\n",
    "    g.ax_marg_x.set_title(f\"$\\\\mathrm{{RMS}}({xlabel or x})={x_rms:.2f}$\", fontsize=10)\n",
    "    g.ax_marg_y.set_title(f\"$\\\\mathrm{{RMS}}({ylabel or y})={y_rms:.2f}$\", fontsize=10)\n",
    "\n",
    "def opplot(name_to_fn: Dict[str, Callable[[Tensor], Tensor]]) -> None:\n",
    "    fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(9, 3))\n",
    "    x = torch.linspace(-2.5, 2.5, int(1e4)).requires_grad_()\n",
    "    for name, fn in name_to_fn.items():\n",
    "        x.grad = None\n",
    "        y = fn(x)\n",
    "        y.backward(torch.ones_like(y))\n",
    "        ax0.plot(x.detach(), y.detach(), label=f\"y={name}\")\n",
    "        ax1.plot(x.detach(), x.grad)\n",
    "    ax0.set_xlabel(\"x\")\n",
    "    ax0.set_ylabel(\"y\")\n",
    "    ax1.set_xlabel(\"x\")\n",
    "    ax1.set_ylabel(\"dy/dx\")\n",
    "    fig.legend(*ax0.get_legend_handles_labels(), loc=\"center left\", bbox_to_anchor=(.9, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our chosen task is to unit-scale `F.hardtanh`, an elementwise nonlinearity provided by PyTorch that looks like a harder/sharper version of `tanh`. It's defined as `F.hardtanh(x, a=-1, b=1) = clip(x, a, b)`, and is suitable as an illustrative example, as it permits both empirical and statistical scaling methods, as we'll see.\n",
    "\n",
    "The op and gradient look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7UAAAEmCAYAAAC9NLfxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCsElEQVR4nO3de1yUZf7/8fcAcvKAGghqKBoqmgqKymInXUnU1s12c91s00htM6mMytRazQ6SlWSZ5nbQjv6ybdPcRExJOklZKB0Bz+IJxFRQVE4zvz/8NhsrGsgMN/fM6/l43I9Hc3HdM++7Wy/5zH3f12Wx2Ww2AQAAAABgQh5GBwAAAAAA4GJR1AIAAAAATIuiFgAAAABgWhS1AAAAAADToqgFAAAAAJgWRS0AAAAAwLQoagEAAAAApkVRCwAAAAAwLYra32Cz2VRSUiKbzWZ0FACAEzHeAwBgThS1v+HEiRMKCAjQiRMnjI4CAHAixnsAAMyJohYAAAAAYFoUtQAAAAAA06KoBQAAAACYFkUtAAAAAMC0KGoBAAAAAKZlqqL2008/1ciRI9WuXTtZLBatWrXqN/fJyMhQ37595ePjo/DwcL322mtOzwkAqB/GewAAUFumKmpLS0sVGRmpRYsW1ar/7t27dd1112nw4MHKzs7W1KlTNXHiRK1bt87JSQEA9cF4DwAAastiM+kq8xaLRStXrtSoUaPO2+fBBx/UmjVr9MMPP9jb/vrXv+r48eNKS0ur1eeUlJQoICBAxcXFatGiRX1jAwDqiPEeAABciJfRAZwpMzNTcXFx1dri4+M1derU8+5TVlamsrIy++uSkhJnxQPc3pJPdmrV1gNGx3BZIyPbacrgcKNjNAjGe9e04adCLUjfpsoqU37/jv/h7+2pOX/sqV6XBhgdBYCLcemitqCgQMHBwdXagoODVVJSotOnT8vPz++cfZKTkzVnzpyGigi4rZxDJZqXlitz3itiDgM6nTE6QoNhvHdNyzfn64cDfNngSj7IPkBRC8DhXLqovRgzZsxQUlKS/XVJSYlCQ0MNTAS4pifXni1oh0S0UcIVnYyO45JCAnyNjtCoMd43flXWs996/f3qzrqqS5DBaVAfK77Zp/98e1BWvsgE4AQuXdSGhISosLCwWlthYaFatGhR47f2kuTj4yMfH5+GiAe4rc+3H9En24rUxNOiWSN7qOMlTY2OBJNjvHdtXYOb68ougUbHQD1s2nnE6AgAXJipZj+uq9jYWKWnp1drW79+vWJjYw1KBMBqtWluao4k6eaYjhS0cAjGewAA3JepitqTJ08qOztb2dnZks4u4ZCdna38/HxJZ28lGzdunL3/HXfcoV27dmnatGnKzc3V4sWL9e677+ree+81Ij4ASauyD+inQyVq7uOlu4d0MToOGinGewAAUFumKmq/+eYb9enTR3369JEkJSUlqU+fPpo1a5Yk6dChQ/ZfeCSpU6dOWrNmjdavX6/IyEjNnz9fr7zyiuLj4w3JD7i7MxVVemZdniTpzsHhat3U2+BEaKwY7wEAQG2Z6pnaQYMG6ULL6r722ms17rN161YnpgJQW8u+2KODxWfULsBXCVeEGR0HjRjjPQAAqC1TXakFYF5HS8u1eOMOSdJ9Q7vJt4mnwYkANHZMlOt6bJxVAE5AUQugQSz8eLtOlFWqR9sWuqFPe6PjAAAAwEVQ1AJwur0/l+qtL/dKkmaO6C4PD4vBiQCYiYUhw/Q4hwCciaIWgNM9tS5PFVU2Xd01iLUmAQAA4FAUtQCcamv+Ma357pAsFmnG8Aij4wAAAMDFUNQCcBqbzaa5qTmSpBv7XqrubVsYnAgAAACuhqIWgNOs/6lQX+85Jt8mHkoa2tXoOAAAAHBBFLUAnKKiyqon03IlSROu7KS2AX4GJwJgNhdaqxjmxCkF4AwUtQCc4p2v92lXUalaN/XWHddcZnQcAAAAuCiKWgAOd7KsUs9t2CZJumdIFzX3bWJwIgBmxnIw5mcRJxGA81DUAnC4lz7ZqSMny9UpsKnGxnQwOg4AAABcGEUtAIcqLDmjlz/bLUl6cFg3NfFkmAEAAIDz8NsmAIdK+WibTldUKbpjK8VfHmJ0HAAAALg4iloADpNXcEL/ytonSZo5IkIWHoQDAACAk1HUAnCYJ9fmyGqThvcMUXTH1kbHAQAAgBugqAXgEJt2HNHGvCJ5eVg0bViE0XEAAADgJihqAdSb1WrT3LU5kqSbYzqoU2BTgxMBcCUsB2N+PI0CwJkoagHU2+pvD+qHAyVq5uOlu4d0MToOAAAA3AhFLYB6OVNRpafX5UmSJg+6TJc08zE4EQAAANwJRS2Aenkjc48OHD+tkBa+uu2KTkbHAQAAgJuhqAVw0Y6fKtcLH++QJN03tKv8vD0NTgQAAAB3Q1EL4KIt/HiHSs5UKiKkuf7U91Kj4wAAAMANUdQCuCj7jp7SG5l7JEkzRnSXpwdTWwJwLJvN6ARwNBsnFYATUNQCuChPrctTRZVNV3UJ1DVdg4yOAwAAADdFUQugzr7dd1z/+fagLBZp+vAIo+MAcHGscWp+nEIAzkRRC6BObDab5qbmSJJu6NNel7cLMDgRAAAA3BlFLYA6Sc85rK92H5W3l4fuH9rN6DgAAABwcxS1AGqtssqq5LVnr9LedkUntWvpZ3AiAAAAuDuKWgC19u43+7WzqFSt/JvozsGXGR0HAAAAoKgFUDulZZVKWb9NknT3kC5q4dvE4EQAXJ1NLP/iajijAJyBohZArbz06S4dOVmmjpf46+aYjkbHAQAAACRR1AKohcMlZ/TyZ7skSdPiI+TtxdCBhrFo0SKFhYXJ19dXMTEx2rx58wX7L1iwQN26dZOfn59CQ0N177336syZMw2UFsB5sS4TACfiN1MAv+nZDdt1qrxKUaEtNaJXiNFx4CZWrFihpKQkzZ49W1u2bFFkZKTi4+N1+PDhGvsvX75c06dP1+zZs5WTk6NXX31VK1as0MyZMxs4OQAAaEgUtQAuaHvhCa34Ol+S9NB13WXh23Y0kJSUFE2aNEkJCQnq0aOHlixZIn9/fy1durTG/ps2bdIVV1yhsWPHKiwsTEOHDtVNN930m1d3AQCAuVHUArigeWm5stqkoT2C1T+stdFx4CbKy8uVlZWluLg4e5uHh4fi4uKUmZlZ4z4DBw5UVlaWvYjdtWuXUlNTNWLEiBr7l5WVqaSkpNoGAADMx8voAAAary93/awNOYfl6WHRg8MjjI4DN3LkyBFVVVUpODi4WntwcLByc3Nr3Gfs2LE6cuSIrrzyStlsNlVWVuqOO+447+3HycnJmjNnjsOzAwCAhsWVWgA1slptmpuaI0m6aUCoLgtqZnAi4MIyMjI0d+5cLV68WFu2bNH777+vNWvW6LHHHqux/4wZM1RcXGzf9u3b18CJ8VtsrP/icjinAJyBK7UAavTh94f03f5iNfX21D1DuhodB24mMDBQnp6eKiwsrNZeWFiokJCaJyv7xz/+oVtuuUUTJ06UJPXq1UulpaW6/fbb9dBDD8nDo/r3uD4+PvLx8XHOAQAAgAbDlVoA5yirrNJTaWdv8bzjmssU1Jxf/NGwvL29FR0drfT0dHub1WpVenq6YmNja9zn1KlT5xSunp6ekiQbl4dMjQnqzI8zCMCZuFIL4BxvZu7V/mOnFdzCRxOv6mx0HLippKQkjR8/Xv369dOAAQO0YMEClZaWKiEhQZI0btw4tW/fXsnJyZKkkSNHKiUlRX369FFMTIx27Nihf/zjHxo5cqS9uAUAAK6HohZANcWnKrTw4x2SpKRru8rPm2IAxhgzZoyKioo0a9YsFRQUKCoqSmlpafbJo/Lz86tdmX344YdlsVj08MMP68CBAwoKCtLIkSP1xBNPGHUIAACgAVDUAqhmUcYOFZ+uULfg5roxOtToOHBziYmJSkxMrPFnGRkZ1V57eXlp9uzZmj17dgMkAwAAjYXpnqldtGiRwsLC5Ovrq5iYGPt6hDV57bXXZLFYqm2+vr4NmBYwl31HT+m1L/ZIkqaPiJCnB09BAQAAoHEzVVG7YsUKJSUlafbs2dqyZYsiIyMVHx+vw4cPn3efFi1a6NChQ/Zt7969DZgYMJf5H+WpvMqqgZddokFdg4yOA8DNMb+X67GJkwrA8UxV1KakpGjSpElKSEhQjx49tGTJEvn7+2vp0qXn3cdisSgkJMS+/fIsFoDqvt9frFXZByVJM0d0Z7ZRAAAAmIJpitry8nJlZWUpLi7O3ubh4aG4uDhlZmaed7+TJ0+qY8eOCg0N1fXXX68ff/zxgp9TVlamkpKSahvg6mw2m+am5kiSbujTXj3bBxicCAD+i6/YzI/vSQE4k2mK2iNHjqiqquqcK63BwcEqKCiocZ9u3bpp6dKl+uCDD/TWW2/JarVq4MCB2r9//3k/Jzk5WQEBAfYtNJSJcuD6MvKKlLnrZ3l7eei+oV2NjgMAAADUmmmK2osRGxurcePGKSoqStdcc43ef/99BQUF6Z///Od595kxY4aKi4vt2759+xowMdDwKqusSl579iptwsAwXdrK3+BEAAAAQO2ZZkmfwMBAeXp6qrCwsFp7YWGhQkJCavUeTZo0UZ8+fbRjx47z9vHx8ZGPj0+9sgJm8l7Wfm0rPKmW/k105+Bwo+MAAAAAdWKaK7Xe3t6Kjo5Wenq6vc1qtSo9PV2xsbG1eo+qqip9//33atu2rbNiAqZyqrxSKeu3SZISB4crwK+JwYkAAACAujHNlVpJSkpK0vjx49WvXz8NGDBACxYsUGlpqRISEiRJ48aNU/v27ZWcnCxJevTRR/W73/1O4eHhOn78uJ5++mnt3btXEydONPIwgEbjlc926/CJMoW29tMtsR2NjgMA1bD8i+thmSYAzmCqonbMmDEqKirSrFmzVFBQoKioKKWlpdknj8rPz5eHx38vPh87dkyTJk1SQUGBWrVqpejoaG3atEk9evQw6hCARqPoRJn++clOSdK0+Aj5eHkanAgAAACoO1MVtZKUmJioxMTEGn+WkZFR7fWzzz6rZ599tgFSAebzXPo2lZZXKfLSAP2hN7fkA2i8WA7G/CwszATAiUzzTC0Ax9lx+KT+3+azM3vPHNFdFn5jBAAAgElR1AJuaF5arqqsNsV1D1ZM50uMjgMAAABcNIpawM1s3n1U638qlKeHRdOHdzM6DgAAAFAvFLWAG7HZbJqbmiNJGtM/VOFtmhucCAAAAKgfilrAjaz5/pCy9x2Xv7enpsZ1MToOAAAAUG8UtYCbKK+06qm0PEnS7Vd3VpvmvgYnAoALY01T18MpBeAMFLWAm3jry73KP3pKQc19NOmqzkbHAQAAAByCohZwA8WnK/T8x9slSUnXdlVTH9MtUQ3AjbHGqfmxchwAZ6KoBdzAixk7dfxUhcLbNNPo6EuNjgMAAAA4DEUt4OIOHD+tpV/sliTNGB4hL0/+2gMAAMB18Nst4OLmr8tTeaVVv+vcWr+PaGN0HAAAAMChKGoBF/bDgWKtzD4gSZo5orssPNQEAAAAF0NRC7gom82mJ9fmymaT/hjZTr0vbWl0JACoE5Z/cT0s0wTAGShqARf1ybYifb7jiLw9PfRAfDej4wAAAABOQVELuKAq69mrtJI0LrajQlv7G5wIAC4eT06YH6cQgDNR1AIu6N9b9iu34IRa+Hop8ffhRscBAAAAnIaiFnAxp8urNP+jPEnSXb/vopb+3gYnAgAAAJyHohZwMUu/2K3CkjK1b+mnW2I7Gh0HAAAAcCqKWsCFHDlZphczdkqSpg3rJt8mngYnAgAAAJyLohZwIc+nb9fJskr1ah+gkb3bGR0HAOqH5V9cECcVgONR1AIuYlfRSS3/Kl+SNGNEhDw8mGsS5rdo0SKFhYXJ19dXMTEx2rx58wX7Hz9+XFOmTFHbtm3l4+Ojrl27KjU1tYHSAgAAI3gZHQCAYzyVlqdKq02/j2ijgZcFGh0HqLcVK1YoKSlJS5YsUUxMjBYsWKD4+Hjl5eWpTZs25/QvLy/XtddeqzZt2ui9995T+/bttXfvXrVs2bLhw8Oh+IrO/FiWCYAzUdQCLuCbPUeV9mOBPCzS9OERRscBHCIlJUWTJk1SQkKCJGnJkiVas2aNli5dqunTp5/Tf+nSpTp69Kg2bdqkJk2aSJLCwsIaMjIAADAAtx8DJmez2TQ3NUeS9Jd+oeoa3NzgRED9lZeXKysrS3FxcfY2Dw8PxcXFKTMzs8Z9Vq9erdjYWE2ZMkXBwcHq2bOn5s6dq6qqqhr7l5WVqaSkpNoGAADMh6IWMLm0Hwq0Jf+4/Jp4KunarkbHARziyJEjqqqqUnBwcLX24OBgFRQU1LjPrl279N5776mqqkqpqan6xz/+ofnz5+vxxx+vsX9ycrICAgLsW2hoqMOPAwAAOB9FLWBi5ZVWzUvLlSRNurqz2rTwNTgRYByr1ao2bdropZdeUnR0tMaMGaOHHnpIS5YsqbH/jBkzVFxcbN/27dvXwIkBAIAj8EwtYGLLv9qrPT+fUmAzb91+dWej4wAOExgYKE9PTxUWFlZrLywsVEhISI37tG3bVk2aNJGn53/XZ+7evbsKCgpUXl4ub2/vav19fHzk4+Pj+PBwGBvLv7gcG6cUgBNwpRYwqZIzFXr+4x2SpKlxXdXMh++o4Dq8vb0VHR2t9PR0e5vValV6erpiY2Nr3OeKK67Qjh07ZLVa7W3btm1T27ZtzyloAQCA66CoBUxqScZOHS0t12VBTfXX/jwLiMbhzJkz5/3ZoUOH6vReSUlJevnll/X6668rJydHkydPVmlpqX025HHjxmnGjBn2/pMnT9bRo0d1zz33aNu2bVqzZo3mzp2rKVOmXNzBoNFgORjzs3ASATgRRS1gQgePn9arn++WJE0f3l1envxVRuPQt29fZWdnn9P+73//W717967Te40ZM0bPPPOMZs2apaioKGVnZystLc0+eVR+fn61Qjk0NFTr1q3T119/rd69e+vuu+/WPffcU+PyPwAAwHVwvyJgQinrt6ms0qoBYa0V172N0XEAu0GDBul3v/ud5syZowcffFClpaWaMmWK3n33XT3xxBN1fr/ExEQlJibW+LOMjIxz2mJjY/Xll1/W+XMAAIB5UdQCJvPTwRL9e8t+SdLM67pzSxcalcWLF+u6667TxIkT9eGHH+rQoUNq1qyZNm/erJ49exodDwAAuCCKWsBkktfmyGaT/tC7raJCWxodBzjH8OHD9ac//UkvvviivLy89J///IeCFgAAOA0P4gEm8um2In22/YiaeFo0LT7C6DjAOXbu3KnY2Fh9+OGHWrdunaZNm6Y//vGPmjZtmioqKoyOB5Nh+RfXwzkF4AwUtYBJVFltSl6bK0m65Xdh6nCJv8GJgHNFRUWpU6dO+vbbb3Xttdfq8ccf18aNG/X+++9rwIABRscDAAAuiKIWMImVWw8o51CJmvt66a7fhxsdB6jR4sWL9c4776hly5b2toEDB2rr1q3q27evccFgcswdAAA4P4pawATOVFRp/kd5kqQpg8PVqqm3wYmAmt1yyy01tjdv3lyvvvpqA6cBAADugImiABNY+sVuHSo+o/Yt/XTrwDCj4wDVrF69ulb9LBaLRo4c6eQ0AAA0rEGDBikqKkoLFixokM8LCwvT1KlTNXXqVKd+zp49e9SpUydt3bpVUVFRF+x7yy23qHv37po5c2at3nv69OkqLS3VwoULHZCUohZo9H4+WaYXN+6UJN0f31W+TTwNTgRUN2rUqGqvLRaLbL+aDebXy05VVVU1VCwAAOpl0KBB+uSTT85pr6iokJdX4ymjLBaLVq5cec6/xw3l22+/VWpqql588cVa73P//ferc+fOuvfee9W5c+d6Z+D2Y6CRW/jxDp0oq9Tl7Vro+sj2RscBzmG1Wu3bRx99pKioKK1du1bHjx/X8ePHlZqaqr59+yotLc3oqAAA1MmkSZN06NChaltDFLRVVVWyWq1O/xxHWLhwoUaPHq1mzZrVep/AwEDFx8fXqRC+EIpaoBHbc6RUb325V5I0c0R3eXgwWQoat6lTp+q5555TfHy8WrRooRYtWig+Pl4pKSm6++67jY4HAHBhb7zxhi655BKVlZVVax81atR553z4Lf7+/goJCam21cRqtWratGlq3bq1QkJC9Mgjj1T7eUpKinr16qWmTZsqNDRUd955p06ePGn/+WuvvaaWLVtq9erV6tGjh3x8fJSfn6/Dhw9r5MiR8vPzU6dOnfT2229Xe9+wsDBJ0g033CCLxWJ/vXPnTl1//fUKDg5Ws2bN1L9/f23YsOGcfefOnavbbrtNzZs3V4cOHfTSSy+dc2y7du3S4MGD5e/vr8jISGVmZtp/VlVVpffee6/a40W5ubny9/fX8uXL7W3vvvuu/Pz89NNPP9nbRo4cqXfeeafG/591ZbqidtGiRQoLC5Ovr69iYmK0efPmC/b/17/+pYiICPn6+qpXr15KTU1toKRA/T21LleVVpsGdQvSFeGBRscBftPOnTurzXz8i4CAAO3Zs6fB88DcWNLU9dg4q6Zls9l0qrzSkM1WywWOR48eraqqqmpzPRw+fFhr1qzRbbfdps8++0zNmjW74Pa/RWNtvf7662ratKm++uorPfXUU3r00Ue1fv16+889PDz0/PPP68cff9Trr7+ujz/+WNOmTav2HqdOndK8efP0yiuv6Mcff1SbNm106623at++fdq4caPee+89LV68WIcPH7bv8/XXX0uSli1bpkOHDtlfnzx5UiNGjFB6erq2bt2qYcOGaeTIkcrPz6/2mfPnz1e/fv20detW3XnnnZo8ebLy8vKq9XnooYd0//33Kzs7W127dtVNN92kyspKSdJ3332n4uJi9evXz94/IiJCzzzzjO68807l5+dr//79uuOOOzRv3jz16NHD3m/AgAHav3+/Q34/aDw3g9fCihUrlJSUpCVLligmJkYLFixQfHy88vLy1KZNm3P6b9q0STfddJOSk5P1hz/8QcuXL9eoUaO0ZcsW9ezZ04AjAGpvS/4xpX5fIA+LNH14hNFxgFrp37+/kpKS9Oabbyo4OFiSVFhYqAceeIB1agHAxE5XVKnHrHWGfPZPj8bL3/u3yxY/Pz+NHTtWy5Yt0+jRoyVJb731ljp06KBBgwbpzJkzys7OvuB7/PJv1y8WL16sV155xf7673//u+bPn3/Ofr1799bs2bMlSV26dNELL7yg9PR0XXvttZJUbVKnsLAwPf7447rjjju0ePFie3tFRYUWL16syMhISdK2bdu0du1abd68Wf3795ckvfrqq+revbt9n6CgIElSy5Ytq11FjoyMtL+PJD322GNauXKlVq9ercTERHv7iBEjdOedd0qSHnzwQT377LPauHGjunXrZu9z//3367rrrpMkzZkzR5dffrl27NihiIgI7d27V56enufUYnfeeadSU1P1t7/9Td7e3urfv7/uuuuuan3atWsnSdq7d6/9CvPFqnNRO378eE2YMEFXX311vT74YqSkpGjSpElKSEiQJC1ZskRr1qzR0qVLNX369HP6P/fccxo2bJgeeOABSWdP5vr16/XCCy9oyZIlDZodqAubzaa5a3IkSTdGX6qIkBYGJwJqZ+nSpbrhhhvUoUMHhYaGSpL27dunLl26aNWqVcaGg2lZePLC9DiHaCiTJk1S//79deDAAbVv316vvfaabr31VlksFvn5+Sk8PLxO73fzzTfroYcesr+u6W4k6WxR+2tt27atdkV1w4YNSk5OVm5urkpKSlRZWakzZ87o1KlT8vf3lyR5e3tXe5+cnBx5eXkpOjra3hYREXHeDL928uRJPfLII1qzZo0OHTqkyspKnT59+pwrtb/+PIvFopCQkGq5/7dP27ZtJZ29Ah4REaHTp0/Lx8en2qSQv1i6dKm6du0qDw8P/fjjj+f08fPzk3T2CnV91bmoLS4uVlxcnDp27KiEhASNHz9e7ds7f/Ka8vJyZWVlacaMGfY2Dw8PxcXFVbuv+9cyMzOVlJRUrS0+Pv6Cv1iVlZVVuw+/pKSkfsGBi7Dux0J9s/eYfJt4KOnabr+9A9BIhIeH67vvvtP69euVm5srSerevbvi4uJq/AcPAGAOfk089dOj8YZ9dm316dNHkZGReuONNzR06FD9+OOPWrNmjSTps88+0/Dhwy+4/z//+U/dfPPN9tcBAQG1KoSbNGlS7bXFYrFP9LRnzx794Q9/0OTJk/XEE0+odevW+vzzzzVhwgSVl5fbi1o/Pz+H/Vt5//33a/369XrmmWcUHh4uPz8/3XjjjSovL6917pr6/JLvlz6BgYE6deqUysvL5e3tXW2/b7/9VqWlpfLw8NChQ4fsBfEvjh49Kum/V5vro85F7apVq1RUVKQ333xTr7/+umbPnq24uDhNmDBB119//Tn/YxzlyJEjqqqqOueWgODgYPsvTv+roKCgxv4FBQXn/Zzk5GTNmTOn/oGBi1RRZdVTaWf/TE+8srNCAnwNTgT8tnHjxun6669XfHy8mjVrpqFDh2ro0KFGxwIAOIjFYqnVLcCNwcSJE7VgwQIdOHBAcXFx9juH+vXrV+fbjx0hKytLVqtV8+fPl4fH2SmN3n333d/cLyIiQpWVlcrKyrLffpyXl6fjx49X69ekSZNzlsz74osvdOutt+qGG26QdPbKrTPmtvhl/dqffvqp2lq2R48e1a233qqHHnpIhw4d0s0336wtW7bYr85K0g8//KAmTZro8ssvr3eOi5ooKigoSElJSfr222/11VdfKTw8XLfccovatWune++9V9u3b693MKPMmDFDxcXF9m3fvn1GR4KbeWdzvnYdKdUlTb3192vqv24X0BDCw8M1d+5cBQUFafjw4XrxxRd14MABo2MBANzQ2LFjtX//fr388su67bbb7O2/3H58oa158+a/+f5DhgzRCy+8UOs84eHhqqio0MKFC7Vr1y69+eabtXoUslu3bho2bJj+/ve/66uvvlJWVpYmTpxYrTCUzj6jm56eroKCAh07dkzS2ed633//fWVnZ+vbb7/V2LFjnbJEUFBQkPr27avPP/+8Wvsdd9yh0NBQPfzww0pJSVFVVZXuv//+an0+++wzXXXVVeccz8Wo1+zHhw4d0vr167V+/Xp5enpqxIgR+v7779WjRw89++yz9Q73a4GBgfL09FRhYWG19sLCwvNOrR0SElKn/pLk4+NjX4bilw1oKCfOVGjBhrNfCk2N66Lmvs658wFwtFmzZikrK0vbt2/XyJEjtWrVKl122WWKjo7Wo48++pvfjAMA4CgBAQH685//rGbNmmnUqFEOf/+dO3fqyJEjte4fGRmplJQUzZs3Tz179tTbb7+t5OTkWu27bNkytWvXTtdcc43+9Kc/6fbbbz9nUqb58+dr/fr1Cg0NVZ8+fSSdnYuoVatWGjhwoEaOHKn4+Hj17du39gdZBxMnTqw2a/Qbb7yh1NRUvfnmm/Ly8lLTpk311ltv6eWXX9batWvt/d555x1NmjTJIRksttrOkf1/KioqtHr1ai1btkwfffSRevfurYkTJ2rs2LH2AnDlypW67bbb7N8UOEpMTIwGDBighQsXSjp7L3eHDh2UmJhY40RRY8aM0alTp/Sf//zH3jZw4ED17t271hNFlZSUKCAgQMXFxRS4cLr5H+Vp4cc71Dmwqdbde7WaeJpu1S3A7sSJE1q7dq0++OADrV27Vs2bN9fIkSM1efJkh9xq5GiM943PnxZ/oS35x/XPW6IVf/n5v5BG47c4Y4eeSsvT6OhL9fToyN/eAainIUOG6PLLL9fzzz9vdBSXd/r0aXXr1k0rVqxQbGxsrfZZu3at7rvvPn333Xfy8qr/be11foe2bdvKarXqpptu0ubNm6vdO/2LwYMH12pWrrpKSkrS+PHj1a9fPw0YMEALFixQaWmpfTbkcePGqX379vZvPu655x5dc801mj9/vq677jq98847+uabb2pcVBgwWkHxGb382S5J0rRhERS0ML3mzZvrL3/5i/7yl7+oqqpKGRkZWr16tTIzMxtlUQsAML9jx44pIyNDGRkZ1ZbLgfP4+fnpjTfeqNPV69LSUi1btswhBa10EUXts88+q9GjR8vX9/yT17Rs2VK7d++uV7CajBkzRkVFRZo1a5YKCgoUFRWltLQ0+wPd+fn59oevpbNXZZcvX66HH35YM2fOtC8pwRq1aIxS1ufpTIVV/Tq2Uvzljp+kAGgIs2fP1m233aaOHTtWa/f09NSQIUM0ZMgQg5LBzJg32/wsnEU0kD59+ujYsWOaN29etbVW4VyDBg2qU/8bb7zRoZ9f56L2lltucWiAukpMTKy2YPCvZWRknNM2evRo++LLQGOVW1Ci97L2S5JmjOjO0icwrQ8++EBPPPGErrnmGk2YMEF//vOf5ePjY3QsAICbcMYMv2j8uL8RaASeXJsrq00a0StE0R1bGR0HuGjZ2dn6+uuvdfnll+uee+5RSEiIJk+erK+//troaAAAwEVR1AIG+2LHEWXkFcnLw6Jp8RFGxwHqrU+fPnr++ed18OBBvfrqq9q/f7+uuOIK9e7dW88995yKi4uNjggAAFwIRS1gIKvVprmpOZKkv/2uo8ICmxqcCHAcm82miooKlZeXy2azqVWrVnrhhRcUGhqqFStWGB0PAAC4CIpawEAffHtAPx4sUXMfL909pIvRcQCHyMrKUmJiotq2bat7771Xffr0UU5Ojj755BNt375dTzzxhO6++26jY8IE6rTmIEyBcwrAGShqAYOcqajSM+u2SZImD75MrZt6G5wIqL9evXopJiZGu3fv1quvvqp9+/bpySefVHh4uL3PTTfdpKKiIgNTAgAAV+KYhYEA1Nnrm/bowPHTahvgq9uu6GR0HMAh/vKXv+i2225T+/btz9snMDBQVqu1AVPB7JgR3vw4hQCciaIWMMCx0nK9sHGHJOm+od3k28TT4ETAxUtKSqr2ev78+eftm5KS4uw4AADAzVDUAgZY+PEOnThTqe5tW+iGPue/ogWYwdatW6u93rJliyorK+2L3m/btk2enp6Kjo42Ih4AAHBxFLVAA8v/+ZTe/HKPJGnG8Ah5enBPFsxt48aN9v9OSUlR8+bN9frrr6tVq7NrLh87dkwJCQm66qqrjIoIAABcGBNFAQ3sqXW5qqiy6aougbq6a5DRcQCHmj9/vpKTk+0FrSS1atVKjz/++AVvSwYAALhYFLVAA8red1wffndIFos0Y3h3o+MADldSUlLjzMZFRUU6ceJEnd9v0aJFCgsLk6+vr2JiYrR58+Za7ffOO+/IYrFo1KhRdf5MNB421n9xOZxTAM5AUQs0EJvNprmpOZKkP/W5VD3atTA4EeB4N9xwgxISEvT+++9r//792r9/v/79739rwoQJ+tOf/lSn91qxYoWSkpI0e/ZsbdmyRZGRkYqPj9fhw4cvuN+ePXt0//33c7szAABugqIWaCAbcg5r8+6j8vHy0P3xXY2OAzjFkiVLNHz4cI0dO1YdO3ZUx44dNXbsWA0bNkyLFy+u03ulpKRo0qRJSkhIUI8ePbRkyRL5+/tr6dKl592nqqpKN998s+bMmaPOnTvX93DQSDDzgPlxDgE4E0Ut0AAqq6x6cu3Zq7QTruyktgF+BicCnMPf31+LFy/Wzz//rK1bt2rr1q06evSoFi9erKZNm9b6fcrLy5WVlaW4uDh7m4eHh+Li4pSZmXne/R599FG1adNGEyZM+M3PKCsrU0lJSbUNAACYD7MfAw1gxTf7tLOoVK2beuuOQZcZHQdwuqZNm6p3794Xvf+RI0dUVVWl4ODgau3BwcHKzc2tcZ/PP/9cr776qrKzs2v1GcnJyZozZ85FZwQAAI0DV2oBJztZVqln12+XJN39+3C18G1icCLA9Zw4cUK33HKLXn75ZQUGBtZqnxkzZqi4uNi+7du3z8kpAQCAM3ClFnCylz7dpSMnyxR2ib/GxnQ0Og5gCoGBgfL09FRhYWG19sLCQoWEhJzTf+fOndqzZ49Gjhxpb7NarZIkLy8v5eXl6bLLqt8l4ePjIx8fHyekBwAADYkrtYATHS45o5c/3SVJenBYhLy9+CsH1Ia3t7eio6OVnp5ub7NarUpPT1dsbOw5/SMiIvT9998rOzvbvv3xj3/U4MGDlZ2drdDQ0IaMDwdh9RfXY+OsAnACrtQCTvTshm06XVGlvh1aaljPc68uATi/pKQkjR8/Xv369dOAAQO0YMEClZaWKiEhQZI0btw4tW/fXsnJyfL19VXPnj2r7d+yZUtJOqcdAAC4FopawEm2FZ7Qiq/PPqP30HXdZbGwoAFQF2PGjFFRUZFmzZqlgoICRUVFKS0tzT55VH5+vjw8uPvBHTB8mh/nEIAzUdQCTjJvba6sNmnY5SGK7tja6DiAKSUmJioxMbHGn2VkZFxw39dee83xgQAAQKPDV9yAE2Tu/FnpuYfl5WHRtGHdjI4DAAAAuCyKWsDBrFab5qbmSJLGxnRQ56BmBicCAAAAXBdFLeBg//nuoL4/UKxmPl66e0gXo+MAAAAALo2iFnCgssoqPb0uT5J0xzWdFdiMNTABAAAAZ6KoBRzojU17tf/YaYW08NWEKzsbHQcAzM3GmqYuh1MKwAkoagEHOX6qXAs/3i5JShraVX7engYnAgAAAFwfRS3gIIs27lDJmUpFhDTXn/teanQcAHAZrHFqfhZxEgE4D0Ut4AD7jp7S65v2SpKmD4+Qpwf/eAMAAAANgaIWcICn1+WpvMqqK8MDdU3XIKPjAAAAAG6Dohaop+/2H9fqbw/KYjl7ldbCfXIAAABAg6GoBerBZrNpbmqOJOmGqPbq2T7A4EQAAACAe6GoBerh49zD+nLXUXl7eei++G5GxwEAl8LqL66HcwrAGShqgYtUWWXVk2tzJUkJV4SpfUs/gxMBAAAA7oeiFrhI/8rar+2HT6qVfxPdOSjc6DgA4LJYDsb8mG4CgDNR1AIXobSsUinrt0mS7vp9FwX4NTE4EQAAAOCeKGqBi/DKZ7tVdKJMHVr762+/62h0HAAAAMBtUdQCdXT4xBn989OdkqRpw7rJ24u/RgAAAIBR+G0cqKMFG7brVHmVIkNb6rpebY2OAwAAALg1ilqgDnYcPqEVX++TJD00orsszHwBAE5jY/0Xl2PjpAJwAopaoA6eXJunKqtN1/YI1oBOrY2OAwAAALg9ilqglr7a9bM25BTK08OiB4dFGB0HANwHN8UAAC7ANEXt0aNHdfPNN6tFixZq2bKlJkyYoJMnT15wn0GDBslisVTb7rjjjgZKDFdis9k0NzVHkvTX/qEKb9PM4EQAAAAAJMnL6AC1dfPNN+vQoUNav369KioqlJCQoNtvv13Lly+/4H6TJk3So48+an/t7+/v7KhwQR9+d0jf7i9WU29PTY3ranQcAAAAAP/HFEVtTk6O0tLS9PXXX6tfv36SpIULF2rEiBF65pln1K5du/Pu6+/vr5CQkIaKChdUVlmlp9blSpL+fs1lCmruY3AiAAAAAL8wxe3HmZmZatmypb2glaS4uDh5eHjoq6++uuC+b7/9tgIDA9WzZ0/NmDFDp06dumD/srIylZSUVNvg3t76Ml/7jp5Wm+Y+mnhVJ6PjAAAAAPgVU1ypLSgoUJs2baq1eXl5qXXr1iooKDjvfmPHjlXHjh3Vrl07fffdd3rwwQeVl5en999//7z7JCcna86cOQ7LDnMrPl2hhR9vlyQlXdtV/t6m+CsDAC7BJpZ/cTWcUQDOYOhv6NOnT9e8efMu2CcnJ+ei3//222+3/3evXr3Utm1bDRkyRDt37tRll11W4z4zZsxQUlKS/XVJSYlCQ0MvOgPMbfHGHTp+qkJdg5vpxuhLjY4DAAAA4H8YWtTed999uvXWWy/Yp3PnzgoJCdHhw4ertVdWVuro0aN1el42JiZGkrRjx47zFrU+Pj7y8eGZSUj7j53Ssk17JEnTh0fIy9MUd+sDgMthRR/zs1g4iwCcx9CiNigoSEFBQb/ZLzY2VsePH1dWVpaio6MlSR9//LGsVqu9UK2N7OxsSVLbtm0vKi/cy/yPtqm80qrYzpdocLc2v70DAAAAgAZniktP3bt317BhwzRp0iRt3rxZX3zxhRITE/XXv/7VPvPxgQMHFBERoc2bN0uSdu7cqccee0xZWVnas2ePVq9erXHjxunqq69W7969jTwcmMAPB4q1cusBSdLMEd35hhkAAABopExR1EpnZzGOiIjQkCFDNGLECF155ZV66aWX7D+vqKhQXl6efXZjb29vbdiwQUOHDlVERITuu+8+/fnPf9Z//vMfow4BJmGz2TQ39eyz3NdHtVOvSwMMTgQAAADgfEwzlWvr1q21fPny8/48LCxMNtt/59QLDQ3VJ5980hDR4GIythVp086f5e3pofuHdjM6DuDWFi1apKeffloFBQWKjIzUwoULNWDAgBr7vvzyy3rjjTf0ww8/SJKio6M1d+7c8/YHAACuwTRXaoGGUGW16cnUXEnSrVeEKbS1v8GJAPe1YsUKJSUlafbs2dqyZYsiIyMVHx9/zsSBv8jIyNBNN92kjRs3KjMzU6GhoRo6dKgOHDjQwMnhKDbWf3E5nFMAzkBRC/zKv7P2K6/whAL8mmjKoHCj4wBuLSUlRZMmTVJCQoJ69OihJUuWyN/fX0uXLq2x/9tvv60777xTUVFRioiI0CuvvCKr1ar09PQGTg4AABoSRS3wf06VV2r++jxJ0l2/D1eAfxODEwHuq7y8XFlZWYqLi7O3eXh4KC4uTpmZmbV6j1OnTqmiokKtW7eu8edlZWUqKSmptqFxYrI+8+MMAnAmilrg/7z62W4VlpQptLWfbontaHQcwK0dOXJEVVVVCg4OrtYeHBysgoKCWr3Hgw8+qHbt2lUrjH8tOTlZAQEB9i00NLTeuQEAQMOjqAUkHTlZpiWf7JQkPRAfIR8vT4MTAaiPJ598Uu+8845WrlwpX1/fGvvMmDFDxcXF9m3fvn0NnBIAADiCaWY/BpzpuQ3bVVpepd6XBugPvdoaHQdwe4GBgfL09FRhYWG19sLCQoWEhFxw32eeeUZPPvmkNmzYcMF1yX18fOTj4+OQvAAAwDhcqYXb21l0Uss350uSZo7oLg8PnvwBjObt7a3o6Ohqkzz9MulTbGzsefd76qmn9NhjjyktLU39+vVriKgAAMBgXKmF23sqLVdVVpviurfR7zpfYnQcAP8nKSlJ48ePV79+/TRgwAAtWLBApaWlSkhIkCSNGzdO7du3V3JysiRp3rx5mjVrlpYvX66wsDD7s7fNmjVTs2bNDDsOAADgXBS1cGtf7zmqdT8WysMiPTgswug4AH5lzJgxKioq0qxZs1RQUKCoqCilpaXZJ4/Kz8+Xh8d/bzh68cUXVV5erhtvvLHa+8yePVuPPPJIQ0aHg7CmqevhlAJwBopauC2bzaa5qTmSpDH9O6hLcHODEwH4X4mJiUpMTKzxZxkZGdVe79mzx/mBAABAo8MztXBba38o0Nb84/L39tS913YxOg4A4DyY6cD8WGoYgDNR1MItlVdaNS8tV5I06arOatO85iU/AAAAADRuFLVwS29/tVd7fz6loOY+uv3qzkbHAQAAAHCRKGrhdkrOVOj59O2SpHvjuqqpD4+WAwAAAGZFUQu382LGTh07VaHwNs30l36XGh0HAAAAQD1Q1MKtHDx+Wks/3y1Jmj4sQl6e/BUAgMaK5V9cj411mgA4Ab/Rw63M/2ibyiqtiunUWkO6tzE6DgAAAIB6oqiF2/jpYIne37pfkjRzRHdZWF8AAEyB4dr8OIUAnImiFm4jeW2ObDZpZGQ7RYa2NDoOAAAAAAegqIVb+HRbkT7bfkRNPC2aFt/N6DgAAAAAHISiFi6vymrT3NQcSdK42DCFtvY3OBEAAAAAR6Gohct7f8t+5RacUAtfL931+3Cj4wAAAABwIIpauLQzFVWa/9E2SVLi78PV0t/b4EQAgNpi+RfXwxkF4AwUtXBpr36+WwUlZ9S+pZ/GxYYZHQcAAACAg1HUwmX9fLJML2bslCQ9EN9Nvk08DU4EALgYFhaEMT2W0QPgTBS1cFkLP96hk2WV6tm+hf4Y2c7oOAAAAACcgKIWLmn3kVK99eVeSdLM4d3l4cE3xAAAAIAroqiFS3oqLVeVVpsGdwvSwPBAo+MAAAAAcBKKWricrL3HtPaHAnlYpOnDuxsdBwAAAIATUdTCpdhsNs1NzZEkjY4OVbeQ5gYnAgAAdqzpA8AJKGrhUtb9WKCsvcfk18RTSUO7Gh0HAAAAgJNR1MJlVFRZNS8tT5I06apOCm7ha3AiAIAjsBqM+XEOATgTRS1cxv/bnK/dR0oV2Mxbt19zmdFxAAAAADQAilq4hBNnKvTchu2SpHviuqqZj5fBiQAAAAA0BIpauIQln+zUz6Xl6hzUVH/tH2p0HAAAAAANhKIWpneo+LRe+Wy3JOnBYRFq4skfawAAAMBd8Ns/TC/lo20qq7Sqf1grDe0RbHQcAICD2Fj+xeXYWNMHgBNQ1MLUcg6V6L0t+yVJM0d0l4XpFQEAAAC3QlELU3tyba5sNum6Xm3Vp0Mro+MAAAAAaGAUtTCtz7cf0SfbitTE06Jpw7oZHQcA4CTcg2N+nEMAzkRRC1OyWm2am5ojSfrb7zqq4yVNDU4EAAAAwAimKWqfeOIJDRw4UP7+/mrZsmWt9rHZbJo1a5batm0rPz8/xcXFafv27c4NigaxKvuAfjpUouY+Xrrr912MjgPASRYtWqSwsDD5+voqJiZGmzdvvmD/f/3rX4qIiJCvr6969eql1NTUBkoKAACMYpqitry8XKNHj9bkyZNrvc9TTz2l559/XkuWLNFXX32lpk2bKj4+XmfOnHFiUjjbmYoqPbMuT5J05+BwtW7qbXAiAM6wYsUKJSUlafbs2dqyZYsiIyMVHx+vw4cP19h/06ZNuummmzRhwgRt3bpVo0aN0qhRo/TDDz80cHIAANCQTFPUzpkzR/fee6969epVq/42m00LFizQww8/rOuvv169e/fWG2+8oYMHD2rVqlXODQunWvbFHh0sPqN2Ab5KuCLM6DgAnCQlJUWTJk1SQkKCevTooSVLlsjf319Lly6tsf9zzz2nYcOG6YEHHlD37t312GOPqW/fvnrhhRcaODkAAGhIXkYHcJbdu3eroKBAcXFx9raAgADFxMQoMzNTf/3rX2vcr6ysTGVlZfbXJSUlF/X5M97/Tlv2Hq/WVtPabDWtwVfTCm62GjrWuNLbeZZ/q8971pyxlsdSy+XoaspyvjxHTp49P/fHd5NvE8/afQAAUykvL1dWVpZmzJhhb/Pw8FBcXJwyMzNr3CczM1NJSUnV2uLj48/7RaajxvusvUf10EquBjvD7p9LjY4AB/ts2xENW/Cp0TFc0qopV/B7EdyWyxa1BQUFkqTg4OBq7cHBwfaf1SQ5OVlz5syp9+fvO3paeYUn6v0+qFlkaEuNimpvdAwATnLkyBFVVVXVOIbn5ubWuE9BQUGdxnxHjfelZVXKLWC8d6Z2Lf2MjoB6urS1vyTpRFklf18AOJyhRe306dM1b968C/bJyclRREREAyWSZsyYUe2b/pKSEoWGhtb5faYPj1Dx6Ypz2muc0t7yvy/P7WWpYcea3stSQ8fa7/vbveqVo1afee7x19QnvE0zeXiwQACAi+eo8b5X+wC9NSHGkdHwK+1b+SkskBnuzW5Q1yCl3n2VjpaWGx3FZTXxNM1ThYDDGVrU3nfffbr11lsv2Kdz584X9d4hISGSpMLCQrVt29beXlhYqKioqPPu5+PjIx8fn4v6zF/r2T6g3u8BAO4qMDBQnp6eKiwsrNZeWFhoH9//V0hISJ36O2q8b9XUW1d2Caz3+wCuzGKxqEe7FkbHAOCiDP1KJygoSBERERfcvL0vbmbbTp06KSQkROnp6fa2kpISffXVV4qNjXXUIQAAnMDb21vR0dHVxnCr1ar09PTzjuGxsbHV+kvS+vXrGfMBAHBxprlPIT8/X9nZ2crPz1dVVZWys7OVnZ2tkydP2vtERERo5cqVks5+Izh16lQ9/vjjWr16tb7//nuNGzdO7dq106hRoww6CgBAbSUlJenll1/W66+/rpycHE2ePFmlpaVKSEiQJI0bN67aRFL33HOP0tLSNH/+fOXm5uqRRx7RN998o8TERKMOAQAANADTTBQ1a9Ysvf766/bXffr0kSRt3LhRgwYNkiTl5eWpuLjY3mfatGkqLS3V7bffruPHj+vKK69UWlqafH19GzQ7AKDuxowZo6KiIs2aNUsFBQWKiopSWlqafTKo/Px8eXj897vZgQMHavny5Xr44Yc1c+ZMdenSRatWrVLPnj2NOgQAANAALLbzraUCSWdvWQ4ICFBxcbFatOBZEABwVYz3AACYk2luPwYAAAAA4H9R1AIAAAAATIuiFgAAAABgWqaZKMoovzxyXFJSYnASAGg4zZs3l8ViMTpGg2K8B+CO3HG8h+uhqP0NJ06ckCSFhoYanAQAGo47TpbEeA/AHbnjeA/Xw+zHv8FqtergwYOm+BarpKREoaGh2rdvn0sPTu5wnO5wjBLH2ZiZYcxzNDON95I5/1zVlTsco8RxuhIzHqNZxjzgQrhS+xs8PDx06aWXGh2jTlq0aGGagbQ+3OE43eEYJY4TjYMZx3vJPf5cucMxShynK3GHYwQaEyaKAgAAAACYFkUtAAAAAMC0KGpdiI+Pj2bPni0fHx+joziVOxynOxyjxHEC9eEOf67c4RgljtOVuMMxAo0RE0UBAAAAAEyLK7UAAAAAANOiqAUAAAAAmBZFLQAAAADAtChqAQAAAACmRVHrgvbs2aMJEyaoU6dO8vPz02WXXabZs2ervLzc6GgO98QTT2jgwIHy9/dXy5YtjY7jMIsWLVJYWJh8fX0VExOjzZs3Gx3JoT799FONHDlS7dq1k8Vi0apVq4yO5HDJycnq37+/mjdvrjZt2mjUqFHKy8szOhZckLuM+Yz35sWYD8DZKGpdUG5urqxWq/75z3/qxx9/1LPPPqslS5Zo5syZRkdzuPLyco0ePVqTJ082OorDrFixQklJSZo9e7a2bNmiyMhIxcfH6/Dhw0ZHc5jS0lJFRkZq0aJFRkdxmk8++URTpkzRl19+qfXr16uiokJDhw5VaWmp0dHgYtxlzGe8Ny/GfADOxpI+buLpp5/Wiy++qF27dhkdxSlee+01TZ06VcePHzc6Sr3FxMSof//+euGFFyRJVqtVoaGhuuuuuzR9+nSD0zmexWLRypUrNWrUKKOjOFVRUZHatGmjTz75RFdffbXRceDiXHnMZ7w3N8Z8AM7AlVo3UVxcrNatWxsdA7+hvLxcWVlZiouLs7d5eHgoLi5OmZmZBiZDfRUXF0sSfw/RIBjzGz/Ge9fGmA80LIpaN7Bjxw4tXLhQf//7342Ogt9w5MgRVVVVKTg4uFp7cHCwCgoKDEqF+rJarZo6daquuOIK9ezZ0+g4cHGM+ebAeO+6GPOBhkdRayLTp0+XxWK54Jabm1ttnwMHDmjYsGEaPXq0Jk2aZFDyurmY4wQasylTpuiHH37QO++8Y3QUmIg7jPmM93BFjPlAw/MyOgBq77777tOtt956wT6dO3e2//fBgwc1ePBgDRw4UC+99JKT0zlOXY/TlQQGBsrT01OFhYXV2gsLCxUSEmJQKtRHYmKiPvzwQ3366ae69NJLjY4DE3GHMZ/xnvHe1TDmA8agqDWRoKAgBQUF1arvgQMHNHjwYEVHR2vZsmXy8DDPRfm6HKer8fb2VnR0tNLT0+2TaFitVqWnpysxMdHYcKgTm82mu+66SytXrlRGRoY6depkdCSYjDuM+Yz3jPeugjEfMBZFrQs6cOCABg0apI4dO+qZZ55RUVGR/Weu9u1vfn6+jh49qvz8fFVVVSk7O1uSFB4ermbNmhkb7iIlJSVp/Pjx6tevnwYMGKAFCxaotLRUCQkJRkdzmJMnT2rHjh3217t371Z2drZat26tDh06GJjMcaZMmaLly5frgw8+UPPmze3PyAUEBMjPz8/gdHAl7jLmM96bF2M+Yz7gdDa4nGXLltkk1bi5mvHjx9d4nBs3bjQ6Wr0sXLjQ1qFDB5u3t7dtwIABti+//NLoSA61cePGGs/b+PHjjY7mMOf7O7hs2TKjo8HFuMuYz3hvXoz5AJyNdWoBAAAAAKZljoduAAAAAACoAUUtAAAAAMC0KGoBAAAAAKZFUQsAAAAAMC2KWgAAAACAaVHUAgAAAABMi6IWAAAAAGBaFLUAAAAAANOiqAUAAAAAmBZFLQAAAADAtChqgUakqKhIISEhmjt3rr1t06ZN8vb2Vnp6uoHJAACOxHgPAI5jsdlsNqNDAPiv1NRUjRo1Sps2bVK3bt0UFRWl66+/XikpKUZHAwA4EOM9ADgGRS3QCE2ZMkUbNmxQv3799P333+vrr7+Wj4+P0bEAAA7GeA8A9UdRCzRCp0+fVs+ePbVv3z5lZWWpV69eRkcCADgB4z0A1B/P1AKN0M6dO3Xw4EFZrVbt2bPH6DgAACdhvAeA+uNKLdDIlJeXa8CAAYqKilK3bt20YMECff/992rTpo3R0QAADsR4DwCOQVELNDIPPPCA3nvvPX377bdq1qyZrrnmGgUEBOjDDz80OhoAwIEY7wHAMbj9GGhEMjIytGDBAr355ptq0aKFPDw89Oabb+qzzz7Tiy++aHQ8AICDMN4DgONwpRYAAAAAYFpcqQUAAAAAmBZFLQAAAADAtChqAQAAAACmRVELAAAAADAtiloAAAAAgGlR1AIAAAAATIuiFgAAAABgWhS1AAAAAADToqgFAAAAAJgWRS0AAAAAwLQoagEAAAAApkVRCwAAAAAwrf8PRNtVLfwH19AAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 900x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "opplot({\"F.hardtanh(x)\": F.hardtanh})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's the _unscaled_ op, but we want to find the equivalent _scaled_ op. To do this:\n",
    "\n",
    " - Replace thresholds `a` and `b` with a `mult` shape parameter, since the output should have zero-mean.\n",
    " - Insert scaling factors separately into the forward and backward passes, to achieve unit scale.\n",
    "\n",
    "In code, that's\n",
    "\n",
    "```python\n",
    "def hardtanh(x: Tensor, mult: float = 1) -> Tensor:\n",
    "    y_scale, grad_scale = ...  # ???\n",
    "    x = scale_bwd(x, grad_scale)\n",
    "    y = F.hardtanh(x, -1/mult, 1/mult)\n",
    "    return scale_fwd(y, y_scale)\n",
    "```\n",
    "\n",
    "which relies on the utilities `scale_fwd` and `scale_bwd` from `unit_scaling.scale`. These apply a muliplicative scaling factor in either the forward or backwards pass (compare with the straight-forward `x * scale` which applies the same scale in both foward and backward passes).\n",
    "\n",
    "**The remaining problem is how to choose `y_scale` and `grad_scale`, so that `y.std` ≈ 1 when `x.std` = 1 and also `x.grad.std` ≈ 1 when `y.grad.std` = 1.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Empirical scaling\n",
    "\n",
    "The simplest way to choose these factors is to feed inputs from an appropriate distribution (typically unit Gaussian), into both forward and backward passes, and measure how the scale changes. Then set scaling factors to counteract this.\n",
    "\n",
    "Let's try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.nn.functional.hardtanh\n",
      "         x.std = 0.999\n",
      "         y.std = 0.718\n",
      "    grad_x.std = 0.828\n"
     ]
    }
   ],
   "source": [
    "def check_scaling(fn: Callable[[Tensor], Tensor], **kwargs: Any) -> None:\n",
    "    x = torch.randn(int(1e6)).requires_grad_()\n",
    "    y = fn(x, **kwargs)\n",
    "    y.backward(torch.randn_like(y))\n",
    "\n",
    "    name = f\"{fn.__module__}.{fn.__name__}\".replace(\"__main__.\", \"\")\n",
    "    print(name + (f\" {kwargs}\" if kwargs else \"\"))\n",
    "    for k, v in {\"x\": x, \"y\": y, \"grad_x\": x.grad}.items():\n",
    "        print(f\"{k:>10}.std = {v.std(correction=0).item():.3f}\")\n",
    "\n",
    "check_scaling(F.hardtanh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unscaled op therefore shrinks the scale in both forward and backward passes.\n",
    "\n",
    "We can pluck these scales, and use them to set `y_scale = 1 / empirical_y_std` and `grad_scale = 1 / empirical_grad_x_std`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hardtanh_scaled_empirical\n",
      "         x.std = 1.000\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 1.001\n"
     ]
    }
   ],
   "source": [
    "def hardtanh_scaled_empirical(x: Tensor) -> Tensor:\n",
    "    y_scale = 1 / 0.718\n",
    "    grad_scale = 1 / 0.826\n",
    "    x = scale_bwd(x, grad_scale)\n",
    "    y = F.hardtanh(x)\n",
    "    return scale_fwd(y, y_scale)\n",
    "\n",
    "check_scaling(hardtanh_scaled_empirical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success! We've unit-scaled hardtanh!\n",
    "\n",
    "However, we still haven't dealt with `mult`, as these static factors do not generalise to different clipping thresholds. We could do this by fitting a curve `f` to `y.std = f(mult)`, but we'll instead use this as an opportunity to highlight an alternative approach based on statistical analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical scaling\n",
    "\n",
    "Our second approach will be to try to compute the output distribution and its standard deviation, given an input distribution.\n",
    "\n",
    "First, let's eyeball the forward-pass distribution for `y = F.hardtanh(x)` when $x \\sim \\mathcal{N}(0, 1)$. Note that this plot shows RMS, which is equal to standard deviation when zero-mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAGaCAYAAADkVo87AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxqklEQVR4nO3deXRUZZ4+8KeqUkvWCiEbCUkIiyIikUTD0rYGjQQn7cCI2Nq/mRPA5ogGu+3YeJRRc5wzmBEXmA4cYBQS+iAt2jXBbh2RpQFRBLQQuwMEQQlQCVnIUpVUUkuq3t8fMVeKm4QslVQqeT7n5Gjdeu+tb5TLU+/93kUhhBAgIiK6htLXBRAR0dDDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw3AgIiIZhgMREckwHMir6urqEB0djfLy8h6v8+ijj+LNN98cuKKIqNcYDsPQ4sWLoVAooFAooFarkZycjOeeew42m002Zvny5bL1c3NzoVAosHjxYmlZbW0tnnzySSQmJkKr1SI2NhZZWVn44osvPNZdvXo15s+fj3HjxvW43hdffBGrV6+G2Wzu9e/anc8++wwPPvgg4uLioFAosGvXrh6tt2HDBowbNw46nQ4zZszA8ePH+zSGyJ8xHIapefPm4cqVK/jhhx+wdu1abN68Gfn5+R5jEhIS8N5776G1tVVaZrPZsGPHDiQmJnqMXbhwIb755hts27YN3333Hf7yl78gIyMDdXV10piWlhZs2bIFjz/+eK9qnTp1KiZMmIDt27f34TftmtVqRUpKCjZs2NDjdXbu3Im8vDzk5+fjxIkTSElJQVZWFmpqano1hsjvCRp2cnJyxPz58z2WPfTQQ2L69OmyMVOnThXbt2+Xlr/77rti2rRpYv78+SInJ0cIIURDQ4MAIA4ePNjt537wwQciKirKY9mOHTuETqcTlZWV0rLFixeL2267TTQ2NkrLXnnlFXHXXXf19lftMQCipKTkhuPS09NFbm6u9Nrlcom4uDhRUFDQqzFE/o4zhxGgtLQUR44cgUajkb23dOlSFBUVSa+3bt2KJUuWeIwJCQlBSEgIdu3aBbvd3uXnHD58GGlpaR7LHn30Udx000149dVXAQD5+fnYt28fPvnkE+j1emlceno6jh8/3un2X331VamGrn4uXbrUs/8Y3XA4HDAajcjMzJSWKZVKZGZm4ssvv+zxGKLhIMDXBdDA+OijjxASEoK2tjbY7XYolUqsX79eNu5f//Vf8cILL+DixYsAgC+++ALvvfceDh48KI0JCAhAcXExli1bhk2bNiE1NRX33HMPHn30UUybNk0ad/HiRcTFxXlsX6FQYPXq1Xj44YcRGxuLwsJCHD58GPHx8R7j4uLi4HA4UFVVhaSkJI/3li9fjkceeaTb3/f6z+2Lq1evwuVyISYmxmN5TEwMysrKejyGaDhgOAxTc+bMwcaNG2G1WrF27VoEBARg4cKFsnFRUVHIzs5GcXExhBDIzs5GZGSkbNzChQuRnZ2Nw4cP4+jRo/jkk0+wZs0avPPOO1LjurW1FTqdTrbuL37xC0yZMgX/8R//gT179uDWW2+VjQkMDATQ3re4XkREBCIiInr7n4CI+oGHlYap4OBgTJw4ESkpKdi6dSuOHTuGLVu2dDp26dKlKC4uxrZt27B06dIut6nT6XD//ffjpZdewpEjR7B48WKPJndkZCQaGhpk6+3evRtlZWWdfuPuUF9fD6A9rK43WIeVIiMjoVKpUF1d7bG8uroasbGxPR5DNBwwHEYApVKJVatW4cUXX/Q4M6nDvHnz4HA44HQ6kZWV1ePtTpkyBVarVXo9ffp0nD592mPMiRMn8Mgjj2DLli2477778NJLL3W6rdLSUowdO7bTWcvy5ctx8uTJbn+8cVhJo9EgLS0N+/fvl5a53W7s378fs2bN6vEYouGAh5VGiEWLFmHlypXYsGEDfv/733u8p1KpcObMGenfr1dXV4dFixZh6dKlmDZtGkJDQ/H1119jzZo1mD9/vjQuKysLL7zwAhoaGjBq1CiUl5cjOzsbq1atwmOPPYbx48dj1qxZOHHiBFJTUz0+4/Dhw5g7d26ntff1sFJzczPOnz8vvb5w4QJOnjyJiIgI6VTd9evXo6SkRPrLPi8vDzk5ObjjjjuQnp6OdevWwWq1ejTpezKGyO/5+nQp8r7OTmUVQoiCggIRFRUlmpubuxzT4dpTWW02m3j++edFamqq0Ov1IigoSNx8883ixRdfFC0tLR7rpaeni02bNom6ujpx8803iyeeeMLj/X/6p38SWVlZHstaW1uFXq8XX375ZZ9+364cOHBAAJD9dPxeQgiRn58vkpKSPNYrLCwUiYmJQqPRiPT0dHH06FHZtnsyhsifKYQQwqfpRMPKxx9/jJUrV6K0tBRKZc+OWm7cuBElJSXYs2fPAFdHRD3Fw0rkVdnZ2Th37hwqKiqQkJDQo3XUajUKCwsHuDIi6g3OHIiISIZnKxERkQzDgYiIZBgOREQkw3AgIiIZhgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMb59BQ8akyVNQWVHRr23ExcfjXNnpGw8kom4xHGjIqKyowINv7u7XNv767DwvVUM0sjEcyCu88a3fZpM/iIiIfIPhQF7hjW/97z91t5eqIaL+YkOaiIhkGA5ERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw9tn0LBis9sRHKrv1zZ4Z1cihgMNM8Lt4p1dibyAh5WIiEiGMwcC0P9bbvN220TDC8OBAPT/ltu83TbR8MLDSkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkwyfBEV3HZrcjOFTfr23ExcfjXNlpL1VENPgYDsNAf5//DPAZ0NcSble/HpkKAH99dp6XqiHyDYbDMNDf5z8DfAY0EXliz4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw8eEEg0Am92O4FB9n9ePi4/HubLTXqyIqHcYDj42afIUVFZU9GsbNlurl6ohbxFuV7+e6/3XZ+d5sRqi3mM4+FhlRUW//hIBgPefuttL1RARtWPPgYiIZBgOREQkw3AgIiIZhgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiGd2UlGoL6+zwIgM+EoP5hOPQDn8VAA6W/z4MA+EwI6h+GQz/wWQxENFyx50BERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGAxERyfA6B6JhildZU3+M6HDo7xXOvLqZhjJeZU39MaLDob9XOPPqZiIarthzICIiGYYDERHJjMjDSh1NOvYMiLp3bVPb2mT2cTU0mEZkOHT0GdgzIOqeN5ra5J9GXDgIIeBstcr+vb/bGunbGAo1DKdtDIUart+GxWJBaGgoFApFv7ZJ/kEhhBC+LmIwWSwW6PX9O/ebaKQym80ICwvzdRk0CEZcOAgh0NTU5LXtWSwWJCQk4PLly0N6p/GHOlmjdwxkjcNh5lBXV4dbbrkFx48fx7hx43q0zqOPPoo777wTzz777MAWN5QI6hez2SwACLPZ7OtSuuUPdbJG7/CHGnNycgQAAUAEBASIcePGiZUrV4rW1lbZmCeeeEK2/lNPPSUAiJycHGlZTU2NWL58uUhISBAajUbExMSIuXPnis8//9xj3d/97nfi17/+da/q/cc//iFGjRolGhsbe/eL9sD69etFUlKS0Gq1Ij09XRw7dqzLsUlJSdJ/t2t/nnrqKWnMq6++Ku644w4REhIioqKixPz580VZWVmv6+KprETkE/PmzcOVK1fwww8/YO3atdi8eTPy8/M9xiQkJOC9995Da+tPZxbabDbs2LEDiYmJHmMXLlyIb775Btu2bcN3332Hv/zlL8jIyEBdXZ00pqWlBVu2bMHjjz/eq1qnTp2KCRMmYPv27X34Tbu2c+dO5OXlIT8/HydOnEBKSgqysrJQU1PT6fivvvoKV65ckX727t0LAFi0aJE05tChQ8jNzcXRo0exd+9eOJ1OzJ07F1ZrL/tPfYo6kvjDtzQh/KNO1ugd/lBjTk6OmD9/vseyhx56SEyfPl02ZurUqWL79u3S8nfffVdMmzZNzJ8/X5o5NDQ0CADi4MGD3X7uBx98IKKiojyWxcfHiw0bNngs++KLL0RgYKAoLy+Xlr3yyivirrvu6s2veUPp6ekiNzdXeu1yuURcXJwoKCjo0fq//e1vxYQJE4Tb7e5yTE1NjQAgDh061KvaOHPoJ61Wi/z8fGi1Wl+X0i1/qJM1eoc/1Hi90tJSHDlyBBqNRvbe0qVLUVRUJL3eunUrlixZ4jEmJCQEISEh2LVrF+x2e5efc/jwYaSlpXksmzFjBr766ivptRACzzzzDH73u98hKSlJWp6eno7jx493uv1XX31VqqGrn0uXLnms43A4YDQakZmZKS1TKpXIzMzEl19+2eXvcO3627dvx9KlS7vtA5nN7denRERE3HCbHnoVJUREXpCTkyNUKpUIDg4WWq1WABBKpVL8+c9/9hgzf/58UVNTI7RarSgvLxfl5eVCp9OJ2tpaj5mDEEL8+c9/FqNGjRI6nU7Mnj1bvPDCC+Lbb7/1+Nz58+eLpUuXeixbs2aNuPXWW6XX27ZtE7GxsaKpqclj3LfffisAeMwmOtTV1Ylz5851++N0Oj3WqaioEADEkSNHPJavXLlSpKen3/C/4c6dO4VKpRIVFRVdjnG5XCI7O1v87Gc/u+H2rjfirnMgoqFhzpw52LhxI6xWK9auXYuAgAAsXLhQNi4qKgrZ2dkoLi6GEALZ2dmIjIyUjVu4cCGys7Nx+PBhHD16FJ988gnWrFmDd955B4sXLwYAtLa2QqfTeaw3c+ZMPP/882huboZCocCqVavwn//5nwgJCfEYFxgYCKC9b3G9iIiI3n8z76ctW7bggQceQFxcXJdjcnNzUVpais8//7zX2+dhJSLyieDgYEycOBEpKSnYunUrjh07hi1btnQ6dunSpSguLsa2bduwdOnSLrep0+lw//3346WXXsKRI0ewePFijyZ3ZGQkGhoaPNZJS0uDUqnEiRMn8NprryEqKkp22AoA6uvrAbSH1fX6clgpMjISKpUK1dXVHsurq6sRGxvb5e8IABcvXsS+ffvw61//ussxK1aswEcffYQDBw5g7Nix3W6vM5w5EJHPKZVKrFq1Cnl5efjVr34lfUvvMG/ePDgcDigUCmRlZfV4u1OmTMGuXbuk19OnT5edcRQUFITbbrsNBoMBb7/9Nv7v//4PSqX8e3NpaSnGjh3b6axl+fLleOSRR7qt5fpv+BqNBmlpadi/fz8WLFgAAHC73di/fz9WrFjR7baKiooQHR2N7Oxs2XtCCDz99NMoKSnBwYMHkZyc3O22usKZAxENCYsWLYJKpcKGDRtk76lUKpw5cwanT5+GSqWSvV9XV4d7770X27dvx9///ndcuHABH3zwAdasWYP58+dL47KysnDq1CnZ7GHmzJkoLCxEVlYWMjIyOq3v8OHDmDt3bqfvRUREYOLEid3+BATIv4vn5eXh7bffxrZt23DmzBk8+eSTsFqtHjOX9evX47777pNeu91uFBUVIScnp9Nt5ubmYvv27dixYwdCQ0NRVVWFqqoqj9OBe4IzByIaEgICArBixQqsWbMGTz75pOz97q72DgkJwYwZM7B27Vp8//33cDqdSEhIwLJly7Bq1Spp3G233YbU1FS8//77eOKJJ6TlKSkpUKvVeP311zvdvs1mw65du7B7t3dvQvjLX/4StbW1ePnll1FVVYXbb78du3fvRkxMjDTm6tWr+P7776XX+/btw6VLl7o8vLZx40YAkIVcUVGR1HvpiRF3+wwiGtk+/vhjrFy5EqWlpdLhozlz5iA1NRVvvvlmp+ts3LgRJSUl2LNnz2CW6lOcORDRiJKdnY1z587h8uXL0Ol02LJlC86dO4cPP/ywy3XUajUKCwsHsUrf48yBiEakgwcP4t5778XkyZNRVFSEGTNm+LqkIYXhQEREMjxbiYiIZBgOREQkM+LCQQgBi8UCHk0j8g3ug/5hxIVDU1MT9Hq9V58GR0Q9x33QP4y4cCAiohtjOBARkQzDgYiIZBgOREQkw3AgIiIZhgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikvGrcCgoKMCdd96J0NBQREdHY8GCBTh79qyvyyIiGnb8KhwOHTqE3NxcHD16FHv37oXT6cTcuXNhtVp9XRoR0bCiEH78lO/a2lpER0fj0KFDuPvuuzsdY7fbYbfbpdcWiwUJCQkwm80ICwsbrFKJRizug/7Jr2YO1zObzQCAiIiILscUFBRAr9dLPwkJCYNVHhGB+6C/8tuZg9vtxj//8z+jsbERn3/+eZfj+K2FyLe4D/qnAF8X0Fe5ubkoLS3tNhgAQKvVQqvVDlJVRHQ97oP+yS/DYcWKFfjoo4/w2WefYezYsb4uh4ho2PGrcBBC4Omnn0ZJSQkOHjyI5ORkX5dERDQs+VU45ObmYseOHfjwww8RGhqKqqoqAIBer0dgYKCPqyMiGj78qiGtUCg6XV5UVITFixf3aBsWiwV6vZ7NMCIf4T7oH/xq5uBHOUZE5Nf8+joHIiIaGAwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw3AgIiIZhgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJ+F04fPbZZ3jwwQcRFxcHhUKBXbt2+bokIqJhx+/CwWq1IiUlBRs2bPB1KUREw1aArwvorQceeAAPPPCAr8ugEchgNKHV2Ya6ZgdCtQHQqJW42mRH/CgdLta1In5UIOxOFwSANpeA0yUQGapGfbMT+kA1Lje0IDkyCAoAP1xtwYTIIJhtLrTY2zA6RItKsw3jInSwOgTsbW4oFECby40gjQoOl4DT5YZGpUSASgGXSwAAmh1tiAzR4mqzHWP0WpgabIgN06HKYkOsXgdHmxv1VgdGBakRpFFBCKDB6oBQKKANUCJQrUKTvQ0ut0CwRoU2l0Czow2xYVpUNNoQP0qH8qstGDc6CPVWJ5ZnTPTp/wMaPH4XDr1lt9tht9ul1xaLxYfVkL8yGE2oaGjBe19dQqXZjli9FlPj9GiytaG8zopqS/tfzvNujcXlhlb8o8KMaosdceE6LJ09Duv2n0O1xY6YMC2W3ZUM48VGfFfdLI2LCdMiaXQw3v/aimU/T8axC/XSe5m3RKO00owqc/tnZN0aC9M1nxETpkXGTVF499glj21drLPigamxKD5yETFhWvz2vkn4W1mNR21ZU2Lw6elqVDbaMEavxa1xeuw7U+OxjaTRwTCcqEDS6OA+hQP3Qf807MOhoKAAr7zyiq/LID9mMJrQ5nLhTFUTKs3tf8lVme24OUZAp1ah2tK+7IrZjlanG06XkJZVNtpQ2+yQXldb7KhtdkCnVnmMq7bYMTk2rP39JofHe06XQJX5p8+wXfcZ1RY7FApFp9uyOd3Sssv1LbLaWp1uVDbapG3fFCNk27j2n33R1T4YGRUNhULRp21S9+Lix+LC9+f6tY1hHw4vvPAC8vLypNcWiwUJCQk+rIj8icFoglu4cK7GiglRwYjVa1H148xBrVKgydaGmDCtNHMI0iihVimkZXHhOkSHaqXXMWFaRIdqcfJyI0J1AR7LbU6X9P73tc3Se2qVQvrcMXotdGrPz4gJ0wJAp9vSqZXSe4kRQThX0+xRW5BGibhwnTRzUKsU0viObVz7z77oah98cM1HUAcG9/P/EHWm5Df39nsbCiGE8EItPqFQKFBSUoIFCxb0eB2LxQK9Xg+z2YywsL59E6KRoSMY3tp7DlfM7X/pPpwaj8YWJyZGh0CjUqK22bPnYHO0/wXqdAm0uQUiQ9RoaHEiTNd1z8Fqb+8bSD0HpxsOpxtQtPcuAjUqOF3tswWNqj0Y2tw/9hzsbYgK0aL2mp7DmDAdrljaew9Olxt11/YcANRbHQB+6jlYbG1wC8+eQ0yYFpUD1HPo2AcfWreX4TBASn5zL+y21n5tY9jPHIj64kBZNZwuF/5RYcEV80+Ha+xtAqnjIrAwdayPKyQaWH4XDs3NzTh//rz0+sKFCzh58iQiIiKQmJjow8poOLnabMMf9p9HQkSQx2GjW+LCGAxe8r/PzAV60XNQKBRQq9UDWNHwERff/z+jfndY6eDBg5gzZ45seU5ODoqLi2+4Pg8r0Y0YjCbsPV2F3aeqAQDpyRGYFB2C1KRRDAYv6OthJW8cKqGe87uZQ0ZGBvwsz8iPGIwmnL5iQUOLU5oxmOpb8Mu0OAYDjSh+Fw5EA8VgNOGbyw0ID1Sj/Mfz+6cnhOP+W6KgUKp8XR7RoPK7w0r9xcNK1BmD0YQ1n5ZJp4I+nBoPW5vAlNhgQKnirMGLOvZBQNG7noMqAEmJif0+f596hjMHGvEMRhPO1zRh3OhgVFvsqLbYYba1YfrYMAbDAHpo3Z5en8rqjfP3qWcYDjSiGYwm7DtThXqrE+V1VqQnR+BinRUpDAYa4RgONGIZjCa8sadMusAtaXQwUhPD8YvbYqFkMNAIx3CgEamj+XztBW6pieFwtLkRrFMzGAZBb69zANr7DskTJrHvMAgYDjTidPQYEkYFelzgNufmKKhUnDEMlr70HAD2HQYLw4FGlOvPSloyOwmXG21IiQ9jMBBdw++eBEfUVx0zhmtvbW2xteFqk53BQHQdhgONCB0zho4rn4H221KPCQvE3FtjGQxE1+FhJRr2DEYTfqhtnzG899Vl/PKOsYgI1sDe5kaQLoDB4CN9aUgDbEoPFoYDDWsdM4anMsZLzeeD39Xit/dNhFbNYPClvjakATalBwPDgYYtg9GE76rbr3w+c6UZv0pPhKXVCX2gmsFAdAMMBxqWDEYT3txzFpVmG2LCtBAAdGol9IFqxEcEMRiIboDhQMNOx/MYKs02AO1nJS24PR7RoVqEB2sYDENEX3sOAPsOg4HhQMOKwWiCo82FaWP1+HuFGZWNNsTptYgM0TAYhpj+9BwA9h0GGsOBhg2D0YRaiw1/PHYRlY02ZN4SjV/eMRYKhQIRIVoGA1Ev8DoHGhYMRhNOXGpAa5sblY3th5P2namBvc2N+FHsMRD1Fh/2Q37PYDTh01NXYG8T0AYoEBceiOIjFxETpsVz8yYzGIaYvj7s53p8+M/A4mEl8msGown1Vju+NZml+yWlJoZj5dybEBseyGAYwvrbcwDYdxhIPKxEfstgNKG0shENLQ6MG93+l0y1xQ5Hm2AwEPUTZw7klwxGE6osrfi0tFq6liE9OQKX6628joHIC9hzIL9jMJrw5t6zmBQdikPf1UrLf31XMm6JC2MwDHHe6jkA7DsMJM4cyK8YjCacuNiAykYbEkYFSfdLigvXMRj8jDd6DgD7DgOF4UB+w2A0odnuRLAuAHOnRGPP6Rpk3hKNzFt0SE0axWAg8iKGA/kFg9GEioYWvHv8kvRYz7z7JyFAqUCMns1nIm/j2Uo05BmMJlyss8LpFtJT3K6Y7ahpsjMYiAYIZw40pBmMJvyjohE2pxtxeh3G6LW4Ym6/nmF6Ig8l+bP+3HjvWgEarReqoesxHGjI6ugxfFJaJV3gljtnAr6rbmYwDANsSA9tPKxEQ5LBaEKrow2X61ulQ0nVFjuqzXYGA9EgYDjQkGMwmmBvc+GL7+vQbG9DTFj7YYOYMC3GR4cwGIgGgV9eBLdhwwa8/vrrqKqqQkpKCgoLC5Gent6jdXkR3NBmMJrQ5nJh3f5zUm8h46YohOrUvI5hmPDmRXAAL4QbKL3qOVy+fBkJCQkDVUuP7Ny5E3l5edi0aRNmzJiBdevWISsrC2fPnkV0dLRPa6P+MRhN2HOqCkpl+9lIQPuhJIutDenjRzMYhhlv9RwA9h0GQq8OK02ePBkvv/wyWlpaBqqeG3rrrbewbNkyLFmyBFOmTMGmTZsQFBSErVu3+qwm6j+D0QSLzYFRwRrUW53SoaQxei0yp8QwGIgGWa/CYe/evfj0008xadIkFBcXD1BJXXM4HDAajcjMzJSWKZVKZGZm4ssvv+x0HbvdDovF4vFDQ4vBaEKrsw3HLzQAAMrrrEgaHYwHpsbi91l8HoO/4z7on3p1WGn27Nk4duwY/vjHP+Lf//3fUVhYiHXr1uHnP//5QNXn4erVq3C5XIiJifFYHhMTg7Kysk7XKSgowCuvvDIY5VEfGIwmnKo045PSKx49hhBtAKbE6xkMw0BX+6C3rnNQKBRIGpfc7+2Qpz43pFtaWvBf//VfeOuttzBv3jy8/vrrSE4e2P9BlZWViI+Px5EjRzBr1ixp+XPPPYdDhw7h2LFjsnXsdjvsdrv02mKxICEhgQ3pIcBgNMHlduGHqy345lIjjl2oBwA8MDWWh5KGka72wYfW7fXadQ52W2u/t0Oe+nUR3Ny5c2GxWFBYWIiPP/4YTz/9NF5++WWEhIR4qz4PkZGRUKlUqK6u9lheXV2N2NjYTtfRarXQankF5VBz4Woz3MKFN/eeky5wS0+OgKneymAYZrgP+qde9Rw2bdqExx9/HNOmTYNer8d9992Hw4cPY/ny5fjv//5vfP3115gyZQq+/vrrASlWo9EgLS0N+/fvl5a53W7s37/fYyZBQ9/X5XU4X9PicYFbamI4nmWPgWhI6NVhpYSEBMyYMQMzZ87EzJkzkZaWhsDAQI8xr776Knbs2IHS0lKvFwu0n8qak5ODzZs3Iz09HevWrcP777+PsrIyWS+iM7zOwfcMRhPO1zRhXGQQ3rpm5vDcPAbDSODt6xwCNFo4bb47g3K48vpFcNXV1YiLi4PL5fLmZj2sX79eugju9ttvxx/+8AfMmDGjR+syHHzLYDRhzadlUiA8c98klNe14KboICy8I8nX5dEg6NgH2XMY2rx+473o6Gj87W9/8/ZmPaxYsQIrVqwY0M8g7zMYTThX0+RxKOlSfXswQKnycXVEdC2v31tJoVDgnnvu8fZmyc8ZjCa8saes/eE819wraWJUezDwcBLR0MJbdtOA6+gxJEYEY/2B75GbMR4uAUxiMIxo3rjOgdc4DByGAw0og9GEfWeqUG914lK9FenJEfjfbyrwVMZEBsMI5417K5X85l7ecG+AMBxowBiMJry59ywqG22ICdMiaXQwZo2PwL/cHgeNmsFANJTxeQ40IAxGE/aerkJlow1Ae/M5IkgNTYCSwUDkBxgO5HUGownVllZEheoQq//p7qpx4YGI1QcyGIj8AA8rkVcZjCZ8euoK/l5hRpXZjsxbonHfzVqMGx2EiFAdg4Ek/W1Isxk9sBgO5DUGowlnrlgwKToUe07XAAD2nanByrk3MRhIpr8NaTajBxbDgbzi2uZzXLgOmVOise90DeLDdYgN56EkIn/DcKB+MxhN+K66SWo+Vzba8NidCUhLHIXoMM4YiPwRw4H6xWA04c09Z3HfLTGICdNK90zSqlUYHaJlMFCX+ttzCNDwNuADieFAfWYwmnDhajMSIoLwXXUTbovXY3KsgDZAwWCgG/JGz4EGDsOB+qSzC9yUCiA1QY+xo4MZDER+jtc5UK+13xKj2uMCt9sT9EgYFcRgIBomOHOgXjEYTTBerEe91SH1GGL1WoRp1RgzimclUc/1p+fAaxwGHsOBesxgNKGisQWx+kD87WwNkkYHIzUxHGP0OgYD9Vp/eg68xmHgMRyoRzqex3DF3H420gNTY9Fsd+GWmFCEh2gYDETDDMOBbqjjCW5XzD89we3C1RYEaVQMBqJhiuFA3TIYTThdaYYbAmP0WmnmoA1QIHNKDIOB+qyvPQf2GwYHw4G6ZDCa0OpogyZAiROXGnH3pCiE6tSICdMigtcxUD/1tefAfsPgYDhQpwxGE+qsdhR9cUGaLQgApoYWPDv3ZgYD0TDHcCCZ9ltilGFSTJhHn+Gh6fH45Z0JDAaiEYAXwZEHg9GEK42tqDTbYXO6EBPWfv+amDAtJsaEMhiIRgiFEEL4uojBZLFYoNfrYTabERYW5utyhhSD0YS6Jjsu1regusmGfWdqkDklGreOCUMir3wmL+nYBwFFrxvSHc1o9hwGHg8rEYD2YLDYHCg+Wo7KRhti9Vr828wkqFUKBgMNiL40pNmMHjwMB/I4XbXjfklVZjuuNtt5uirRCMWewwhnMJpQWtmIj0uv4HRlk9RjiNPrGAxEIxhnDiOYwWhCbZMNVxptqDLbUWW2Iz05Av8yPR6T2HymAdbbi+B48dvgYjiMUAajCTVNNigVCjS0OKU7rJrqW3i6Kg2K3vYc2G8YXAyHEchgNOGtvWdR8WPj+bZ4/Y93WB3FQ0lEBIA9hxGnY8ZQcU3jWR+owU0xIQwGIpJw5jCCdJyu2uJwYe6UaOw5XYOYMC0srQ7MmjCGwUCDqtc9B1UAtLrAAaxoeImLH9uvw3B+FQ6rV6/Gxx9/jJMnT0Kj0aCxsdHXJfkNg9GEPaeqcNLUiGqLHWP0WuTdPwkBSgVi9HxQDw2+/jzsh26s5Df39mt9vwoHh8OBRYsWYdasWdiyZUu/tvWnY+Vwq4OgVABhOjVsbS402doQE6ZFU6sLTrcbCghEBGvQbHfB3uZGmC4ALQ4XVAoF3AAcLjdCNCo02V3QBSihVSvR6nChzS0QEaxGrcWBUSEaNLY4EaxWorXNjfBANa4226FTqxAeFIAqsx1hgWq0OFwIVCvR6nQjVBsAh8uNZlsbIoI10KoVqG92ok0Ao4La148K0aLV4YLF5kRcuA5Whwstjvbt1zTZEB+uw+UGGxJG6WBqtCEiSI2xowLx6elqAMAVsx3VFjtSk0YxGIhIxq/C4ZVXXgEAFBcX93gdu90Ou90uvbZYLACAtw9fwPj4aOgDA6AA8K3JjGqLHXOnRKO00oLKRhtiwtqbtbfF67F23zmM0WuRdWssTA2t+EeFWfoGnhARjIt1VjwwNRa7T1Whyty+fGHqWGw4cF7aVsZNUTj4Xa303OWpcXrsO9N+aCfj5mgcPFuDcaODEaoL8Nj+o3cmYsfxS9LrhaljceBsjVTzGL0WU+P1sLS2obzOimpL+11Uk0YH4/2vrci4KQo7v7qM/5eeKD2TYYxey2CgQdHVPkhDm1+FQ18UFBRIoXKtmiYHblWr4Ghrv7VUtaX9D6+97aerhKstdkyOFbC0OgG0f9u2Od1wuoQ0/orZjptiwlBtaX+vyvzT8laHy2NbCoVCWq/KbMfNMT99tkL6vDDZ9i2tTo/XrQ4X7G3X1yCgU6ukZR3buvZzK802/NuMJNhdbt4SgwZNV/vgX5/7BRR9eNgP9UxcfP/272EfDi+88ALy8vKk1xaLBQkJCYgO1cDmdCHsx5lDx3n+2gAF4sN1qPjx275apYA+SA0AGKPXQqdWQq1SSOPH6LXS3Ut1aiVi9Vpp5hCoUSEuXCfNHHDN58Tq27fdsazjnzanC6G6AI/t6wPVHq8DNSpoAzxr0AYoYG5tk5Z1bOvabevUSrQJwWCgQdXVPni1toY3vxzCfH5X1ueffx6vvfZat2POnDmDyZMnS6+Li4vxzDPP9Kkh3XFHyE17vpV6DvpANVod1/QcbG1wukV7zyFEg2Zbe88hVBeAFrsLASoF3ELA0SYQrFXBandB+2PPocXugkt09BzsGBWkQaOtDcGa9n5CeJAaV5t+6jlUm+0I/bHnoAtQwtbmRog2AM42N5rt1/QcrE60uYGIIDVqm+2I/LHn0GRzYky4Di12F1qcboQHBqCmyY64cB1M1/QcRgeroVGpoFGrGAzkU7wzsn/w+czh2WefxeLFi7sdM378eK9/7mMzxvEPJhFRF3weDlFRUYiKivJ1GUREdA2fh0NvXLp0CfX19bh06RJcLhdOnjwJAJg4cSJCQkJ8WxwR0TDiV+Hw8ssvY9u2bdLr6dOnAwAOHDiAjIwMH1VFRDT8+LwhPdjYDCPyLe6D/oE33iMiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw3AgIiIZhgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZPwmHMrLy/H4448jOTkZgYGBmDBhAvLz8+FwOHxdGhHRsBPg6wJ6qqysDG63G5s3b8bEiRNRWlqKZcuWwWq14o033vB1eUREw4pCCCF8XURfvf7669i4cSN++OGHHq9jsVig1+thNpsRFhY2gNURUWe4D/oHv5k5dMZsNiMiIqLbMXa7HXa7XXptsVgGuiwiugb3Qf/kNz2H650/fx6FhYV44oknuh1XUFAAvV4v/SQkJAxShUQEcB/0Vz4/rPT888/jtdde63bMmTNnMHnyZOl1RUUF7rnnHmRkZOCdd97pdt3OvrUkJCRwSks0SLgP+iefh0NtbS3q6uq6HTN+/HhoNBoAQGVlJTIyMjBz5kwUFxdDqezd5IfHO4l8i/ugf/B5zyEqKgpRUVE9GltRUYE5c+YgLS0NRUVFvQ4GIiLqGZ+HQ09VVFQgIyMDSUlJeOONN1BbWyu9Fxsb68PKiIiGH78Jh7179+L8+fM4f/48xo4d6/GeH5+NS0Q0JPnNcZnFixdDCNHpDxEReZffhAMREQ0ehgMREckwHIiISIbhQEREMgwHIiKSYTgQEZEMw4GIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikvGbW3Z7S8ddXPmQc6LeCw0NhUKh8HUZNAhGXDg0NTUBAB9yTtQHfLTnyOHzZ0gPNrfbjcrKSq99A+p4WPrly5eH9E7jD3WyRu8YyBq9sd8IIdDU1MRZyBA34mYOSqVS9iQ5bwgLCxuyf1lcyx/qZI3eMVRrVCgUQ7Iu8sSGNBERyTAciIhIhuHQT1qtFvn5+dBqtb4upVv+UCdr9A5/qJGGvhHXkCYiohvjzIGIiGQYDkREJMNwICIiGYYDERHJMBy8qLy8HI8//jiSk5MRGBiICRMmID8/Hw6Hw9eleVi9ejVmz56NoKAghIeH+7ocAMCGDRswbtw46HQ6zJgxA8ePH/d1SR4+++wzPPjgg4iLi4NCocCuXbt8XZJMQUEB7rzzToSGhiI6OhoLFizA2bNnfV0W+SmGgxeVlZXB7XZj8+bNOHXqFNauXYtNmzZh1apVvi7Ng8PhwKJFi/Dkk0/6uhQAwM6dO5GXl4f8/HycOHECKSkpyMrKQk1Nja9Lk1itVqSkpGDDhg2+LqVLhw4dQm5uLo4ePYq9e/fC6XRi7ty5sFqtvi6N/JGgAbVmzRqRnJzs6zI6VVRUJPR6va/LEOnp6SI3N1d67XK5RFxcnCgoKPBhVV0DIEpKSnxdxg3V1NQIAOLQoUO+LoX8EGcOA8xsNiMiIsLXZQxZDocDRqMRmZmZ0jKlUonMzEx8+eWXPqzM/5nNZgDgnz/qE4bDADp//jwKCwvxxBNP+LqUIevq1atwuVyIiYnxWB4TE4OqqiofVeX/3G43nnnmGfzsZz/D1KlTfV0O+SGGQw88//zzUCgU3f6UlZV5rFNRUYF58+Zh0aJFWLZs2ZCskYav3NxclJaW4r333vN1KeSnRtwtu/vi2WefxeLFi7sdM378eOnfKysrMWfOHMyePRv/8z//M8DVtettjUNFZGQkVCoVqqurPZZXV1cjNjbWR1X5txUrVuCjjz7CZ599NiC3p6eRgeHQA1FRUYiKiurR2IqKCsyZMwdpaWkoKiqCUjk4k7Pe1DiUaDQapKWlYf/+/ViwYAGA9kMi+/fvx4oVK3xbnJ8RQuDpp59GSUkJDh48iOTkZF+XRH6M4eBFFRUVyMjIQFJSEt544w3U1tZK7w2lb8GXLl1CfX09Ll26BJfLhZMnTwIAJk6ciJCQkEGvJy8vDzk5ObjjjjuQnp6OdevWwWq1YsmSJYNeS1eam5tx/vx56fWFCxdw8uRJREREIDEx0YeV/SQ3Nxc7duzAhx9+iNDQUKlno9frERgY6OPqyO/4+nSp4aSoqEgA6PRnKMnJyem0xgMHDvispsLCQpGYmCg0Go1IT08XR48e9VktnTlw4ECn/81ycnJ8XZqkqz97RUVFvi6N/BBv2U1ERDI8W4mIiGQYDkREJMNwICIiGYYDERHJMByIiEiG4UBERDIMByIikmE4EBGRDMOBiIhkGA5ERCTDcCAiIhmGA/ncn/70JwQGBuLKlSvSsiVLlmDatGnSoy6JaHDxxnvkc0II3H777bj77rtRWFiI/Px8bN26FUePHkV8fLyvyyMakfg8B/I5hUKB1atX4+GHH0ZsbCwKCwtx+PBhBgORD3HmQENGamoqTp06hT179uCee+7xdTlEIxp7DjQk7N69G2VlZXC5XIiJifF1OUQjHmcO5HMnTpxARkYGNm/ejOLiYoSFheGDDz7wdVlEIxp7DuRT5eXlyM7OxqpVq/DYY49h/PjxmDVrFk6cOIHU1FRfl0c0YnHmQD5TX1+P2bNnIyMjA5s2bZKWZ2dnw+VyYffu3T6sjmhkYzgQEZEMG9JERCTDcCAiIhmGAxERyTAciIhIhuFAREQyDAciIpJhOBARkQzDgYiIZBgOREQkw3AgIiIZhgMREcn8fxGAHuxONzt0AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 400x400 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = torch.randn(int(1e5)).requires_grad_()\n",
    "y = F.hardtanh(x)\n",
    "y.backward(torch.ones_like(y))\n",
    "df = pd.DataFrame.from_dict(dict(x=x.detach(), y=y.detach(), grad_x=x.grad))\n",
    "\n",
    "jointplot(df, x=\"x\", y=\"y\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recalling that `hardtanh(x, mult)` is defined as `clip(x, -1/mult, 1/mult)`, we can see that the output distribution is a mixture of three components:\n",
    "\n",
    " - truncated Gaussian with weight `p(-1/mult <= x <= 1/mult)`\n",
    " - spike at `y=-1/mult` with weight `p(x < -1/mult)`\n",
    " - spike at `y=1/mult` with weight `p(x > 1/mult)`\n",
    "\n",
    "So we can write out the pdf of $Y$ as a mixture distribution (where $\\alpha = \\mathrm{mult}$):\n",
    "\n",
    "$\\mathrm{P}(Y=y) = \\begin{cases}\\frac{1-Z}{2}\\, \\delta(y - \\alpha^{-1}) + \\frac{1-Z}{2}\\, \\delta(y + \\alpha^{-1}) + \\varphi(y) & \\textrm{if } {-\\alpha^{-1}} \\leq y \\leq \\alpha^{-1} \\\\\n",
    "0 & \\textrm{otherwise}\n",
    "\\end{cases}\n",
    "$\n",
    "\n",
    "where $Z \\coloneqq \\mathrm{erf}(\\sqrt{\\frac{1}{2}}\\, \\alpha^{-1})$ is the probability that $X \\sim \\mathcal{N}(0,1)$ falls in the range $[-\\alpha^{-1}, \\alpha^{-1}]$, $\\delta(\\cdot)$ is the Dirac delta function and $\\varphi(\\cdot)$ is the Gaussian pdf (note that the $Z$ normaliser for a truncated Gaussian cancels exactly with the mixture weight).\n",
    "\n",
    "Next, by symmetry, we can observe that $\\mathrm{E}(Y) = 0$. Therefore the scale, $\\sigma_Y = \\sqrt{\\mathrm{E}(Y^2) - \\mathrm{E}(Y)^2} = \\sqrt{\\mathrm{E}(Y^2)}$.\n",
    "\n",
    "This expands to:\n",
    "\n",
    "$\\sigma_Y = \\sqrt{(1-Z)\\, \\alpha^{-2} + Z\\,(1 - 2 e^{-1/(2\\alpha^2)} / (Z \\alpha \\sqrt{2 \\pi}))}$\n",
    "\n",
    "where the first term is from the pair of spikes, and the second term from the variance of a symmetric [truncated Gaussian](https://en.wikipedia.org/wiki/Truncated_normal_distribution).\n",
    "\n",
    "Leading to the **forward scale**:\n",
    "\n",
    "$\\sigma_Y = \\sqrt{\\alpha^{-2} + (1 - \\alpha^{-2})\\,\\mathrm{erf}(\\sqrt{\\frac{1}{2}}\\, \\alpha^{-1}) - \\sqrt{\\frac{2}{\\pi}}\\, \\alpha^{-1}\\, e^{-\\frac{1}{2}\\alpha^{-2}}}$\n",
    "\n",
    "> Note: when $\\alpha\\!=\\!1$, this simplifies to $\\sqrt{1 - \\sqrt{2 / (\\pi e)}}$\n",
    "\n",
    "Let's test this rule by sweeping `mult` over a logarithmic range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAEqCAYAAAB3BAsnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAaElEQVR4nO3deZxN9ePH8de5m5lhZhhjbEPIJLKMtShCspYkS3ZCkREt6kcJbaikvW8pxhrZpZQlS7QIIRGSLXuYwQxzt/P742ZKUTOamTPL+/l43MfMvXPvOe/PjJm3c+45n2OYpmkiIiIi/8pmdQAREZGcQqUpIiKSRipNERGRNFJpioiIpJFKU0REJI1UmiIiImmk0hQREUmjPF2apmly5swZdKqqiIikRZ4uzbNnzxIeHs7Zs2etjiIiIjlAni5NERGR9FBpioiIpJFKU0REJI1UmiIiImmU40tz8eLFVKhQgZiYGN5//32r44iISC5m5ORLg3m9XipVqsTKlSsJDw+nZs2afPXVVxQuXDhNrz9z5gzh4eEkJiYSFhaWyWlFRCSnc1gd4L9Yv349N9xwAyVLlgSgRYsWLF26lE6dOmX+ys8nQPLJzF9PdmQYgPH3j5f7ms0ONsefPjrAsAfuG4ZlQxARuRrZvjRHjx7NvHnz+OmnnwgODqZevXqMHTuWChUqcPjw4dTCBChZsiSHDh3KmmDb5sInj2TNunIr409FerFYnSHgyg+uEHAVCHye+liB3x+/+HkBKFAUQotBaHEIiVARi0imyvaluXr1agYMGEDt2rXxer0MGzaMpk2bsn37dqujAbDKV5UDZlGrY2QpAzP1Y+D218f+uG83/DjwYceHHT92Lt7//aPhw46JHR8OfISQQohxgvykEMIFnIYv7cHsLihQDMKK/1Gkf/4YFg0RZQPlLCJyFbJ9aX722WeX3I+PjycqKoqNGzdSokSJS7YsDx06RJ06da64rJSUFFJSUlLvnzlz5j/n+8jXiE/9N/7n5cjlufAQwgXyc4EQI1Ck+Y0LhHCBUM4TZZymqJFAMeMURf2nKZpwmqiEDbiuVLaOYIiqCMUqQ9Eqv3+8AYLCs3ZgIpIjZfvS/KvExEQAIiIiuP7669m2bRuHDh0iPDycJUuWMHz48Cu+dvTo0YwaNSpD89Sw7U7dqsoLzN9He+nHv257/vG4HwMfdrymDT82vNjx/f4xcN8W+Dp2vKad87hIJgg3TgDcOHHjJIFQUr/Nafh2R5JIlHE6UKbGaYoapyltHKeifz/XHtqC6/CmS19QsPSfSrRy4GPBMmDL8QeYi0gGylFHz/r9flq3bk1CQgJr164FYNGiRTz22GP4/X4ef/xx7r///iu+/nJbmqVKlbq6o2e/+0DvaWYit2nnPEEkkY9kM4gkgkgyg0j+02OJ5OeYWYjjZiGOmoU4akZwnEJ4/uX/gk68xBi/UtHYTyXbfioaB6hk209BI+nSJwYVhGsbQfnbofxtgV28IpKn5ajS7N+/P0uWLGHt2rVER0f/5+X9p1NOjm2H/ev+c4YcyQxsS176kcs8ZoLpB78X/L7fP3qvfN/nAU8yuJMCN08yuM/9fj8ZPElXjHSR3zQ4TQGOmYU4ZkZw1CzEMQKf7/EXZ4d5DWcJuexrS/AbFW0HqGTsp6JtPzVtuyhqJPzxhGJVAgUacztE1wa78799H0Ukx8kxpRkXF8fChQtZs2YNZcuWzZBl6jzNHMbvB+/530v0HJw/DWePwdkjcPbonz7+/nnyb39bhGnCr2YkO8xr2G5eww5/ababZThoRl12ldcb+2lo20JD+2ZqGrv/ODApXziUuzVQoOWbQFiJzBy5SLazb98+ypYty/fff09sbKzVcbJMti9N0zQZOHAg8+fPZ9WqVcTExGTYslWauZzXDeeOBUr0zCE48RMc/QGObYPT+y556hkzmJ/M0uzwB8p0m78M281rMP80aVYoydxs20ZD22ZutW+luHHqjwVE3QAVWkBsZyh8bRYNUMQ6Pp+PEydOEBkZicNh7eExI0eOZMGCBWzevDnT15XtDwQaMGAAM2bMYOHChYSGhnL06FEAwsPDCQ4OtjidZGsOFxQsFbhR+9KvXTgDx7enlmjY0W3UOb6dOp6dqU85aYbypb8qq3zVWOOvyinC+Mxfh8/8dcAL1xsHuNW2mYa2LdQ89hOu4z/Cly9DmfpQowdUvBOcQVk7ZpEs4Ha7cblcFCuW997nz/aHBr7zzjskJibSsGFDihcvnnqbNWuW1dEkJwsKg9I3QZ2+cOdr0HcFDP0V4jZAu0lw0wAKFylBG/s6XnW9zXf5+rPANZyHHbOpbuzGwM9PZmne9bWmk2c4NVLe5XFPXzb6YzD3fgnz+sC4CrDkCTj2o9Wjlb8wTZNkt9eSW3p37vn9fkaPHk3ZsmUJDg6mWrVqzJkzB9M0adKkCc2aNUtd5qlTp4iOjubpp58GYNWqVRiGwSeffELVqlUJCgripptuYtu2bZesY+3atdSvX5/g4GBKlSrFQw89RFLSH8cQlClThmeffZbu3bsTFhbG/fffz759+zAMI3Xr7uK6Pv/8c6pXr05wcDCNGzfm+PHjLFmyhIoVKxIWFkbnzp1JTk7+1/FddHG5K1asoFatWoSEhFCvXj127gz8Bzc+Pp5Ro0axZcsWDMPAMAzi4+PT9T1Oj2y/ezYzafes/KvT+2HPCti9HPauDryXCpw2C7DGX4XVvlhW+6tykj/O84wxfqWjfSVt7WuJMM4GHixZE2p0h8r3QL5QK0Yif5Ls9lLp6c8tWff2Z5oR4kr7Tr7nn3+eadOm8eqrrxITE8OaNWvo168fn3/+OeXLl6dKlSqMGDGCQYMG0aFDB/bv38+6detwOBysWrWKRo0aUbFiRV577TWKFSvGsGHD2LZtG7t27cLpdLJnzx6qVavGc889R6tWrThx4gRxcXFUq1aNSZMmAYHSPH36NE8//TRt2rQBwG63X/Ke5sV13XTTTbz88suEhITQoUMHSpYsSb58+RgzZgznzp3j7rvvZsiQITzxxBP/Or5bb701dbk33ngjY8eOpUiRIvTr1w+fz8e6des4f/48w4cP57PPPmP58uVA5u6JVGmqNCWtvG448DX8vDxwOx6YlcpvGmw0r2OWryGLfTdxgXxAYGKGprYN3GtfST3bj9gME5z5oXLbwO7b6Fqa9s8iOaU0U1JSiIiIYPny5dStWzf18T59+pCcnMyMGTOYPXs23bt3Z/Dgwbzxxht8//33qcd+XCycmTNn0rFjR+CPrdH4+Hg6dOhAnz59sNvtvPvuu6nLX7t2LbfeeitJSUkEBQVRpkwZqlevzvz581Of89cDgS6ua/ny5dx2220AjBkzhqFDh7Jnzx7KlSsHQL9+/di3bx+fffZZmsZ3ueV++umntGrVivPnzxMUFKT3NEWyJYcrcMRsuVuh6bOQ+Cv8vALb7qXU3r2U2radPO2YwiJfPWb5GvGDWY7F/ros9tcl2jhOR/sq2purKfb9VPh+auAUlgaPw/V3aBKFLBbstLP9mWaWrTutfv75Z5KTk7n99tsvedztdlO9enUA2rdvz/z58xkzZgzvvPPOZQ+W/HMhRUREUKFCBXbs2AHAli1b2Lp1K9OnT099jmma+P1+9u7dS8WKFQGoVatWmjJXrVo19fOiRYsSEhKSWpgXH1u/fn2ax3e55RYvXhyA48ePU7p06TTlyigqTZGrFR4NNXsEbkknYetMwjZNoeuJFXR1rGCb/xpm+RqxwHczv5pRjPN2YLy3HY1sm7nX/gW3Hfke20fdAkfe3vo4VGyt8swihmGkaxepVc6dC7wd8Mknn1xycQqAfPkCezSSk5PZuHEjdrud3bt3X9U6HnjgAR566KG/fe3PhZQ/f/40Lc/p/OP8ZcMwLrl/8TG/35+6bvjn8V1puUDqcrJS9v9XI5IT5C8MdQfATQ/Cr9/BpslU3jaPyp54hjlmsMRfh5neRqw3K7LCX4MV/hpUMA4w0DGflsfWY5vdA6IqQYMhUKmNylMAqFSpEvny5ePAgQPceuutl33Oo48+is1mY8mSJbRs2ZJWrVrRuHHjS57zzTffpBbg6dOn2bVrV+oWZI0aNdi+fTvly5fP3MFcRlrGlxYulwufLx0Xd/gPVJoiGckwoFSdwK3ZaNg2l+BNU2h7eC1t7WvZ4y/OR76GzPA1ZqdZmjjPIGKMXxnomE+rY99gn9MLiowNlOcNd+uKLHlcaGgojz32GA8//DB+v59bbrmFxMRE1q1bR1hYGJGRkUycOJGvv/6aGjVqMGTIEHr06MHWrVspVKhQ6nKeeeYZChcuTNGiRXnyySeJjIxMPaDniSee4KabbiIuLo4+ffqQP39+tm/fzrJly3jzzTctHV+PHj3StJwyZcqwd+9eNm/eTHR0NKGhoX/bUs0o+u+sSGYJCoNaveD+ldBvLdR5gGtDkhnq/JC1+QYx2DGHMJLYbUbzkGcgzdxjWeiri+/4TpjbG96uCz/MCUwxKHnWs88+y/Dhwxk9ejQVK1akefPmfPLJJ5QpU4bevXszcuRIatSoAcCoUaMoWrQo/fr1u2QZY8aMYdCgQdSsWZOjR4/y8ccf43K5gMB7hatXr2bXrl3Ur1+f6tWr8/TTT1OiRNbMcnWl8aVn5rd77rmH5s2b06hRI4oUKcKHH36YaXl19KyOnpWs5DkPWz8KTIKQcIAzZjDxvua8723JGQLvGV1rHGKgYwF32r7CbpgQeV1gy7PyPdrylHS5eOTp6dOnKViwoNVxcgWVpkpTrODzwJaZsOYlSNjPGTOYyb5mvO9tSSIFAChnHCbOsYDWtq9wGH4oUhFavgRl61scXnIKlWbGU2mqNMVKPg9snQVrXobTezlrBjPF15QJ3paBa4gCZY0jPOSYx122rwLnela+B25/FsJL/svCJa9TaWY8laZKU7IDnxd++AhWvwin93LODGKyrynve1tx+vfyrGnsZJQznsq2/YFJEm4dAjcNCJw/KiJZQqWp0pTsxOeFH2YHdtue2vN7eTbjLe9dJBOEgZ8u9hU85vgocNHswuWhxdjA5clEJNOpNFWakh35vLBtLqx5EU7+zFGzEC94OrPIfzMAhTjLEMcsOtpXBg4Wuv4OaPYCFLrG4uAiuZtKU6Up2ZnPC9+9Dyufh5QzfO2ryAhvT3aZpQCoauxhlDOe6rY94AiCWx6Bmx8Cpy6bJ5IZVJoqTckJzh2H5SNh83Q8pp2pvtsZ723HWUIA6GBfyROOmRQ2zkLBa6D5mMBFsTUhvEiGUmmqNCUnObgePn0MjmzhhBnGWG8n5vgC04+FkcSjjtl0sS8PnKJSoRXc+SoUiLI2s0guohmBRHKSUnWg70q4YzxFQuy87HyXua4R3GDs5Qz5GeHtyR3u59noj4Gdn8DbN8H2hVanlmyuYcOGDB48OM3Pj4+Pz7OnsKg0RXIamx1q3QcDN0HNXtS0/cwi11M85/iAcM7xk3kN7dwjGO25lwtJZ+Cj7jC3D5w/bXVykRxPpSmSU4VEBHa/3r8Se6ladHWsYGW+R7nHthoTG+/6WtPa/Rzb/GUCp7G8XRd2L7M6tUiOpquciOR0JarDfUthy4dELHuaccnv0tz3HUM9fdhllqKN+xniHAsYcGYhzuntoEYPaPY85Au1Ork1Fg2E4zuszRBVEVq/8a9Pa9iwIVWqVMFutzN58mRcLhfPPfccnTt3Ji4ujjlz5lC0aFHeeOMNWrRoAcDq1asZMmQIW7ZsISIigh49evDcc8/hcAT+3CclJdG/f3/mzZuXepWRv0pJSeHJJ5/kww8/JCEhgcqVKzN27FgaNmyYod+GnEilKZIb2GxQvQtc1wwWD+b2HR9T0/YET3nu41P/jbzqbccKXw1ecb5DzKbJ8MtKaPMOlLnF6uRZ7/iOwDVPc4jJkyfz+OOPs379embNmkX//v2ZP38+d999N8OGDWP8+PF069aNAwcOcPr0aVq2bEnPnj2ZMmUKP/30E3379iUoKIiRI0cCMGTIEFavXs3ChQuJiopi2LBhbNq0idjY2NR1xsXFsX37dmbOnEmJEiWYP38+zZs354cffiAmJsaab0Q2oaNndfSs5DamGdgd++ljmOcT+dhfl+GeXiRSABduHnPMprf908CkCDc9CLc9nbfO63y/ifWlGV0b+iz/16c1bNgQn8/Hl19+CYDP5yM8PJy2bdsyZcoUAI4ePUrx4sX5+uuv+fjjj5k7dy47duzA+P10o7fffpsnnniCxMREkpOTKVy4MNOmTaN9+/YAnDp1iujoaO6//35effVVDhw4QLly5Thw4MAllwdr0qQJderU4YUXXiA+Pp7BgweTkJCQwd+Y7E/vaYrkNoYBVTvAg99gxDShtf1rluZ7nEa273Hj4gVvF+51D2e/Pwq+eRv+Vx9+3Wh1armCqlWrpn5ut9spXLgwVapUSX2saNGiABw/fpwdO3ZQt27d1MIEuPnmmzl37hy//vore/bswe12c+ONN6Z+PSIiggoVKqTe/+GHH/D5fFx33XUUKFAg9bZ69Wr27NmTmUPNEbR7ViS3CisBXebAxniKfv4kE3mJj3wNecbbje/M62nuHsMwxwy6/rYc44PboeFQqP9oYFevZBtOp/OS+4ZhXPLYxYL0+/0Zsr5z585ht9vZuHEjdvul128tUKBAhqwjJ1NpiuRmhgG1ekG5hhgLB9Bx/yrq2bYxxPsA3/hvYLj3Ppb6a/GS812KrXwODnwFbSdA/kirk2eeqIpWJ8i0DBUrVmTu3LmYpplapuvWrSM0NJTo6GgiIiJwOp18++23lC5dGoDTp0+za9cubr01MElG9erV8fl8HD9+nPr1de3Wv1JpiuQFEWWhx2L49h1KLR/FDOMFJvuaMtZ7L1/6q9IyZTSvOt+iwZ4vArtr202Ea+panTpzpOGo1ZzqwQcf5NVXX2XgwIHExcWxc+dORowYwSOPPILNZqNAgQL07t2bIUOGULhwYaKionjyySex/WnvwnXXXUeXLl3o3r0748aNo3r16pw4cYIVK1ZQtWpVWrVqZeEIraf9MCJ5hc0GdQdAvy+xlYyll+NzPnEN4wZjL6cIo4fnCV7xtMN35gjEt4J1r0EG7fKTrFGyZEk+/fRT1q9fT7Vq1ejXrx+9e/fmqaeeSn3OSy+9RP369bnzzjtp0qQJt9xyCzVr1rxkOZMmTaJ79+48+uijVKhQgTZt2vDdd9+lbp3mZTp6VkfPSl7k88LaV2DVGC74bTzj7cYMX+CanHVtP/Ka802ijES4rgW0eTswkYKIqDRVmpKn7f8a5vSCs0dY6KvLUE9fkgkikgRed75JPft2CC8N7eMhuua/Lk4kt9PuWZG87Jq60G8tXNuYu+xfs8j1FBWMA/xGQbp6hvGGtw3+hIMwsRl887/AOaAieZi2NLWlKRJ47/LLcbDqBc77HTzt7clsX0MA6tu28qrzrcC1Oiu2hrvehKBwa/OKWESlqdIU+cMvqwNXREk6zhxffZ7y3McF8lGMk7zhepPatp1QqCx0mAzFq1mdViTLqTRVmiKXOnsM5vaGfV+y0x/Ng55B7DFLYsfHEMcs7rd/gs3hClxhJbaz1WlFspTe0xSRS4UWhW4LoMEQKtgOscj1FG1sa/FhZ4y3M309j5LotcOC/vD5k+D3WZ1YJMtoS1NbmiJX9vNymHc/ZtJJZvoaMcLbAzcuyhmHmeAcx7W2I1C+CdzzAQQXtDqtSKZTaao0Rf5Z4iGYcx8c/IZt/mt4wP0IhyhCKEm87nyTRvYtUDgGOs2EyPJWpxXJVNo9KyL/LLwk9FwMdeOobNvPwnzDqWXs5Cz56e0ZwgRvS8zfdsOExoEtU5FcTKUpIv/O7oRmz8Pd7xLpSGGG6zk62lfix8bz3q486unHhQvJML09fP2WzueUXEu7Z7V7ViR9ft0AM7tgnj3KZF9TnvV2w4edWGM377rGU9RIgNiucMcr4MhndVqRDKUtTRFJn+hacP9KjJLV6elYyhTnGMI5x2YzhtYpz7HFXw42T4PJd8K541anFclQKk0RSb+wEtBrCVRpz832H1nkeooY41eOEUF799Ms8N0MB7+F9xrC4c1WpxXJMCpNEbk6zuDABaubjOQa2wnmuUZwm20TblwM9gxgjOdefImHYWJz2DbP6rQiGUKlKSJXzzDgloeh00xC8zl4zzmOB+0LAfifrzV9PY9yxkPgSipfjtMBQpLj6UAgHQgkkjGO74AP74XT+1joq8vjngdIwUV541cmOV+klO03qNkLWr4MdofVaUWuirY0RSRjRFWEviuhbAPusn/NbNcoinGSn81o7nY/ww/+srBxEszsBCnnrE4rclVUmiKScUIioOs8qHM/VW17mZ9vBNcb+/mNgnRwD+cLXyzsXgrxLQMTw4vkMCpNEclYdie0fAlavkxxWwKzXc9Q37aV8wTRx/MY072N4cgWeL8JnNhpdVqRdFFpikjmqNMX7p1BqMtgovMl2ttX4cfGk94+jPV0xJ9wED64HfattTqpSJqpNEUk81RoAT0X4ywQwYuO93jYMRuAd3x3MdjzICnnk2Dq3fDDHIuDiqSNSlNEMlfJmtB7GUZkeQY55vOy8x0ceFnkv5nu7v8j0esMXPR67XidkiLZnkpTRDJfRFnovQxK16Wd/UvinS9SgGS+NStxj3sEB/2RsHwkfPIo+LxWpxW5IpWmiGSNkAjotgBuuJtb7Nsuf0rKhg9gVhdwJ1mdVuSyVJoiknWcQXDPRKg3kIq2g5c/JWXXZxDfCpJOWp1W5G9UmiKStWw2aPoctHiJ4sbpv52SMsPbGA5/DxObQcJBq9OKXEKlKSLWuPF+uHc6oU4uOSVlmLcPb3rvwvxtd6A4T+yyOqlIKpWmiFjn+laBU1LyF+JFx3vE2ecD8LK3I895u+JPPBwozkMbLQ4qEpArSvPuu++mUKFCtGvXzuooIpJe0bWgzzKMiLI85pzNU46pAHzga8ljnn54khNhcmv4ZZW1OUXIJaU5aNAgpkyZYnUMEblaEeXgvs+haBX6OJYwzvkOdnzM89enn+dhLqSkwPT2sH2R1Uklj8sVpdmwYUNCQ0OtjiEi/0VoUei5GErX5R77l7zrHE8+3Kzw1/h9EgQHzO4BGydbnVTyMMtLc/To0dSuXZvQ0FCioqJo06YNO3dqEmeRPCm4YOAqKTHNaGLfxFTXaEJJZr1ZkXvdT3HcHwofP6TZg8Qylpfm6tWrGTBgAN988w3Lli3D4/HQtGlTkpICJzfHxsZSuXLlv90OHz6c7nWlpKRw5syZS24iks24QuDe6VC1I3VsO5npepZIEthhlqG9ewQH/UUCswctfUrFKVnOMM3s9a/uxIkTREVFsXr1aho0aJDm161atYo333yTOXOuPPHzyJEjGTVq1N8eT0xMJCws7Kryikgm8fvh82Hw7Tvs8xelq2cov5pRRHGaKa4xXG87CLFd4M7Xwe6wOq3kEZZvaf5VYmIiABERERm+7KFDh5KYmJh6O3hQJ06LZFs2GzQfDY2eooztGHNdI6lgHOA4hejgfpqN/hjYPB0+6g6eC1anlTwiW21p+v1+WrduTUJCAmvXpv0ae02aNGHLli0kJSURERHB7NmzqVu37r++7syZM4SHh2tLUyS7++59+OQxEs0Q7nM/xkazAkGk8I7zVRrZt0CZ+tBpJuQrYHVSyeWyVWn279+fJUuWsHbtWqKjozN9fSpNkRxk2zyYdz/JPhsPegaxyh+LAy/jnW9zp/0bKHUjdJkNQeFWJ5VcLNvsno2Li2Px4sWsXLkySwpTRHKYym2h80xCXHYmOMdxl20dXhwM8sQx29sADn4bmAQh+ZTVSSUXs7w0TdMkLi6O+fPn88UXX1C2bFmrI4lIdlW+CXRfhDM4lPHOt+lkX4EfG0O8/ZjqbQJHNgeukHLuuNVJJZeyvDQHDBjAtGnTmDFjBqGhoRw9epSjR49y/vx5q6OJSHZUqjb0WoKtQBFecHxAL/sSAIZ772OCtyUc3w6TWkDiIYuDSm5k+XuahmFc9vFJkybRs2fPTF233tMUycF++xmmtMZMPMRL3o687bsLgEccsxlon49R6BrosQgKlbE2p+QqlpemlVSaIjnc6f0w+U5I2M+b3rt42dsRgP72hTzumIURXhK6L4LI8hYHldzC8t2zIiJXrdA1cN9nUDiGOMdCnnJMA+Ad312M8nYPXFpsUgs4tt3ioJJbqDRFJGcLKwG9PoWoG+jj+JRnHRMBiPc1Z5i3N75zJwIHBx3ebG1OyRVUmiKS8xWIClwhpUR1ujmW87LzHWz4melrzKOe/niTEwKnoxxcb3VSyeFUmiKSO4REQPeFUOom2tm/5HXnGzjwssB/CwM9A3FfOAdT2sDeL61OKjmYSlNEco+gcOg6F8o24A77t7zjfBUXHpb4b+QBzyNccLthejv4eYXVSSWHUmmKSO6SrwB0/ghimnK7fRPvO18miBRW+qvT2zOEZI8JH3aC3cutTio5kEpTRHIfZzB0nA4V76SB/QfiXWPJz3nW+SvTyz2EZC8wsxPsWmp1UslhVJoikjs5XNAuHqq05ybbT0xxjaEAyXxrVqKn+3GSvAbM6gK7Prc6qeQgKk0Ryb3sDrj7XajelZq23UxxjSGUZNabFenpfoJzXhvM7AI7l1idVHIIlaaI5G42O9z5BtToTg3bz0x1jSaUJL4zrw8Up88Os7rBT59anVRyAJWmiOR+Nhvc8RrU7EmsbQ/Tfi/ODWYFerif4KzPAR91h58+sTqpZHMqTRHJG2w2aDUeavaimu0XprteIIwkNpoV6O7+P85cLM4dH1udVLIxlaaI5B02G7R6BWr1pqptLzNczxPOOb43Y34vTifM7gnbF1mdVLIplaaI5C02G7QaB7X7Utm2j+mu5ynIWTabMXRzDyXR5woU548LrE4q2ZBKU0TyHsOAli9BnQeobNvPdNcLFOQsW8zygeL0B8Gc++DH+VYnlWxGpSkieZNhQIuxcGN/brDtZ4breQpxlq3mtXRNLc7esG2u1UklG0nTRairV6+OYRhpWuCmTZv+c6isootQiwimCZ8Pg2/eZoe/FF3cT3KKMCobe5nmeoGCtvNwzwdQua3VSSUbSNOWZps2bbjrrru46667aNasGXv27CFfvnw0bNiQhg0bEhQUxJ49e2jWrFlm5xURyViGAc1egLpxVLQd5EPXcxQmkW1mWbq4h5HgD4a5fXRwkABp3NL8sz59+lC8eHGeffbZSx4fMWIEBw8eZOLEiRkaMDNpS1NEUpkmLBsOX73BLn9JOruf4jfCqWrsYaprNOF2N3SYCte3tDqpWCjdpRkeHs6GDRuIiYm55PHdu3dTq1YtEhMTMzRgZlJpisglTBOWj4B1r7HTH00n91OcIoxYYzdTXWMItXvh3hlwXVOrk4pF0n0gUHBwMOvWrfvb4+vWrSMoKChDQomIWMIwoMkoqDeQCrZfA+9p/n46Si/34yT5bDCrq67HmYc50vuCwYMH079/fzZt2kSdOnUA+Pbbb5k4cSLDhw/P8IAiIlnKMOD2Z8HnpdK37zDNNZrO7ifZYFagl/tx4nmRkJmdofMsKNfQ6rSSxdK9exbgo48+4rXXXmPHjh0AVKxYkUGDBtGhQ4cMD5iZtHtWRK7INOHTIfDdBLb4y9HVPYyzhFDX9iMTnS8R7LRDl9lQtr7VSSULXVVp5hYqTRH5R34/fPIwbIxnoz8w1V4SwdS3bWWCcxxBLhd0nQvX1LU6qWSRdL+nWa5cOU6ePPm3xxMSEihXrlyGhBIRyRYuTvL++/U4410vEsIFvvRX5QHPw6S4U2B6Ozi43uqkkkXSXZr79u3D5/P97fGUlBQOHTqUIaFERLINmw3ufB2q3ktt204mul4kiBRW+2N50DMYd8p5mHYP/LrR6qSSBdJ8INCiRX+c2Pv5558THh6eet/n87FixQrKlCmToeFERLIFmx3avA2mj5t+mM1E50v08jzOCn8NBnoG8iZv4Jx2N3RfBCVirU4rmSjN72nabIGNUsMw+OtLnE4nZcqUYdy4cdxxxx0ZnzKT6D1NEUkXnxfm9YEf57PGV4U+nkdx46KV7Rtec76JIzgMei6GYlWsTiqZJN0HApUtW5bvvvuOyMjIzMqUZVSaIpJuPg/M6QU7PmalL5YHPA/jxklr2zrGO9/Gnj8Cen4KUddbnVQyQbrf09y7d+/fCjMhISGj8oiIZG92J9wzEa5rQSP7Zt52voYDL4v8NzPE8wD+pFMwpTWc3GN1UskE6S7NsWPHMmvWrNT77du3JyIigpIlS7Jly5YMDSciki05XNBhMsQ0pYl9E286X8eOj3n+Bjzl7YV59hhMbg0JB6xOKhks3aX5v//9j1KlSgGwbNkyli9fzmeffUaLFi0YMmRIhgcUEcmWHPkCE7hf25jm9g2Md76NgZ8ZviY86+2KmfhroDjPHLE6qWSgdJfm0aNHU0tz8eLFdOjQgaZNm/L444/z3XffZXhAEZFsyxkEHadDmfq0tn/NWMcEACb6WjLO2x5O7w3sqj13wuKgklHSXZqFChXi4MGDAHz22Wc0adIEANM0L3v+pohIruYKgU4fQnRtOjhW84xjEgBv+u7mLe9d8NsumHo3JJ+yOKhkhHSXZtu2bencuTO33347J0+epEWLFgB8//33lC9fPsMDiohke/lCocscKFaV7o5lDHXMAOAlb0c+8DaHYz8EZg66cMbioPJfpbs0x48fT1xcHJUqVWLZsmUUKFAAgCNHjvDggw9meEARkRwhuCB0WwBFKvKAYzGDHXMAeNbbnRnexnBoI8zoCO4kS2PKf6MJ23WepohkpLPHYFILzJN7GOPtxLu+OzHwM875P9ra1wYuJ9ZpVuD9UMlx0r2leSVHjhzhwAEdXi0ieVxoUeixCKNgaf7P8SHd7UsxsfGYpx+f+urAL6tgdg/wuq1OKlch3aXZo0cP1qxZ87fHGzduTNmyZTMklIhIjhYeHSjOsOKMdEymvX0Vfmw85InjC18s7PoM5vUNTMsnOUq6SzMxMZEmTZoQExPDCy+8kHplkylTpvDFF19keEARkRwpoix0X4Qtf2HGOCZwp+0rvDjo5xnMWl9l2L4AFg4IXLNTcox0l+aCBQs4dOgQ/fv3Z9asWZQpU4YWLVqwb98+6tWrlxkZRURypiLXQfeF2IPDecX5DrfbNuDGRV/PI3znrwBbZ8Inj0DePbQkx7mq9zSLFCnCI488wpYtW/j2228pX7483bt3p0SJEjz88MPs3r07o3OKiORMxSpDt3k484XwpvN1Gti2cJ4germHsMVfDjZOguUjVJw5xH86EOjIkSMsW7aMZcuWYbfbadmyJT/88AOVKlVi/PjxGZVRRCRnK1kTuswmn8vFu87x3GT7kXOE0MP9BDv90bDuNfhynNUpJQ3SfcqJx+Nh0aJFTJo0iaVLl1K1alX69OlD586dU0/bmD9/Pvfddx+nT5/OlNAZRaeciEiW+mUVTO/AOa9BF/cwtpjlKcJp5rhGcY3tOLR4EW58wOqU8g/SXZqRkZH4/X46depE3759iY2N/dtzEhISqF69Onv37s2onJlCpSkiWW7nZzCrCwm+fNzrHs5PZmmijePMdj1DceMUtHkHYjtbnVKuIN2lOXXqVNq3b09QUM4/MVelKSKW+GEOzO3DcTOMDu4R7DOLca1xiI9cz1DYlgTtJ0Ol1lanlMvQjEAqTRGxwsZ4+HgQv5qRtE8ZwREKc4Oxlxmu5wm3e6DzTCjfxOqU8hcZNiOQiIikQ82e0PQ5oo3fmOZ6gcIk8qNZlt7uIST7bDCzK+z/2uqU8hcqTRERq9QbCA0e51rbEaa4xhBKEhvMCjzgeZgUjwdmdIDDm61OKX+i0hQRsVKjYXBjf26w7Sfe9SLBXOBLf1UGeeLwXjgH09rCiZ1Wp5TfqTRFRKxkGNDsBYjtSk3bbt5zvoILD5/56/CEpy/+pFMwpQ2c3md1UkGlKSJiPZsNWr8Ole6ivn0bbzjfwI6Puf5becbbDfPMYZhyF5w5YnXSPE+lKSKSHdjs0PZ9KN+EZvYNvOR8F4B4X3PGedsHtjSntoHkU5bGzOtUmiIi2YXDBR2mQul6tLWv5VnHRADe9N3Nu9474MRPML0dpJy1OGjepdIUEclOXCGBczSLx9LNsZwhjpkAjPZ25kNvIzi0EWZ2Bs8Fi4PmTTm+NA8ePEjDhg2pVKkSVatWZfbs2VZHEhH5b4LCoes8KHI9AxyL6GdfBMAwb28+8d0Ie9fA3N66iLUFcvyMQEeOHOHYsWPExsZy9OhRatasya5du8ifP/+/vlYzAolItnbmCExshnl6P8O8ffjQ1xgnXj5wvkQD+w8Q2xXuejNwBK5kiRy/pVm8ePHUSeOLFStGZGQkp07pjXIRyQXCikP3BRihRXnO8QGtbN/gwcEDnofZ6I+BzdNg6VO6FmcWsrw0R48eTe3atQkNDSUqKoo2bdqwc+fVnci7ceNGfD4fpUqVyuCUIiIWiSgHXedhDw5jvPMt6tu2cp4g7nMPCVyL8+s3Ye0rVqfMMyzfPdu8eXPuvfdeateujdfrZdiwYWzbto3t27eTP39+YmNj8Xr/vt9+6dKllChRIvX+qVOnqF+/PhMmTKBevXqXXVdKSgopKSmp98+cOUOpUqW0e1ZEsr8D38LUNiS7fXR1D2WTeR1Rv1+Ls7TtOLR6BWr3tjplrmd5af7ViRMniIqKYvXq1TRo0CBNr0lJSeH222+nb9++dOvW7YrPGzlyJKNGjfrb4ypNEckRfl4OM+4lweeio3s4O83SlDaOMcc1iigjEdp9AJXvsTplrmb57tm/SkxMBCAiIiJNzzdNk549e9K4ceN/LEyAoUOHkpiYmHo7ePDgf84rIpJlyjeBtu9R0EhmqmsMpY1jHDCL0t39fySaITDvfti93OqUuVq22tL0+/20bt2ahIQE1q5dm6bXrF27lgYNGlC1atXUx6ZOnUqVKlX+9bU6elZEcqQNk2DxYA74o7jHPYITFKKGsYtprtGEOG3QfQGUvsnqlLlStirN/v37s2TJEtauXUt0dHSmr0+lKSI51pevwIpR/OQvRQf305whPw1sW3jf+TKu4ALQ81MoVtnqlLlOttk9GxcXx+LFi1m5cmWWFKaISI52y8NQ7yGutx1k0u+XFFvjr8YjngfxnT8DU++Gk3usTpnrWF6apmkSFxfH/Pnz+eKLLyhbtqzVkUREsj/DgNufgerdqGnbzbvO8Tjxsthfl+HeXpjnjgcmeNeVUTKU5aU5YMAApk2bxowZMwgNDeXo0aMcPXqU8+fPWx1NRCR7Mwy48zWo2JoG9h941fkWBn5m+JrwkrcjJBwIXMT6/Gmrk+Yalr+naVxh+qdJkybRs2fPTF233tMUkVzBmwIzOsAvq/jQ24ih3r4APOWYSh/HEih1E3SbH5gMXv4Ty0vTSipNEck1Us4FLlR9aANve+/kRW8nAF5xvk1b+1qIaQb3Tge70+KgOZvlu2dFRCQD5CsAXWZDkevpb/+Y3vZPARjieYAvfLGw+3NYGAd+v7U5cziVpohIbhESAV3nYRQsxZOO6bS1fYkPOw96BrHBfx1snakJ3v8jlaaISG4SXhK6zceWP4KxzvdobNvEBfJxn3sIP/lLwTdvwdrxVqfMsVSaIiK5TWQMdJmDM18wbzlfp5axkzPkp7v7/zjoLwIrRsHGyVanzJFUmiIiuVHJGnDvdIId8IHrZSoYBzhOIbp5hvKbGQaLB8P2RVanzHFUmiIiuVW5htB2AuFGMlNcY4g2jrPPLEYP9xOc9eeDub1h7xqrU+YoKk0RkdzshjbQahxFjQSmOscQSSI/mmW53/MIF7wmfNgZDm+2OmWOodIUEcntaveGRk9S1naUeNdYCpDM1/4bGOwZgC/lHEy7R/PUppFKU0QkL2gwBOo8QGXbPt5zvoILD5/56/Cktzdm0m+apzaNVJoiInmBYUDzMVC5HfXs23nd+SY2/Mz0NeZlbwfNU5tGKk0RkbzCZoM278C1t9Hc/h3POz4A4C1fGz7wNofj2+HDTuDRBTOuRKUpIpKXOFzQcSqUrEUnx0qGOGYC8Ky3Owt8N8OBr2HOfeDzWhw0e1JpiojkNa78gXlqIyvwoH0RvexLAHjM8wBrfFVg56fwycOabu8yVJoiInlRSAR0m48RHs1wxzRa29bhxUE/z8Ns8ZeDTVNg5fNWp8x2VJoiInlVeEnoNg9bSEFedv6P+ratJBNEL/fj7PEXhzUvwbfvWp0yW1FpiojkZUUqQOfZuJwu3nG+SlVjD6cIo7v7/zhmFoQlT8C2uVanzDZUmiIieV2p2tBhCgVsHia6XqKscYRDFKGH+/9ININh3gOwZ6XVKbMFlaaIiMB1TeGut4g0zjDFOYYinOYnszR93Y9ywQfM6qrp9lBpiojIRbGd4PZnKGU7wRTXGEJJZr1ZkYc8cXhTkmF6uzw/3Z5KU0RE/lDvIagbR0XbQSa4XsaFm6X+2gz33od57kRg1qCzx6xOaRmVpoiI/MEw4PZnoWpHbrL9xOvOt7Dh50NfY8Z728HpfYEJ3i8kWp3UEipNERG5lM0Gd70F5ZvQ3P4dzzomAvC6ry1TvLfDsR9gZhfwXLA4aNZTaYqIyN/ZndB+MpSsSRfHFzzsmA3ACG8PPvHdCPu+hPn3g99ncdCspdIUEZHLy1cAOs+GwjE8ZJ9PN/tSTGw87HmQr3yVYPvCwHmceWi6PZWmiIhcWf7C0G0eRlhxRjom09L2LW6c3O95hB/918B3E2DtK1anzDIqTRER+WcFS0PXudiDw3jF+TY32X7kHCH0dD/OQX8RWPEMfD/d6pRZQqUpIiL/rugN0GkmQQ4b7zlf4XpjPycoRHfP/3HSDIVFA2HXUqtTZjqVpoiIpM019aDdB4TZUpjsGktJTrDXLM597iEk+x0wuwf8usHqlJlKpSkiImlX8U5o+TJFjQQmu8ZSkLNsMcszwPMQHncKTG8Pv/1sdcpMo9IUEZH0qd0bGjxOedthPnC9TBAprPRXZ6i3D2byKZh2N5w9anXKTKHSFBGR9Gs0DGp0p6ZtN285X8eOjzm+W3nZ2wESDsC0drly1iCVpoiIpJ9hQKvxcF0LbrN/zwuODwB4y9eGeG/TwKxBs7qCN8XioBlLpSkiIlfH7oB2EyG6Dh0dq3jU8REAo7zdA7MG7V0D8x8Av9/ioBlHpSkiIlfPFQKdZ0HkdcTZF9DVvix11qCvfRXhx/nw+dBcM2uQSlNERP6bkAjoOhcjrDijHPE0t63/fdagR9nhLwXf/g/WvWZ1ygyh0hQRkf/u4qxBQWG86nyLOsYOzhJCD/f/8asZCctHwOYPrU75n6k0RUQkYxS9ATrNIMhhY4LrFa4zDnKcQnR3/x+nzQKwKA5+XmF1yv9EpSkiIhmnzC1wzwTCjWQmu8ZSgt/4xSxBb/djnPfZ4KPucHiz1SmvmkpTREQyVqW7oOVLFDdOEe8aSxhJbDKvY6AnDm9KcmDWoFN7rU55VVSaIiKS8er0hfqPcp3tEO+7XsaFm+X+Wjzt7Yl57jhMuweSTlqdMt1UmiIikjkaD4dqnalj28nrzrcw8DPD14Q3fW3g1B6Y0QHcyVanTBeVpoiIZA7DgNavw7W30dz+HSMdUwAY5+3AR95b4dAGmNMLfF6Lg6adSlNERDKP3QkdpkDxWHo4ltLfvhCAod4+rPRVg12fwSeP5JjJD1SaIiKSufIVgC6zoVAZHnfMoq1tDT7sPOgZxBZ/Odg0GVaPtTplmqg0RUQk8xWIgq7zMPIXZqxzAvVtWzlPEPe5h7DPXxRWjYaNk61O+a9UmiIikjUKXwudZ+N05eMd56tUNvZyknB6eJ7gNzMMFj8MOz+zOuU/UmmKiEjWia4J7eMpYPMw0fUipYzj7DeLcZ97CEl+B8zuCb9usDrlFak0RUQka13XDO58jSgjkcnOMURwhq3mtQzwDMLjcQdORfntZ6tTXpZKU0REsl6NbtDoScrZjvKB6yWCSGGVP5ah3j6YSSdhWls4e8zqlH+j0hQREWs0GAI1e1Ldtoe3nK9jx8cc36284m0PCfsDW5wp56xOeQmVpoiIWMMwoOU4qNCS2+zf87zjAwDe8N3NdG9jOLIZZvcAn8fanH+i0hQREevYHXDPBxBdm3sdqxjsmAPAcO99LPPVgJ+Xw8eDs83kBypNERGxlisEOs2CwuUZZJ9HR/tK/NgY6BnIJn952DwtcB5nNqDSFBER6+UvDF3nYhSI4jnHRBraNnOBfPRxP8Zef7HAjEEb461OqdIUEZFsolAZ6PIRTlcQbzlfo4rxC6cIo4fnCU6YYbD4Edj1uaURc3xpJiQkUKtWLWJjY6lcuTITJkywOpKIiFytEtWhwxTy27xMdL1IaeMYB8yi9P7z5AeHNloWzzDNbPLu6lXy+XykpKQQEhJCUlISlStXZsOGDRQuXPhfX3vmzBnCw8NJTEwkLCwsC9KKiEiafD8dFj7IL/5i3OMexWlCaWT7ngnOcTjyR0DvpYFp+bJYjt/StNvthISEAJCSkoJpmuTw/weIiEj1Ln+b/GClvzpPentjJv0G09tB0m9ZHsvy0hw9ejS1a9cmNDSUqKgo2rRpw86dO9O1jISEBKpVq0Z0dDRDhgwhMjIyk9KKiEiWaTAEavSghu1n3nC+gQ0/s3yNeM3XFk79Epj8wJ2UpZEs3z3bvHlz7r33XmrXro3X62XYsGFs27aN7du3kz9/fmJjY/F6/35V76VLl1KiRIlLHjt27Bht27Zl3rx5FC1a9G+vSUlJISUlJfX+mTNnKFWqlHbPiohkVz4vzOwMuz9nmvc2nvL2BmCs4z06OlbBdc2h4/TA+Z5ZwPLS/KsTJ04QFRXF6tWradCgQbpf/+CDD9K4cWPatWv3t6+NHDmSUaNG/e1xlaaISDbmToL4O+DwJl7ydOAtXxvs+HjfOY5G9s1Qsyfc8WpghqFMZvnu2b9KTEwEICIiIk3PP3bsGGfPnk197Zo1a6hQocJlnzt06FASExNTbwcPHsyY0CIiknlc+aHzR1CoLI85PqKtbQ0+7DzoeYit/rKB8zfXvJwlUbLVlqbf76d169YkJCSwdu3aNL1m/fr13H///akHAA0YMIAHHnggTa/V0bMiIjnIyT3wwe24kxLo7RnCl/6qRJLIPNcIStuOw11vBw4gykTZqjT79+/PkiVLWLt2LdHR0Zm+PpWmiEgO8+sGiL+Dsx7o6B7OdrMMMcavLHH9Hw6bAd3mQ7lbM2312Wb3bFxcHIsXL2blypVZUpgiIpIDRdeC9vGE2lKY5HqRCsYBnnJMw2H4oVQdKFYlU1dv+ZamaZoMHDiQ+fPns2rVKmJiYrJs3drSFBHJoTZMgsWD8ZkGdsOEiq2h7QRwBmXqarPmGN1/MGDAAGbMmMHChQsJDQ3l6NGjAISHhxMcHGxxOhERyZZq9YIzh7CveQlu7AfNXgCbPdNXa/mWpnGFQ4QnTZpEz549M3Xd2tIUEcnBTDNwvc3yTbLkdBPIBlua2eg4JBERyUkMA2Juz9JVZpsDgURERLI7laaIiEgaqTRFRETSSKUpIiKSRpYfCGSliwchnTlzxuIkIiJitdDQ0Cue0XFRni7NixO9lypVyuIkIiJitbScfmj5eZpW8vv9HD58OE3/u7iSi9fkPHjwYK4/11NjzZ3yyljzyjhBY71a2tL8FzabLcPmuQ0LC8v1/zgv0lhzp7wy1rwyTtBYM4MOBBIREUkjlaaIiEgaqTT/o3z58jFixAjy5ctndZRMp7HmTnllrHllnKCxZqY8fSCQiIhIemhLU0REJI1UmiIiImmk0hQREUkjlaaIiEgaqTRFRETSSKWZiRISEqhVqxaxsbFUrlyZCRMmWB0p0xw8eJCGDRtSqVIlqlatyuzZs62OlKnuvvtuChUqRLt27ayOkmEWL15MhQoViImJ4f3337c6TqbKjT+/y8krv5dZ+bdWp5xkIp/PR0pKCiEhISQlJVG5cmU2bNhA4cKFrY6W4Y4cOcKxY8eIjY3l6NGj1KxZk127dpE/f36ro2WKVatWcfbsWSZPnsycOXOsjvOfeb1eKlWqxMqVKwkPD6dmzZp89dVXufLfKuS+n9+V5JXfy6z8W6stzUxkt9sJCQkBICUlBdM0ya3/RylevDixsbEAFCtWjMjISE6dOmVtqEzUsGFDQkNDrY6RYdavX88NN9xAyZIlKVCgAC1atGDp0qVWx8o0ue3ndyV55fcyK//WqjT/wejRo6lduzahoaFERUXRpk0bdu7cma5lJCQkUK1aNaKjoxkyZAiRkZGZlPa/yYixXrRx40Z8Pl+2veRaRo41J/mncR8+fJiSJUumPrdkyZIcOnTIqqj/WV76Gad1rNn99/Lf/Ns4s+pvrUrzH6xevZoBAwbwzTffsGzZMjweD02bNiUpKQkgdf/5X2+HDx9OXUbBggXZsmULe/fuZcaMGRw7dsyq4fyjjBgrwKlTp+jevTvvvfeeFcNIk4waa07zb+POTTTWS8eaE34v/82/jTPL/taakmbHjx83AXP16tVX9fr+/fubs2fPzuBUmeNqxnrhwgWzfv365pQpUzIxWca72p/rypUrzXvuuSeTUmW+P4973bp1Zps2bVK/NmjQIHP69OkWpstYl/sZ5/Sf35X8daw59ffy3/zT721m/q3VlmY6JCYmAhAREZGm5x87doyzZ8+mvnbNmjVUqFAh0/JlpPSO1TRNevbsSePGjenWrVtmRstw6R1rbvHncdepU4dt27Zx6NAhzp07x5IlS2jWrJnFCTNOXvoZ/3msOfn38t/8eZxZ+rc2U6o4F/L5fGarVq3Mm2++Oc2v+fbbb81q1aqZVatWNatUqWL+73//y8SEGedqxvrll1+ahmGY1apVS71t3bo1E1NmjKsZq2ma5m233WZGRkaawcHBZsmSJc2vvvoqkxJmjsuNe+HChWZMTIx57bXXmu+++66F6TLW5caa039+V/LXsebU38t/89dxZuXfWpVmGvXr18+85pprzIMHD1odJdNprLlfXhq3xpr7WDlOlWYaDBgwwIyOjjZ/+eUXq6NkOo0198tL49ZYcx+rx6nS/Ad+v98cMGCAWaJECXPXrl1Wx8lUGmvul5fGrbHmPtllnI7Meac0dxgwYAAzZsxg4cKFhIaGcvToUQDCw8MJDg62OF3G0lhz51j/LC+NW2PNfWPNNuO0rK5zAOCyt0mTJlkdLcNprLlzrH+Wl8atsea+sWaXcWruWRERkTTSeZoiIiJppNIUERFJI5WmiIhIGqk0RURE0kilKSIikkYqTRERkTRSaYqIiKSRSlNERCSNVJoiIiJppNIUkSsaOXIksbGxVscQyTZUmiKSZj179qRNmzZWxxCxjEpTREQkjVSaIrlEw4YNGThwIIMHD6ZQoUIULVqUCRMmkJSURK9evQgNDaV8+fIsWbIEgPj4eAoWLHjJMhYsWIBhGJdd/siRI5k8eTILFy7EMAwMw2DVqlWZPCqR7EWlKZKLTJ48mcjISNavX8/AgQPp378/7du3p169emzatImmTZvSrVs3kpOT073sxx57jA4dOtC8eXOOHDnCkSNHqFevXiaMQiT7UmmK5CLVqlXjqaeeIiYmhqFDhxIUFERkZCR9+/YlJiaGp59+mpMnT7J169Z0L7tAgQIEBweTL18+ihUrRrFixXC5XJkwCpHsS6UpkotUrVo19XO73U7hwoWpUqVK6mNFixYF4Pjx41meTSQ3UGmK5CJOp/OS+4ZhXPLYxfcr/X4/NpuNv16D3uPxZH5IkRxMpSmSRxUpUoSzZ8+SlJSU+tjmzZv/8TUulwufz5fJyUSyL5WmSB514403EhISwrBhw9izZw8zZswgPj7+H19TpkwZtm7dys6dO/ntt9+0ZSp5jkpTJI+KiIhg2rRpfPrpp1SpUoUPP/yQkSNH/uNr+vbtS4UKFahVqxZFihRh3bp1WRNWJJswzL++qSEiIiKXpS1NERGRNFJpioiIpJFKU0REJI1UmiIiImmk0hQREUkjlaaIiEgaqTRFRETSSKUpIiKSRipNERGRNFJpioiIpJFKU0REJI3+Hw/1Mk9qy1UJAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = torch.randn(int(1e6)).requires_grad_()\n",
    "mults = 2**torch.linspace(-3, 3, 31)\n",
    "scales, grad_scales = [], []\n",
    "for mult in mults:\n",
    "    x.grad = None\n",
    "    y = F.hardtanh(x, -1/mult, 1/mult)\n",
    "    y.backward(torch.randn_like(y))\n",
    "    scales.append(y.std(correction=0).item())\n",
    "    grad_scales.append(x.grad.std(correction=0).item())\n",
    "\n",
    "model_scales = [\n",
    "    sqrt(mult**-2 + (1 - mult**-2) * erf(1/(mult*sqrt(2))) - sqrt(2/pi) * mult**-1 * exp(-1/2 * mult**-2))\n",
    "    for mult in mults\n",
    "]\n",
    "_, ax = plt.subplots(figsize=(5, 3))\n",
    "ax.plot(mults, scales, label=\"experiment\", zorder=1)\n",
    "ax.plot(mults, model_scales, lw=4, label=\"model\", zorder=0)\n",
    "ax.set_xscale(\"log\", base=2); ax.set_yscale(\"log\", base=2); ax.legend()\n",
    "ax.set_xlabel(\"mult\"); ax.set_ylabel(\"y.std\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks pretty solid.\n",
    "\n",
    "Now for the backwards pass. Note that in this case we just feed in `y.grad = 1` in order to obtain the partial derivatives $\\partial y / \\partial x$ that define the scaling behaviour:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAGaCAYAAADKE66pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAvXUlEQVR4nO3df3hU5Z3//9fMZDKTnxMiISGQgIhiEaGCxootQkVBsyxuERe/7W4i1oof9LMWay+wSKq7mFZttR+gYKsSukqx2kVbqFxYFKUiYkW6RkRFiGggAQLM5Ockmbm/f4SMDEkQuAOThOfjuua6cu5z32fek5mTV865Z844jDFGAABYcMa6AABA90eYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJzriqqir16dNHZWVlJzxm2rRp+sUvfnH6igJghTA5SxUWFsrhcMjhcMjtduvcc8/Vj3/8YzU0NLTpM2PGjDbjZ86cKYfDocLCwkjb/v37dccddyg3N1cej0dZWVmaMGGC3nzzzaix8+fP1+TJkzVw4MATrnfu3LmaP3++/H7/ST/W43njjTc0adIkZWdny+Fw6MUXXzyhcYsWLdLAgQPl9Xp1+eWXa/PmzafUB+gpCJOz2MSJE7V3717t3LlTjz32mJ544gkVFRVF9cnJydGKFStUX18faWtoaNDy5cuVm5sb1XfKlCl67733tGzZMn388cf605/+pLFjx6qqqirSp66uTk899ZRuvfXWk6p12LBhOu+88/TMM8+cwiPtWG1trUaMGKFFixad8JjnnntOs2bNUlFRkbZs2aIRI0ZowoQJ2rdv30n1AXoUg7NSQUGBmTx5clTbd77zHXPJJZe06TNs2DDzzDPPRNqfffZZM3z4cDN58mRTUFBgjDHm0KFDRpJZv379ce/3+eefNxkZGVFty5cvN16v1+zZsyfSVlhYaC6++GJz+PDhSNsDDzxgvvnNb57sQz1hkszKlSu/sl9eXp6ZOXNmZDkUCpns7GxTXFx8Un2AnoQjE0iSSktLtXHjRsXHx7dZN336dC1dujSy/PTTT+uWW26J6pOcnKzk5GS9+OKLCgaDHd7Phg0bNGrUqKi2adOm6YILLtBDDz0kSSoqKtJf//pXvfzyy/L5fJF+eXl52rx5c7vbf+ihhyI1dHTbvXv3if0yjqOxsVHvvvuuxo8fH2lzOp0aP3683nrrrRPuA/Q0cbEuALGzatUqJScnq7m5WcFgUE6nUwsXLmzT73vf+57mzJmjzz77TJL05ptvasWKFVq/fn2kT1xcnEpKSnTbbbdpyZIlGjlypK666ipNmzZNw4cPj/T77LPPlJ2dHbV9h8Oh+fPn68Ybb1RWVpYWLFigDRs2qF+/flH9srOz1djYqIqKCg0YMCBq3YwZM3TTTTcd9/Eee7+n4sCBAwqFQsrMzIxqz8zM1Pbt20+4D9DTECZnsXHjxmnx4sWqra3VY489pri4OE2ZMqVNv4yMDOXn56ukpETGGOXn56t3795t+k2ZMkX5+fnasGGDNm3apJdfflkPP/ywnnzyychEfX19vbxeb5ux//RP/6ShQ4fqwQcf1Nq1a3XRRRe16ZOQkCCpZd7lWOnp6UpPTz/ZXwGATsJprrNYUlKSBg8erBEjRujpp5/W22+/raeeeqrdvtOnT1dJSYmWLVum6dOnd7hNr9era665Rvfff782btyowsLCqEn93r1769ChQ23GrVmzRtu3b2/3P/pWBw8elNQSbsc6U6e5evfuLZfLpcrKyqj2yspKZWVlnXAfoKchTCCp5Zz+fffdp7lz50a9c6vVxIkT1djYqKamJk2YMOGEtzt06FDV1tZGli+55BJt27Ytqs+WLVt000036amnntLVV1+t+++/v91tlZaWqn///u0eFc2YMUNbt2497q0zTnPFx8dr1KhRWrduXaQtHA5r3bp1uuKKK064D9DTcJoLEVOnTtW9996rRYsW6Uc/+lHUOpfLpQ8//DDy87Gqqqo0depUTZ8+XcOHD1dKSor+/ve/6+GHH9bkyZMj/SZMmKA5c+bo0KFD6tWrl8rKypSfn6/77rtPN998swYNGqQrrrhCW7Zs0ciRI6PuY8OGDbr22mvbrf1UT3PV1NRox44dkeVdu3Zp69atSk9Pj7z1eeHChVq5cmUkHGbNmqWCggJdeumlysvL0+OPP67a2tqoNyWcSB+gR4n128kQG+29NdgYY4qLi01GRoapqanpsE+ro98a3NDQYGbPnm1GjhxpfD6fSUxMNEOGDDFz5841dXV1UePy8vLMkiVLTFVVlRkyZIi5/fbbo9Zff/31ZsKECVFt9fX1xufzmbfeeuuUHm9HXnvtNSOpza31cRljTFFRkRkwYEDUuAULFpjc3FwTHx9v8vLyzKZNm9ps+0T6AD2FwxhjYppmOOusXr1a9957r0pLS+V0ntiZ1sWLF2vlypVau3btaa4OwKngNBfOuPz8fH3yyScqLy9XTk7OCY1xu91asGDBaa4MwKniyAQAYI13cwEArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACscTkVdDvnXzhUe8rLT3l8dr9++mT7tq/uCOCEESbodvaUl2vSL9ac8vg/3zOxE6sBIBEmOMNsjyokqaGh7Zd3AYgtwgRnlO1RhST94f+M6aRqAHQWJuABANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1LqeCs05DMKikFJ/VNrjyMBCNMMFZx4RD1tcH48rDQDROcwEArHFkghPG5eMBdIQwwQnj8vEAOsJpLgCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDW+aRE4BQ3BoJJSfFbbyO7XT59s39ZJFQGxRZicJfj+9s5lwiHrrzD+8z0TO6kaIPYIk7ME398O4HRizgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMAEAWONre4EYaQgGlZTis9pGdr9++mT7tk6qCDh1hEk3cf6FQ7WnvPyUxzc01HdiNegMJhzSpF+ssdrGn++Z2EnVAHYIk25iT3m51R+eP/yfMZ1YDQBEY84EAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNqwYD3RjfiYKugjA5A2y/i0Ti+0jQPr4TBV0FYXIG2H4XicT3kQDo2pgzAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDU+ZwKc5Ww/Rc8n6CERJl+JT6+jp7P9FD2foIdEmHwlPr0OAF+NORMAgDXCBAA6sGzZMo0ZM0b333+/wuFwrMvp0giTDiSl+JSU4mO+A/gKrRP4tpfC74quu+46/c///I+2bNmi5cuX67333tOECRMi61966SX94Ac/iGGFXQdzJh1onSdhvgM4vs64DH5X1adPH0nSHXfcodWrV2vatGnatq3lnWvNzc168MEHtWrVqliW2GUQJu0wxqipvrbNz52xvVhtoyvU0FW20RVq6EnbOHp8IBBQSkqKHA6HVU1dQXV1taZMmaKamhpdfPHFSk5OVlxcnHJyclRWVqa//OUvys/PV9++fWNdapfgMMaYWBfR1QQCAfl8Pe+QHTgT/H6/UlNTY13GKamqqtLXvvY1bd68WatXr1YgENCcOXM0bdo0TZo0Sd/97nd19913a8SIEfrlL3+pjRs3KiUlJWob06ZN02WXXaZ77rknRo8iNjgyaUdKSor8fn+nbjMQCCgnJ0eff/55l93RqLHzdIc6T1eNx/5xPVmFhYVatmyZJCkuLk79+/fX1KlT9eCDD8rr9Ub1uf3227VkyZKo8TNnztSvf/1rFRQUqKSkRJK0f/9+zZs3T6tXr1ZlZaV69eqlESNGaN68ebryyisjY+fPn6/Jkydr4MCBqqio0IgRI3T48GHt3LlT+fn5kqS8vDzdddddevDBB9t9rHPnztWYMWP0/e9/v9P/KV20aJEeeeSRSG0LFixQXl5eh/1DoZB++tOf6plnnlFFRYWys7NVWFiouXPnRo4eFy9erMWLF6usrEySdNFFF2nevHm67rrrTq44gzPC7/cbScbv98e6lA5RY+fpDnV21RoLCgrMxIkTzd69e83u3bvNypUrTWpqqvnxj38c1ScnJ8f4fD5TV1cXaa+vrzdpaWkmNzfXFBQURNq/9a1vmcsvv9y8+uqrpqyszLz99tvmoYceMi+99FKkT21trUlNTTVvvfWWMcaYjz/+2Hzzm980eXl5Zs2aNZF+77zzjjn//PNNU1NTh4/h0ksvNQsXLuyMX0fEihUrTHx8vHn66afNBx98YG677TaTlpZmKisrOxwzf/58c84555hVq1aZXbt2meeff94kJyebX/3qV5E+f/rTn8zq1avNxx9/bD766CNz3333GbfbbUpLS0+qPsLkDOmqO+7RqLHzdIc6u2qNBQUFZvLkyVFt3/nOd8wll1zSps+wYcPMM888E2l/9tlnzfDhw83kyZMjYXLo0CEjyaxfv/649/v888+bjIyMr6zvlltuMS+++OJx+zzwwAPmm9/85ldu62Tk5eWZmTNnRpZDoZDJzs42xcXFHY7Jz88306dPj2r7zne+Y7773e8e97569eplnnzyyZOqj7cGA+jSSktLtXHjRsXHx7dZN336dC1dujSy/PTTT+uWW26J6pOcnKzk5GS9+OKLCgaDHd7Phg0bNGrUqA7Xf/rppxoyZIiSkpI0efLk49acl5enzZs3t3t/Dz30UKSmjm67d++OGtPY2Kh3331X48ePj7Q5nU6NHz9eb731Vod1jB49WuvWrdPHH38sSfrHP/6hv/3tbx2ewgqFQlqxYoVqa2t1xRVXHPcxHos5kzPE4/GoqKhIHo8n1qV0iBo7T3eosyvXuGrVKiUnJ6u5uVnBYFBOp1MLFy5s0+973/ue5syZo88++0yS9Oabb2rFihVav359pE9cXJxKSkp02223acmSJRo5cqSuuuoqTZs2TcOHD4/0++yzz5Sdnd1hTeedd54++uijE6o/OztbjY2Nqqio0IABA6LWzZgxQzfddNNXjj/agQMHFAqFlJmZGdWemZmp7du3d7id2bNnKxAI6MILL5TL5VIoFNL8+fP13e9+N6rf+++/ryuuuEINDQ1KTk7WypUrNXTo0BN5qBGEyRni8Xj005/+NNZlHBc1dp7uUGdXrnHcuHFavHixamtr9dhjjykuLk5Tpkxp0y8jI0P5+fkqKSmRMUb5+fnq3bt3m35TpkxRfn6+NmzYoE2bNunll1/Www8/rCeffFKFhYWSpPr6+sgEv62EhARJUl1dXZt16enpSk9P75T7+Sp/+MMf9Oyzz2r58uW66KKLtHXrVt19993Kzs5WQUFBpN+QIUO0detW+f1+vfDCCyooKNDrr79+UoHCaS4AXU5SUpIGDx6sESNG6Omnn9bbb7+tp556qt2+06dPV0lJiZYtW6bp06d3uE2v16trrrlG999/vzZu3KjCwkIVFRVF1vfu3VuHDh2KLDscjpO+tTp48KCklrA71qmc5urdu7dcLpcqKyuj2isrK5WVldXhY7733ns1e/ZsTZs2TRdffLH+7d/+TT/84Q9VXFwc1S8+Pl6DBw/WqFGjVFxcrBEjRuhXv/pVh9ttD2ECoEtzOp267777NHfuXNXXt7280cSJE9XY2KimpqaoS518laFDh6q29ssPa15yySWRT7dLLR/GPNlbq9LSUvXv37/do6QZM2Zo69atx70de5orPj5eo0aN0rp16yJt4XBY69atO+7cRl1dnZzO6D/zLpfrK68zFg6Hjzu/1B7CBECXN3XqVLlcLi1atKjNOpfLpQ8//FDbtm2Ty+Vqs76qqkrf/va39cwzz+h///d/tWvXLj3//PN6+OGHoybSJ0yYoA8++CDq6ORUL/S4YcMGXXvtte2uS09P1+DBg497i4trOwMxa9Ys/fa3v9WyZcv04Ycf6o477lBtbW3UGw4WLlyoq6++OrI8adIkzZ8/X6tXr1ZZWZlWrlypX/7yl/qXf/mXSJ85c+bojTfeUFlZmd5//33NmTNH69evbzOv8pVO6r1fAHCatffWYGOMKS4uNhkZGaampqbDPq2OfmtwQ0ODmT17thk5cqTx+XwmMTHRDBkyxMydOzfqMyrGtLz9dsmSJZHlyspKs3//fnP99deb//7v/zbGGDN69GizadMmY4wx06dPN7/85S+jtlFfX298Pl/k8yqdacGCBSY3N9fEx8ebvLy8SB2tioqKzIABAyLLgUDA/Md//IfJzc01Xq/XDBo0yPzkJz8xwWAw0mf69OlmwIABJj4+3mRkZJirr77arF279qRr43IqAHDE6tWrde+996q0tDTq9NCqVau0evVqLV68WGvWrNETTzyhb33rW/rwww/129/+Nmobixcv1sqVK7V27dozXX5M8W4uADgiPz9fn3zyicrLy5WWltbmQo9SyxzNT37yE61evVpr1rS9WrLb7daCBQvOdOkxx5wJABzl7rvvVk5Ojn73u99p3Lhx2rhxo/x+v0aOHClJeuedd3Tw4EH5fD653e4247///e9ryJAhZ7rsmCNMAKAdFRUVOv/886Mu9FheXq7vf//7evXVV1VWVqbS0tJYl9llMGcCAO345JNPNH36dDU2NurBBx/UmDFjdPXVV+tnP/uZxowZo+eff14vvPCCnnvuuViX2iUQJgAAa5zmAgBYI0zaYYxRIBAQB21AbLAPdj+ESTuqq6vl8/lUXV0d61KAsxL7YPdDmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKz1yDApLi7WZZddppSUFPXp00c33HCDPvroo1iXBQA9Vo8Mk9dff10zZ87Upk2b9Morr6ipqUnXXnutamtrY10aAPRIDmOMiXURp9v+/fvVp08fvf766xozZkyb9cFgUMFgMLIcCASUk5Mjv9+v1NTUM1kqcFZiH+z+euSRybH8fr8kKT09vd31xcXF8vl8kVtOTs6ZLA8467EPdn89/sgkHA7rn//5n3X48GH97W9/a7cP/xUBscU+2P3FxbqA023mzJkqLS3tMEgkyePxyOPxnMGqAByNfbD769Fhcuedd2rVqlV644031L9//1iXAwA9Vo8ME2OM7rrrLq1cuVLr16/XueeeG+uSAKBH65FhMnPmTC1fvlwvvfSSUlJSVFFRIUny+XxKSEiIcXUA0PP0yAl4h8PRbvvSpUtVWFj4leMDgYB8Ph+Tf0CMsA92Pz3yyKQH5iMAdGlnxedMAACnF2ECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKz1yDB54403NGnSJGVnZ8vhcOjFF1+MdUkA0KP1yDCpra3ViBEjtGjRoliXAgBnhbhYF3A6XHfddbruuutiXUa388d3v9Cew3XqnexRoL5ZYRklul2qawzJ6XSod1KcdlbVq39agg7XNynB7ZTD4VAobFTT0KzsNI8q/I3KTI3XZwfr1S8tQftrgkpPild1Q7M8cU4le1wqq6rTwHMS1RSWqmqCSvbEKdnj0mdV9erfK0GSFAoZVdUF5UtwK8nj0q4DdTqvd6IO1YdUF2xWls+jkJEq/A3qlehWnMupxuaw/PVNyu3l1c4DX26rJtisppBRWqJbB2qCSnC7lJYYpy8ONSi3l1e7quqVm56gvf6g+vo8OljTKE+8SzUNzUrxxsnhcCje5dQXh+s14JwEfXGoQX19XtUFQzKSmkJhnZPs1mdH7jNkjPZXB5Xbyys5XZoysn8Mn1XgzOiRYXKygsGggsFgZDkQCMSwmtj447tf6NG125WbnqQUb5zeL/erMhBUZqpHA85J0heH6jS0b6qqG5r1QtUXqgwElZ3m1bVDM/VyaYUqAy1/iAtHD9Sjr3wSGTv2ggz9fvNu7fUHVTh6QKRvZqpH/19erh776ye6ZmgflZb7tdff0j5z7Hla/Pqn2utvuY+hfVP11w/3qa/Po4v7+bR22z71S/NqaHaqXtm2T5mpHl1/cZZeLq1Qhf/Lmv/nvS804aIsrfkguv2zqloN7+/TkMwUPbz2y1pnjBmktdv2SZJK9/gjY64f1rKN1vrGXpChbXsCcjqkf3zh18BzklRWVRv5HQzr59Mr21rqnTX+fP1xy+eaMjInxs9w18Y+2P0RJpKKi4v1wAMPxLqMmPp0f7X2+oO6IDNVTSGjykDLjl0ZCOrCrFTtOdyg8/ukyOt2RdbtOdyghqZwZHmvP6iDtU1RYx0Oh/b6W5aP7lsZCCpQ3yRJamw2kT6VgaD2+Bsiy63327r9CzKNJKn8cIMGH2mvDARV3xhWhT+65r3+oBqa2rZXBoJqbDaqbwxF1VNZHVRTqGX7R4+pbwpH1edwOCL9jt7msTXu9Qe180CdHI5OeIJ6uI72wd4ZfeTgF3jaZPfrr12fftIp2yJMJM2ZM0ezZs2KLAcCAeXknF3/SZ7XJ0V9fR41NIWU4o1TZqon8h97Q1NI2WleuV0OVTc0R9Zlp3nldTsjy319HqUnuaPGSlJfn0d7/cGovpmpHvkS3ZKk+DhHpE9mqkf9fAmR5db7bd2O190yzdcvzStPXEt7ZqpHifFOZfk8kaOJhqZQpP+x7ZmpHnniHEqId0XV09fn0Y59NZIUNSYx3hVVnyS5XQ45HYraZuvvwBv3Zb2DMxIlp+vMPZHdVEf74KSHV8mdkBTDynq2lf/32522LYcxxnTa1rogh8OhlStX6oYbbjjhMYFAQD6fT36/X6mpqaevuC7mj1s+155D9S1zJg1NChspMd6pusawnI6WOZNdR+Y1DtUdmTORQyFz9JxJUH1SPdp91JzJOYnxCgSb5XE5leR16bPWOZOQUVVtY9s5E4fUFDI6WBNUWqJbSfEu7apqnTNpVl0wpCyfR+GwtDdwZM7E6VRjqO2ciZFU09CsZmPk87pVVXu8OZOWuZCq2kZ541yqCR4zZ3LoyJzJ4ZZ+tcFmSS21pie5tftI/eGw0b4a5kxstO6D7ngPRyanEUcmOC04r4+u5sD+fWfVP3TdWY8Mk5qaGu3YsSOyvGvXLm3dulXp6enKzc2NYWUA0DP1yDD5+9//rnHjxkWWW8/FFhQUqKSkJEZVAUDP1SPDZOzYserhU0EA0KX0yE/AAwDOLMIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABg7bSEicvlOh2bBQB0UScVJjNmzNC+ffu+sp8xJvLzr371q5OvqpMsWrRIAwcOlNfr1eWXX67NmzfHrBYA6MlOKkyuu+46XX/99frpT3+q2traDvs5HI7Iz++//75uv/12hUIhSdK2bdt08803n2K5J+65557TrFmzVFRUpC1btmjEiBGaMGHCCYUhAODkOMzRhxEnIBQK6Te/+Y2WLFmiO+64Qz/4wQ/kdEZnksvlioSHJD322GP6y1/+Ip/Pp7KyMs2ePVs33nhj5zyCDlx++eW67LLLtHDhQklSOBxWTk6O7rrrLs2ePfu4YwOBgHw+n5a99r4ONLrV1+dVXWNITqfU0BRWotulsDGqawor3uWQ2+WQw+FQXWNIaQluVTc0K9Ht0r7aoPr5vKqsDirJHaekBJf2BYLyJbhV1xiSL8GtAzVBpSW45XY5tb86qIR4l1K9cTpQ26Rkz5H7aQwrOd6luqaQzkly62BNozzxLlU3NCsj2aOKQIMykj1yOKV9gaD6+rza629Q/15eBepDCjQ0KaeXVzUNIR2qb1JWqle1wZCawmHFO51yOKU4p0O1jSE5HQ71TvFoysj+p/X56en++O4X2n2wVr0S3XI4HAobo8N1Tcrt5dXBumY1h41SvHHaXx1URopHoXBYdY1hpSW69cXBeg3onaBKf1CZqR7VBkOqbmhWls+jg7VNMpLSEtxqDIUVbArLHedQXTCkFG+cqoMhJbidCh3ZfmOz0f6aoHJ6eeWvb5bL4VB9U0i+RLf2VweV28urmsawDtY2qk+KR9UNTeqdHK+dB+o0qHeiyqrqlZOeoNpgy+uof5pXB2qaIttPiHdqrz+opPg4eeKc8tc3aea3z7f+/bXug+54T9Q/p11Ndr/+2vXpJ7Euo0uIO9kBLpdL+fn5SkhI0I9+9CM9/vjjeuSRRzRp0qR2+7/zzjvasGGDDh06pJ07d+rVV1/VgAEDrAs/nsbGRr377ruaM2dOpM3pdGr8+PF666232vQPBoMKBoOR5UAgIEn6f69+qkH9+ijVGyenQ/rHF35VBoK6dmgfle4JaM/hBmWmenRxP58cDslf36wvDtXpXy/N0VNv7tLAc5JUVlWrykAw0u+cZI+efXu3KgNB9fV5lJOepCvPO0cr3tmtvf6gsnweDcv26a8f7lNfn0cXHfk5M9WjAeckKdUbJ4dD+t8v/FHb7+vzaFg/n17Z1tL3ltED9cq2fZGaj14//mt99MEev/b6v6xLkgINzdp9sFbDsn2SjKaMzDmtz1NP9cd3v9Cja7drrz+o8V/rI5dD2nrM83BOkkevfbQv8tq4bliWkj1xKtlYFmmbfuVArd22T++Xfzm29fXQL82ra4Zmqvxwvd4v92tAevRrYcJFWXI4pL+8XxFpm3hRlv5SWhH12vvDwVpNvChLSzd+psxUj/7j6vP1yNpPNPCcJP1xS3mb187RNWSneTUsO1Vrj7zmhvf3yV/ffEph0tE+OOnhVXInJHXac9PZVv7fb8e6hC7jpMJk4sSJ2rZtm3Jzc5WXl6cFCxboggsu0K9//WutW7dOjz/+eJsxP/zhDzVv3jxde+21euedd3TDDTdo4cKFuvLKKzvrMbRx4MABhUIhZWZmRrVnZmZq+/btbfoXFxfrgQceaNO+r7pRF7ldagq1HLxVBlpe7MFmoz2HGyJtF2a1rE9wu7TncIMC9U1H2lMjY1r7OY7azl5/UBdkpipQ36S9/pa2Cn9QQzLNUevNUeNTo2o5evvH9j1U16Rgs2l3fVPIRO7v2Ppb7//TfTUn+uvGMT7dXx35/TaFjJp07HMe/XqqDATV0BSWyxGKajtY26SmUPvPYfnhBjU0hdXYbI48Z9GvhYamcNR97PUHVd8UbvPaa21v7fv5wbrjvraO/nnP4Qad3yclMrax2SjBfWpvvuloH/zzj/+pyx+ZoMVJhcnPfvYzXXzxxW3erfXUU0/pwgsvbHfM3/72t8jPl112mVatWqWbbrpJb7755imUe3rMmTNHs2bNiiwHAgHl5OSoT0q8GppaTh84HVJmqkeVgaA8cQ5lp3kjRyZul0POI0cm2Wle+RLcykz1qKEpFBnT2k/6cjt9fS19fAlu9fV5Ikcmrf36+qLHtNbicKjN9vv6PPLEfdk3PcktT5wjar33yHq3yxG5v6Praj2V4nY5dN6RPxI4eef1SYn8ft0uh1wOdfg8tb42EuOdSoh3RbWlJ7nldkU/h63PVb80r7xup+LjHMrytX0teN3OqNdsX1/LfRz72mtpd0XqyU1PPO5r6+gastO8kddUZmpLH3998yn9zjraBw/s36fU1NRTeyJwRp3UnMny5cv15ptvKjExUSNHjlR+fn7kid65c6cGDRok6cs5E2OMfv/737cZEx8fL6/Xe3oekVpOcyUmJuqFF17QDTfcEGkvKCjQ4cOH9dJLLx13/LFzJlk+r+oaW843NzSHlRjvVDgs1TeF5W5nziTQ0KyEeJeqaoLKTvOqMnBkzsTj0r4jcyS1jSGlet2qqj16zqRBCfFxSk2I04GaI3Mm4SNzJl6X6hpb5kyqahvljXOpOtisjBSPKvxH5kwk7asJKsvnVcWRORN/fbOqG5qV08ur6oaQDkfmTJrVFDaKdzrkcDrkcrbUz5xJ5/jjls+1u6pOvRLdkhwyxuhwfcvc1cHaZoWMUYonTvtrWuZMmkNh1TeF1SvRrc8P1mvAOQmqDATVJ9WjumBI1Q1NyvJ5daiuSWHTMmcSbA4r2Nwyb1cXDCkloZ05k1BY+6sbldPLq8N1zYpzOVTfGD1nUhsMq6ru+HMmdcGQ/EfmTKpqm9QcamfOxH1kzmRc582Z+P1+wqSbOOEwaX1X1IwZM/TnP/9Zfr9fu3bt0n/+53/qmmuuierbGibz5s3Te++912bMf/3Xf2n8+PGn5QG1uvzyyyOn4qSWCfjc3FzdeeedJzwBzwsZiA32wW7InKAxY8aYqqoqY4wxV155pTHGmEOHDplvfOMbbfo6nc6THtPZVqxYYTwejykpKTHbtm0zP/jBD0xaWpqpqKj4yrF+v99IMn6//7TXCaAt9sHu54TnTG699VaFwy0TdUlJSfr5z3+u888/X263u1PHdJZ//dd/1f79+zVv3jxVVFTo61//utasWdNmUh4AYO+kP2ciSTU1Nfrd736niooK/fu//7sGDx4ctd7pdEZC5ETHdCUcYgOxxT7Y/ZxSmPR0vJCB2GIf7H64ajAAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACs9bgwmT9/vkaPHq3ExESlpaXFuhwAOCv0uDBpbGzU1KlTdccdd8S6FAA4a8TFuoDO9sADD0iSSkpKTnhMMBhUMBiMLAcCgc4uC8BxsA92fz3uyORUFBcXy+fzRW45OTmxLgk4q7APdn+EiaQ5c+bI7/dHbp9//nmsSwLOKuyD3V+3CJPZs2fL4XAc97Z9+/ZT3r7H41FqamrUDcCZwz7Y/XWLOZN77rlHhYWFx+0zaNCgM1MMAKCNbhEmGRkZysjIiHUZAIAOdIswORm7d+/WwYMHtXv3boVCIW3dulWSNHjwYCUnJ8e2OADooXpcmMybN0/Lli2LLF9yySWSpNdee01jx46NUVUA0LM5jDEm1kV0NYFAQD6fT36/n4lAIAbYB7ufbvFuLgBA10aYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrhAkAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArBEmAABrPS5MysrKdOutt+rcc89VQkKCzjvvPBUVFamxsTHWpQFAjxUX6wI62/bt2xUOh/XEE09o8ODBKi0t1W233aba2lo9+uijsS4PAHokhzHGxLqI0+2RRx7R4sWLtXPnzhPqHwgE5PP55Pf7lZqaepqrA3As9sHup8cdmbTH7/crPT29w/XBYFDBYDCyHAgEzkRZAI5gH+z+etycybF27NihBQsW6Pbbb++wT3FxsXw+X+SWk5NzBisEwD7Y/XWb01yzZ8/Wz3/+8+P2+fDDD3XhhRdGlsvLy3XVVVdp7NixevLJJzsc195/RTk5ORxiA2cI+2D3123CZP/+/aqqqjpun0GDBik+Pl6StGfPHo0dO1bf+MY3VFJSIqfzxA/COF8LxBb7YPfTbeZMMjIylJGRcUJ9y8vLNW7cOI0aNUpLly49qSABAJy8bhMmJ6q8vFxjx47VgAED9Oijj2r//v2RdVlZWTGsDAB6rh4XJq+88op27NihHTt2qH///lHruskZPQDodnrc+Z/CwkIZY9q9AQBOjx4XJgCAM48wAQBYI0wAANYIEwCANcIEAGCNMAEAWCNMAADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYK3HXYK+M7ReYTgQCMS4EqD7SUlJkcPhiHUZOMMIk3ZUV1dLknJycmJcCdD98FW7Z6du8x3wZ1I4HNaePXs69T+sQCCgnJwcff755112R6PGztMd6jxdNXbGfmOMUXV1NUc53QhHJu1wOp1tvqWxs6SmpnbZPy6tqLHzdIc6u2KNDoejy9WE42MCHgBgjTABAFgjTM4Qj8ejoqIieTyeWJfSIWrsPN2hzu5QI7oPJuABANY4MgEAWCNMAADWCBMAgDXCBABgjTA5w8rKynTrrbfq3HPPVUJCgs477zwVFRWpsbEx1qVFmT9/vkaPHq3ExESlpaXFupyIRYsWaeDAgfJ6vbr88su1efPmWJcU5Y033tCkSZOUnZ0th8OhF198MdYlRSkuLtZll12mlJQU9enTRzfccIM++uijWJeFHoAwOcO2b9+ucDisJ554Qh988IEee+wxLVmyRPfdd1+sS4vS2NioqVOn6o477oh1KRHPPfecZs2apaKiIm3ZskUjRozQhAkTtG/fvliXFlFbW6sRI0Zo0aJFsS6lXa+//rpmzpypTZs26ZVXXlFTU5OuvfZa1dbWxro0dHcGMffwww+bc889N9ZltGvp0qXG5/PFugxjjDF5eXlm5syZkeVQKGSys7NNcXFxDKvqmCSzcuXKWJdxXPv27TOSzOuvvx7rUtDNcWTSBfj9fqWnp8e6jC6tsbFR7777rsaPHx9pczqdGj9+vN56660YVta9+f1+SeL1B2uESYzt2LFDCxYs0O233x7rUrq0AwcOKBQKKTMzM6o9MzNTFRUVMaqqewuHw7r77rt15ZVXatiwYbEuB90cYdJJZs+eLYfDcdzb9u3bo8aUl5dr4sSJmjp1qm677bYuWSN6rpkzZ6q0tFQrVqyIdSnoAbgEfSe55557VFhYeNw+gwYNivy8Z88ejRs3TqNHj9ZvfvOb01xdi5OtsSvp3bu3XC6XKisro9orKyuVlZUVo6q6rzvvvFOrVq3SG2+8cdq+bgFnF8Kkk2RkZCgjI+OE+paXl2vcuHEaNWqUli5dKqfzzBwgnkyNXU18fLxGjRqldevW6YYbbpDUcppm3bp1uvPOO2NbXDdijNFdd92llStXav369Tr33HNjXRJ6CMLkDCsvL9fYsWM1YMAAPfroo9q/f39kXVf6D3v37t06ePCgdu/erVAopK1bt0qSBg8erOTk5JjUNGvWLBUUFOjSSy9VXl6eHn/8cdXW1uqWW26JST3tqamp0Y4dOyLLu3bt0tatW5Wenq7c3NwYVtZi5syZWr58uV566SWlpKRE5pt8Pp8SEhJiXB26tVi/nexss3TpUiOp3VtXUlBQ0G6Nr732WkzrWrBggcnNzTXx8fEmLy/PbNq0Kab1HOu1115r9/dWUFAQ69KMMabD197SpUtjXRq6OS5BDwCwxru5AADWCBMAgDXCBABgjTABAFgjTAAA1ggTAIA1wgQAYI0wAQBYI0wAANYIEwCANcIEAGCNMEG38Pvf/14JCQnau3dvpO2WW27R8OHDI189CyB2uNAjugVjjL7+9a9rzJgxWrBggYqKivT0009r06ZN6tevX6zLA856fJ8JugWHw6H58+frxhtvVFZWlhYsWKANGzYQJEAXwZEJupWRI0fqgw8+0Nq1a3XVVVfFuhwARzBngm5jzZo12r59u0KhkDIzM2NdDoCjcGSCbmHLli0aO3asnnjiCZWUlCg1NVXPP/98rMsCcARzJujyysrKlJ+fr/vuu08333yzBg0apCuuuEJbtmzRyJEjY10eAHFkgi7u4MGDGj16tMaOHaslS5ZE2vPz8xUKhbRmzZoYVgegFWECALDGBDwAwBphAgCwRpgAAKwRJgAAa4QJAMAaYQIAsEaYAACsESYAAGuECQDAGmECALBGmAAArP3/B20GlKpwOJQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 400x400 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "jointplot(df, x=\"x\", y=\"grad_x\", ylabel=r\"\\frac{\\partial y}{\\partial x}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivatives are a square pulse, where $\\frac{\\partial y}{\\partial x}=1$ when ${-\\alpha^{-1}} \\leq x \\leq \\alpha^{-1}$ and $0$ otherwise.\n",
    "\n",
    "We assume that the output gradient $\\dot{Y}$ is independent of the inputs $X$, so that the input gradient is the product of two random variables: $\\dot{X} = \\dot{Y} \\Delta$, where $\\dot{Y} \\sim N(0, 1)$ and $\\Delta \\sim \\mathrm{Bernoulli}(Z)$, where $Z$ is defined as previously.\n",
    "\n",
    "Since they're independent, the expectation of the product is the product of the expectations, so $\\mathrm{E}(\\dot{X})=0$, and\n",
    "\n",
    "$\\sigma_{\\dot{X}} = \\sqrt{\\mathrm{E}((\\dot{Y} \\Delta)^2)} = \\sqrt{Z\\, \\mathrm{E}(\\dot{Y}^2) + (1-Z)\\,0} = \\sqrt{Z}$\n",
    "\n",
    "Therefore, we have the **backward scale**:\n",
    "\n",
    "$\\sigma_{\\dot{X}} = \\sqrt{\\mathrm{erf}(\\sqrt{\\frac{1}{2}}\\, \\alpha^{-1})}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAEqCAYAAAB3BAsnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAhklEQVR4nO3deZxN9QPG8c+52yyMse9E0VjHLpEoEvklEkW2kMiSLBUilUKrdlR2UslSQqjslSIhW4nsW5gZM8zdzu+PyzAl3WFmzizP+/W6r7n3zp0zz7dp5vE995zvMUzTNBEREZH/ZLM6gIiISGah0hQREQmSSlNERCRIKk0REZEgqTRFRESCpNIUEREJkkpTREQkSNm6NE3TJDY2Fp2qKiIiwcjWpRkXF0dkZCRxcXFWRxERkUwgW5emiIhISqg0RUREgqTSFBERCZJKU0REJEiZujQXLlxIVFQUZcuW5YMPPrA6joiIZHFGZr00mNfrpUKFCnz77bdERkZSo0YN1q1bR758+YLeRmxsLJGRkcTExJArV640TCsiIlmBw+oAV2v9+vVUrFiRYsWKAdCsWTOWLl1Ku3bt0ieAOx4859Lne2UkhvG3+8aVPxp2sDnAlql3aoiIABm8NEePHs3cuXPZsWMHYWFh1K1bl7FjxxIVFcWhQ4eSChOgWLFiHDx4MP3CrXwJ1o5Lv++X6Rnny9MBNvv5m+PizbCD3QHOHOAKB1eO87ec4Ay/eN91yf2QCIgoDBFFIEeBwDZFRNJQhi7NlStX0rt3b2rVqoXX62Xo0KE0adKEbdu2WR0tyWZ/aRIIwYkPBz7s+JLuO/DhMAKP7ecfO/EB4MOGDxte7En3faY92eML9wMf7XhNW+AjFz7ak7/ODHw0CcwGTQwu7HsP3L84S7z0/oVX2fBjnH9su+SrbUbgeRt+bJjnx+JP9tGOH7uR/Hkbfuz4L27Pbybdv7At8GEzvOe36ycnBwkjMdmENiiG/WKBRhSGXEXP3y8CuYpARFGILBYoXBGRq5ShS3PJkiXJHk+ZMoWCBQuyYcMGihYtmmxmefDgQWrXrn3F7SUmJpKYmJj0ODY29pozPu3pymbzhmvejlxkx0cECeQyEogggQgjgQjOkot4Ioyz5Dr/XCTx5Ddikm75Yo4SEnulvQ0G5LsBCkdDkejzH6tAjvzpNjYRydwydGn+XUxMDAB58+alXLlybN26lYMHDxIZGcnixYsZPnz4Fb9+9OjRPPvss6maqYRxnHhC8WLHe36meOnNgx3v+XnXv7EnzUQDM7O/z+AcxoVZrP9vs7yLn7fjT3p86TzSuGTGeGEWeen9C581Mc7PCS88DswD/ck+Bl7j+9uMN/nM+PxjM/D44jYDX+u/ZA7rPz+3vfD5wOsD2zxNBKfNiMAgUnCoWgTxFDBiyE8M+Y3YQJmef1zaOEKVE7vJ8dfv8Ovci1+Uq9jfijQaIkuQ8umuiGR1meboWb/fT4sWLTh9+jRr1qwB4PPPP2fQoEH4/X6eeOIJevToccVtXG6mWaJEias7enbZMyl6T9NvGnix4T3/75QLpWfD1N/m80wTEgghjnDizHBiCSfWDD//OIxYchBnhhNHGLFmDk6TgxNmJH+ZuThBZNJ/2yux4SfK2Ec12+9Ut/1GdeM3ShtH/vkzCMsDxWpC2SYQ1RRyl0ybQYtIppJpSrNXr14sXryYNWvWULx48VTZ5jWdcrJnFexfnyo5MqbL/G9hXnLHNC/z8TKfM33gv3DzXnK75PGF1/jc4E4A9xnwJASOUHafCXz0e6+c1oSY8yV63IzkBOfL9Pz942ZutvtLcoh/7orNQxzVbL8llWgV225yGInJX1SwItx4J0Q1g2I1dNCRSDaVKUqzT58+LFiwgFWrVlG6dOlU267O08xEvO6LBepJgMQzcPYkxB6CuCMQdwhiD1/8mHDisps5YubhZ38ZNvrLstFfli1mady4kr3mwmz0JtsO6tp+5SbbdiKNhIsvCM9/cQZ6w+2Bo3hFspm9e/dSunRpfv75Z6pWrWp1nHSToUvTNE369u3LvHnzWLFiBWXLlk3V7as0szCvG84cuVikMQfh6K9wZDMc35E0c000HWwzr0sq0Z/9Zf8xG7Xhp7Kxh7q2rdS1/UpN2y7CDPf5Tzqh1C2BGWi55hCZOntBRDI6n8/H8ePHyZ8/Pw6HtYfHjBw5kvnz57Np06Y0/14ZujQfffRRZs2axYIFC4iKikp6PjIykrCwsGvevkozm/Kcg+Pb4fDmQIke3gxHtwZmsARmoxv8N/KdvwLr/BX5wyya7MtdeKhm/EY9+6/Utf1KFWM3TsMHGHBjU6jVPTAD1YIOkkW53W5cLtd/vzCdpGdpZujf6vfee4+YmBgaNmxIkSJFkm4ff/yx1dEkM3OGQtFqUKMzNH8Vui+DIQeg94/Q+kMK1+tI85JeRjmn8E3IIL4L6cOrzve417aKwvyFGyc/mBV4zduG+9wjqZo4ka7uQSz1Vce3cwnMbA1vVYd1b0HCSatHK5dhmiYJbq8lt5TOU/x+P6NHj6Z06dKEhYVRpUoV5syZg2maNG7cmDvvvDNpmydPnqR48eKMGDECgBUrVmAYBl9++SXR0dGEhoZSp04dtm7dmux7rFmzhvr16xMWFkaJEiXo168f8fHxSZ8vVaoUzz//PJ06dSJXrlz06NGDvXv3YhhGUlFd+F5fffUV1apVIywsjNtvv51jx46xePFiypcvT65cuWjfvj0JCRff7vi38V1wYbtff/01NWvWJDw8nLp167Jz504gcCris88+yy+//IJhGBiGwZQpU1L03zglMvRMM61ppilXdOYY7PoKdi2B3d+CJx7ThD1mYdb6K/GdvyLf+StwiovvaZYyjtDNvojW9tWEG4ngCIVKrQOzz2LVLRyMXCrB7aXCiK8s+d7bnruTcFfwuzNfeOEFZsyYwbhx4yhbtiyrVq2iZ8+efPXVV5QpU4bKlSvzzDPP8Nhjj9G2bVv+/PNP1q5di8PhYMWKFdx2222UL1+eN954g8KFCzN06FC2bt3Krl27cDqd7N69mypVqjBq1CiaN2/O8ePH6dOnD1WqVGHy5MlAoDRPnTrFiBEjaNmyJQB2uz3Ze5oXvledOnV45ZVXCA8Pp23bthQrVoyQkBDGjBnDmTNnaNWqFYMHD+bJJ5/8z/E1aNAgabs33XQTY8eOpUCBAvTs2ROfz8fatWs5e/Ysw4cPZ8mSJSxfvhxIvb2Rl6PSVGlKMDznYO+aQIHuWgIx+4HAqUTbzZJ84buZWb5GxBJYcSg3cTxo/5pOjqUUMk4HtlG0eqA8K90LzrT5hZbgZJbSTExMJG/evCxfvpybb7456fnu3buTkJDArFmz+PTTT+nUqRP9+/fnrbfe4ueff046/uNC4cyePZv7778fuDgbnTJlCm3btqV79+7Y7XYmTJiQtP01a9bQoEED4uPjCQ0NpVSpUlSrVo158+YlvebvBwJd+F7Lly+nUaNGAIwZM4YhQ4awe/durr/+egB69uzJ3r17WbJkSVDju9x2Fy1aRPPmzTl79iyhoaHpuns2Uy1uIGIZZyiUbRy43fVy4KCiXUuw7VpCxQM/UdH2J30d8/jU14BJvmbsMwvxjq8lE33/427bOro7FlHh0EZY8CgsHQbVOkCthyHPdVaPLFsKc9rZ9tydln3vYP3+++8kJCRwxx13JHve7XZTrVo1ANq0acO8efMYM2YM77333mUPmLy0kPLmzUtUVBTbt28H4JdffmHz5s3MnDkz6TWmaeL3+9mzZw/ly5cHoGbNmkFljo6OTrpfqFAhwsPDkwrzwnPr168PenyX226RIkUAOHbsGCVLpu851CpNkZQyDChcKXC7dVDgCN2N08ixYTJd4pbS0b6MZf4afOBtzk9mFHP9tzLXfSv1bFvpbl9Eg4RfsK17C75/D2p0gVsHB9bLlXRjGEaKdpFa5cyZMwB8+eWXyS5QARASEgJAQkICGzZswG6389tvv13V93jkkUfo16/fPz53aSHlyBHcus1OpzPpvmEYyR5feM7v9yd9b7jy+P5tu0DSdtJTxv+/RiSjy1UEGj4J9QfAzkXYf/yApntW0dT+Ez/7b+AD710s9t/EWn8l1vorUcY4QHf7IlrZ1xDy4wfw80yo0wvq9QusRCRyXoUKFQgJCWHfvn00aNDgsq8ZOHAgNpuNxYsXc9ddd9G8eXNuv/32ZK/5/vvvkwrw1KlT7Nq1K2kGWb16dbZt20aZMmXSdjCXEcz4guFyufD5fKmY7N+pNEVSi90JFe4J3I7vhJ8mUW3TLN5JfIv9/o+Y6ruT2b7b+N0szlPeHrzhbc0jji94wPyW0DWvwU8fQr3+cFPPwCXQJNuLiIhg0KBBPP744/j9fm655RZiYmJYu3YtuXLlIn/+/EyaNInvvvuO6tWrM3jwYDp37szmzZvJk+fiP8Cee+458uXLR6FChRg2bBj58+dPOqDnySefpE6dOvTp04fu3buTI0cOtm3bxrJly3j77bctHV/nzp2D2k6pUqXYs2cPmzZtonjx4kRERPxjpppaMvQpJyKZVoEoaDYWBmyH/42jRJHCPO2cybqQvgxzzKAgpzhMPkZ6u3BL4puM9/6PM2cT4etn4c1q8OMH4PNYPQrJAJ5//nmGDx/O6NGjKV++PE2bNuXLL7+kVKlSdOvWjZEjR1K9euDI7GeffZZChQrRs2fPZNsYM2YMjz32GDVq1ODIkSN88cUXSedZRkdHs3LlSnbt2kX9+vWpVq0aI0aMoGjRov/Ikp7jS8nqb61bt6Zp06bcdtttFChQgI8++ijN8uroWR09K+nBNANrFf/4Pvw6j3M+gzm+W3nP24KDFAAgkjN0dSymi30pkUY85CkFtw2DSvdpoQS5KheOPD116hS5c+e2Ok6WoNJUaUp6+2s3fPsibJ2Dx7Qz31ePd333sMcMHBGYkwQ62ZfRzbGIfEYcFKoEtw8PLBivS+JICqg0U59KU6UpVjm8Gb55Hn5bis80+NJfh3e897DTDBywEcY52tu/oYdjYeBcz1L14X/jIH/6H7AhmZNKM/WpNFWaYrU/18HyZ2H/9/hNg2X+GrztbckWM3BumwsPHe3LGOj4lHAH0GAw1H0MHBln7U+R7EKlqdKUjMA04bel8PVzcHQrpgkr/dG85W3FBjNwsYKSxlFeck6gjm0HFCgPd78BJW+yOLhI9qLSVGlKRuL3w69z4ZtRcGoPpgkr/FUY5umWdMmyzvaveNIxm3DDDTW7QuNnIDTS4uAi2YNKU6UpGZHPAxunwcqX4MwRYs0wXvQ+yGxf4KT1ksZRxjomcrN9O+QsHFjar/zdOlBIJI2pNFWakpElxgVmnT9MAExW+SrzlOfhZLPOJxyzyWEkQlTzQHlGFrvyNkXkqqk0VZqSGRzYAF/0g6NbiTs/6/zo/KyzhHGMlxwTArNOV05oNCJwNRVb8AuDi0hwdMa0SGZQvAb0WAGNnyXCCaOdHzDNOZqinGC/WZB2nuGM8HQhPtEDi5+AD++AY9utTi2ZRMOGDenfv3/Qr58yZUq2PYVFpSmSWdidcEt/ePQ7uP42brVv4auQJ2ln/xqAab4mNHWP5TtfeTi4ASY2DCzHl313JomkOpWmSGaTtzR0nAf3vk9EjnBGOz9khvNFinE8adb5rKcj5zw++HIgzH4Q4v+yOrVIlqCrnIhkRoYB0W2hTGNY+jS3bJrJEttTjPa2Y5avMZN9zfjeX563nG9TZueXMH4jtJoA11/95ZeyjM/7Wr/rumB5aPHWf76sYcOGVK5cGbvdztSpU3G5XIwaNYr27dvTp08f5syZQ6FChXjrrbdo1qwZACtXrmTw4MH88ssv5M2bl86dOzNq1CgcjsCf+/j4eHr16sXcuXOTrjLyd4mJiQwbNoyPPvqI06dPU6lSJcaOHUvDhg1T9T9DZqTSFMnMwvNCy3ch+n4iFvbnxZOTuMO2kUGeR9huluJ/7hcY4ZhOu9hvMKbdE9i9e9uwwK7e7OrYdjjwo9UpgjZ16lSeeOIJ1q9fz8cff0yvXr2YN28erVq1YujQobz++ut07NiRffv2cerUKe666y66dOnCtGnT2LFjBw8//DChoaGMHDkSgMGDB7Ny5UoWLFhAwYIFGTp0KBs3bqRq1apJ37NPnz5s27aN2bNnU7RoUebNm0fTpk3ZsmULZcuWteY/RAaho2d19KxkFZ6z8PXz8P07HDNzM9DTk9X+aACa2tYzxvk+uY14KFod7vsQ8l5vcWCLfNDY+tIsXgu6L//PlzVs2BCfz8fq1asB8Pl8REZGcu+99zJt2jQAjhw5QpEiRfjuu+/44osv+Oyzz9i+fTvG+XN23333XZ588kliYmJISEggX758zJgxgzZt2gBw8uRJihcvTo8ePRg3bhz79u3j+uuvZ9++fckuD9a4cWNq167Niy++yJQpU+jfvz+nT59O5f8wGZ/e0xTJKpxh0PRFePAzCuZ0MtU5lqGOmTjxssRfm2aJY/jBXw4ObYTx9eGX2VYnliBER0cn3bfb7eTLl4/KlSsnPVeoUCEAjh07xvbt27n55puTChOgXr16nDlzhgMHDrB7927cbjc33XRx+cW8efMSFRWV9HjLli34fD5uvPFGcubMmXRbuXIlu3fvTsuhZgraPSuS1ZRtDL3WYZvfix6/f8nNtm308/Rhj1mEdu6n6WOfTz9zLo55j8DvX0PzVyFUe1oyKqcz+a50wzCSPXehIP1+f6p8vzNnzmC329mwYQN2e/JzfXPmzJkq3yMzU2mKZEU5C0L7T+GH8VRe/gwLjaGM9HbmU19D3vTdy1p/RcY536HElk9g/w/Q+kMoUcvq1OmjYHmrE6RZhvLly/PZZ59hmmZSma5du5aIiAiKFy9O3rx5cTqd/PDDD5QsGbgE3alTp9i1axcNGgQOEqtWrRo+n49jx45Rv379NMmZmak0RbIqmw1ufhRK1SPHnG68/NdE6tu2MMzTjQ1mFHe5R/Oi80PuPv09TLoT7ngObu6d9devDeKo1czq0UcfZdy4cfTt25c+ffqwc+dOnnnmGQYMGIDNZiNnzpx069aNwYMHky9fPgoWLMiwYcOw2S6+U3fjjTfy4IMP0qlTJ1599VWqVavG8ePH+frrr4mOjqZ58+YWjtB6ek9TJKsrUgUeWQnVO9HC/h2LXE9R3dhFHDno6+nHIM8jJPgdsHQYfNYd3AlWJ5arVKxYMRYtWsT69eupUqUKPXv2pFu3bjz99NNJr3n55ZepX78+d999N40bN+aWW26hRo0aybYzefJkOnXqxMCBA4mKiqJly5b8+OOPSbPT7ExHz+roWclOfp0PX/TDezaON7338ravJX5s3Gjs5z3nOG6wHYZCleGBGZCnlNVpRTIclaZKU7Kb0/thbg/Yt47v/eXo5+7LMfKQkwReck7kLvt6CMsTeJ+zTCOr04pkKNo9K5Ld5C4BXRZCwyHUse1kYchQbjK2cYZwHvX0Z5TnQTwJsTDzPljzutauFbmEZpqaaUp29tsy+Kwb3rNxvOxtywRfCwBqGTt42/UmhYzTUKEl3PMOhOh0AxGVpkpTsruTfwQWdT+2jSW+mgz29CSOcPJzmrddb1LHtgMKlIcHZkK+G6xOK2Ip7Z4Vye7yXg/dlkHFVjS1/8TnrqcpZ/zJCXLzoHsYE7z/wzy2Hd6/DXYttTqtiKVUmiIS2PV632S44zlK248xz/UM99pW4cPOaG97HvE8TuxZN8xqC6tehlRafUYks9HuWe2eFUlu9zcwpytmwik+8t3OSG9n3DgpZRzhPefrlLfth3L/C1xqTO9zSjaj0lRpivzTqb3wcQc4soXN/tL0cvfnIAUIJZEXnJNobV8NhaOh/ceQq+h/bk4kq9DuWRH5pzyloOtSqNyWaNseFoYMo4FtE+cIYaCnF6M9D+A/vAXebwSHN1udViTdqDRF5PJc4XDvRLhzNHlsZ5nsfJl+9rkATPC14FHPY5yNPQGTmsLOJRaHFUkfKk0R+XeGEVj0vdMCbDnyMsA5h3HOd3DhYYm/Nve7h3PM7YTZ7eCHCVanFUlzKk0R+W+l68PD30D+KFra1zLT9SJ5iGOzeQMtE59ju68YLH4CFj0Bfp/VaUXSjEpTRIKTpxR0WwqlG1DLtpP5ruFcbxziEPm5zz2Sb31VYf0EmN0eEs9YnVYkTag0RSR4Ybmhw2dQrSPX2QLnc95s+5V4wujmGcRUbxPYtQQmN4PYQ1anFUl1Kk0RSRm7M3Ah50bPEGnEM9U5hrb2b/Fj4xlvF0Z6OuHTkbWSRak0RSTlDAPqD4A2U3A5nYx1vM+Tjo8AmOJrysOegZyJPakjayXLUWmKyNWr2Ao6L8TIkZ9eji941zmOENx846/Ofe5nOOQOOX9k7USrk4qkCpWmiFybErXg4a8hfxR32dfzset58nOaHeZ13JP4PJt918HiwbB8pK7NKZmeSlNErt0lR9ZWte1mfsgIoox9HCcPD7iHs85XIXBB6wW9wee1Oq3IVVNpikjquOTI2uLGCea4nqWebSsJhNLF8wTLfNVh00z4+EFwJ1idVuSqqDRFJPVccmRthHGWD50vc4ftJ9y46Ol5nAW+uoFTUqbdAwknrU4rkmIqTRFJXReOrL33fULtJu85x3GvbTU+7PT3PMoMbyM4sD5wZG3MAavTiqSISlNE0kZ0W2j/MQ5XGK84x9PRvhQTG097u/Ge9244sRM+bALHdlidVCRoKk0RSTtlGkPnL7DlyMtzjin0ts8HYKy3HWM992PGHIRJd8L+9dbmFAmSSlNE0lbxGtB1KUbukgx2fsJTjlkAvOe7hxHeLvjPxsDUFloEQTIFlaaIpL38ZQKnpBSsSE/HQkY5PsTAz3RfEwZ6euL1JAYWev95ptVJRa5IpSki6SNXEXhoEVxXjw6OrxnnfBc7Pub569PL059zfhsseDRwPqcWQZAMSqUpIuknLDd0mAvl/sc99nVMcL6OCzfL/DXp5hlMvBkSWDnoq2EqTsmQVJoikr6codB2GtToQmP7RqY4XyIHZ1nrr0QH91BizBzw/TvweV9d0FoyHJWmiKQ/mx3+Nw4aPEld+zZmuF4kkjP8bJalrXs4R83c8PN0+KwbeN1WpxVJotIUEWsYBtw2FJq/SjXbH3zsep6CnGKnWZLW7pHs9ReCX+cFlt3znLU6rQig0hQRq9XqDm2mUM5xhM9cI7nOOMIBsyD3uUey1X8d/LYUZtwHiXFWJxVRaYpIBlCxJbSfTQnXGea4RlLe2MsJImnnHs73/nLw5xqtVysZgkpTRDKGMo2h41wKhMLHruepbWwnjnA6uZ8KXCHl4AaY0hzijlqdVLIxlaaIZBzX1YXOn5MrPIxprjE0vuQKKZ96b4Vj22ByUzi9z+qkkk2pNEUkYylaDR5aTGhEPsY7x9HathIfdgZ7ezLR2xxO/gGTmsGJ361OKtmQSlNEMp6C5aDrEhx5ivOycyIP2xcC8KL3QUZ7HsCMORCYcR7ZYnFQyW5UmiKSMeUtDV2/wlagLMOcs5IWep/ga8FT3ofxnvkr8B7n/h8tDirZiWGa/71W1eeffx70Blu0aHFNgdJTbGwskZGRxMTEkCtXLqvjiMjlxJ+A6a3gyGY+9jZkiLc7fmw0sf3Im863CXW5oP1sKH2r1UklGwiqNG225BNSwzC49MsMw0i67/NlnmWvVJoimcTZ0zDrftj/PUt8Nenn6YsbJ3Vsv/K+8zUinCY8MAvKNLI6qWRxQe2e9fv9SbelS5dStWpVFi9ezOnTpzl9+jSLFi2ievXqLFmi6+GJSBoIyw0d58L1t9HU/hNTnGPJSQLf+yvSwT2EGI8dPnoAdn1ldVLJ4oKaaV6qUqVKjB8/nltuuSXZ86tXr6ZHjx5s3749VQOmJc00RTIZbyLM6Qo7FrLFX5qO7qc4TQQVjT1Md40hr/0ctJkM5e+2OqlkUSk+EGj37t3kzp37H89HRkayd+/eVIgkIvIvHCHQZipUbktl2x4+co0iHzH8apamnXsYx31h8Eln2DrX6qSSRaW4NGvVqsWAAQM4evTiqhxHjx5l8ODB1K5dO1XDiYj8g90BrcZD1Q6Ut+1PttD7/e4RHPHnClwdZfMnVieVLCjFpTlp0iQOHz5MyZIlKVOmDGXKlKFkyZIcPHiQDz/8MC0yiogkZ7NDi7egZlfK2A7xies5inKCP8yitHWP4IA/L8ztAT/PsDqpZDEpfk8TwDRNli1bxo4dOwAoX748jRs3TnYUbWag9zRFMjnThCVPwQ/j2e/Pz4OeYewzC1GM48xyvcB1tmOB63bWfMjqpJJFpLg0p02bxv33309ISEiy591uN7Nnz6ZTp06pGjAtqTRFsgDThGXDYd1bHDHz0N49jD/MohTiJDNdL1LGdgiavQw39bA6qWQBKS5Nu93O4cOHKViwYLLn//rrLwoWLKjzNEUk/ZkmfDMKVr/CMTOSDu6h7DJLkJ8YZrhepJxtPzQZBXX7Wp1UMrkUv6dpmuZld8MeOHCAyMjIVAklIpIihgGNhkPDoRQ0YpjtGkVFYw8niOQB99Ns9ZeCpU/DqlesTiqZnCPYF1arVg3DMDAMg0aNGuFwXPxSn8/Hnj17aNq0aZqEFBEJSsMnwe4k79fPMsv1Ip3dT7DJLEs79zCmucZQ7ZvnweeBhk8FilYkhYIuzZYtWwKwadMm7rzzTnLmzJn0OZfLRalSpWjdunWqBxQRSZH6A8ARQuRXQ5nuGkNX92B+NMvRwT2Uya6XqL1yDJg+uG2YilNSLMXvaU6dOpUHHnjgHwcCZUZ6T1MkC1v/PiwaRIIZQnfPQNb5KxHGOSY5X+Zm+3a4dbCKU1Isxe9p3n777Rw/fjzp8fr16+nfvz8TJ05M1WAiItek9sPwv3GEG24mOV+mgW0TZwmlq2cw3/nKw6qXAwcPpfysO8nGUlya7du359tvvwXgyJEjNG7cmPXr1zNs2DCee+65VA8oInLVaj4ELd8l1PAywfk6t9l+5iyhPOR5gnW+CrD6FRWnpEiKS3Pr1q1Jy+V98sknVK5cmXXr1jFz5kymTJmS2vlERK5N1fbQagKhhpf3nOO4zfYz5wihq2fwJcX5vIpTgpLi0vR4PEnvZy5fvjzpotPlypXj8OHDqZtORCQ1VLk/UJw2H+PPzziTF+erKk4JSopLs2LFiowfP57Vq1ezbNmypNNMDh06RL58+VI9oIhIqqhyP7QcT4jNz3jn69xu25hUnGt9FVWcEpQUl+bYsWOZMGECDRs2pF27dlSpUgWAzz//XFc5EZGM7fyMM8Tm5z3nOBpdrji/fk7FKf/qqhZs9/l8xMbGkidPnqTn9u7dS3h4+D+W18vIdMqJSDa1+ROY9wiJfhuPevrztb86Ibj50PkKt9i3wi0DoNEInY4i/3BVpZlVqDRFsrHNn8K8HiT6bfT2PMZyf42/Fefj0OgZFackk+Lds/9m6NChdO3aNbU2JyKStqLbQKuJhNj8vON8g8a2DSTioptnEKt9lWDN6/D1s9pVK8mkWmkeOHCAvXv3ptbmRETS3iXF+a5zHI1tP5GIi+6eQazyVQ4U5/KRKk5Jot2z2j0rIlvmwNyHcfsNHvU8xnJ/TVy4+cD5Krfat+g9TkmSajNNEZFMq/J9cO/7uGwm7zrf4A7bT7hx8bBn4Pldta/BitFWp5QMIKiZ5ptvvhn0Bvv163dNgdKTZpoikswlM87ensdY5q9JCIG1a+vZf4WGQwOXH5NsK6jSLF26dLLHx48fJyEhgdy5cwNw+vTppNNN/vjjjzQJmhZUmiLyD8l21fZnub8GoSQyyfkyde3b4PbhcOsgq1OKRYLaPbtnz56k2wsvvEDVqlXZvn07J0+e5OTJk2zfvp3q1avz/PPPp3VeEZG0Vfk+aDURl83kHecbyVYO+s5XPrBq0JpxVqcUi6T4QKAbbriBOXPmUK1atWTPb9iwgfvuu489e/akasC0pJmmiPyrTR/B/F4kmnZ6eh7nW381wjjHFNdL3GTbAXe+CDf3tjqlpLMUHwh0+PBhvF7vP573+XwcPXo0VUKJiFiuaju4521Czl8d5cL1OB9yP8F6fxR8NRR+mGB1SklnKS7NRo0a8cgjj7Bx48ak5zZs2ECvXr1o3LhxqoYTEbFUtQ5w9xuEGh4mOF+nvm0zCeeL8yf/jbD4CVj/vtUpJR2luDQnTZpE4cKFqVmzJiEhIYSEhFC7dm0KFSrEBx98kBYZRUSsU6MLNH+NUMPD+85XucW2hXjC6Ox+kg3+srBoEPw02eqUkk6uenGDXbt2sWPHDiBwLc0bb7wxVYOlB72nKSJB+2EiLB7MWTOw1N46fyVyksA01xiq236HFm9D9Y5Wp5Q0phWBVJoiEqzv3oWvhnDWdAWOpvVXJIIEprtGU9X2B7R8F6q2tzqlpKGrKs0DBw7w+eefs2/fPtxud7LPvfbaa6kWLq2pNEUkxda9BUufJsEM4SH3YH4wKxBBPDNco6li2wOtJgSu2ylZUopL8+uvv6ZFixZcf/317Nixg0qVKrF3715M06R69ep88803aZU11ak0ReSqnF/IPcEMoYv7Cdab5YkgnlmuF6ls/xNafwiV7rU6paSBFB8INGTIEAYNGsSWLVsIDQ3ls88+Y//+/TRo0IA2bdqkRUYRkYzllsfh9qcJNxKZ7HqJWsYO4shBJ/dT/OYrAp91h+0LrU4paSDFpbl9+3Y6deoEgMPh4OzZs+TMmZPnnnuOsWPHpnpAEZEM6dbB0HAIOc4XZxXjd04RQQf3EPb78sKnXWDXUqtTSipLcWnmyJEj6X3MIkWKsHv37qTPnThxIvWSiYhkdA2ehFsHk9MIrBQUZezjKHl50DOUo74c8HEH2J153rKS/5bi0qxTpw5r1qwB4K677mLgwIG88MILdO3alTp16qR6QBGRDMsw4LZhULcfeYwzTHeNoaRxlH1mITq4h3LK64SP2sPeNVYnlVSS4gOB/vjjD86cOUN0dDTx8fEMHDiQdevWUbZsWV577TWuu+66tMqa6nQgkIikCtOEJU/BD+PZ7y9AG/cIjpCPaGM3M10vEuGyQcd5UPImq5PKNUpRafp8PtauXUt0dHTSZcEyM5WmiKQa04SFj8OGyfzuL0ob9zOcIoKbjG1MdY0lNDQMOs2HYjWsTirXIEW7Z+12O02aNOHUqVNplUdEJHMyDGj+GlR9kDK2Q0xzjSGCBH4wK/Co5zHc5+Jheis4/IvVSeUapPg9zUqVKmWqC02LiKQbmw1avAWV21DZtocPXS8TSiLf+KszwPMovrOxMK0lHN1mdVK5SikuzVGjRjFo0CAWLlzI4cOHiY2NTXYTEcnWbHZoOR7Kt6C2bSfjna/jxMtC/8087e2GmXASprWA47usTipXIcUHAtlsF3vWMIyk+6ZpYhgGPp8v9dKlMb2nKSJpxuuGTzrBrsV86buJvp6++LHRw76QIY5ZGBGF4aFFkO8Gq5NKCjhS+gXffvttWuS4Jq1atWLFihU0atSIOXPmWB1HRAQcLmg7FWa3p/nvy4k3Q3nC+wgTff8jlxFPnzMLYGqLQHHmyTxnHWR3WeIqJytWrCAuLo6pU6emqDQ10xSRNOc5C7Pawp5VfOBtxihv4PJhIx1T6OJYCrmvg65LIFdRi4NKMFI809y8efNlnzcMg9DQUEqWLElISMg1B0uJhg0bsmLFinT9niIiQXGGQbvZMOM+uu9bTJwZzhu+1oz0diGncZb7Tq++OOPMWdDqtPIfUlyaVatWTfZe5t85nU7uv/9+JkyYQGho6H9ub/To0cydO5cdO3YQFhZG3bp1GTt2LFFRUSmNJiKSMblywIOfwPRW9N//GXGEMcl3F094HiGcRO76a33gqNouCyE8r9Vp5QpSfPTsvHnzKFu2LBMnTmTTpk1s2rSJiRMnEhUVxaxZs/jwww/55ptvePrpp4Pa3sqVK+nduzfff/89y5Ytw+Px0KRJE+Lj44FASVeqVOkft0OHDqU0uoiIdUIi4ME5GEWrMtwxg/vt3+LHxmOePnzrqwrHfg2cx3kuxuqkcgUpfk+zdu3aPP/889x5553Jnv/qq68YPnw469evZ/78+QwcODDZYu7BOn78OAULFmTlypXceuutQX/dihUrePvtt6/4nmZiYiKJiYlJj2NjYylRooTe0xSR9JNwEqY0x3d0O495+rDQfzMhuJnqGkMd2w4ocRN0mAshOa1OKpeR4pnmli1bLru+7HXXXceWLVuAwOzw8OHDVxUoJibwr6y8eVN/F8Xo0aOJjIxMupUoUSLVv4eIyBWF54VOC7DnL8PrzndpZNtIIi66uQezyX8D7P8BPnogcACRZDgpLs1y5coxZsyYpMuDAXg8HsaMGUO5cuUAOHjwIIUKFUpxGL/fT//+/alXrx6VKlUK+usaN25MmzZtWLRoEcWLF+e777677OuGDBlCTExM0m3//v0pzigics1yFoTOn+PMW4J3nG9Q17aVeMLo7H6S7f4SsHd14LJi3sT/3pakqxTvnl23bh0tWrTAZrMRHR0NBGafPp+PhQsXUqdOHaZPn86RI0cYPHhwisL06tWLxYsXs2bNGooXL56ir70aOuVERCx16k+Y3Iz4mBN0dA9ho3kj+YnhE9ezXG87AuX+B22mgN1pdVI576rO04yLi2PmzJns2hVYBioqKor27dsTERFx1UH69OnDggULWLVqFaVLl77q7aSESlNELPfXbpjcjJi4ONq5n2abWYqinOCTkOcobpyASq3h3vcDy/OJ5Sxf3MA0Tfr27cu8efNYsWIFZcuWTbfvrdIUkQzh2HaYfBd/JXho6x7BbrMY1xlH+NT1HAWN01D1QWjxdmBBeLFUqv0EDh8+zL59+1L8db1792bGjBnMmjWLiIgIjhw5wpEjRzh7Vm+Ci0g2UbA8dJpPvlAbM1yjKW4c40+zMB3cQzhl5oRNM2Hx4MA1O8VSqTbTLF++PLt27Urxgu3/tlDC5MmT6dKlSyok+3eaaYpIhrL/R5jekn3nwmnjHsFR8hJt7Gam60UijLNwcx9oMipw7U6xRKqV5o8//khCQgINGjRIjc2lC5WmiGQ4e9fAjPv43Z2Htu4RnCQXtY3tTHWNJcxwQ4Mn4bahVqfMtlK8e/bfrnJSq1YtduzYcc2BRESytVK3wAMzKeM8wTTXaCKIZ71Znh6eASSaDlg5Fta+YXXKbCvFpdm0aVMGDx6Mx+NJeu7EiRPcfffdPPXUU6kaTkQkWyrTCNpOo5LjIFNcLxHOOVb7o+nn6YPXtMGyEbD+fatTZktXNdOcN28etWrVYtu2bXz55ZdUqlSJmJgYNm3alAYRRUSyoahm0PoDath3877zVVy4+cpfm8GeR/CbBiwaBJtmWZ0y20lxadatW5dNmzZRqVIlqlevTqtWrXj88cdZuXLlZZfXExGRq1SxFdzzDvXsv/Ku800ceJnnr89w70OBA2kX9IZf51mdMlu5qlNOdu3axU8//UTx4sVxOBzs3LmThISE1M4mIiJV28Ndr9DYvpHXnO9h4GemrzGjve0x/X74rDvs+srqlNlGiktzzJgx3Hzzzdxxxx1s3bqV9evX8/PPPxMdHf2va76KiMg1qP0w3PE8LezfMcbxAQATff/jTV8r8Hvh447wx0qLQ2YPKT7lpEiRIkyaNIlmzZolPefxeBg6dChvvvlmsktvZXQ65UREMpVvR8PKMUzyNuU5bycAnnZMp7tjMThzQMd5UPImi0NmbSkuzRMnTpA/f/7Lfm7lypU6T1NEJK2YJix9Gr57m7e8LXnV2xaA0Y73aef4FkIiofPnULSqtTmzMMvXnrWSSlNEMh3ThC8HYP44iTHeB5jga4GBn3HOd7nHvg7C8sJDiwJL80mq0+q/IiKZiWHAXa9iVHmApxyz6WhfiomNAZ5eLPXVgLMnYdo9gaunSKpTaYqIZDY2G9zzDkaFFjzrmMq9tlX4sNPH04/Vvkpw5migOE/vtzpplqPSFBHJjOwOaP0htrKNeck5kWa2H3DjpIdnAD/6oyBmP0xrAXFHrU6apag0RUQyK4cL7p+Oo3Q93nC+TQPbJs4SSlf3YLb4S8PJP2B6S0g4aXXSLEOlKSKSmTnDoN1HuEpUZ7xzHLWN7cQRTgf3ELb6S8GxbTDjXjgXa3XSLEGlKSKS2YVEwINzCCsSxYeuV6hm/EYMOWnvHsYv/uvh0M/w0QPg1spt10qlKSKSFYTlho7ziShQkmmuMdQ0dhJLDjq4h7LRXwb+XAufdARv5lmAJiNSaYqIZBU58kOn+UTkKchU15ikXbWd3E/xk/9G+H05fNYNfF6rk2ZaKk0RkawkV1Ho/Dk5cuVliuslbrb9ypnzxfmDvxxs/yJwdRS/3+qkmZJKU0Qkq8lTCjotIDxHTiY5X6a+bTMJhNLF/QTrfBVg82xYPDiwupCkiEpTRCQrKhAFHecRFhrG+85Xk05HecjzRGABhB8/gOUjVZwppNIUEcmqilSBDnMIdbmY4Hyd220bScRFN88gVviiYe04WP2K1SkzFZWmiEhWVqI2tJtFqMPGe85x3GH7CTcuengG8o2vKnwzCr5/z+qUmYZKU0Qkq7u+IbSdSogd3nG+QVPbetw4ecQzILDI+5KnYON0q1NmCipNEZHsIKoZtJqAy/DzlvMtmtu+w4ODRz2PscRXC77oB1s/szplhqfSFBHJLirfBy3exGn4eMP5Di1sa/HioLenH4u9NWBuD9j1ldUpMzSVpohIdlK9E9w5Gofh53Xnu9xrW40PO/08fVntKQefdII9q61OmWGpNEVEspubH4XbhmE3TF52jucu2w94cPCIZwA/u4sF1qk9sMHqlBmSSlNEJDu6dTDU7YfdMHnd+c4lCyA8ya5zkYEroxz91eqUGY5KU0QkOzIMuOM5qNmVEMPLeOfrVD1/dZSO7iHsT3DAtJbw126rk2YoKk0RkezKMOCuV6FyW3IYiUxxvcSNxn6OkpeOnqEcP3MOpt0DMQesTpphqDRFRLIzmw1avgtRzcltxDPdNZrixjH2moXp5H6KmNN/BYrzzDGrk2YIKk0RkezO7oT7JkHpBhQyTjPDOZr8nGa7WYru7kGcPbEPpreCs6esTmo5laaIiIAzFB6YBcVrU8p2lGmuMUQQz49mOXp7+uE5sh1mtoHEM1YntZRKU0REAkJywoOfQqHKVLDtY5LrFUJJ5Bt/dQZ5euLf/xPMbgeec1YntYxKU0RELgrLDR3nQb6y1LLt5D3nGzjwssBfj2e9nTD/WAWfdgGfx+qkllBpiohIcjkLQKf5EFmS2+ybeNU5HgM/U313Ms7bGnYthnk9we+zOmm6U2mKiMg/RRYPFGfOQtxjX8ezjqkAvOFrzWTvnbB1Dnw5INtdxFqlKSIil5fvBug4H8Ly0MmxjAGOTwF41tuZ+b56sGEKLBuRrYpTpSkiIv+uUAXo8Bm4ctLXPo+H7IsBGOR5hBW+aFj3Jqx+xeKQ6UelKSIiV1asBrT/GMMZynDHDO45f0mxXp7+bPSXgW9GwQ8TrE6ZLlSaIiLy30rdAm2nY7Pbedk5nga2TZwllK7uwfzmLwaLn4BNs6xOmeZUmiIiEpwbm8C9E3EZft5zvkFV4zdOE0En91McNPPBgt6w7XOrU6YplaaIiASvUmu4+w3CjUQmu16mjHGAw+Sjk/spTvpzwJyu8PvXVqdMMypNERFJmRqdockL5DHOMM01hqKcYLdZjIfcg4n32WD2g7Dve6tTpgmVpoiIpFzdPtDgSYoaJ5nmGkNu4vjFLEMvT3/cHndgndrDv1idMtWpNEVE5Oo0HAI39aKM7RCTXS8RxjlW+asE1qk9Fxe4MsrxXVanTFUqTRERuTqGAXe+CFU7UM22m/HOcTjw8rm/Hs95O2LGn78W56k/rU6aalSaIiJy9Ww2aPEmVLiHBvbNvOp8D4Apvqa847sH4g4FijPuiMVBU4dKU0REro3NDvd+AGUac4/9O545v07tK977meW9HU7tCeyqTThpcdBrp9IUEZFr53BB2+lQsi4POb6ir30eAE97u7LEVwuObYOZ90FinMVBr41KU0REUocrHNrPhiJVGOD4lHb2r/Fjo5+nD+t8FeDgBvgoc1/EWqUpIiKpJzQSOszDKBDFKMckmtrW48bJw56B/OK/HvauhjkPZdqLWKs0RUQkdeXIB53mY89Tgjecb1PPtpV4wujifpLf/UVh5yKY/yj4/VYnTTGVpoiIpL5cRaHTAkIi8jPB+RpVjN2cIoKO7iGBdWq3fAKLBmW6a3GqNEVEJG3kvR46ziNneBhTXGOT1qnt6B7CCTMX/PQhfP2c1SlTRKUpIiJpp1AFePAz8oTAdNcYinGcP8yidHE/SZwZBmtegzWvW50yaCpNERFJW8VrQLuPKOKIZ7prNPmIYatZmu7ugZwznbB8JPz4odUpg6LSFBGRtFf6Vmg7lesdJ5jqGkMECfxgVqCPpy9e0wZfDoTNn1qd8j+pNEVEJH1ENYNWE6hk28cHrlcIwc1yf02e8PTAbwLzHoGdi61OeUUqTRERST+V74Pmr3KTbQfvON/Ejo+5/lsZ5X0Q0++DTzrDnlVWp/xXKk0REUlftbpB45E0tm/kZecEACb57uJtX0vwJQZWDTqwwdqM/0KlKSIi6e+Wx+GWx7nXvoYRjmkAvOpty3RvY3CfgRn3wtFtFof8J5WmiIhYo9EzULMbXR1L6GefC8AIbxcW+G6Gc6cDV0Y5+Ye1Gf9GpSkiItYwDLjrFajchscdc+hkX4qJjQGeR1nsqwVnjsC0lhB7yOqkSVSaIiJiHZsNWr6HUe4uRjqmcq9tFT7s9PX0ZamvBpz+MzDjjP/L6qSASlNERKxmd8J9k7GVvoWXnRO4x7YWLw56ex7jG19VOL4DZraGc7FWJ1VpiohIBuAMhXYfYS9WjVed79Hc9h0eHPT0PM5KXzQc+vn8tTjPWhpTpSkiIhlDSAR0+AxHwSjGOd9NuhZnD88A1voqwp9r4NMull6LU6UpIiIZR3he6DgPZ94SvOl8i8a2DSTioptnEN/7y8GuJTCvJ/h9lsRTaYqISMaSqwh0WoArV0Hecb5BQ9smzhFCV/cT/OiPgq1zLLsWp0pTREQynjyloOM8QsIjGO98nfq2zSQQShf3E2z0l4GfJsHXz6Z7LJWmiIhkTAXLQ4fPCA0JYaLzNW62/Uo8YXR2P8Uv/usD1+FM52txqjRFRCTjKlYD2s0mzGHwofMVahvbiSOcju4hbPWXClyL86dJ6RZHpSkiIhlb6frQdhrhdh+TXC9Tw9hJLDno4B7CNn9JWDgAtsxJlygqTRERyfiimkKrCeQ0Epnieokqxu+cJoIO7qHs9Bc7fy3OJWkeQ6UpIiKZw/lrcUYYZ5nmGkNl4w9OkotnvZ3A74VPO8PhzWkaQaUpIiKZx/lrcUYaCUx3jaaVbTVvOd8KfK5iKyhYIU2/vWGaFpzokkHExsYSGRlJTEwMuXLlsjqOiIgEa9kzsHbcxce1H4GmYwILwKchzTRFRCTzaTwSajwUuN/gSWg2Ns0LEzTT1ExTRCSz8vvg9+Vw453p9i010xQRkczJZk/XwgSVpoiISNBUmiIiIkFSaYqIiARJpSkiIhIkh9UBrHThwOHY2FiLk4iIiNUiIiIwDOOKr8nWpRkXFwdAiRIlLE4iIiJWC+b0w2x9nqbf7+fQoUNB/evi38TGxlKiRAn279+f5c/11FiznuwyTtBYs6rUHKtmmv/BZrNRvHjxVNlWrly5svz/nBdorFlPdhknaKxZVXqNVQcCiYiIBEmlKSIiEiSV5jUKCQnhmWeeISQkxOooaU5jzXqyyzhBY82q0nus2fpAIBERkZTQTFNERCRIKk0REZEgqTRFRESCpNIUEREJkkpTREQkSCrNNHT69Glq1qxJ1apVqVSpEu+//77VkdLM/v37adiwIRUqVCA6OppPP/3U6khpplWrVuTJk4f77rvP6iipauHChURFRVG2bFk++OADq+Okqaz6M/y77PJ7mZ5/a3XKSRry+XwkJiYSHh5OfHw8lSpV4qeffiJfvnxWR0t1hw8f5ujRo1StWpUjR45Qo0YNdu3aRY4cOayOlupWrFhBXFwcU6dOZc6cOVbHSRVer5cKFSrw7bffEhkZSY0aNVi3bl2W/H8VsubP8HKyy+9lev6t1UwzDdntdsLDwwFITEzENE2y6r9RihQpQtWqVQEoXLgw+fPn5+TJk9aGSiMNGzYkIiLC6hipav369VSsWJFixYqRM2dOmjVrxtKlS62OlWay4s/wcrLL72V6/q1VaV7B6NGjqVWrFhERERQsWJCWLVuyc+fOFG3j9OnTVKlSheLFizN48GDy58+fRmmvTWqM9YINGzbg8/ky5CXXUnOcmc2Vxn7o0CGKFSuW9NpixYpx8OBBq6Jes+z0cw52rBn59zIY/zXO9Ppbq9K8gpUrV9K7d2++//57li1bhsfjoUmTJsTHxwMk7T//++3QoUNJ28idOze//PILe/bsYdasWRw9etSq4VxRaowV4OTJk3Tq1ImJEydaMYz/lFrjzIz+a+xZicaafKwZ/fcyGP81znT7W2tK0I4dO2YC5sqVK6/q63v16mV++umnqZwqbVzNWM+dO2fWr1/fnDZtWhomS11X+zP99ttvzdatW6dRqvRx6djXrl1rtmzZMulzjz32mDlz5kwL06Wuy/2cs8LP8HL+PtbM+HsZjCv97qbl31rNNFMgJiYGgLx58wb1+qNHjxIXF5f0tatWrSIqKirN8qWmlI7VNE26dOnC7bffTseOHdMyWqpK6TizkkvHXrt2bbZu3crBgwc5c+YMixcv5s4777Q4YerJTj/nS8eaWX8vg3HpONP1b22aVHEW5PP5zObNm5v16tUL+mt++OEHs0qVKmZ0dLRZuXJlc/z48WmYMPVczVhXr15tGoZhVqlSJem2efPmNEx57a5mnKZpmo0aNTLz589vhoWFmcWKFTPXrVuXRgnTzuXGvmDBArNs2bLmDTfcYE6YMMHCdKnrcmPNCj/Dy/n7WDPj72Uw/j7O9Pxbq9IMUs+ePc3rrrvO3L9/v9VR0lx2GWt2GeflZKexa6xZj5XjVGkGoXfv3mbx4sXNP/74w+ooaS67jDW7jPNystPYNdasx+pxqjSvwO/3m7179zaLFi1q7tq1y+o4aSq7jDW7jPNystPYNdasJ6OM05E275RmDb1792bWrFksWLCAiIgIjhw5AkBkZCRhYWEWp0td2WWs2WWcl5Odxq6xZr2xZphxWlbXmQBw2dvkyZOtjpbqsstYs8s4Lyc7jV1jzXpjzSjj1NqzIiIiQdJ5miIiIkFSaYqIiARJpSkiIhIklaaIiEiQVJoiIiJBUmmKiIgESaUpIiISJJWmiIhIkFSaIiIiQVJpisi/GjlyJFWrVrU6hkiGodIUkaB16dKFli1bWh1DxDIqTRERkSCpNEWyiIYNG9K3b1/69+9Pnjx5KFSoEO+//z7x8fE89NBDREREUKZMGRYvXgzAlClTyJ07d7JtzJ8/H8MwLrv9kSNHMnXqVBYsWIBhGBiGwYoVK9J4VCIZi0pTJAuZOnUq+fPnZ/369fTt25devXrRpk0b6taty8aNG2nSpAkdO3YkISEhxdseNGgQbdu2pWnTphw+fJjDhw9Tt27dNBiFSMal0hTJQqpUqcLTTz9N2bJlGTJkCKGhoeTPn5+HH36YsmXLMmLECP766y82b96c4m3nzJmTsLAwQkJCKFy4MIULF8blcqXBKEQyLpWmSBYSHR2ddN9ut5MvXz4qV66c9FyhQoUAOHbsWLpnE8kKVJoiWYjT6Uz22DCMZM9deL/S7/djs9n4+zXoPR5P2ocUycRUmiLZVIECBYiLiyM+Pj7puU2bNl3xa1wuFz6fL42TiWRcKk2RbOqmm24iPDycoUOHsnv3bmbNmsWUKVOu+DWlSpVi8+bN7Ny5kxMnTmhmKtmOSlMkm8qbNy8zZsxg0aJFVK5cmY8++oiRI0de8WsefvhhoqKiqFmzJgUKFGDt2rXpE1YkgzDMv7+pISIiIpelmaaIiEiQVJoiIiJBUmmKiIgESaUpIiISJJWmiIhIkFSaIiIiQVJpioiIBEmlKSIiEiSVpoiISJBUmiIiIkFSaYqIiATp/3t2pkBpvOQgAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_grad_scales = [sqrt(erf(1/(mult*sqrt(2)))) for mult in mults]\n",
    "\n",
    "_, ax = plt.subplots(figsize=(5, 3))\n",
    "ax.plot(mults, grad_scales, label=\"experiment\", zorder=1)\n",
    "ax.plot(mults, model_grad_scales, lw=4, label=\"model\", zorder=0)\n",
    "ax.set_xscale(\"log\", base=2); ax.set_yscale(\"log\", base=2); ax.legend()\n",
    "ax.set_xlabel(\"mult\"); ax.set_ylabel(\"x.grad.std\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting these rules together, we can test our new, 'fancy statistical' version of scaled hardtanh:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hardtanh_scaled_statistical\n",
      "         x.std = 1.000\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 1.001\n",
      "\n",
      "hardtanh_scaled_statistical {'mult': 0.25}\n",
      "         x.std = 1.000\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 0.999\n",
      "\n",
      "hardtanh_scaled_statistical {'mult': 4}\n",
      "         x.std = 1.000\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 0.999\n"
     ]
    }
   ],
   "source": [
    "def hardtanh_scaled_statistical(x: Tensor, mult: float = 1.0) -> Tensor:\n",
    "    Z = erf(1 / (mult * sqrt(2)))\n",
    "    y_scale = 1 / sqrt(Z + (1 - Z) / mult**2 - sqrt(2/pi) / mult * exp(-1/2 / mult**2))\n",
    "    grad_scale = 1 / sqrt(Z)\n",
    "    x = scale_bwd(x, grad_scale)\n",
    "    y = F.hardtanh(x, -1/mult, 1/mult)\n",
    "    return scale_fwd(y, y_scale)\n",
    "\n",
    "check_scaling(hardtanh_scaled_statistical); print()\n",
    "check_scaling(hardtanh_scaled_statistical, mult=1/4); print()\n",
    "check_scaling(hardtanh_scaled_statistical, mult=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supporting scaling constraints\n",
    "\n",
    "The version we've got is great for maintaining scale in the forward and backward passes. Unit scaling, however, sometimes requires these scales to be kept consistent. This is enforced by the user according to the cut edge rule (see the [unit scaling](https://arxiv.org/abs/2303.11257) paper for more detail) and supported in the op via a string argument `constraint`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hardtanh {'constraint': 'to_output_scale'}\n",
      "         x.std = 1.000\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 1.149\n",
      "\n",
      "hardtanh {'constraint': 'to_grad_input_scale'}\n",
      "         x.std = 1.001\n",
      "         y.std = 0.870\n",
      "    grad_x.std = 0.999\n",
      "\n",
      "hardtanh {'constraint': 'gmean'}\n",
      "         x.std = 1.000\n",
      "         y.std = 0.932\n",
      "    grad_x.std = 1.073\n",
      "\n",
      "hardtanh {'constraint': None}\n",
      "         x.std = 0.999\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 1.002\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# (Final version)\n",
    "def hardtanh(x: Tensor, mult: float = 1.0, constraint: Optional[str] = \"to_output_scale\") -> Tensor:\n",
    "    Z = erf(1 / (mult * sqrt(2)))\n",
    "    y_scale = 1 / sqrt(Z + (1 - Z) / mult**2 - sqrt(2/pi) / mult * exp(-1/2 / mult**2))\n",
    "    grad_scale = 1 / sqrt(Z)\n",
    "    y_scale, grad_scale = apply_constraint(constraint, y_scale, grad_scale)\n",
    "    x = scale_bwd(x, grad_scale)\n",
    "    y = F.hardtanh(x, -1/mult, 1/mult)\n",
    "    return scale_fwd(y, y_scale)\n",
    "\n",
    "check_scaling(hardtanh, constraint=\"to_output_scale\"); print()\n",
    "check_scaling(hardtanh, constraint=\"to_grad_input_scale\"); print()\n",
    "check_scaling(hardtanh, constraint=\"gmean\"); print()\n",
    "check_scaling(hardtanh, constraint=None); print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see the trade-off implied by constraints. When constraining the scaling factors, either the forward or backward passes can be well-scaled (or some trade-off between them), but in general it isn't possible for both to have good scale when `y_scale == grad_scale`.\n",
    "\n",
    "With larger `mult`, the constrained scaling rule must relax the unit scale requirement, as the ideal forward and backward scales are more different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hardtanh {'mult': 4, 'constraint': 'to_output_scale'}\n",
      "         x.std = 0.999\n",
      "         y.std = 1.000\n",
      "    grad_x.std = 1.906\n",
      "\n",
      "hardtanh {'mult': 4, 'constraint': 'to_grad_input_scale'}\n",
      "         x.std = 1.001\n",
      "         y.std = 0.524\n",
      "    grad_x.std = 0.999\n"
     ]
    }
   ],
   "source": [
    "check_scaling(hardtanh, mult=4, constraint=\"to_output_scale\"); print()\n",
    "check_scaling(hardtanh, mult=4, constraint=\"to_grad_input_scale\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default constraint should generally be `\"to_output_scale\"`, which keeps forward and backward passes consistent while prioritising forward-pass scaling. An exception is for arguments that are typically trainable parameters, where the default constraint should be `None`.\n",
    "\n",
    "Let's take a look at the final scaled op (with default constraint):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAEpCAYAAAAAiAjcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABtFUlEQVR4nO3dd3gU5drH8e8mIY0USholFKW3JFSD5whIJIAiqAcVOFIUfEVQESuKIFiwUUQRsIGgHLCBBUWKhCIISBKagIJCAFNAQgKB1N33j5XVhSRskk12N/l9rmsuZmefZ+Zelgy552kGk8lkQkREREREREQcys3RAYiIiIiIiIiIEnQRERERERERp6AEXURERERERMQJKEEXERERERERcQJK0EVEREREREScgBJ0ERERERERESegBF1ERERERETECShBFxEREREREXECStBFREREREREnIASdBEREREREREn4FIJ+saNG+nXrx9169bFYDCwYsWKYsvHxcVhMBgu21JSUiomYBEREREREREbuVSCnpWVRUREBHPmzClRvYMHD5KcnGzZQkJCbK5rMpnIzMzEZDKVNFwREXEhut+LiIiIo3k4OoCS6NOnD3369ClxvZCQEGrUqFGqa549e5bAwEAyMjIICAgo1TlERMT56X4vIiIijuZSLeilFRkZSZ06dbjhhhv44Ycfii2bk5NDZmam1SYiIiIiIiJS3ip1gl6nTh3mzZvHZ599xmeffUZ4eDjdu3cnPj6+yDrTpk0jMDDQsoWHh1dgxCIiIiIiIlJVGUwuOtjOYDCwfPlyBgwYUKJ63bp1o0GDBixevLjQ93NycsjJybG8zszMJDw8XF0eRUQquczMTHVxFxEREYdyqTHo9tC5c2c2b95c5PteXl54eXlVYEQiIiIiIiIilbyLe2ESExOpU6eOo8MQERERERERseJSLejnzp3j0KFDlte///47iYmJ1KpViwYNGjBhwgROnDjBokWLAJg1axaNGzemdevWZGdn8+677/L999+zevVqR30EERERERERkUK5VIL+008/0aNHD8vr8ePHAzBs2DAWLlxIcnIySUlJlvdzc3N55JFHOHHiBL6+vrRr1461a9danUNERERERETEGbjsJHEVRZMGiYhUDbrfi4iIiKO5VAu6iFQym2fBnk9AzwnLR5tb4bpHHR2FSKWTlZfF4xsfJzkr2dGhFKpVrVY8d+1zGAwGR4ciIiIlpARdRBwjZS+sfRZQcl5uznZ1dAQilVJCWgIbj290dBhF+jX9V0ZHjqaeXz1HhyIiIiWkBF1EHGPtZMAEzXpDl/scHU3lFKBfzkXKg9FkBKBhQEOe7vK0g6Ox9sD3D5BTkGOJUUREXIsSdBGpeIfXw6G14FYNek+DWlc5OiIRkRLzq+ZHdN1oR4dhxd3g7ugQRESkDKrcOugi4mBGI6x5xrzf6R4l5yIiIiIif1GCLiIVa8/HkLIHvALguscdHY2IiIiIiNNQgi4iFSfvAqx7zrz/7/FQvbZj4xERERERcSJK0EWk4mybB5nHIaC+JoYTEREREbmEEnQRqRhZf8KmGeb96ydCNR/HxiMiUkomkwssD+kCIYqIyOWUoItIxdj4KuRkQlhbaHeHo6MREREREXE6StBFpPyd/g12vGvev+E5cNOtR0RcnwGDo0O4jMHgfDGJiIjt9FuyiJS/dVPBmAdX94Srezg6GhERERERp6QEXUTK1/GfYN9ywAA3THV0NCIiIiIiTksJuoiUH5MJVk8070cOgbA2jo1HRERERMSJKUEXkfJz8BtI2goePtDjKUdHIyIiIiLi1JSgi0j5KMiDNZPN+9H3Q2A9x8YjImInJhdYw8wVYhQRkcspQReR8hH/Afz5K/jWhmvHOToacTFz586lXbt2BAQEEBAQQHR0NN9++22xdT755BNatGiBt7c3bdu25ZtvvqmgaEVERETsQwm6iNhfzlmIe8m83+1J8A5wbDzicurXr89LL73Ezp07+emnn7j++uvp378/+/btK7T8li1bGDRoEPfccw8JCQkMGDCAAQMGsHfv3gqOXKoSZ1zSzBmXfhMREdspQRcR+/thNmSdhFpXQ8cRjo5GXFC/fv3o27cvTZs2pVmzZrzwwgv4+fnx448/Flr+9ddfp3fv3jz22GO0bNmS5557jvbt2/Pmm29WcOQiIiIipacEXUTsKzMZtv6VFMU8C+7VHBqOuL6CggKWLl1KVlYW0dHRhZbZunUrMTExVsdiY2PZunVrRYQoIiIiYhcejg5ARCqZ9S9A3nkI7wIt+zk6GnFhe/bsITo6muzsbPz8/Fi+fDmtWrUqtGxKSgqhoaFWx0JDQ0lJSSny/Dk5OeTk5FheZ2Zm2idwERERkVJSC7qI2E/qz5D4kXn/hufACcdniuto3rw5iYmJbNu2jdGjRzNs2DB+/vlnu51/2rRpBAYGWrbw8HC7nVtERESkNNSCLlLJpWenM/b7saSdTyv/i2Wdgvph5nXPt0+E7eV/SSnazVffzANRDzg6jFLz9PSkSZMmAHTo0IEdO3bw+uuvM3/+/MvKhoWFkZqaanUsNTWVsLCwIs8/YcIExo8fb3mdmZmpJF1sYjI5/xJmWmZNRMQ1KUEXqeTWJq1l98ndFXdBDw8gD7KK7losFSMzp3J12TYajVZd0v8pOjqadevWMW7cOMuxNWvWFDlmHcDLywsvLy97hykiIiJSakrQRSq5xLREAAY2G8htzW4rn4sYTbD8Xjj1C7S+Ff71UPlcR0qkllctR4dQahMmTKBPnz40aNCAs2fPsmTJEuLi4vjuu+8AGDp0KPXq1WPatGkAPPTQQ3Tr1o3p06dz4403snTpUn766SfefvttR34MqeSccUkzZ4xJRERspwRdpJKLT40H4PoG19O6duvyucjuj+GPfeDpDzEvQvWg8rmOVBlpaWkMHTqU5ORkAgMDadeuHd999x033HADAElJSbi5/T2NSteuXVmyZAkTJ07kqaeeomnTpqxYsYI2bdo46iOIiIiIlJgSdJFK7NSFUxw/dxwDBiKCI8rnInnZsG6qef9f45Sci1289957xb4fFxd32bGBAwcycODAcopIREREpPy51CzuGzdupF+/ftStWxeDwcCKFSuuWCcuLo727dvj5eVFkyZNWLhwYbnHKeIsEtISAGhasyn+nv7lc5Htb0PGMfCvC9fcXz7XEBERERGpAlwqQc/KyiIiIoI5c+bYVP7333/nxhtvpEePHiQmJjJu3DhGjhxpGcMoUtld7N4eFRJVPhc4fxo2vWbev34iePqWz3VERERERKoAl+ri3qdPH/r06WNz+Xnz5tG4cWOmT58OQMuWLdm8eTMzZ84kNja2vMIUcRoXJ4grtwR942uQnQEhrSHizhJXz803kpCUjlGrAZWLsEBvGgdVd3QYIuIArrAUnIiIXM6lEvSS2rp1KzExMVbHYmNjrZbhuVROTo7VMj6ZmZVrmSKpOs7nnWf/6f1AOSXo6UfM3dsBek0FN/cSVc/KyefWt7ZwMPWs/WMTAIZGN2Rqf02SJmJvWmNcRETKS6VO0FNSUggNDbU6FhoaSmZmJhcuXMDHx+eyOtOmTWPKlCkVFaJIudl7ai8FpgJCfUOpU72O/S+wbioY8+CqHtAk5srl/8FkMvHk53s4mHoWPy8PwgK97R+fEOynNb5FypVWNBMRETur1Al6aUyYMIHx48dbXmdmZhIeHu7AiERK5+IEcVEhURgMdv4t8sRO2PsZYIAbppa4+oc/HuWrXX/g7mZg4YhOdGzkuut1i4g4FT00EBFxaZU6QQ8LCyM1NdXqWGpqKgEBAYW2ngN4eXnh5aVWJ3F9/0zQ7cpkgtWTzPsRd0KddiWqnnjsDFO//hmACX1aKDkXEREREfmLS83iXlLR0dGsW7fO6tiaNWuIjo52UEQiFaPAWMCuk7uAckjQf1kFRzeDu5d55vYSOHM+lzEfxZNXYCK2dSj3/KuxfWMTEREREXFhLpWgnzt3jsTERBITEwHzMmqJiYkkJSUB5u7pQ4cOtZS/7777+O2333j88cc5cOAAb731Fh9//DEPP/ywI8IXqTCHzhziXN45fD18aVqzqf1OXJAPa/5qPb9mNATWt7mq0Whi/Me7OHHmAg1r+/LqwAj7d70XEREREXFhLpWg//TTT0RFRREVZW4RHD9+PFFRUUyaZE4YkpOTLck6QOPGjVm5ciVr1qwhIiKC6dOn8+6772qJNan0LnZvjwiOwMPNjiNZEhbDqV/Apxb8e/yVy//D3A2H+f5AGl4ebrw1pD0B3tXsF5eIiFjRTPMiIq7Jpcagd+/evdh1PRcuXFhonYSEhHKMSsT5lMv485xzsP5F8363J8A70OaqWw6fYvrqgwA8178NrevaXldExNko+RURkfLiUi3oImIbS4IeascEfcsbkJUGNRtDx7ttrpaamc2D/0vAaIKBHepzeyetiiAilYNBU6aLiIidKUEXqWRSslJIzkrG3eBOu6CSzbBepLMp5gQdIGYyeHjaVC2/wMgDSxI4dS6XFmH+TO3fxj7xiIhIofTQQETEtSlBF6lkLraeN6vZDN9qvvY5adw0yMuCeh2h1QCbq726+iDbj5zGz8uDuf/tgI+nu33iERERERGphJSgi1QyFxP09qHt7XPCtAMQv8i83+t5sHHm9dX7Upi/4TcAXvlPOxoHVbdPPCIiIiIilZQSdJFKJjEtEYDIkEj7nHDts2AyQouboGG0TVWS/jzPI5+Y12G/+9rG9G1bxz6xiIiIiIhUYkrQRSqRrLwsDqabZ0uPCrbDBHFHNsMv34LBHWKetalKdl4B9y/ZydnsfNo3qMGTfVqUPQ4RESkRzTQvIuKalKCLVCK7Tu7CaDJSz68eodVDy3YyoxFWTzTvdxgOQU1tqjb165/ZeyKTWtU9eXNwezw9dJsRkUpGua+IiJQT/eYsUolcHH9ul+7t+z6HPxLA0w+6P2lTlc/jj7NkWxIGA8y6I5K6NXzKHoeIiJPSjOkiImJvStBFKhHLBHEhZZwgLj8H1k0x7187DvxCrljlYMpZnl6+F4CHejblumbBZYtBRERKzGDjRJ4iIuKclKCLVBL5xnx2n9wN2KEFffs7cCYJ/OtA9JgrFj+Xk8/oj3ZyIa+AfzcN4oHrbesOLyIiIiIif1OCLlJJHEw/yIX8C/hX86dJjSalP9GFdNj4qnm/x1PgWfxa6iaTiSc+281vJ7MIC/Bm1h2RuLupBUfKZtq0aXTq1Al/f39CQkIYMGAABw8eLLbOwoULMRgMVpu3t3cFRSwiIiJSdkrQRSqJi8urRYRE4GYow4/2pumQfQZCWkHkkCsWX7T1KCt3J+PhZmDOkChq+3mV/toif9mwYQNjxozhxx9/ZM2aNeTl5dGrVy+ysrKKrRcQEEBycrJlO3r0aAVFLOJkNJGdiIhL8nB0ACJiH/Gp8QBEhZRhebX0o7Btvnn/hqng5l5s8YSkdJ5f+TMAE/q2pEPDWqW/tsg/rFq1yur1woULCQkJYefOnVx33XVF1jMYDISFhZV3eCIiIiLlQi3oIpWAyWSytKCXKUH//nkoyIXG10GTmGKLpmflMnZJAnkFJvq0CePuaxuV/roiV5CRkQFArVrFPwQ6d+4cDRs2JDw8nP79+7Nv374iy+bk5JCZmWm1idhCa4yLiEh5UYIuUgn8kfUHaRfS8DB40CaoTSlPkgB7Pjbv3/AcFDMTsNFo4uGPEzlx5gKNg6rzyn/aaeZgKTdGo5Fx48Zx7bXX0qZN0f++mzdvzvvvv88XX3zBhx9+iNFopGvXrhw/frzQ8tOmTSMwMNCyhYeHl9dHkEpK9z0REbE3JegilcDF7u2tarfCx6MUa4+bTLD6GfN+uzugbmSxxd+KO0TcwZN4ebjx1pD2+HtXK/k1RWw0ZswY9u7dy9KlS4stFx0dzdChQ4mMjKRbt258/vnnBAcHM3/+/ELLT5gwgYyMDMt27Nix8ghfpEJpbXYREdemMegilcDF7u2lXl7t1zVwZBO4e8H1E4st+sOhU8xY8wsAzw9oQ8s6AaW7pogNxo4dy9dff83GjRupX79+iepWq1aNqKgoDh06VOj7Xl5eeHlpUkMRERFxHmpBF6kEEk4mAKUcf16QD2smmfe7/B/UaFBk0ZSMbB5amoDRBLd3rM/AjuoSLOXDZDIxduxYli9fzvfff0/jxo1LfI6CggL27NlDnTp1yiFCEREREftTC7qIi8vMzeRQurmFsFQt6Ikfwcn94FMT/v1IkcXyCow88L94Tp3LpUWYP1P7l3Ksu4gNxowZw5IlS/jiiy/w9/cnJSUFgMDAQHx8zMM4hg4dSr169Zg2bRoAU6dO5ZprrqFJkyacOXOGV199laNHjzJy5EiHfQ4RR9FEdiIirkkJuoiL25W2CxMmGvg3IMgnqGSVc7Ng/Yvm/eseA58aRRZ99buD7DiSjr+XB3P/2wHvasUvwSZSFnPnzgWge/fuVscXLFjA8OHDAUhKSsLN7e+OYOnp6YwaNYqUlBRq1qxJhw4d2LJlC61ataqosEVERETKRAm6iItLSCtD9/atc+BcCtRoCJ2KbmX8bl8Kb2/8DYBXB7ajcVD1UsUqYiuT6cqtf3FxcVavZ86cycyZM8spIpG/XWyd1oRsIiJibxqDLuLiSp2gn0uDH14378dMBo/CJ8s6+mcWj36yC4CR/2pM7zYazysiIiIiUh6UoIu4sLyCPPae2guUIkGPewlyz0Hd9tD61kKLZOcVMPrDeM5m59OhYU2e6NOirCGLiEg5Uqu+iIhrU4Iu4sL2n95PdkE2Nbxq0DiwBLNcn/wFdi407/d6HgyF/0I35at9/JycSe3qnswZ3J5q7rpliIiIiIiUF/22LeLCLnZvjwyOxFBEkl2otc+CqQCa94VG1xZa5NOdx/nf9mMYDPD6nVGEBXrbIWIRERERESmKEnQRF2YZfx5agu7tR7fAwZVgcIeYZwstciAlk4kr9gAwrmcz/tW0hLPDi4iIQ9ky0aKIiDgfl0vQ58yZQ6NGjfD29qZLly5s3769yLILFy7EYDBYbd7eagWUysFkMpV8gjiTCVY/Y95vPxSCm19W5Gx2HqM/jCc7z8h1zYJ54Pom9gpZRERERESK4VIJ+rJlyxg/fjyTJ08mPj6eiIgIYmNjSUtLK7JOQEAAycnJlu3o0aMVGLFI+Tl29hins09Tza0arWrbuM7zvuVw4ieoVh26T7jsbZPJxJOf7eH3U1nUCfRm1h2RuLlpwiERERERkYrgUgn6jBkzGDVqFCNGjKBVq1bMmzcPX19f3n///SLrGAwGwsLCLFtoaGgFRixSfuLT4gFoE9QGL/fCl0izkp8L66aY9699EPwv/1lYuOUIK/ck4+FmYM6Q9tSq7mnPkEVEKgV1HxcRkfLiMgl6bm4uO3fuJCYmxnLMzc2NmJgYtm7dWmS9c+fO0bBhQ8LDw+nfvz/79u2riHBFyl1iWiIAkSGRtlX46T1IPwJ+oRA99rK345PSefGb/QA8fWNL2jeoaZ9ARURERETEJi6ToJ86dYqCgoLLWsBDQ0NJSUkptE7z5s15//33+eKLL/jwww8xGo107dqV48ePF3mdnJwcMjMzrTYRZ3SxBb19SPsrF75wBja8bN7v8RR4+Vm9fTorl7EfxZNXYOLGtnUY3rWRfYMVEZEKUaIVPURExOm4TIJeGtHR0QwdOpTIyEi6devG559/TnBwMPPnzy+yzrRp0wgMDLRs4eHhFRixiG3OZJ/h94zfAfMSa1e0eSZcSIeg5hD5X6u3jEYT45Yl8kdGNlcFVeel29rqFzwREREREQdwmQQ9KCgId3d3UlNTrY6npqYSFhZm0zmqVatGVFQUhw4dKrLMhAkTyMjIsGzHjh0rU9wi5SHxZCIAjQMbU8O7RvGFzxyDH+ea92+YCu4eVm+/uf4QG385iXc1N976b3v8vavZP2AREalQJjROXkTEFblMgu7p6UmHDh1Yt26d5ZjRaGTdunVER0fbdI6CggL27NlDnTp1iizj5eVFQECA1SbibErUvf3756EgBxr9G5rFWr21+ddTzFz7CwDPD2hLizD9excRERERcRSPKxdxHuPHj2fYsGF07NiRzp07M2vWLLKyshgxYgQAQ4cOpV69ekybNg2AqVOncs0119CkSRPOnDnDq6++ytGjRxk5cqQjP4ZImdk8QVzyLti9zLx/w1T4R9f15IwLPLg0AZMJ7uwUzn861C+fYEVEKikNBxIREXtzqQT9jjvu4OTJk0yaNImUlBQiIyNZtWqVZeK4pKQk3Nz+7hSQnp7OqFGjSElJoWbNmnTo0IEtW7bQqpWNa0aLOKGcghz2ntoLXKEF3WSCNZMAE7T5D9T7u2xegZGxSxI4nZVLqzoBPHtz63KOWkSk8lD3cRERKS8ulaADjB07lrFjL18iCiAuLs7q9cyZM5k5c2YFRCVScX7+82fyjHnU8q5FuH8xkxgeWge/xYG7J/R8xuqtl789wM6j6fh7ezD3v+3xruZevkGLiIiIiMgVucwYdBExS0hLACAqJKro7pXGgr9az4HO90LNRpa3Vu1N5t3N5hngXxsYQcPa1cszXBERERERsZESdBEXk5D6d4JepF3/g7R94B0I/37EcvjIqSwe+2Q3APdedxWxrW1bAUFERERERMqfEnQRF2I0GS1LrBWZoOeeN8/cDnDdY+BbC4DsvAJGfxTP2Zx8OjWqyWOxzSsgYhERcQSNkxepON27d2fcuHEVdr1GjRoxa9ascr/OkSNHMBgMJCYmXrHsXXfdxYsvvmjzuZ988kkeeOCBMkRXeSlBF3EhRzKOcCbnDN7u3rSs1bLwQj++BWeTIbABdBplOTz5i33sT86kdnVP3hjUnmru+vEXERERKUr37t0xGAyXbfn5+Y4OzYrBYGDFihUOu/6uXbv45ptvePDBB22u8+ijj/LBBx/w22+/lWNkrkm/oYu4kIvjz9sEtaGae7XLC5w7CZtnmfd7ToJq3gB8/NMxlv10DIMBZg+KIizQu4IiFhGpvAxomTWRym7UqFEkJydbbR4e5T/PdkFBAUajsdyvYw9vvPEGAwcOxM/Pz+Y6QUFBxMbGMnfu3HKMzDUpQRdxIf+cIK5QG16G3LNQJxLa3AbAz39k8swK87Js42OacW2ToIoIVaRMpk2bRqdOnfD39yckJIQBAwZw8ODBK9b75JNPaNGiBd7e3rRt25ZvvvmmAqKVqkbdx0Wc06JFi6hduzY5OTlWxwcMGMBdd91VqnP6+voSFhZmtRXGaDTy+OOPU6tWLcLCwnj22Wet3p8xYwZt27alevXqhIeHc//993Pu3DnL+wsXLqRGjRp8+eWXtGrVCi8vL5KSkkhLS6Nfv374+PjQuHFjPvroI6vzNmrUCIBbbrkFg8FgeX348GH69+9PaGgofn5+dOrUibVr115W98UXX+Tuu+/G39+fBg0a8Pbbb1/22X777Td69OiBr68vERERbN261fJeQUEBn376Kf369bMcO3DgAL6+vixZssRy7OOPP8bHx4eff/7Zcqxfv34sXbq00L/PqkwJuogLKTZBP3UIdi4w7/d6DtzcOJudx5gl8eTkG+nePJgxPZpUYLQipbdhwwbGjBnDjz/+yJo1a8jLy6NXr15kZWUVWWfLli0MGjSIe+65h4SEBAYMGMCAAQPYu3dvBUYuIlI5mUwmzufmO2QzmWx7KDZw4EAKCgr48ssvLcfS0tJYuXIld999N5s2bcLPz6/Y7dIE2FYffPAB1atXZ9u2bbzyyitMnTqVNWvWWN53c3Nj9uzZ7Nu3jw8++IDvv/+exx9/3Ooc58+f5+WXX+bdd99l3759hISEMHz4cI4dO8b69ev59NNPeeutt0hLS7PU2bFjBwALFiwgOTnZ8vrcuXP07duXdevWkZCQQO/evenXrx9JSUlW15w+fTodO3YkISGB+++/n9GjR1/2QPzpp5/m0UcfJTExkWbNmjFo0CBLN//du3eTkZFBx44dLeVbtGjBa6+9xv33309SUhLHjx/nvvvu4+WXX6ZVq1aWcp07d+b48eMcOXKkVH/nlZXBZOu/+CoqMzOTwMBAMjIyCAgIcHQ4UoWdunCKHh/3wICBzYM2E+B5yb/HZf+F/V9B01gY8jEmk4kxS+L5Zk8KdQO9Wfngv6lZ3dMxwYuU0cmTJwkJCWHDhg1cd911hZa54447yMrK4uuvv7Ycu+aaa4iMjGTevHlXvIbu92KrVUdW8diGx+gY2pEFvRc4Ohwr3Zd158/sP/ns5s9oVrOZo8ORSuR8bj6tJn3nkGv/PDUWX0/bupXff//9HDlyxNKDasaMGcyZM4dDhw6RnZ3NiRMniq0fGhqKv78/YB6DvmXLFjw9//796f/+7/+YPn26VZ3u3btTUFDApk2bLMc6d+7M9ddfz0svvVTodT799FPuu+8+Tp06BZhb0EeMGEFiYiIREREA/PLLLzRv3pzt27fTqVMnwNw63bJlS2bOnGmZmM5gMLB8+XIGDBhQ7Gdr06YN9913H2PHjgXMLej//ve/Wbx4MWB+CBMWFsaUKVO47777OHLkCI0bN+bdd9/lnnvuAeDnn3+mdevW7N+/nxYtWrBixQr+85//kJeXd9nyvzfddBOZmZl4enri7u7OqlWrrMpc/H83Li6Obt26FRt7VVL+AyhExC4S0xIBaFKzyeXJedKP5uTc4AY3TAFgwQ9H+GZPCtXcDcwZ0l7Jubi0jIwMAGrVqlVkma1btzJ+/HirY7GxsUVOnJOTk2PVDTIzM7PsgYqIiEONGjWKTp06ceLECerVq8fChQsZPnw4BoMBHx8fmjQpWW/CIUOG8PTTT1te16hRo9By7dq1s3pdp04dq5butWvXMm3aNA4cOEBmZib5+flkZ2dz/vx5fH19AfD09LQ6z/79+/Hw8KBDhw6WYy1atCgyhn86d+4czz77LCtXriQ5OZn8/HwuXLhwWQv6P69nMBgICwuzivvSMnXq1AHMPRNatGjBhQsX8PLyuiw5B3j//fdp1qwZbm5u7Nu377IyPj4+gLnngPxNCbqIi7B0bw++pHu7yQSrnzHvR/0XQlqy82g6L36zH4CJN7YiqkHNigxVxK6MRiPjxo3j2muvpU2bNkWWS0lJITQ01OpYaGgoKSkphZafNm0aU6ZMsWusIs5CHSTF3nyqufPz1FiHXdtWUVFRREREsGjRInr16sW+fftYuXIlAJs2baJPnz7F1p8/fz5DhgyxvA4MDLQpqa9WzXryXoPBYJnk7ciRI9x0002MHj2aF154gVq1arF582buuececnNzLQm6j49PoYluaTz66KOsWbOG1157jSZNmuDj48N//vMfcnNzbY67sDIX47tYJigoiPPnz5Obm2vV0wDMs7tnZWXh5uZGcnKyJbm/6PTp0wAEBweX4ZNWPkrQRVzExRb0yJBI6zf2fwnHt0M1X+jxNH+ey2HsknjyjSZubFeHodENKzxWEXsaM2YMe/fuZfPmzXY974QJE6xa3DMzMwkPD7frNUREKguDwWBzN3NHGzlyJLNmzeLEiRPExMRY7u0dO3a84prelz7otYedO3diNBqZPn06bm7mKcA+/vjjK9Zr0aIF+fn57Ny509LF/eDBg5w5c8aqXLVq1SgoKLA69sMPPzB8+HBuueUWwNyiXh5jvSMjIwFz1/eL+2BOvocPH87TTz9NcnIyQ4YMIT4+3tJqDrB3716qVatG69at7R6XK9MkcSIu4EL+BX7+0zzrZfvQ9n+/kZ8La58173d9gILqoYxblkhyRjZXBVfn5dva2e1JrIgjjB07lq+//pr169dTv379YsuGhYWRmppqdSw1NbXIGXe9vLwICAiw2kRKQvdXEec0ePBgjh8/zjvvvMPdd99tOX6xi3tx28Xx58Xp2bMnb775ps3xNGnShLy8PN544w1+++03Fi9ebNPcKM2bN6d379783//9H9u2bWPnzp2MHDnSKskF81jydevWkZKSQnp6OgBNmzbl888/JzExkV27djF48OByWbYtODiY9u3bX/YQ/b777iM8PJyJEycyY8YMCgoKePTRR63KbNq0iX//+9+XfZ6qTgm6iAvYe2ov+aZ8QnxCqFu97t9v7FwAp3+D6sHQ9QHe+P5XNv16Cu9qbswd0gE/L9d40i1yKZPJxNixY1m+fDnff/89jRs3vmKd6Oho1q1bZ3VszZo1REdHl1eYUlWp97iIUwsMDOS2227Dz8/vihOnlcbhw4ctk7vZIiIighkzZvDyyy/Tpk0bPvroI6ZNm2ZT3QULFlC3bl26devGrbfeyr333ktISIhVmenTp7NmzRrCw8OJijIPhZwxYwY1a9aka9eu9OvXj9jYWNq3b1/YJcps5MiRVrPfL1q0iG+++YbFixfj4eFB9erV+fDDD3nnnXf49ttvLeWWLl3KqFGjyiUmV6ZZ3K9As/qKM3hn9zvMTphNbKNYXuv2mvlgdgbMjoLzf8KNM9gYeDPDFmzHZIIZt0dwa/viWxtFnNn999/PkiVL+OKLL2jevLnleGBgoOVJ+9ChQ6lXr57ll5wtW7bQrVs3XnrpJW688UaWLl3Kiy++SHx8fLFj1y/S/V5ster3VTy28TE6hXXi/dj3HR2OlYuzuH/a71Oa12p+5QoilVTPnj1p3bo1s2fPdnQold6FCxdo3rw5y5Yts/mh+LfffssjjzzC7t278fBQg9I/qQVdxAXEp8UDl6x/vnmWOTkPakZyk9sZtywRkwkGdW6g5FwcJjs7u8j3kpOTbT7P3LlzycjIoHv37tSpU8eyLVu2zFImKSnJ6pxdu3ZlyZIlvP3220RERPDpp5+yYsUKm5JzkcpC3e6lqktPT2f58uXExcUxZswYR4dTJfj4+LBo0aIS9SrIyspiwYIFSs4Lob8RESdnNBnZlbYL+McEcRnH4ce3AMi/fjJj/reb01m5tK4bwOR+rRwUqQi0b9+eJUuWWE0UA/DZZ59x3333cfLkSZvOY0vnrri4uMuODRw4kIEDB9p0DRERqXyioqJIT0/n5ZdftuqBJeWre/fuJSr/n//8p3wCqQTUgi7i5A6dOcTZvLP4ePjQvOZf/9GsfxHys6FBV6Ydbkx80hn8vT2YO6QD3iVYikTE3rp3784111zDyy+/DJifkA8fPpy77rqLp556ysHRiYhIZXfkyBEyMjIum5BMxFWoBV3EyV1cXq1dcDs83DwgZQ8kLgFgy9XjeO/bIwBMHxhBg9q+DopSxOytt97ixhtvZOTIkXz99dckJyfj5+fH9u3b1dVcRERE5AqUoIs4uYS0BADah/w18+aaSYCJc01v5t7vzYf+77qr6NW68KWkRCpanz59uPXWW5k7dy4eHh589dVXSs6lUjKg8d4iImJf6uIu4uQuJuiRIZFwaB0c/h6TWzXGpvTjXE4+nRvV4tFYjbES53D48GGio6P5+uuv+e6773j88ce5+eabefzxx8nLy3N0eCJ2YdI6ayIiUk6UoIs4sdSsVE6cO4GbwY2I2m1gzWQANtUYQNzJ6gT5efLG4CiquetHWZxDZGQkjRs3ZteuXdxwww08//zzrF+/ns8//5zOnTs7OjwRERERp6bf6kWcWMJJc+t585rNqb7/a0jdQ66HPw/+EYObAWYPiiI0wNvBUYr87a233mLp0qXUqFHDcqxr164kJCTQvn17xwUmUkWo272IiGtTgi7ixC5OEBcZ1Aa+fx6A13P6cQZ/HunVnK5XBzkwOpHL3XXXXYUe9/f357333qvgaERERERciyaJE3FilgnizqRB5glSDMG8m9eLHs2DGd3tagdHJ2L25Zdf2lTOYDDQr1+/co5GREDj5EXsoVGjRowbN45x48ZVyPUMBgPLly9nwIAB5XqduLg4evToQXp6ulWPt8Jcd9113HfffQwePNimc99555106tSJRx55xA6RVk1K0EWc1Pm88xw8fRCAiN3mBOilnIEE1Qhkxu2RuLmpG6M4h0t/kTAYDJhMJqvXFxUUFFRUWCIiIi7ryJEjNG7cmISEBCIjIx0Sw5dffklqaip33nmnzXUmTpzIddddx8iRIwkMDCzH6CovdXEXcVK7T+2mwFRAHTdv6lzIYK+xEd8YrmXOkPbUrO7p6PBELIxGo2VbvXo1kZGRfPvtt5w5c4YzZ87wzTff0L59e1atWuXoUEXsSuO9RaSkcnNzHR2CzWbPns2IESNwc7M9ZWzTpg1XX301H374YTlGVrkpQRdxUgmp5u7tUZmnAXgxfzATb2pDZHgNB0YlUrxx48bx+uuvExsbS0BAAAEBAcTGxjJjxgwefPBBR4cnIiKV2KJFi6hduzY5OTlWxwcMGFDkHClXcv78ee6++278/f1p0KABb7/9ttX7TzzxBM2aNcPX15errrqKZ555xmpZ0WeffZbIyEjeffddGjdujLe3eXLfX3/9leuuuw5vb29atWrFmjVrrM7buHFjAKKiojAYDHTv3h2AHTt2cMMNNxAUFERgYCDdunUjPj7eqq7BYODdd9/llltuwdfXl6ZNmxY6HG3nzp107NgRX19funbtysGDBy3vnTx5ku+//95qaFpcXByenp5s2rTJcuyVV14hJCSE1NRUy7F+/fqxdOlSm/5+5XIul6DPmTOHRo0a4e3tTZcuXdi+fXux5T/55BNatGiBt7c3bdu25ZtvvqmgSEXK5uL486jsbNYXRFC7bS/uuqahg6MSKd7hw4cLHc8WGBjIkSNHKjwekfLwzyEcIlWGyQS5WY7ZbPyZGzhwIAUFBVbJaFpaGitXruTuu+9m06ZN+Pn5Fbt99NFHVuecPn06HTt2JCEhgfvvv5/Ro0dbJbL+/v4sXLiQn3/+mddff5133nmHmTNnWp3j0KFDfPbZZ3z++eckJiZiNBq59dZb8fT0ZNu2bcybN48nnnjCqs7FHGft2rUkJyfz+eefA3D27FmGDRvG5s2b+fHHH2natCl9+/bl7NmzVvWnTJnC7bffzu7du+nbty9Dhgzh9OnTVmWefvpppk+fzk8//YSHhwd333235b3Nmzfj6+tLy5YtLce6d+/OuHHjuOuuu8jIyCAhIYFnnnmGd999l9DQUEu5zp07s3379sselIhtXGoM+rJlyxg/fjzz5s2jS5cuzJo1i9jYWA4ePEhISMhl5bds2cKgQYOYNm0aN910E0uWLGHAgAHEx8fTpk0bB3wCEdvkG/PZlWZ+GhqRnct0/7uZdWtbq7G8Is6oU6dOjB8/nsWLF1v+s05NTeWxxx7TOugiIq4s7zy8WNcx137qD/CsfsViPj4+DB48mAULFjBw4EAAPvzwQxo0aED37t3Jzs4mMTGx2HP8M9EE6Nu3L/fffz9gbi2fOXMm69evp3nz5oB5zPVFjRo14tFHH2Xp0qU8/vjjluO5ubksWrSI4OBgAFavXs2BAwf47rvvqFvX/Hf64osv0qdPH0udi2Vr165NWFiY5fj1119vFd/bb79NjRo12LBhAzfddJPl+PDhwxk0aJDl3LNnz2b79u307t3bUuaFF16gW7duADz55JPceOONZGdn4+3tzdGjRwkNDb2se/vzzz/PmjVruPfee9m7dy/Dhg3j5ptvtipTt25dcnNzSUlJoWFDNS6VVIlb0IcNG8bGjRvLI5YrmjFjBqNGjWLEiBG0atWKefPm4evry/vvv19o+ddff53evXvz2GOP0bJlS5577jnat2/Pm2++WcGRi5TMr6d/4XxBDn5GI3tzuvL4sNvw83Kp52lSRb3//vskJyfToEEDmjRpQpMmTWjQoAEnTpzQMmsiFUDj4qWqGzVqFKtXr+bEiRMALFy4kOHDh2MwGPDx8bH831TU5u/vb3W+du3aWfYNBgNhYWGkpaVZji1btoxrr72WsLAw/Pz8mDhxIklJSVbnaNiwoSXhBti/fz/h4eGW5BwgOjraps+XmprKqFGjaNq0KYGBgQQEBHDu3LnLrvnPuKtXr05AQIBV3JeWqVOnDoClzIULFyzd8f/J09OTjz76iM8++4zs7OzLeguA+UEJmIcHSMmV+Df+jIwMYmJiaNiwISNGjGDYsGHUq1evPGKzkpuby86dO5kwYYLlmJubGzExMWzdurXQOlu3bmX8+PFWx2JjY1mxYkWR18nJybHqjpGZmVm2wEVK4Zsf3gWgTXYe/r0n0yzU/wo1RJxDkyZN2L17N2vWrOHAgQMAtGzZkpiYGPUAEalA6oYvdlfN19yS7ahr2ygqKoqIiAgWLVpEr1692LdvHytXrgRg06ZNVq3UhZk/fz5Dhgz5+9LVqlm9bzAYMBqNgDnXGDJkCFOmTCE2NpbAwECWLl3K9OnTrepUr37l1n9bDRs2jD///JPXX3+dhg0b4uXlRXR09GWTzxUXd2FlLv4ffbFMUFAQ6enphcawZcsWAE6fPs3p06cv+3wXu9L/86GE2K7ECfqKFSs4efIkixcv5oMPPmDy5MnExMRwzz330L9//8v+MdjLqVOnKCgouKzbSWhoqOWXwEulpKQUWj4lJaXI60ybNo0pU6aUPWCRUvrjz0wOH/8O/Nyo7dGc3l3bOzokkSsaOnQo/fv3JzY2Fj8/P3r16kWvXr0cHZaIiNiLwWBTN3NnMHLkSGbNmsWJEyeIiYkhPDwcgI4dO5a4i3txtmzZQsOGDXn66actx44ePXrFei1btuTYsWMkJydbWq5//PFHqzKenuYVey5dnvSHH37grbfeom/fvgAcO3aMU6dO2RyzraKiokhJSSE9PZ2aNWtajh8+fJiHH36Yd955h2XLljFs2DDWrl1r1RV+79691K9fn6CgILvHVRWUapK44OBgxo8fz65du9i2bRtNmjThrrvuom7dujz88MP8+uuv9o6zwkyYMIGMjAzLduzYMUeHJFVIbr6RrxZM4xdv89PLm3qOc2xAIjZq0qQJL774IsHBwfTp04e5c+dauheKVFrqFCLilAYPHszx48d55513rCY+K00X9+I0bdqUpKQkli5dyuHDh5k9ezbLly+/Yr2YmBiaNWvGsGHD2LVrF5s2bbJK8gFCQkLw8fFh1apVpKamkpGRYbnm4sWL2b9/P9u2bWPIkCGWLuX2FBUVRVBQED/88IPlWEFBAf/973+JjY1lxIgRLFiwgN27d1/WY2DTpk16SF8GZZrFPTk5mTVr1rBmzRrc3d3p27cve/bsoVWrVoWORyiLoKAg3N3drabwB/M4jH9OnPBPYWFhJSoP4OXlZVka6OImUlGmf/0T/zr/EakeHrhjoH24beORRBxt0qRJ7Ny5k19//ZV+/fqxYsUKrr76ajp06MDUqVOv2GIhIiJiL4GBgdx22234+fkxYMCAcrvOzTffzMMPP8zYsWOJjIxky5YtPPPMM1es5+bmxvLly7lw4QKdO3dm5MiRvPDCC1ZlPDw8mD17NvPnz6du3br0798fgPfee4/09HTat2/PXXfdxYMPPljoZNll5e7uzogRI6xmtX/hhRc4evQo8+fPB8zj1t9++20mTpzIrl27AMjOzmbFihWMGjXK7jFVFQZTCQcp5eXl8eWXX7JgwQJWr15Nu3btGDlyJIMHD7Yks8uXL+fuu+8uctxCaXXp0oXOnTvzxhtvAOYxEg0aNGDs2LE8+eSTl5W/4447OH/+PF999ZXlWNeuXWnXrh3z5s2z6ZqZmZkEBgaSkZGhZF3K1crdyRz++EmuDlzNkyFBtKndmv/dpDUkxXWdPXuWb7/9li+++IJvv/0Wf39/+vXrx+jRo2ndurWjw7uM7vdiq5W/reTJTU/SpU4X3u31rqPDsdLz456kXUjj45s+pmXtlleuIFJJ9ezZk9atWzN79mxHh+KyUlJSaN26NfHx8TbPxj537lyWL1/O6tWryzm6yqvEY9Dr1KmD0Whk0KBBbN++ncjIyMvK9OjRo9B1cMtq/PjxDBs2jI4dO9K5c2dmzZpFVlYWI0aMAMxjIOvVq8e0adMAeOihh+jWrRvTp0/nxhtvZOnSpfz000+8/fbbdo9NpCwOnzzHa5+u5xv3b5jubR7fFRkS5eCoRMrG39+f22+/ndtvv52CggLi4uL48ssv2bp1q1Mm6CIi4vrS09OJi4sjLi6Ot956y9HhuLSwsDDee+89kpKSbE7Qq1WrZmlMldIpcYI+c+ZMBg4cWOi0+xfVqFGD33//vUyBFeaOO+7g5MmTTJo0iZSUFCIjI1m1apVlMoekpCSrCQq6du3KkiVLmDhxIk899RRNmzZlxYoVWgNdnMqF3ALu/zCe+4zL8PHIJcG/PpBP+1BNDieuZ/Lkydx9992X/Ufu7u5Oz5496dmzp4MiE6kiNC5eqrioqCjS09N5+eWXLWuVS+mVdIjAyJEjyyeQKqTEY9DvuuuuYpPz8jZ27FiOHj1KTk4O27Zto0uXLpb34uLiWLhwoVX5gQMHcvDgQXJycti7d69lxkMRZ2AymXh6xR5I28d/PDZy1mDgV4N5ts4otaCLC/riiy+4+uqr6dmzJ0uWLLFatrIkNm7cSL9+/ahbty4Gg6HY5THBfP83GAyXbcWt2iFSmZnQMmtSNR05coSMjAweffRRR4ciUiplmiRORMpm2Y5jfB5/ggke/8MdI7ubdceEifp+9Qny0dIU4noSExPZsWMHrVu35qGHHiIsLIzRo0ezY8eOEp0nKyuLiIgI5syZU6J6Bw8eJDk52bKVx8Q5IiIiIuWlxF3cRcQ+9v2RwaQv99HVbS/d3XeBmwfxDdvDr4fVvV1cWlRUFFFRUUyfPp2vvvqKBQsWcO2119KiRQvuuecehg8fTmBgYLHn6NOnD3369CnxtUNCQsplDhSRwhjUn1xEROxMLegiDpBxIY/7P4onLz+faX4fmw92vIfEs0cBiAyJdFxwInZiMpnIy8sjNzcXk8lEzZo1efPNNwkPD2fZsmXlcs3IyEjq1KnDDTfcYLV2a2FycnLIzMy02kREREQcSQm6SAUzmUw89skujv55nuH+O2iYewi8Asj79yPsObUHgPYhakEX17Vz507Gjh1LnTp1ePjhh4mKimL//v1s2LCBX3/9lRdeeIEHH3zQrtesU6cO8+bN47PPPuOzzz4jPDyc7t27Ex8fX2SdadOmERgYaNnCw8PtGpNUXhrfLSIi5UVd3EUq2Lubfmf1z6n4ueczwetTyAP+9TAHc05yIf8CAZ4BNA5s7OgwRUqlbdu27N+/n9jYWN577z369euHu7u7VZlBgwbx0EMP2fW6zZs3t5qtt2vXrhw+fJiZM2eyePHiQutMmDCB8ePHW15nZmYqSRcRERGHUoIuUoF2HDnNS6sOAPBBmwQ8D56AgHpwzWgSfv0UMHdvdzOoc4u4pttvv527776bevXqFVkmKCgIo9FY7rF07tyZzZs3F/m+l5cXXl5e5R6HSEXSuHgREdemBF2kgpw8m8OYj+IpMJoY1LY67Y+8Z37j+olQzYeEtARAy6uJ6/lnKzTA9OnTiyw7Y8aM8g7HIjExkTp16lTY9UScibrhi9hHo0aNGDduHOPGjauQ6xkMBpYvX17i9cdLKi4ujh49epCenn7FyVWvu+467rvvPgYPHlyuMdlD9+7diYyMZNasWRV2zdzcXJo1a8ann35Kx44dy3w+JegiFaDAaOKhpQmknc2hSYgfU2t8g+HXTAhtC+3uwGQyKUEXl5WQkGD1Oj4+nvz8fEuX819++QV3d3c6dOhg8znPnTvHoUOHLK9///13EhMTqVWrFg0aNGDChAmcOHGCRYsWATBr1iwaN25M69atyc7O5t133+X7779n9erVdviEIiIiFe/IkSM0btyYhIQEIiMjHRLDl19+SWpqKnfeeadDrl9W9njAMnfuXObOncuRI0cAaN26NZMmTbKsNuPp6cmjjz7KE088wbp168ocsxJ0kQowa+0vbDn8J76e7rzbrzbV/vdX6/kNU8DNneNnj3HqwimquVWjTVAbxwYrUkLr16+37M+YMQN/f38++OADatasCUB6ejojRozg3//+t83n/Omnn+jRo4fl9cVW+mHDhrFw4UKSk5NJSkqyvJ+bm8sjjzzCiRMn8PX1pV27dqxdu9bqHOKCcs/Dkc1gzHN0JNZOJQJgyPoTDqx0bCyXys82/3lkM5w84tBQLuNZHRr+C9z166dUbbm5uXh6ejo6DJvMnj2bESNG4OZWdYdf1q9fn5deeommTZtiMpn44IMP6N+/PwkJCbRu3RqAIUOG8Mgjj7Bv3z7LsdIymEwm9YEqRmZmJoGBgWRkZBAQEODocMQFrT+YxogFOwB4/c5I+v86EfZ9DldfD3ctB+DLw1/y9OaniQiO4MO+HzoyXJEyqVevHqtXr77sP6e9e/fSq1cv/vjjDwdFdmW63zuhLx+E+A8cHcVlvq7uy4SQIKIvXODtlJOODsdKTHhdUj08WHoimda5TvZgA6DX89D1AUdHIZXUokWLePjhh/njjz+s5hgZMGAA/v7+RU4aWpxGjRpx7733cujQIT755BNq1qzJxIkTuffeey1lnnjiCZYvX87x48cJCwtjyJAhTJo0iWrVqgHw7LPPsmLFCsaOHcsLL7zA0aNHMRqN/Prrr9xzzz1s376dq666itdff51evXpZurgbDNZzSnTr1o24uDh27NjBU089RUJCAnl5eURGRjJz5kzat/97FSCDwcA777zDypUr+e6776hXrx7Tp0/n5ptvBv7u4r527VqeeOIJfv75ZyIjI1mwYIGlB9zJkycJDQ1lz549lv/X4+Li6NWrF+vWrbM8eH/llVd47bXX2LNnD6GhoTb9vRoMBubNm8dXX33F999/T8OGDXn//fcJDg5m5MiR7Nixg4iICBYvXszVV18NwPDhwzlz5gwrVqywnGfcuHEkJiYSFxcHWHdx7969Oxs2bLC6rr1S31q1avHqq69yzz33WI5df/31XHvttTz33HNlOrceYYqUoxNnLvDwskQA/ntNA/oHpcCKzwED3DDVUk7d26WyyMzM5OTJyxOWkydPcvbsWQdEJC4t84T5z5qNoHqwQ0P5J5PhApAJXoFQv5Gjw7HmfhIwQkhroJqjo/lbxnE4mwyZzvuQTopnMpm4kH/BIdf28fC5LFktzMCBA3nwwQf58ssvGThwIABpaWmsXLnSMuRp06ZNlq7JRZk/fz5DhgyxvJ4+fTrPPfccTz31FJ9++imjR4+mW7dulkTW39+fhQsXUrduXfbs2cOoUaPw9/fn8ccft5zj0KFDfPbZZ3z++ee4u7tjNBq59dZbCQ0NZdu2bWRkZFzWDXv79u107tyZtWvX0rp1a0ur+9mzZxk2bBhvvPEGJpOJ6dOn07dvX3799Vf8/f0t9adMmcIrr7zCq6++yhtvvMGQIUM4evQotWrVspR5+umnmT59OsHBwdx3333cfffd/PDDDwBs3rwZX19fWrZsaSnfvXt3xo0bx1133cWuXbv47bffeOaZZ/jkk09sTs4veu6555gxYwYzZszgiSeeYPDgwVx11VVMmDCBBg0acPfddzN27Fi+/fbbEp33os8//5yIiAjuvfdeRo0aZTmelJREq1atiq371FNP8dRTT112vKCggE8++YSsrCyio6Ot3uvcuTObNm0qVaz/pARdpJzk5hu5/6N4zpzPo139QJ65sSV8aH5qScQgCGtrKZuQqgRdKodbbrmFESNGMH36dDp37gzAtm3beOyxx7j11lsdHJ24rG5PQuQgR0fxt8NfweanoF4H6PW2o6Ox9kkMnE+FW96C2mXrZmlXa6fA5oqbJFLs70L+Bbos6eKQa28bvA3far5XLOfj48PgwYNZsGCBJUH/8MMPadCgAd27dwegY8eOJCYmFnueSxPNvn37cv/99wPm1vKZM2eyfv16S4I+ceJES9lGjRrx6KOPsnTpUqsEPTc3l0WLFhEcbH7YuHr1ag4cOMB3331H3bp1AXjxxRetHh5cLFu7dm3CwsIsx6+//nqr+N5++21q1KjBhg0buOmmmyzHhw8fzqBBgyznnj17Ntu3b6d3796WMi+88ALdunUD4Mknn+TGG28kOzsbb29vjh49Smho6GXd259//nnWrFnDvffey969exk2bJilZb4kRowYwe233w6Y/16jo6N55plniI2NBeChhx5ixIgRJT7vRbVq1cLd3R1/f3+rv7+6dete8d/APx9iAOzZs4fo6Giys7Px8/Nj+fLllyX5devW5ejRo6WO9yIl6CLl5MVv9rPr2BkCfaoxZ3B7vA6vhqM/gIe3eeb2v2TkZHA44zBgXmJNxJXNmzePRx99lMGDB5OXZ+5e6+HhwT333MOrr77q4OjE5WgUXonZ0sroUPpOpZyNGjWKTp06ceLECerVq8fChQsZPny45WfDx8eHJk2alOic7dq1s+wbDAbCwsJIS0uzHFu2bBmzZ8/m8OHDnDt3jvz8/MuGSjVs2NCScAPs37+f8PBwS3IOXNYiW5TU1FQmTpxIXFwcaWlpFBQUcP78eau5WS6Nu3r16gQEBFjFfWmZiyufpKWl0aBBAy5cuIC3t/dl1/f09OSjjz6iXbt2NGzYkJkzZ9oU96X+ee2LD0Xatm1rdSw7O5vMzEy7Dj3z8PAo8b+B5s2bk5iYSEZGBp9++inDhg1jw4YNVkm6j48P58+fL3t8ZT6DiFzmq11/sHDLEQBm3B5BeKAnLJlsfvOa+yHw7zWid53cBUCjgEbU8q516alEXIqvry9vvfUWr776KocPmx88XX311VSvXt3BkYlUMcqDxc58PHzYNnibw65tq6ioKCIiIli0aBG9evVi3759rFz592SOpenifnEs+UUGgwGj0QjA1q1bGTJkCFOmTCE2NpbAwECWLl162ZKj9vx/cNiwYfz555+8/vrrNGzYEC8vL6Kjo8nNzbUqV1zchZW5+BDjYpmgoCDS09MLjWHLli0AnD59mtOnT5fq8xV27eLicXNzu2wM+cXGgJIoTRd3T09PS1LfoUMHduzYweuvv878+fMtZU6fPm31EKa0lKCL2NmhtHM8+dluAO7vfjU9W4bCT+/DqV/Atzb8a5xV+fjUeEDd26VyqV69utWTcZEycfZWYbkyfYcuz2Aw2NTN3BmMHDmSWbNmceLECWJiYggPD7e8V5ou7sXZsmULDRs25Omnn7Ycs6Wbc8uWLTl27BjJycmWlusff/zRqszFMecFBQVWx3/44Qfeeust+vbtC8CxY8c4deqUzTHbKioqipSUFNLT0y0rswAcPnyYhx9+mHfeeYdly5YxbNgw1q5dW+4zvQcHB7N3716rY4mJiZc9iPgnT0/Py/7+StPF/VJGo5GcnByrY3v37iUqquy/zytBF7Gj87n53P/RTrJyC7jmqlqMv6EZ5JyF9dPMBbo9Ad6BVnU0QZyISFGcuxnY6buTOyXn/k6lchg8eDCPPvoo77zzDosWLbJ6rzRd3IvTtGlTkpKSWLp0KZ06dWLlypUsX778ivViYmJo1qwZw4YN49VXXyUzM9MqyQcICQnBx8eHVatWUb9+fby9vQkMDKRp06YsXryYjh07kpmZyWOPPYaPj+29DGwVFRVFUFAQP/zwg2Vse0FBAf/973+JjY1lxIgR9O7dm7Zt2zJ9+nQee+wxu8fwT9dffz2vvvoqixYtIjo6mg8//PCKSXGjRo3YuHEjd955J15eXgQFBZW4i/uECRPo06cPDRo04OzZsyxZsoS4uDi+++47q3KbNm0q8wzuAFV3QTsROzOZTExcvpdfUs8R7O/F7EFReLi7wZY3ICsNal0FHawnusgtyGXvKfOTQCXoIiIiImUXGBjIbbfdhp+fHwMGDCjXa9188808/PDDjB07lsjISLZs2cIzzzxzxXpubm4sX76cCxcu0LlzZ0aOHMkLL7xgVcbDw4PZs2czf/586tatS//+/QF47733SE9Pp3379tx11108+OCDhISE2P2zubu7M2LECD766CPLsYvLxF3s2l2nTh3efvttJk6cyK5d5mGbCxcuLJcHmLGxsTzzzDM8/vjjdOrUibNnzzJ06NBi60ydOpUjR45w9dVXl7r7eVpaGkOHDqV58+b07NmTHTt28N1333HDDTdYymzdupWMjAz+85//lOoa/6R10K9A6+KKrf63PYkJn+/BzQBLRl3DNVfVhrMpMDsK8s7D7YugVX+rOolpidz17V3U8q5F3O1xao0RcSDd753Q4lvg8Pdwy9sQcYejo7H46vBXPLX5KbrW7cr8G+ZfuUIFuuHTG0jJSmHpjUtpHeREs7ivew42vQZd7oM+Lzs6GqkCevbsSevWrZk9e7ajQ3FpKSkptG7dmvj4eBo2bGhTncmTJ7NhwwbL2uRVwR133EFEREShS7OVlFrQRexg74kMJn+5D4DHYluYk3OA9S+ak/P6naHl5ctPJKYlAhAZHKnkXETkUmpDqHz0nUo5S09PZ/ny5cTFxTFmzBhHh+PywsLCeO+99y6bIb443377La+88ko5RuVccnNzadu2LQ8//LBdzqcx6CJllHE+j9Ef7SQ330hMyxD+77qrzG+k7YeExeb9Xs8XOkFOfJomiBMREfsxoIe9UrVFRUWRnp7Oyy+/bFmnXMqmpMMEtm/fXj6BOClPT08mTpx45YI2UoIuUgYmk4lHPtnFsdMXqF/Th+kDI3Fz++uXo7XPgskILftBgy6F1rW0oGv9cxGRoqmHUYmZnG0yNn2HUkGOHDni6BBEykRd3EXK4O2Nv7F2fyqe7m7MHdKBQN+/lnn4fRP8sgrcPKDns4XWPZJ5hPScdLzcvWhVu/i1GEVEqiYnSzLFDvSdiogURwm6SClt++1PXvnuIACTb25F2/p/LZ9mNMLqv7q5dBgBQYUv43Cx9bx17dZ4unuWd7giImJn6k4uIiL2pgRdpBROns3hgf8lUGA0cUtUPQZ3bvD3m/s+h+RE8PQ3r3tehIvrn7cPbV/O0YqIuDolwq5P36GIiC2UoIuUUIHRxIP/SyDtbA5NQ/x44ZY2f8/Anp8D66aY9//1EPgVvd7ixQRdE8SJiBRBM35XPvpORUSKpQRdpIRmrvmFrb/9ia+nO3P/2x5fz3/Mtbj9bTiTBP514Zqil/Y4nX2aI5lHAIgIjijniEVExJ6cbgI2ERGpNJSgi5TA9wdSeXP9IQBeuq0dTUL8/37z/GnY+Kp5//qnwdO3yPNcbD1vUqMJgV6B5RaviEiloBnAXZ++QxERm7hMgn769GmGDBlCQEAANWrU4J577uHcuXPF1unevTsGg8Fqu++++yooYqlsjqef5+FluwAYGt2QmyPqWhfYNB2yMyCkNUQMKvZcWl5NRMQWaqkuqYsT15mctiu5s8YlIuIcXGYd9CFDhpCcnMyaNWvIy8tjxIgR3HvvvSxZsqTYeqNGjWLq1KmW176+RbdqihQlJ7+AMR/Fk3Ehj4j6gTx9Y0vrAulHzN3bAW6YCm7uxZ7PMkFciCaIExERERERM5dI0Pfv38+qVavYsWMHHTt2BOCNN96gb9++vPbaa9StW7fIur6+voSFhVVUqFJJvbByP7uOZxDoU405Q9rj5XFJAr7uOSjIhau6Q5OexZ4rOz+bfX/uA9SCLiLi0tRruwT0lyUiYguX6OK+detWatSoYUnOAWJiYnBzc2Pbtm3F1v3oo48ICgqiTZs2TJgwgfPnzxdbPicnh8zMTKtNqrYvEk+waOtRAGbdEUn9mpf0wjgRD3s/BQzm1vMrjLPb9+c+8o35BPkEUd+vfjlFLSJSCThtN20pNX2nIiLFcokEPSUlhZCQEKtjHh4e1KpVi5SUlCLrDR48mA8//JD169czYcIEFi9ezH//+99irzVt2jQCAwMtW3h4uF0+g7imQ2lnmfD5HgDG9LiaHi2s/x1iMsGaSeb9dndAnSvPyP7P5dUMmjRHpFAbN26kX79+1K1bF4PBwIoVK65YJy4ujvbt2+Pl5UWTJk1YuHBhuccpIiIiYk8OTdCffPLJyyZxu3Q7cOBAqc9/7733EhsbS9u2bRkyZAiLFi1i+fLlHD58uMg6EyZMICMjw7IdO3as1NcX13Y+N5/RH8ZzPreA6Ktq83BMs8sL/fIdHNkE7l5w/USbzqv1z0WuLCsri4iICObMmWNT+d9//50bb7yRHj16kJiYyLhx4xg5ciTfffddOUcqFcLJHmY67wRsTszJvkMREWfl0DHojzzyCMOHDy+2zFVXXUVYWBhpaWlWx/Pz8zl9+nSJxpd36dIFgEOHDnH11VcXWsbLywsvLy+bzymVk8lk4qnP9/Br2jlC/L14fVAkHu6XPM8qyIe1k83719wHNa7c28JoMlpmcNcEcSJF69OnD3369LG5/Lx582jcuDHTp08HoGXLlmzevJmZM2cSGxtbXmFKOcsrMFIN2HL4T45dSHJ0OBa7zvwJQMqZbJbtcJ64ALJy8wFYsz+Fn484zzKerU9k0AbINxpdYwIkEREHceg9Mjg4mODg4CuWi46O5syZM+zcuZMOHToA8P3332M0Gi1Jty0SExMBqFOnTqnilapjyfYkViT+gbubgTcGRRHi7315ocQP4eQB8KkF/xpv03l/O/MbmbmZ+Hj40KxWIS3yIlIqW7duJSYmxupYbGws48aNK7JOTk4OOTk5lteac8T5HP3zPE2AJduS+Nq4x9HhWHgEHsenLhxMPcsTPzlPXADVr87DzRPe/P4QxuxcR4dj8ZB7Km2qwYGUTNo4OhgRESfmEg8xW7ZsSe/evRk1ahTz5s0jLy+PsWPHcuedd1pmcD9x4gQ9e/Zk0aJFdO7cmcOHD7NkyRL69u1L7dq12b17Nw8//DDXXXcd7dq1c/AnEme253gGU778GYDHY5vT5aralxfKOQfrXzTvd3scfGrYdO6Ek+bu7W2D2lLNrZo9whURzHOVhIaGWh0LDQ0lMzOTCxcu4OPjc1mdadOmMWXKlIoKUUohr8AIQL2avsRcMheNI50kgN+A2n6etGjpPHEBJOJODtC5cU38cJ7YAo97Qh5k5xodHYqIiFNziQQdzLOxjx07lp49e+Lm5sZtt93G7NmzLe/n5eVx8OBByyztnp6erF27llmzZpGVlUV4eDi33XYbEyfaNk5YqqaM83mM/mgnuQVGbmgVyr3XXVV4wa1z4Fwq1GwEHe+x+fwXu7dr/LmI402YMIHx4//u/ZKZmamJQZ2Oeax3jxYhTOjXycGx/O2LQ38w8QdoW68Gc2OcJy6A3p95ceIcPNW3Fe2CnadBYvO7y+A4XPxORUSkcC6ToNeqVYslS5YU+X6jRo2sJm0JDw9nw4YNFRGaVBJGo4nxHydyPP0C4bV8eG1gROGzrJ9NhR9eN+/3nAwenjZfIz41HlCCLmJvYWFhpKamWh1LTU0lICCg0NZz0JwjIiIi4nxcYpk1kYowf+NvrDuQhqeHG3OHdCDQp4gu6HHTIC8L6nWA1rfYfP6T509y/Nxx3AxuRARfeTk2EbFddHQ069atszq2Zs0aoqOjHRSR2JOWpKwE9B2KiNhECboI8ONvf/Lqd+Yl/Z7t15o29YqY+fbkQYhfZN7v9XyJfuG4uLxa0xpN8fP0K1O8IpXduXPnSExMtEzu+fvvv5OYmEhSknnG7AkTJjB06FBL+fvuu4/ffvuNxx9/nAMHDvDWW2/x8ccf8/DDDzsifLETg5N2hzY5aVyuQX93IiLFUYIuVV7a2Wwe+F8CRhPcGlWPQZ2LGYO69lkwFUDzG6Fh1xJd52KCHhkSWfpgRaqIn376iaioKKKizMNBxo8fT1RUFJMmTQIgOTnZkqwDNG7cmJUrV7JmzRoiIiKYPn067777rpZYExEREZfiMmPQRcpDfoGRB/+XwMmzOTQL9eP5W9oU3ZXyyA9w8BswuEPMsyW+ltY/F7Fd9+7dreYVudTChQsLrZOQkFCOUYnDqHd0iTlfK7++RBERW6gFXaq0GWt+4cffTlPd0525/+2Ar2cRz6xMJlj91woAHYZBcMnWMD+fd579p/cDmiBORMR2zpZkSlkZinnwJiIiStClClu3P5W34g4D8NJt7bg6uJhx4fs+hz/iwdMPuk8o8bX2ntpLgamAsOph1PGrU9qQRUSqlr9yOYOTtr46a1zOTOm5iEjxlKBLlXTs9HkeXpYIwPCujegXUbfowvk5sHaKef/ah8AvpMTXi0/7a3m1YLWei4iUnBJhl6dZ3EVEbKIEXaqcnPwC7v8onszsfCLDa/BU35bFV9jxHpw5Cn5hED2mVNe8OP5cE8SJiJSE2lsrH32nIiLFUYIuVc5zX//MnhMZ1PCtxpwh7fH0KObH4MIZ2PiKeb/HU+BZvcTXKzAWsOvkLgDah2qCOBERV1fcBIYiIiJloQRdqpQvEk/w4Y/mpZlm3hFJvRo+xVfYNB0upENwS4gcUqprHjpziHN556herTpNazQt1TlERKo0dY+uBP76lVPPNkREiqUEXaqMX1PPMuHzPQA8cH0TejS/wljyM0mwbb55/4Yp4F66VQkvrn8eERyBu5t7qc4hIlIVKS0vuYsT1zlvK7+zxiUi4hyUoEuVkJWTz+iP4jmfW0DXq2szLsaGZdK+fx4KcqDRv6Fpr1Jf++IEcRp/LiJSMhdTOSXqIiJSVShBl0rPZDIx4fM9HEo7R2iAF7MHReHudoVf9/5IhN3LzPu9nitT98qLE8Rp/XMRkdIxOWkXd4OTxuWU9FclImITJehS6X24LYkvd/2Bu5uBNwe3J8jPq/gKJhOseca833Yg1C19Yp2SlUJyVjLuBnfaBbUr9XlERKoiw1/dtJXbub6/v0N1cRcRKY4SdKnUdh07w3Nf/QzAk71b0KlRrStXOrQWft8I7p5w/TNluv7F8efNazXHt5pvmc4lIiIiIiKVmxJ0qbTOnM/l/o/iyS0w0qtVKCP/3fjKlYwFsGaSeb/L/0HNhmWK4WKC3j5Ey6uJiJSaupJXAvqVU0TEFrpbSqVkNJoY//EuTpy5QINavrw6MMK2sYKJSyDtZ/CuAf9+pMxxXEzQNUGciEhpqDt0peO0s8uLiDgHJehSKc3dcJjvD6Th6eHGW0PaE+hT7cqVcrNg/Qvm/eseA5+aZYohKy+LX9J/ATRBnIiIVAxNXCci4tqUoEuls/Xwn0xffRCAqTe3pk29QBsrvgVnk6FGQ+g8qsxx7Dq5C6PJSD2/eoT4XmHNdRERKZJB08S5Pn2FIiI2UYIulUpaZjYP/C8Bowlua1+fOzqF21bx3En4YZZ5v+ck8LjCTO82uNi9Xa3nIiKlY3DyldD14EBEROxNCbpUGvkFRsb+L4FT53JoEebP8wPa2N7Vb8NLkHvOvKRa61vtEo8SdBERO1EeXIloDLqISHGUoEul8drqX9j++2n8vDx4a0h7fDzdbat46lf4aYF5v9fz4Fb2H4t8Yz67T+4GlKCLiIiIiIhtlKBLpbDm51TmbTgMwMu3teOqYD/bK699FkwF0KwPNPqXXeI5mH6QC/kX8Pf05+oaV9vlnCIiVY+5tVVdySuBv3q0GTSLu4hIsZSgi8tL+vM8j3ycCMDwro24sV0d2ysf3QoHvgaDG8Q8a7eYEtPM8UQGR+Jm0I+ZiEhlYlI3bRERKSfKHMSlZecVcP+SnWRm5xMZXoOn+ra0vbLJBGueMe+3HwohLewWV3xqPKDu7SJlNWfOHBo1aoS3tzddunRh+/btRZZduHAhBoPBavP29q7AaKXcqAG9xJztIcLFXhDOFZWIiPNRgi4u7bmvf2bviUxq+lZjzpD2eHqU4J/0z1/A8R1QrTp0f8puMZlMJksLuhJ0kdJbtmwZ48ePZ/LkycTHxxMREUFsbCxpaWlF1gkICCA5OdmyHT16tAIjFnE8px0OoPXZRURsogRdXNbyhON8tC0JgwFm3hFJvRo+tlfOzzWPPQfo+gD4h9otrj+y/iDtQhoebh60CWpjt/OKVDUzZsxg1KhRjBgxglatWjFv3jx8fX15//33i6xjMBgICwuzbKGh9vvZFgewNLc6Z3LntMmwU1MbuohIcVwmQX/hhRfo2rUrvr6+1KhRw6Y6JpOJSZMmUadOHXx8fIiJieHXX38t30ClQvySepanPt8LwAPXN6V785CSneCn9yH9d/ALNSfodnSxe3urWq3w9lD3WpHSyM3NZefOncTExFiOubm5ERMTw9atW4usd+7cORo2bEh4eDj9+/dn3759RZbNyckhMzPTahMRERFxJJdJ0HNzcxk4cCCjR4+2uc4rr7zC7NmzmTdvHtu2baN69erExsaSnZ1djpFKeTuXk899H+7kQl4B/2oSxEM9m5bsBNkZsOFl8373CeBVghnfbaDu7SJld+rUKQoKCi5rAQ8NDSUlJaXQOs2bN+f999/niy++4MMPP8RoNNK1a1eOHz9eaPlp06YRGBho2cLDw+3+OaSs/prFXd2jKw81oIuIFMvD0QHYasqUKYB5EiBbmEwmZs2axcSJE+nfvz8AixYtIjQ0lBUrVnDnnXeWV6gArD+QRkrm3w8CLl1V5NLJWy5//xKXFDAV/RamYsracq1L618hlBJ/lpLWv9SPv/3JbyezCAvw5vU7I3F3K+EvbptnwoXTENQcou4qWV0bxKdpgjgRR4iOjiY6OtryumvXrrRs2ZL58+fz3HPPXVZ+woQJjB8/3vI6MzNTSbqTUVpe+eg7FREpnssk6CX1+++/k5KSYtU9MjAwkC5durB169YiE/ScnBxycnIsr0vb5fG9zb+z+dCpUtWVK3N3M/Dm4Chq+3mVrGLGcfhxrnn/hingbt8fgczcTA6fMa/HHhkSaddzi1QlQUFBuLu7k5qaanU8NTWVsLAwm85RrVo1oqKiOHToUKHve3l54eVVwnuICFd+kC0iIlJalTZBv9gFsiTdI8Hc5fFia31ZtG9QA+9q7lbH/tlD79InyJf23rt04pnL3jcUXfYKLy/rKmiweq/0dQu/9t8Hrnzuouv+832DAW5qV5eOjWpdevUr+/4FyM+Ghv+CZr1LXv8KdqXtwoSJhgENqe1T2+7nF6kqPD096dChA+vWrWPAgAEAGI1G1q1bx9ixY206R0FBAXv27KFv377lGKlUCHVxLzGne4hgMI+qdLKoREScjkMT9CeffJKXX3652DL79++nRQv7rU99Jfbq8ji+V3N7hiX2kLIHdv3PvN9rarn8wpeQlgBAZHCk3c8tUtWMHz+eYcOG0bFjRzp37sysWbPIyspixIgRAAwdOpR69eoxbdo0AKZOnco111xDkyZNOHPmDK+++ipHjx5l5MiRjvwYUiZK50rK2cfrG/SdiogUy6EJ+iOPPMLw4cOLLXPVVVeV6twXu0CmpqZSp04dy/HU1FQiIyOLrKcuj5XY6mcAE7S5Dep1KJdLXEzQ24e2L5fzi1Qld9xxBydPnmTSpEmkpKQQGRnJqlWrLD2jkpKScHP7e67T9PR0Ro0aRUpKCjVr1qRDhw5s2bKFVq1aOeojiJ04bcrptIE5L6XnIiLFc2iCHhwcTHBwcLmcu3HjxoSFhbFu3TpLQp6Zmcm2bdtKNBO8VBKH1sFv68GtGvScVC6XyCvIY8+pPYDGn4vYy9ixY4vs0h4XF2f1eubMmcycObMCopIK5+StwmILfYciIrZwmWXWkpKSSExMJCkpiYKCAhITE0lMTOTcuXOWMi1atGD58uWAuYvXuHHjeP755/nyyy/Zs2cPQ4cOpW7dupbxjFJFGAtgzV9Jeed7oWajcrnM/tP7ySnIoYZXDRoHNC6Xa4iIVCXqDl356DsVESmey0wSN2nSJD744APL66go8xJW69evp3v37gAcPHiQjIwMS5nHH3+crKws7r33Xs6cOcO//vUvVq1ahbe3d4XGLg62aymk7gXvQLju0XK7jGX8eUik048BFBERERER5+MyCfrChQuvuAb6pTOWGgwGpk6dytSpU8sxssK9suMV9p3aV+HXLYlL1x93VmWaidZkhORdUCcUajSAuIfsF9gljp09Bmj9cxERe3O2h56u8v+nM7F8hfqrExEplssk6K7mUPoh4tPiHR2GAFQzQDUvyE41b+Wsa92u5X4NEZEqwdmWCnMhzvsQwVnjEhFxDkrQy8n/RfwfA5sPdNj1L1sbvSpd/+Kls8/C1w9D/gW45n5odG25XzrMN4wWtSpuWUAREZF/cvT//0Vz1rhERJyLEvRy0iG0fJbxkhL45nHIPA11IuBfE8HNZeZEFBGRf3KyLu4XOW8y7ISc9DsUEXE2ylikcvrzMPz0nnn/hueUnIuIuCCD5U8ld5WHuriLiBRHWYtUTmufBWM+NO0FV3VzdDQiIiIiIiJXpARdKp9j22H/l2Bwg5gpjo5GRETKSt2jKwF9hyIitlCCLpWLyQSrJ5r3I4dAaCvHxiMiImXgnN2hnXeGdBegmflFRIqlBF0ql/1fwbFtUM0Xejzt6GhEREQcwqREWETEJSlBl8qjIM889hwgeiwE1HFoOCIiIvIXDVMQEbGJEnSpPHYuhNOHoXowXPugo6MREZEyMjh5V3LNLl8azv2diog4mhJ0qRyyMyHuJfN+9yfBy9+x8YiISJldTOUMan11efoGRURsowRdKocfZsH5U1C7KbQf5uhoRETErpTeuTqTvkMREZsoQRfXl3ECts4x78c8C+7VHBqOiIjYh7N3cRfbWTpBaPI6EZFiKUEX17f+RcjPhgbR0OJGR0cjIiKVnGZIFxGR8qIEXVxbyl5I/Mi83+t5zRIrIlKZ/JUHawx6yTnfWu36DkVEbKEEXVzb2smACVoNgPodHR2NiIjYkVK6ykfDFkREiqcEXVzX4fVwaC24VYOYyY6ORkREyo1zpurO2LLvjDGBOriJiNhKCbq4JqMR1jxj3u80Empd5dh4RESk/Ci5c3maxV1ExDZK0MU17fkYUvaAVwBc95ijoxGRcjJnzhwaNWqEt7c3Xbp0Yfv27cWW/+STT2jRogXe3t60bduWb775poIilfJh7g6t1M71qQVdRMQ2StDF9eRdgHXPmff/PR6q13ZsPCJSLpYtW8b48eOZPHky8fHxREREEBsbS1paWqHlt2zZwqBBg7jnnntISEhgwIABDBgwgL1791Zw5CJSNI1BFxEpjhJ0cT3b5kHmcQioD13uc3Q0IlJOZsyYwahRoxgxYgStWrVi3rx5+Pr68v777xda/vXXX6d379489thjtGzZkueee4727dvz5ptvVnDkYndqfq0E9B2KiNjCw9EBVFpfPgjHd1gfK3Td1EKO2bucI69daDi2nq+Isuf+aj3r+QxU8yminoi4stzcXHbu3MmECRMsx9zc3IiJiWHr1q2F1tm6dSvjx4+3OhYbG8uKFSsKLZ+Tk0NOTo7ldWZmZqli/SLubd759Y1S1ZXiVQvLw2AII/vYbLy/XOjocCwysjMcHcIVPfPDM/hW83V0GBbnc07iWy8Mo+kQ+e+0dXQ4ldJHgzYQ6FfL0WGISBkpQS8vZ45C2s+OjqLyqtcB2t7u6ChEpJycOnWKgoICQkNDrY6HhoZy4MCBQuukpKQUWj4lJaXQ8tOmTWPKlClljvVc9hmOepb5NFKoauY/clPNm5Op61fX0SFcpp5fPX7P+J0T5044OpTLeeoHpTyZTEZHhyAidqAEvbzETIHsM4W8UUgXr8u67tlSpgLK2f2ahRQrzd+HwQBBzcBNIzREpPQmTJhg1eKemZlJeHh4ic/zr3YDmLDH3Z6hyT/41a5DcN3Gjg7jMp7unrQLbufoMC4zo/sMdp/cjdHZkjWTieQjB8g+l+7oSCotXy9/R4cgInagBL281I10dAQiIi4rKCgId3d3UlOtW01TU1MJCwsrtE5YWFiJynt5eeHl5VXmWBvWbUbDuo+U+Twi9uDj4UOXOl0cHUbh6nV1dAQiIk5PTZAiIuJ0PD096dChA+vWrbMcMxqNrFu3jujo6ELrREdHW5UHWLNmTZHlRURERJyNyyToL7zwAl27dsXX15caNWrYVGf48OEYDAarrXfv3uUbqIiI2MX48eN55513+OCDD9i/fz+jR48mKyuLESNGADB06FCrSeQeeughVq1axfTp0zlw4ADPPvssP/30E2PHjnXURxAREREpEZfp4p6bm8vAgQOJjo7mvffes7le7969WbBggeW1PboziohI+bvjjjs4efIkkyZNIiUlhcjISFatWmWZCC4pKQm3f8xF0bVrV5YsWcLEiRN56qmnaNq0KStWrKBNmzaO+ggiIiIiJWIwmYpc38opLVy4kHHjxnHmzJkrlh0+fDhnzpwpcokdW2RmZhIYGEhGRgYBAQGlPo+IiDg33e9FRETE0Vymi3tpxcXFERISQvPmzRk9ejR//vlnseVzcnLIzMy02kRERERERETKW6VO0Hv37s2iRYtYt24dL7/8Mhs2bKBPnz4UFBQUWWfatGkEBgZattIsuSMiIiIiIiJSUg5N0J988snLJnG7dDtw4ECpz3/nnXdy880307ZtWwYMGMDXX3/Njh07iIuLK7LOhAkTyMjIsGzHjh0r9fVFREREREREbOXQSeIeeeQRhg8fXmyZq666ym7Xu+qqqwgKCuLQoUP07Nmz0DKXrot7cYi+urqLSFXi7++PwWBwdBgVSvd7EamKquL9XsSZOTRBDw4OJjg4uMKud/z4cf7880/q1Kljc52zZ88CqKu7iFQpVXGiNN3vRaQqqor3exFn5jLLrCUlJXH69GmSkpIoKCggMTERgCZNmuDn5wdAixYtmDZtGrfccgvnzp1jypQp3HbbbYSFhXH48GEef/xxmjRpQmxsrM3XrVu3LseOHXOJp4uZmZmEh4dz7NixSn2jrQqfsyp8RtDndGb+/v6ODqHCudL9Hlzz31VJVYXPCPqclYkrfsaqeL8XcWYuk6BPmjSJDz74wPI6KioKgPXr19O9e3cADh48SEZGBgDu7u7s3r2bDz74gDNnzlC3bl169erFc889V6K10N3c3Khfv779PkgFCAgIcJn/FMqiKnzOqvAZQZ9TnIMr3u+havy7qgqfEfQ5K5Oq8BlFpHy4TIK+cOFCFi5cWGyZfy7p7uPjw3fffVfOUYmIiIiIiIjYR6VeZk1ERERERETEVShBr0S8vLyYPHlyibrwu6Kq8DmrwmcEfU6RsqgK/66qwmcEfc7KpCp8RhEpXwbTP/uFi4iIiIiIiIhDqAVdRERERERExAkoQRcRERERERFxAkrQRURERERERJyAEnQRERERERERJ6AEvRI6cuQI99xzD40bN8bHx4err76ayZMnk5ub6+jQ7O6FF16ga9eu+Pr6UqNGDUeHYzdz5syhUaNGeHt706VLF7Zv3+7okOxq48aN9OvXj7p162IwGFixYoWjQ7K7adOm0alTJ/z9/QkJCWHAgAEcPHjQ0WFJJVRV7vm637su3fNFRGynBL0SOnDgAEajkfnz57Nv3z5mzpzJvHnzeOqppxwdmt3l5uYycOBARo8e7ehQ7GbZsmWMHz+eyZMnEx8fT0REBLGxsaSlpTk6NLvJysoiIiKCOXPmODqUcrNhwwbGjBnDjz/+yJo1a8jLy6NXr15kZWU5OjSpZKrKPV/3e9ele76IiO20zFoV8eqrrzJ37lx+++03R4dSLhYuXMi4ceM4c+aMo0Mpsy5dutCpUyfefPNNAIxGI+Hh4TzwwAM8+eSTDo7O/gwGA8uXL2fAgAGODqVcnTx5kpCQEDZs2MB1113n6HCkkqvM93zd712b7vkiIsVTC3oVkZGRQa1atRwdhlxBbm4uO3fuJCYmxnLMzc2NmJgYtm7d6sDIpKwyMjIA9HMoFUL3fOen+33lpnu+iJSWEvQq4NChQ7zxxhv83//9n6NDkSs4deoUBQUFhIaGWh0PDQ0lJSXFQVFJWRmNRsaNG8e1115LmzZtHB2OVHK657sG3e8rL93zRaQslKC7kCeffBKDwVDsduDAAas6J06coHfv3gwcOJBRo0Y5KPKSKc3nFHFmY8aMYe/evSxdutTRoYgLqQr3fN3vpTLSPV9EysLD0QGI7R555BGGDx9ebJmrrrrKsv/HH3/Qo0cPunbtyttvv13O0dlPST9nZRIUFIS7uzupqalWx1NTUwkLC3NQVFIWY8eO5euvv2bjxo3Ur1/f0eGIC6kK93zd73W/r2x0zxeRslKC7kKCg4MJDg62qeyJEyfo0aMHHTp0YMGCBbi5uU5niZJ8zsrG09OTDh06sG7dOssEOkajkXXr1jF27FjHBiclYjKZeOCBB1i+fDlxcXE0btzY0SGJi6kK93zd73W/ryx0zxcRe1GCXgmdOHGC7t2707BhQ1577TVOnjxpea+yPZVPSkri9OnTJCUlUVBQQGJiIgBNmjTBz8/PscGV0vjx4xk2bBgdO3akc+fOzJo1i6ysLEaMGOHo0Ozm3LlzHDp0yPL6999/JzExkVq1atGgQQMHRmY/Y8aMYcmSJXzxxRf4+/tbxpQGBgbi4+Pj4OikMqkq93zd712X7vm654tICZik0lmwYIEJKHSrbIYNG1bo51y/fr2jQyuTN954w9SgQQOTp6enqXPnzqYff/zR0SHZ1fr16wv93oYNG+bo0OymqJ/BBQsWODo0qWSqyj1f93vXpXu+iIjttA66iIiIiIiIiBNwjUFqIiIiIiIiIpWcEnQRERERERERJ6AEXURERERERMQJKEEXERERERERcQJK0EVEREREREScgBJ0ERERERERESegBF1ERERERETECShBFxEREREREXECStBFREREREREnIASdBEREREREREnoARdxImcPHmSsLAwXnzxRcuxLVu24Onpybp16xwYmYiI2JPu9yIiUhiDyWQyOToIEfnbN998w4ABA9iyZQvNmzcnMjKS/v37M2PGDEeHJiIidqT7vYiIXEoJuogTGjNmDGvXrqVjx47s2bOHHTt24OXl5eiwRETEznS/FxGRf1KCLuKELly4QJs2bTh27Bg7d+6kbdu2jg5JRETKge73IiLyTxqDLuKEDh8+zB9//IHRaOTIkSOODkdERMqJ7vciIvJPakEXcTK5ubl07tyZyMhImjdvzqxZs9izZw8hISGODk1EROxI93sREbmUEnQRJ/PYY4/x6aefsmvXLvz8/OjWrRuBgYF8/fXXjg5NRETsSPd7ERG5lLq4iziRuLg4Zs2axeLFiwkICMDNzY3FixezadMm5s6d6+jwRETETnS/FxGRwqgFXURERERERMQJqAVdRERERERExAkoQRcRERERERFxAkrQRURERERERJyAEnQRERERERERJ6AEXURERERERMQJKEEXERERERERcQJK0EVEREREREScgBJ0ERERERERESegBF1ERERERETECShBFxEREREREXECStBFREREREREnIASdBEREREREREn8P+gznY0M/gGfgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 900x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "opplot({\"F.hardtanh(x)\": F.hardtanh,\n",
    "        \"hardtanh(x)\": hardtanh,\n",
    "        \"hardtanh(x, mult=3)\": lambda x: hardtanh(x, mult=3)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summing up\n",
    "\n",
    "That's it; unit-scaling an op can be a relatively systematic process, based on either empirical or statistical analysis.\n",
    "\n",
    "> Note: The main thing our `hardtanh` example missed is any dependence on input tensor shapes. Since it is an elementwise nonlinearity, these do not change the scaling behaviour. In general, a scaling rule would have to consider input shapes.\n",
    "\n",
    "To unit-scale an op for u-μP:\n",
    "\n",
    " 1. If needed, add a `mult` hyperparameter to control the shape of the op when the input scale is unit.\n",
    " 1. Make distributional assumptions about the forward and backward pass inputs (typically: IID unit Gaussian).\n",
    " 1. _Either:_ feed in data from these distributions and measure the change in `std`.\n",
    " 1. _Or:_ do some maths to work out the theoretical change in `std`.\n",
    " 1. Add the `apply_constraint` boilerplate to support constrained scales.\n",
    " 1. Test it over a range of shapes and `mult` (if applicable), by feeding in artificial inputs & grad-outputs, measuring `std`.\n",
    "\n",
    "To use the compendium of ops for which we have proposed scaling rules, or to develop your own rules, see https://github.com/graphcore-research/unit-scaling.\n",
    "\n",
    "Thanks for reading, well done for reaching the end & happy scaling!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}