KAIST_AI605_Assignment_3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ReonT_YasRSx"
      },
      "source": [
        "# KAIST AI605 Assignment 3: Transformer\n",
        "\n",
        "TA in charge: Miyoung Ko (miyoungko@kaist.ac.kr)\n",
        "\n",
        "**Due date**:  May 17 (Tue) 11:00pm, 2022  \n",
        "\n",
        "\n",
        "## Your Submission\n",
        "If you are a KAIST student, you will submit your assignment via [KLMS](https://klms.kaist.ac.kr). If you are a NAVER student, you will submit via [Google Form](https://forms.gle/FSng5HUwtQinTFAU8). \n",
        "\n",
        "You need to submit both (1) .ipynb file (needs to be fully executable on CoLab), and (2) a pdf of the file.\n",
        "\n",
        "Use in-line LaTeX (see below) for mathematical expressions. Collaboration among students is allowed but it is not a group assignment so make sure your answer and code are your own. Make sure to mention your collaborators in your assignment with their names and their student ids.\n",
        "\n",
        "## Grading\n",
        "The entire assignment is out of 20 points.. For every late day, your grade will be deducted by 2 points (KAIST students only). You can use one of your no-penalty late days (7 days in total). Make sure to mention this in your submission. You will receive a grade of zero if you submit after 7 days.\n",
        "\n",
        "\n",
        "## Environment\n",
        "You will need Python 3.7+ and PyTorch 1.9+, which are already available on Colab:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9qwta269rqLQ",
        "outputId": "bf9a74cc-bd4b-41bb-db21-e2cdce658173"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "python 3.7.13\n",
            "torch 1.11.0+cu113\n"
          ]
        }
      ],
      "source": [
        "from platform import python_version\n",
        "import torch\n",
        "\n",
        "print(\"python\", python_version())\n",
        "print(\"torch\", torch.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTm5eq4NwQZs"
      },
      "source": [
        "## 1. Attention Layer\n",
        "\n",
        "We will first start with going over a few concepts that you learned in your high school statistics class. The variance of a random variable $X$, $\\text{Var}(X)$ is defined as $\\text{E}[(X-\\mu)^2]$ where $\\mu$ is the mean of $X$. Furthermore, given two independent random variables $X$ and $Y$ and a constant $a$,\n",
        "$$ \\text{Var}(X+Y) = \\text{Var}(X) + \\text{Var}(Y),$$\n",
        "$$ \\text{Var}(aX) = a^2\\text{Var}(X),$$\n",
        "$$ \\text{Var}(XY) = \\text{E}(X^2)\\text{E}(Y^2) - [\\text{E}(X)]^2[\\text{E}(Y)]^2.$$\n",
        "\n",
        "> **Problem 1.1** *(3 points)* Suppose we are given two sets of $n$ random variables, $X_1 \\dots X_n$ and $Y_1 \\dots Y_n$, where all of these $2n$ variables are mutually independent and have a mean of $0$ and a variance of $1$. Prove that\n",
        "$$\\text{Var}\\left(\\sum_i^n X_i Y_i\\right) = n.$$\n",
        "\n",
        "ANSWER\n",
        "\n",
        "$$ Var(\\sum_i^n X_i Y_i) = Var(X_1Y_1) + Var(X_2Y_2) + ........ Var(X_nY_n)  $$\n",
        "$$ Var(XY) = E[X^2]E[Y^2] - [E(X)]^2 [E(Y)]^2 $$\n",
        "\n",
        "the expected value E(X) = 0\n",
        "$$ Var(X) = E(X^2) – [E(X)]^2 $$\n",
        "$$E[X^2]E[Y^2] = (Var(X) + [E(X)]^2)(Var(Y)+[E(Y)]^2) $$\n",
        "\n",
        "\n",
        "$$ Var(XY) = (Var(X) + [E(X)]^2)(Var(Y)+[E(Y)]^2) - [E(X)]^2 [E(Y)]^2 $$\n",
        "$$Var(XY) = Var(X)*Var(Y) \\;\\;\\;\\; - \\;\\;\\;\\; eq \\; (1)$$\n",
        "$$ Var(X) and Var(Y) = 1  $$\n",
        "$$Var(XY) = 1$$\n",
        "$$ Var(\\sum_i^n X_i Y_i) = 1 + 1 +...... +1 $$\n",
        "$$ Var(\\sum_i^n X_i Y_i) = n$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KIelEM56jn_"
      },
      "source": [
        "In Lecture 11 and 12, we discussed how the attention is computed in Transformer via the following equation,\n",
        "$$ \\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}\\right)V.$$\n",
        "> **Problem 1.2** *(3 points)*  Suppose $Q$ and $K$ are matrices of independent variables each of which has a mean of $0$ and a variance of $1$. Using what you learned from Problem 1.1., show that (lazily defining the equality to be element-wise comparison)\n",
        "$$\\text{Var}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}\\right) = 1.$$\n",
        "\n",
        "\n",
        "$QK^T$ is a matrix multiplication, where each element is computed by sum of product of indenpendent random variables hence using from problem 1.1 Var of each element of matrxi of n x n is n. \n",
        "\n",
        "$$ \\frac{Var(QK^T)}{Var(\\sqrt d_k)} = \\frac{n}{d_k} = \\frac {n}{n} = 1  \\;\\;\\;\\; - \\;\\;\\;\\; eq \\; (2)$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DU3a3FEu6loq"
      },
      "source": [
        "\n",
        "> **Problem 1.3** *(2 points)* What would happen if the assumption that the variance of $Q$ and $K$ is $1$ does not hold? Consider each case of it being higher and lower than $1$ and conjecture what it implies, respectively.\n",
        "\n",
        "\n",
        "Using the following equation:  \n",
        "$$Var(XY) = Var(X)*Var(Y) \\;\\;\\;\\; - \\;\\;\\;\\; eq \\; (1)$$\n",
        "\n",
        "we can see that if variance of random variable is less than 1 or greater than 1 the varianace of the sum of products will be\n",
        "$n \\times Var(X) \\times Var(Y)$\n",
        "\n",
        "Hence eq 2 wont hold, and\n",
        "$$\\text{Var}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}\\right) = Var(X) \\times Var(Y)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjIPCG9xHxiK"
      },
      "source": [
        "> **Problem 1.4** *(2 points)* Now it is time to experimentally verify the theory! Create the random variables $X$ and $Y$ with the mean of zero and the variance of one (using `torch.randn`) and verify the equation in Problem 1.2. Then experiment with higher and lower variance to verify your finding in Problem 1.3. Briefly explain your results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fa1OHbIR7Fsd",
        "outputId": "0062c55a-10a9-4b85-d137-9c5646939882"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(1.0000)\n",
            "tensor(9.9986e-05)\n",
            "tensor(9998.7422)\n"
          ]
        }
      ],
      "source": [
        "n = 100\n",
        "\n",
        "Q_unitVar = torch.empty([n, n]).normal_(mean=0, std=1)\n",
        "K_unitVar = torch.empty([n, n]).normal_(mean=0, std=1)\n",
        "\n",
        "print(torch.var(Q_unitVar@K_unitVar.T)/n)\n",
        "\n",
        "Q_lowVar = torch.empty([n, n]).normal_(mean=0, std=0.1)\n",
        "K_lowVar = torch.empty([n, n]).normal_(mean=0, std=0.1)\n",
        "\n",
        "print(torch.var(Q_lowVar@K_lowVar.T)/n)\n",
        "\n",
        "Q_highVar = torch.empty([n, n]).normal_(mean=0, std=10)\n",
        "K_highVar = torch.empty([n, n]).normal_(mean=0, std=10)\n",
        "\n",
        "print(torch.var(Q_highVar@K_highVar.T)/n)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Results prove the given statment that with lower than 1 var, sum of product varriance goes down as well, and greater than 1 var, sum of product variance goes up as well."
      ],
      "metadata": {
        "id": "j2bvyQXa4rAj"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afEcnAG8Q1Jo"
      },
      "source": [
        "## 2. Transformer for Spelling Error Correction\n",
        "\n",
        "In this section, you will implement Transformer for a few tasks that are simpler than machine translation. Feel free to copy and paste from [The Annotated Transformer](http://nlp.seas.harvard.edu/annotated-transformer/) (note that this is a new version released in 2022 recently), though make sure to mention that you copied the code from it. Note that we do not provide a separate training or evaluation data, so it is your job to be able to create these in a reasonable manner.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDaIrTN1FtSb"
      },
      "source": [
        "### Copied Code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gmRZG6ozEnmq"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from os.path import exists\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn.functional import log_softmax, pad\n",
        "import math\n",
        "import copy\n",
        "import time\n",
        "from torch.optim.lr_scheduler import LambdaLR\n",
        "import pandas as pd\n",
        "import altair as alt\n",
        "from torchtext.data.functional import to_map_style_dataset\n",
        "from torch.utils.data import DataLoader\n",
        "from torchtext.vocab import build_vocab_from_iterator\n",
        "import torchtext.datasets as datasets\n",
        "import spacy\n",
        "import warnings\n",
        "from torch.utils.data.distributed import DistributedSampler\n",
        "import torch.distributed as dist\n",
        "import torch.multiprocessing as mp\n",
        "from torch.nn.parallel import DistributedDataParallel as DDP\n",
        "\n",
        "\n",
        "# Set to False to skip notebook execution (e.g. for debugging)\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "RUN_EXAMPLES = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pwgrAl9TEwVy"
      },
      "outputs": [],
      "source": [
        "class EncoderDecoder(nn.Module):\n",
        "    \"\"\"\n",
        "    A standard Encoder-Decoder architecture. Base for this and many\n",
        "    other models.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):\n",
        "        super(EncoderDecoder, self).__init__()\n",
        "        self.encoder = encoder\n",
        "        self.decoder = decoder\n",
        "        self.src_embed = src_embed\n",
        "        self.tgt_embed = tgt_embed\n",
        "        self.generator = generator\n",
        "\n",
        "    def forward(self, src, tgt, src_mask, tgt_mask):\n",
        "        \"Take in and process masked src and target sequences.\"\n",
        "        return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)\n",
        "\n",
        "    def encode(self, src, src_mask):\n",
        "        return self.encoder(self.src_embed(src), src_mask)\n",
        "\n",
        "    def decode(self, memory, src_mask, tgt, tgt_mask):\n",
        "        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZA8vexbgEyYp"
      },
      "outputs": [],
      "source": [
        "class Generator(nn.Module):\n",
        "    \"Define standard linear + softmax generation step.\"\n",
        "\n",
        "    def __init__(self, d_model, vocab):\n",
        "        super(Generator, self).__init__()\n",
        "        self.proj = nn.Linear(d_model, vocab)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return log_softmax(self.proj(x), dim=-1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cI7k6fMZE2F_"
      },
      "outputs": [],
      "source": [
        "def clones(module, N):\n",
        "    \"Produce N identical layers.\"\n",
        "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "POfzriILE4Rd"
      },
      "outputs": [],
      "source": [
        "class Encoder(nn.Module):\n",
        "    \"Core encoder is a stack of N layers\"\n",
        "\n",
        "    def __init__(self, layer, N):\n",
        "        super(Encoder, self).__init__()\n",
        "        self.layers = clones(layer, N)\n",
        "        self.norm = LayerNorm(layer.size)\n",
        "\n",
        "    def forward(self, x, mask):\n",
        "        \"Pass the input (and mask) through each layer in turn.\"\n",
        "        for layer in self.layers:\n",
        "            x = layer(x, mask)\n",
        "        return self.norm(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5k_qeTWTE757"
      },
      "outputs": [],
      "source": [
        "class LayerNorm(nn.Module):\n",
        "    \"Construct a layernorm module (See citation for details).\"\n",
        "\n",
        "    def __init__(self, features, eps=1e-6):\n",
        "        super(LayerNorm, self).__init__()\n",
        "        self.a_2 = nn.Parameter(torch.ones(features))\n",
        "        self.b_2 = nn.Parameter(torch.zeros(features))\n",
        "        self.eps = eps\n",
        "\n",
        "    def forward(self, x):\n",
        "        mean = x.mean(-1, keepdim=True)\n",
        "        std = x.std(-1, keepdim=True)\n",
        "        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yJ23VaGlE-MB"
      },
      "outputs": [],
      "source": [
        "class SublayerConnection(nn.Module):\n",
        "    \"\"\"\n",
        "    A residual connection followed by a layer norm.\n",
        "    Note for code simplicity the norm is first as opposed to last.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, size, dropout):\n",
        "        super(SublayerConnection, self).__init__()\n",
        "        self.norm = LayerNorm(size)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x, sublayer):\n",
        "        \"Apply residual connection to any sublayer with the same size.\"\n",
        "        return x + self.dropout(sublayer(self.norm(x)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nwq_h1t4FA0C"
      },
      "outputs": [],
      "source": [
        "class EncoderLayer(nn.Module):\n",
        "    \"Encoder is made up of self-attn and feed forward (defined below)\"\n",
        "\n",
        "    def __init__(self, size, self_attn, feed_forward, dropout):\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        self.self_attn = self_attn\n",
        "        self.feed_forward = feed_forward\n",
        "        self.sublayer = clones(SublayerConnection(size, dropout), 2)\n",
        "        self.size = size\n",
        "\n",
        "    def forward(self, x, mask):\n",
        "        \"Follow Figure 1 (left) for connections.\"\n",
        "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))\n",
        "        return self.sublayer[1](x, self.feed_forward)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7y_-CkqCFDBP"
      },
      "outputs": [],
      "source": [
        "class Decoder(nn.Module):\n",
        "    \"Generic N layer decoder with masking.\"\n",
        "\n",
        "    def __init__(self, layer, N):\n",
        "        super(Decoder, self).__init__()\n",
        "        self.layers = clones(layer, N)\n",
        "        self.norm = LayerNorm(layer.size)\n",
        "\n",
        "    def forward(self, x, memory, src_mask, tgt_mask):\n",
        "        for layer in self.layers:\n",
        "            x = layer(x, memory, src_mask, tgt_mask)\n",
        "        return self.norm(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xyLwgRZ-FFUk"
      },
      "outputs": [],
      "source": [
        "class DecoderLayer(nn.Module):\n",
        "    \"Decoder is made of self-attn, src-attn, and feed forward (defined below)\"\n",
        "\n",
        "    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):\n",
        "        super(DecoderLayer, self).__init__()\n",
        "        self.size = size\n",
        "        self.self_attn = self_attn\n",
        "        self.src_attn = src_attn\n",
        "        self.feed_forward = feed_forward\n",
        "        self.sublayer = clones(SublayerConnection(size, dropout), 3)\n",
        "\n",
        "    def forward(self, x, memory, src_mask, tgt_mask):\n",
        "        \"Follow Figure 1 (right) for connections.\"\n",
        "        m = memory\n",
        "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))\n",
        "        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))\n",
        "        return self.sublayer[2](x, self.feed_forward)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vhD0AZz1FJEX"
      },
      "outputs": [],
      "source": [
        "def subsequent_mask(size):\n",
        "    \"Mask out subsequent positions.\"\n",
        "    attn_shape = (1, size, size)\n",
        "    subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1).type(\n",
        "        torch.uint8\n",
        "    )\n",
        "    return subsequent_mask == 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6YlTfVTUFL0V"
      },
      "outputs": [],
      "source": [
        "def attention(query, key, value, mask=None, dropout=None):\n",
        "    \"Compute 'Scaled Dot Product Attention'\"\n",
        "    d_k = query.size(-1)\n",
        "    scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)\n",
        "    if mask is not None:\n",
        "        scores = scores.masked_fill(mask == 0, -1e9)\n",
        "    p_attn = scores.softmax(dim=-1)\n",
        "    if dropout is not None:\n",
        "        p_attn = dropout(p_attn)\n",
        "    return torch.matmul(p_attn, value), p_attn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HQbxOVAEFZl0"
      },
      "outputs": [],
      "source": [
        "class MultiHeadedAttention(nn.Module):\n",
        "    def __init__(self, h, d_model, dropout=0.1):\n",
        "        \"Take in model size and number of heads.\"\n",
        "        super(MultiHeadedAttention, self).__init__()\n",
        "        assert d_model % h == 0\n",
        "        # We assume d_v always equals d_k\n",
        "        self.d_k = d_model // h\n",
        "        self.h = h\n",
        "        self.linears = clones(nn.Linear(d_model, d_model), 4)\n",
        "        self.attn = None\n",
        "        self.dropout = nn.Dropout(p=dropout)\n",
        "\n",
        "    def forward(self, query, key, value, mask=None):\n",
        "        \"Implements Figure 2\"\n",
        "        if mask is not None:\n",
        "            # Same mask applied to all h heads.\n",
        "            mask = mask.unsqueeze(1)\n",
        "        nbatches = query.size(0)\n",
        "\n",
        "        # 1) Do all the linear projections in batch from d_model => h x d_k\n",
        "        query, key, value = [\n",
        "            lin(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n",
        "            for lin, x in zip(self.linears, (query, key, value))\n",
        "        ]\n",
        "\n",
        "        # 2) Apply attention on all the projected vectors in batch.\n",
        "        x, self.attn = attention(\n",
        "            query, key, value, mask=mask, dropout=self.dropout\n",
        "        )\n",
        "\n",
        "        # 3) \"Concat\" using a view and apply a final linear.\n",
        "        x = (\n",
        "            x.transpose(1, 2)\n",
        "            .contiguous()\n",
        "            .view(nbatches, -1, self.h * self.d_k)\n",
        "        )\n",
        "        del query\n",
        "        del key\n",
        "        del value\n",
        "        return self.linears[-1](x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KprVLG6IFdJK"
      },
      "outputs": [],
      "source": [
        "class PositionwiseFeedForward(nn.Module):\n",
        "    \"Implements FFN equation.\"\n",
        "\n",
        "    def __init__(self, d_model, d_ff, dropout=0.1):\n",
        "        super(PositionwiseFeedForward, self).__init__()\n",
        "        self.w_1 = nn.Linear(d_model, d_ff)\n",
        "        self.w_2 = nn.Linear(d_ff, d_model)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.w_2(self.dropout(self.w_1(x).relu()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pvJGTP4tFfSY"
      },
      "outputs": [],
      "source": [
        "class Embeddings(nn.Module):\n",
        "    def __init__(self, d_model, vocab):\n",
        "        super(Embeddings, self).__init__()\n",
        "        self.lut = nn.Embedding(vocab, d_model)\n",
        "        self.d_model = d_model\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.lut(x) * math.sqrt(self.d_model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ga-TV4OhFihx"
      },
      "outputs": [],
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    \"Implement the PE function.\"\n",
        "\n",
        "    def __init__(self, d_model, dropout, max_len=5000):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        self.dropout = nn.Dropout(p=dropout)\n",
        "\n",
        "        # Compute the positional encodings once in log space.\n",
        "        pe = torch.zeros(max_len, d_model)\n",
        "        position = torch.arange(0, max_len).unsqueeze(1)\n",
        "        div_term = torch.exp(\n",
        "            torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model)\n",
        "        )\n",
        "        pe[:, 0::2] = torch.sin(position * div_term)\n",
        "        pe[:, 1::2] = torch.cos(position * div_term)\n",
        "        pe = pe.unsqueeze(0)\n",
        "        self.register_buffer(\"pe\", pe)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x + self.pe[:, : x.size(1)].requires_grad_(False)\n",
        "        return self.dropout(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KhXHh6JfeB0i"
      },
      "outputs": [],
      "source": [
        "class Batch:\n",
        "    \"\"\"Object for holding a batch of data with mask during training.\"\"\"\n",
        "\n",
        "    def __init__(self, src, tgt=None, pad=2):  # 2 = <blank>\n",
        "        self.src = src\n",
        "        self.src_mask = (src != pad).unsqueeze(-2)\n",
        "        if tgt is not None:\n",
        "            self.tgt = tgt[:, :-1]\n",
        "            self.tgt_y = tgt[:, 1:]\n",
        "            self.tgt_mask = self.make_std_mask(self.tgt, pad)\n",
        "            self.ntokens = (self.tgt_y != pad).data.sum()\n",
        "\n",
        "    @staticmethod\n",
        "    def make_std_mask(tgt, pad):\n",
        "        \"Create a mask to hide padding and future words.\"\n",
        "        tgt_mask = (tgt != pad).unsqueeze(-2)\n",
        "        tgt_mask = tgt_mask & subsequent_mask(tgt.size(-1)).type_as(\n",
        "            tgt_mask.data\n",
        "        )\n",
        "        return tgt_mask"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KT8y9tezeKKL"
      },
      "outputs": [],
      "source": [
        "class TrainState:\n",
        "    \"\"\"Track number of steps, examples, and tokens processed\"\"\"\n",
        "\n",
        "    step: int = 0  # Steps in the current epoch\n",
        "    accum_step: int = 0  # Number of gradient accumulation steps\n",
        "    samples: int = 0  # total # of examples used\n",
        "    tokens: int = 0  # total # of tokens processed"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I6T2eKvpejJM"
      },
      "outputs": [],
      "source": [
        "def rate(step, model_size, factor, warmup):\n",
        "    \"\"\"\n",
        "    we have to default the step to 1 for LambdaLR function\n",
        "    to avoid zero raising to negative power.\n",
        "    \"\"\"\n",
        "    if step == 0:\n",
        "        step = 1\n",
        "    return factor * (\n",
        "        model_size ** (-0.5) * min(step ** (-0.5), step * warmup ** (-1.5))\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z-qFvH6Dep6P"
      },
      "outputs": [],
      "source": [
        "class LabelSmoothing(nn.Module):\n",
        "    \"Implement label smoothing.\"\n",
        "\n",
        "    def __init__(self, size, padding_idx, smoothing=0.0):\n",
        "        super(LabelSmoothing, self).__init__()\n",
        "        self.criterion = nn.KLDivLoss(reduction=\"sum\")\n",
        "        self.padding_idx = padding_idx\n",
        "        self.confidence = 1.0 - smoothing\n",
        "        self.smoothing = smoothing\n",
        "        self.size = size\n",
        "        self.true_dist = None\n",
        "\n",
        "    def forward(self, x, target):\n",
        "        assert x.size(1) == self.size\n",
        "        true_dist = x.data.clone()\n",
        "        true_dist.fill_(self.smoothing / (self.size - 2))\n",
        "        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)\n",
        "        true_dist[:, self.padding_idx] = 0\n",
        "        mask = torch.nonzero(target.data == self.padding_idx)\n",
        "        if mask.dim() > 0:\n",
        "            true_dist.index_fill_(0, mask.squeeze(), 0.0)\n",
        "        self.true_dist = true_dist\n",
        "        return self.criterion(x, true_dist.clone().detach())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4cFFMSPte0r1"
      },
      "outputs": [],
      "source": [
        "def loss(x, crit):\n",
        "    d = x + 3 * 1\n",
        "    predict = torch.FloatTensor([[0, x / d, 1 / d, 1 / d, 1 / d]])\n",
        "    return crit(predict.log(), torch.LongTensor([1])).data\n",
        "\n",
        "\n",
        "def penalization_visualization():\n",
        "    crit = LabelSmoothing(5, 0, 0.1)\n",
        "    loss_data = pd.DataFrame(\n",
        "        {\n",
        "            \"Loss\": [loss(x, crit) for x in range(1, 100)],\n",
        "            \"Steps\": list(range(99)),\n",
        "        }\n",
        "    ).astype(\"float\")\n",
        "\n",
        "    return (\n",
        "        alt.Chart(loss_data)\n",
        "        .mark_line()\n",
        "        .properties(width=350)\n",
        "        .encode(\n",
        "            x=\"Steps\",\n",
        "            y=\"Loss\",\n",
        "        )\n",
        "        .interactive()\n",
        "    )\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D5sRRHWZhEw4"
      },
      "outputs": [],
      "source": [
        "# Some convenience helper functions used throughout the notebook\n",
        "\n",
        "\n",
        "def is_interactive_notebook():\n",
        "    return __name__ == \"__main__\"\n",
        "\n",
        "\n",
        "def show_example(fn, args=[]):\n",
        "    if __name__ == \"__main__\" and RUN_EXAMPLES:\n",
        "        return fn(*args)\n",
        "\n",
        "\n",
        "def execute_example(fn, args=[]):\n",
        "    if __name__ == \"__main__\" and RUN_EXAMPLES:\n",
        "        fn(*args)\n",
        "\n",
        "\n",
        "class DummyOptimizer(torch.optim.Optimizer):\n",
        "    def __init__(self):\n",
        "        self.param_groups = [{\"lr\": 0}]\n",
        "        None\n",
        "\n",
        "    def step(self):\n",
        "        None\n",
        "\n",
        "    def zero_grad(self, set_to_none=False):\n",
        "        None\n",
        "\n",
        "\n",
        "class DummyScheduler:\n",
        "    def step(self):\n",
        "        None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l_JVxr463gv7"
      },
      "outputs": [],
      "source": [
        "def make_model(src_vocab, tgt_vocab, N=6, d_model=512, d_ff=2048, h=8, dropout=0.1):\n",
        "    \"Helper: Construct a model from hyperparameters.\"\n",
        "    c = copy.deepcopy\n",
        "    attn = MultiHeadedAttention(h, d_model)\n",
        "    ff = PositionwiseFeedForward(d_model, d_ff, dropout)\n",
        "    position = PositionalEncoding(d_model, dropout)\n",
        "    model = EncoderDecoder(\n",
        "        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),\n",
        "        Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), N),\n",
        "        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),\n",
        "        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),\n",
        "        Generator(d_model, tgt_vocab),\n",
        "    )\n",
        "\n",
        "    # This was important from their code.\n",
        "    # Initialize parameters with Glorot / fan_avg.\n",
        "    for p in model.parameters():\n",
        "        if p.dim() > 1:\n",
        "            nn.init.xavier_uniform_(p)\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZxYOgzTK3XeQ"
      },
      "outputs": [],
      "source": [
        "def run_epoch(\n",
        "    data_iter,\n",
        "    model,\n",
        "    loss_compute,\n",
        "    optimizer,\n",
        "    scheduler,\n",
        "    mode=\"train\",\n",
        "    accum_iter=1,\n",
        "    train_state=TrainState(),\n",
        "    epochID=None\n",
        "):\n",
        "    \"\"\"Train a single epoch\"\"\"\n",
        "    start = time.time()\n",
        "    total_tokens = 0\n",
        "    total_loss = 0\n",
        "    tokens = 0\n",
        "    n_accum = 0\n",
        "    for i, batch in enumerate(data_iter):\n",
        "        out = model.forward(\n",
        "            batch.src, batch.tgt, batch.src_mask, batch.tgt_mask\n",
        "        )\n",
        "        loss, loss_node = loss_compute(out, batch.tgt_y, batch.ntokens)\n",
        "        # loss_node = loss_node / accum_iter\n",
        "        if mode == \"train\" or mode == \"train+log\":\n",
        "            loss_node.backward()\n",
        "            train_state.step += 1\n",
        "            train_state.samples += batch.src.shape[0]\n",
        "            train_state.tokens += batch.ntokens\n",
        "            if i % accum_iter == 0:\n",
        "                optimizer.step()\n",
        "                optimizer.zero_grad(set_to_none=True)\n",
        "                n_accum += 1\n",
        "                train_state.accum_step += 1\n",
        "            scheduler.step()\n",
        "\n",
        "        total_loss += loss\n",
        "        total_tokens += batch.ntokens\n",
        "        tokens += batch.ntokens\n",
        "        if i % 40 == 1 and (mode == \"train\" or mode == \"train+log\"):\n",
        "            lr = optimizer.param_groups[0][\"lr\"]\n",
        "            elapsed = time.time() - start\n",
        "            print(\n",
        "                (\n",
        "                    \"Epoch Step: %6d | Batch Number: %6d | Accumulation Step: %3d | Loss: %6.2f \"\n",
        "                    + \"| Tokens / Sec: %7.1f | Learning Rate: %6.1e\"\n",
        "                )\n",
        "                % (epochID, i, n_accum, loss / batch.ntokens, tokens / elapsed, lr)\n",
        "            )\n",
        "            start = time.time()\n",
        "            tokens = 0\n",
        "        del loss\n",
        "        del loss_node\n",
        "    return total_loss / total_tokens, train_state"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KXjep4nBe3r0"
      },
      "outputs": [],
      "source": [
        "def data_gen(V, batch_size, nbatches, len=32):\n",
        "    \"Generate random data for a src-tgt copy task.\"\n",
        "    for i in range(nbatches):\n",
        "        data = torch.randint(0, V, size=(batch_size, len)).to(device)\n",
        "        data[:, 0] = 0\n",
        "        src = data.requires_grad_(False).clone().detach()\n",
        "        tgt = data.requires_grad_(False).clone().detach()\n",
        "        yield Batch(src, tgt, 0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wnO7_nale8y2"
      },
      "outputs": [],
      "source": [
        "class SimpleLossCompute:\n",
        "    \"A simple loss compute and train function.\"\n",
        "\n",
        "    def __init__(self, generator, criterion):\n",
        "        self.generator = generator\n",
        "        self.criterion = criterion\n",
        "\n",
        "    def __call__(self, x, y, norm):\n",
        "        x = self.generator(x)\n",
        "        sloss = (\n",
        "            self.criterion(\n",
        "                x.contiguous().view(-1, x.size(-1)), y.contiguous().view(-1)\n",
        "            )\n",
        "            / norm\n",
        "        )\n",
        "        return sloss.data * norm, sloss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-_1F26hYe_kc"
      },
      "outputs": [],
      "source": [
        "def greedy_decode(model, src, src_mask, max_len, start_symbol, end_symbol=None):\n",
        "    memory = model.encode(src, src_mask)\n",
        "    ys = torch.zeros(1, 1).fill_(start_symbol).type_as(src.data)\n",
        "    i = 0\n",
        "    next_word = 0\n",
        "    while next_word != end_symbol and i < max_len-1:\n",
        "      out = model.decode(memory, src_mask, ys, subsequent_mask(ys.size(1)).type_as(src.data))\n",
        "      prob = model.generator(out[:, -1])\n",
        "      _, next_word = torch.max(prob, dim=1)\n",
        "      next_word = next_word.data[0]\n",
        "      ys = torch.cat([ys, torch.zeros(1, 1).type_as(src.data).fill_(next_word)], dim=1)\n",
        "      i += 1\n",
        "    return ys"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPDDq-X7Fpv2"
      },
      "source": [
        "### **Problem 2.1** *(3 points)* \n",
        "> Create a model that takes a random set of input symbols from a vocabulary of digits (i.e. 0, 1, ... , 8, 9) as the input and generate back the same symbols. Instead of varying length, we fix the length to 32. Make sure to report that your model's sequence-level (not token level) accuracy goes above 90%. Note that a similar problem is also in Annotated Transformer, and copying the code is allowed."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "metadata": {
        "id": "F5_49dXKk2un"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YjbQocMdg405",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e194c15f-b8e5-4eac-c129-cd60f9549ae6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch Step:      1 | Batch Number:      1 | Accumulation Step:   2 | Loss:   3.05 | Tokens / Sec:  9058.6 | Learning Rate: 5.5e-06\n",
            "Epoch Step:      2 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.17 | Tokens / Sec: 12228.1 | Learning Rate: 8.8e-05\n",
            "Epoch Step:      3 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.05 | Tokens / Sec: 12272.9 | Learning Rate: 1.7e-04\n",
            "Epoch Step:      4 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.99 | Tokens / Sec: 12303.4 | Learning Rate: 2.5e-04\n",
            "Epoch Step:      5 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.55 | Tokens / Sec: 12305.7 | Learning Rate: 3.4e-04\n",
            "Epoch Step:      6 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.86 | Tokens / Sec: 10657.7 | Learning Rate: 4.2e-04\n",
            "Epoch Step:      7 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.27 | Tokens / Sec: 12218.3 | Learning Rate: 5.0e-04\n",
            "Epoch Step:      8 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.15 | Tokens / Sec: 12048.1 | Learning Rate: 5.9e-04\n",
            "Epoch Step:      9 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 12248.8 | Learning Rate: 6.7e-04\n",
            "Epoch Step:     10 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 12175.8 | Learning Rate: 7.5e-04\n",
            "Epoch Step:     11 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 12197.8 | Learning Rate: 8.3e-04\n",
            "Epoch Step:     12 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 12336.6 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     13 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 12216.5 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     14 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 12250.1 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     15 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 12328.8 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     16 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12198.5 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     17 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12221.0 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     18 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12337.6 | Learning Rate: 9.8e-04\n",
            "Epoch Step:     19 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 12356.1 | Learning Rate: 9.5e-04\n",
            "Epoch Step:     20 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12274.6 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     21 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 12133.5 | Learning Rate: 9.0e-04\n",
            "Epoch Step:     22 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12361.5 | Learning Rate: 8.8e-04\n",
            "Epoch Step:     23 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12257.9 | Learning Rate: 8.6e-04\n",
            "Epoch Step:     24 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 12330.3 | Learning Rate: 8.4e-04\n",
            "Epoch Step:     25 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 12322.1 | Learning Rate: 8.2e-04\n",
            "Epoch Step:     26 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 12220.0 | Learning Rate: 8.1e-04\n",
            "Epoch Step:     27 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 12210.3 | Learning Rate: 7.9e-04\n",
            "Epoch Step:     28 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12282.9 | Learning Rate: 7.8e-04\n",
            "Epoch Step:     29 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 12255.7 | Learning Rate: 7.6e-04\n",
            "Epoch Step:     30 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 12206.6 | Learning Rate: 7.5e-04\n"
          ]
        }
      ],
      "source": [
        "# Train the simple copy task.\n",
        "V = 10\n",
        "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
        "model = make_model(V, V, N=2)\n",
        "\n",
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(), lr=0.5, betas=(0.9, 0.98), eps=1e-9\n",
        ")\n",
        "lr_scheduler = LambdaLR(\n",
        "    optimizer=optimizer,\n",
        "    lr_lambda=lambda step: rate(\n",
        "        step, model_size=model.src_embed[0].d_model, factor=1.0, warmup=400\n",
        "    ),\n",
        ")\n",
        "\n",
        "batch_size = 80\n",
        "model = model.to(device)\n",
        "\n",
        "for epoch in range(30):\n",
        "    model.train()\n",
        "    run_epoch(\n",
        "        data_gen(V, batch_size, 30, 32),\n",
        "        model,\n",
        "        SimpleLossCompute(model.generator, criterion),\n",
        "        optimizer,\n",
        "        lr_scheduler,\n",
        "        mode=\"train\",\n",
        "        epochID=epoch+1\n",
        "    )\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZA19OQHcuMQn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9eebda3-5358-4d65-f82c-cc76c94f633c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(1., device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "model.eval()\n",
        "max_len = 32\n",
        "src_mask = torch.ones(1, 1, max_len).to(device)\n",
        "\n",
        "ite = 100\n",
        "TP = 0\n",
        "for i in range(ite):\n",
        "  src = torch.randint(1, V, size=(1, max_len)).to(device)\n",
        "  src[0][0] = 0\n",
        "  predicted = greedy_decode(model, src, src_mask, max_len=max_len, start_symbol=0)\n",
        "\n",
        "  TP += torch.all(src == predicted)\n",
        "\n",
        "print(TP/ite)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "\n",
        "### **Problem 2.2** *(4 points)* \n",
        ">Now, we will implement a bit more useful function, so-called spelling error correction. Your job is to create a model whose input is a word with spelling errors, and the output is the spelling-corrected word. Here, your vocabulary will be character instead of word. You can create your own training data by using an existing text corpus as the target and inject noise into it to use it as the input. You are free to use whichever text corpus you like. If you can't think of one, please use context data in SQuAD Dataset (see Assignment 2). Report accuracy in your own evaluation data (you will receive full credit as long as both the evaluation data and the accuracy are reasonable), and also show 3 examples where it succeeds at correcting spelling."
      ],
      "metadata": {
        "id": "FkriXFh4ZZrZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q datasets"
      ],
      "metadata": {
        "id": "-kYpnRx7_gq5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bf0a10c1-3909-4580-c6e5-8a0569ec7498"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K     |████████████████████████████████| 342 kB 5.3 MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 44.6 MB/s \n",
            "\u001b[K     |████████████████████████████████| 212 kB 45.4 MB/s \n",
            "\u001b[K     |████████████████████████████████| 136 kB 52.9 MB/s \n",
            "\u001b[K     |████████████████████████████████| 84 kB 2.7 MB/s \n",
            "\u001b[K     |████████████████████████████████| 127 kB 44.3 MB/s \n",
            "\u001b[K     |████████████████████████████████| 144 kB 43.2 MB/s \n",
            "\u001b[K     |████████████████████████████████| 94 kB 3.1 MB/s \n",
            "\u001b[K     |████████████████████████████████| 271 kB 39.7 MB/s \n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "maxWordLen = 20\n",
        "batchSize = 64"
      ],
      "metadata": {
        "id": "uxS6Gvj7vdbu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "import torchtext\n",
        "from torchtext.data import get_tokenizer\n",
        "\n",
        "tokenizer = get_tokenizer(\"basic_english\") \n",
        "\n",
        "squad_dataset = load_dataset('squad')\n",
        "dataSet = squad_dataset['validation']\n",
        "\n",
        "string = ' '.join([str(elem).strip('\\n') for elem in dataSet['context']])\n",
        "\n",
        "words = set(tokenizer(string))\n",
        "words = [i for i in words if len(i) > 2 and len(i) <= maxWordLen - 2]\n",
        "\n",
        "alphabatedWords = [list(i) for i in words]\n",
        "\n",
        "vocab = ['SOW', 'EOW', 'PAD', 'UNK'] + list(set(''.join([str(elem).strip('\\n') for elem in words])))\n",
        "word2id = {word: id_ for id_, word in enumerate(vocab)}\n",
        "\n",
        "noisyWords = []\n",
        "\n",
        "for i in alphabatedWords:\n",
        "  replaceID = torch.randint(1, len(i), [1])[0]\n",
        "  replaingID = torch.randint(4, len(vocab), [1])[0]\n",
        "  temp = i.copy()\n",
        "  temp[replaceID] = vocab[replaingID]\n",
        "  noisyWords.append(temp)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296,
          "referenced_widgets": [
            "37a1c3f646f34a69a351789eb5058154",
            "f6a4cf97a61c4589a16dd2484690e042",
            "f40f0e17a44d46d8aea607479fe19ca9",
            "de809c06234742af8cb09a122abc7793",
            "d9a132afc1d64534bd8a2af52ed68ebc",
            "e2ad502ac00f42478d8ec91292477c0b",
            "bd5a726b8115452ebdc95ec055156709",
            "be0372fe2e354da683de2fc44cbdf53a",
            "011e721dd0474dadb22d4f2542702565",
            "7a0b6ff42a634d8385547a90b67d8478",
            "2150053d072540f498852f02c8ad2ce0",
            "90e52321f2f042739987e5e4689b9584",
            "3631fc5e601f4f949926142280103f7b",
            "8bb5f3908e25416d92e60b760c09a985",
            "cb7c358c324a4f48be877dd8f1a366d4",
            "7fda162eb92b4ec8af5858ebaaa0b530",
            "85d42260600646c29f651906d07b0614",
            "56c12d049c674101be93d2d58e365194",
            "145c60d16a6d43c397a5aec8bbbb88b1",
            "e341996a7fa84bf998cd724691e8f4a1",
            "18d173973a9a471eb4214c0bcaa05f67",
            "cfe4aae8f0cf4cccbe9a3381838e6737",
            "1ed385fe271c426aa3ebe78fabd818fa",
            "97cc0d91c6aa4214992de1e9a4422cb9",
            "8f38e1ba93b844cdab487fe952a50e91",
            "dc979228c3cf42229c921f9ebb544051",
            "312266c9c3e14c50a02ff96b7c71ceaf",
            "0e04f3664880435198f1a50fa8c1ee92",
            "1003465b490a46a5a4a3a74993d93ed5",
            "a2a23cbdb65c46d6baf033d87e688622",
            "5f784fc250d04dec93928b2c23b98530",
            "0be2c167de3242199af03b341cb5e87e",
            "aa64c77fa6654dacbe2c59b77d3931dc",
            "636eb2f3d0ee4dfba757f5ab3f82dfba",
            "21c861a5de814336934aba3a7f5633cb",
            "5cc3ab4ac7ff43d5a5916a0152bf241e",
            "82de3dd9c58640a3b8454ac5a1ef3cf2",
            "60f0bc965af84127a6caff61f60eb29c",
            "3a8a88035be34d67b46e8e1f6eaa65f6",
            "94c12462d0784a46b573f58bd0ac809f",
            "fc9b554fb3bb4fbe9c678d27199963a4",
            "38286e8e71fc46e585feba5535d6c56a",
            "c9fb158b66e943a799c9a84f2339c045",
            "790cdfe161714e9295cd46b1fa6e7765",
            "c4fdfcfb6a36475cbbaebb09f5572231",
            "3fa66a14af0c4bce9da4241a662de37c",
            "3154bc23f2bf4f60ab1249478880fd2f",
            "ad81a932fa2c450897896ef774a380a8",
            "3eb3debc46e34bc0a0d186d4d38f047b",
            "3fa79e95bc2c49dbac7d90f77e81399a",
            "5fc89844e65a4dd9b4122376876e023b",
            "f1e1f121a7e448c28a7671b28b5fbf9b",
            "9c4aafd33c0c483cbb7367eaa38f1ec2",
            "e01e5e853b7b4fbc883edc12254bac30",
            "6f289e2a8cba495394b6ed58b4b9fa13",
            "32fd852d8f21436f910a2afc600273e3",
            "47cf2dfd8e7b43f295246bb1a31068ae",
            "7ae1101a8b17496ba0fc7665910684db",
            "bd866563605d4b9795e156f7058d33c8",
            "0537e872515a441a87c1e61944b283b7",
            "e030e39f46a44d8499bf1cf970bd5c14",
            "b41c1351af7349d48849fa871e898690",
            "57f23c622fde491a90a9e4f27c4f0d46",
            "921c15f474474996891162e5ed4b769c",
            "57d403f7b7ac40a595c79d09b1099711",
            "c781a4e8a56c46068ac82e6c29a7d256",
            "1de15df05d1d4286985253db63f0e8e7",
            "ff0ea9831b7c458bb01ea9b185b65679",
            "f9927354daf34230b8108ca46e54c25c",
            "bef24215718748d5980285299e4c7d97",
            "a72a7b1fc8e84bec8542dc0a003186bb",
            "b5607a20871c471cbcb4d7dc74fa3415",
            "5e2373d2396445fbac09bbb8db774aa5",
            "73f83e5fdc854821bca5b96a4c43fb7d",
            "8cdcd0b034744325a98686b6a4990a39",
            "5a8d6b909a2346869c6f13fb6244cdd3",
            "90fb40158daa4eb58a1f90ef27c47b23",
            "00eb8bc6166c42069c261c2789b48b20",
            "21bbecdeae51471ba2fd4ea49a72f94f",
            "84912be0495945c4810113cd4b78eebc",
            "34ac6894b4124ec3b086f4b126c5f6f0",
            "a241157c8bb14cdea1b8d93542e12d62",
            "dbd3c6a402754be98995540d76cb4bb6",
            "e0652c0246cb4ce1aa4c7050f9b98762",
            "1af0b0bebc424cb6858c0d7bf1683b1f",
            "fb84f9f4ce5a499c9adcacf1cb6bee91",
            "60609b0928dd4b7b89423a5caaf5a1a5",
            "cbdb79ae1478465d94c55bbeeeef6a8f",
            "79b99b9a258f49e09ad27b402854240d",
            "daa82bc51780498b9d3a381a936e2d7e",
            "b8d51a6cceac462da3af8dc6d45d5982",
            "3891cd26ee414eb380ce38aca5884fdc",
            "5739fedfcd414558be05f7673c9e0556",
            "14916a09e75748abaa413a6633926aa7",
            "0b43f0a6e13d4ba8b63bd04799ef93da",
            "d8bab05b445f4211bf4a442acec8a09a",
            "74bc092eb82a48eaadfb8515b490f6ab",
            "e97f623a655946d0a67abd3900f2d4bf",
            "68669a5c4aa94cc69ce80b442cbfc0b7"
          ]
        },
        "id": "yuVefnes8tkG",
        "outputId": "78024c9e-48a0-4597-c854-d41ba8dda8be"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/1.97k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "37a1c3f646f34a69a351789eb5058154"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/1.02k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "90e52321f2f042739987e5e4689b9584"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset squad/plain_text (download: 33.51 MiB, generated: 85.63 MiB, post-processed: Unknown size, total: 119.14 MiB) to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1ed385fe271c426aa3ebe78fabd818fa"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/8.12M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "636eb2f3d0ee4dfba757f5ab3f82dfba"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/1.05M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c4fdfcfb6a36475cbbaebb09f5572231"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Extracting data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "32fd852d8f21436f910a2afc600273e3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/87599 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1de15df05d1d4286985253db63f0e8e7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating validation split:   0%|          | 0/10570 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "00eb8bc6166c42069c261c2789b48b20"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset squad downloaded and prepared to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "79b99b9a258f49e09ad27b402854240d"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "numberOfWords = (len(alphabatedWords)//batchSize)*batchSize\n",
        "alphabetDataSet = torch.zeros([numberOfWords, maxWordLen], device=device).fill_(2)\n",
        "alphabetDataSet[:, 0] = 0\n",
        "alphabetDataSet[:, -1] = 1\n",
        "\n",
        "for i in range(numberOfWords):\n",
        "  for j, alphabet in enumerate(alphabatedWords[i]):\n",
        "    if alphabet not in word2id:\n",
        "      alphabetDataSet[i][j+1] = 3\n",
        "    else:\n",
        "      alphabetDataSet[i][j+1] = word2id[alphabet]\n",
        "\n",
        "alphabetDataSet = alphabetDataSet.reshape(len(alphabatedWords)//batchSize, batchSize, maxWordLen)\n",
        "\n",
        "\n",
        "\n",
        "noisyAlphabetDataSet = torch.zeros([numberOfWords, maxWordLen], device=device).fill_(2)\n",
        "noisyAlphabetDataSet[:, 0] = 0\n",
        "noisyAlphabetDataSet[:, -1] = 1\n",
        "\n",
        "for i in range(numberOfWords):\n",
        "  for j, alphabet in enumerate(noisyWords[i]):\n",
        "    if alphabet not in word2id:\n",
        "      noisyAlphabetDataSet[i][j+1] = 3\n",
        "    else:\n",
        "      noisyAlphabetDataSet[i][j+1] = word2id[alphabet]\n",
        "\n",
        "noisyAlphabetDataSet = noisyAlphabetDataSet.reshape(len(alphabatedWords)//batchSize, batchSize, maxWordLen)"
      ],
      "metadata": {
        "id": "uTi5E5pV7uAX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def originalTraining():\n",
        "  for i in range(len(alphabatedWords)//batchSize):\n",
        "      data = alphabetDataSet[i, :, :].long()\n",
        "      src = data.requires_grad_(False).clone().detach()\n",
        "      tgt = data.requires_grad_(False).clone().detach()\n",
        "      yield Batch(src, tgt, 0)\n",
        "\n",
        "def noisyFineTuning():\n",
        "  for i in range(len(noisyAlphabetDataSet)//batchSize):\n",
        "      sData = noisyAlphabetDataSet[i, :, :].long()\n",
        "      tData = alphabetDataSet[i, :, :].long()\n",
        "      src = sData.requires_grad_(False).clone().detach()\n",
        "      tgt = tData.requires_grad_(False).clone().detach()\n",
        "      yield Batch(src, tgt, 0)"
      ],
      "metadata": {
        "id": "S3hO5PjrKp3_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the simple copy task.\n",
        "V = len(vocab)\n",
        "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
        "model = make_model(V, V, N=2)\n",
        "\n",
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(), lr=0.5, betas=(0.9, 0.98), eps=1e-9\n",
        ")\n",
        "lr_scheduler = LambdaLR(\n",
        "    optimizer=optimizer,\n",
        "    lr_lambda=lambda step: rate(\n",
        "        step, model_size=model.src_embed[0].d_model, factor=1.0, warmup=400\n",
        "    ),\n",
        ")\n",
        "\n",
        "model = model.to(device)\n",
        "\n",
        "#for epoch in range(10):\n",
        "#    model.train()\n",
        "#    run_epoch(\n",
        "#        originalTraining(),\n",
        "#        model,\n",
        "#        SimpleLossCompute(model.generator, criterion),\n",
        "#        optimizer,\n",
        "#        lr_scheduler,\n",
        "#        mode=\"train\",\n",
        "#        epochID=epoch+1\n",
        "#    )\n",
        "\n",
        "    \n",
        "for epoch in range(100):\n",
        "    model.train()\n",
        "    run_epoch(\n",
        "        noisyFineTuning(),\n",
        "        model,\n",
        "        SimpleLossCompute(model.generator, criterion),\n",
        "        optimizer,\n",
        "        lr_scheduler,\n",
        "        mode=\"train\",\n",
        "        epochID=epoch+1\n",
        "    )"
      ],
      "metadata": {
        "id": "wxp5mRP6Cr73",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ebc81cc6-e830-476a-cb9c-586af1f6af6b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch Step:      1 | Batch Number:      1 | Accumulation Step:   2 | Loss:   6.30 | Tokens / Sec:  9866.3 | Learning Rate: 5.5e-06\n",
            "Epoch Step:      2 | Batch Number:      1 | Accumulation Step:   2 | Loss:   4.03 | Tokens / Sec:  9329.4 | Learning Rate: 2.2e-05\n",
            "Epoch Step:      3 | Batch Number:      1 | Accumulation Step:   2 | Loss:   3.48 | Tokens / Sec:  9250.0 | Learning Rate: 3.9e-05\n",
            "Epoch Step:      4 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.32 | Tokens / Sec:  9914.2 | Learning Rate: 5.5e-05\n",
            "Epoch Step:      5 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.09 | Tokens / Sec: 10984.3 | Learning Rate: 7.2e-05\n",
            "Epoch Step:      6 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.82 | Tokens / Sec:  9701.6 | Learning Rate: 8.8e-05\n",
            "Epoch Step:      7 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.58 | Tokens / Sec:  9816.8 | Learning Rate: 1.0e-04\n",
            "Epoch Step:      8 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.47 | Tokens / Sec: 13742.6 | Learning Rate: 1.2e-04\n",
            "Epoch Step:      9 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.39 | Tokens / Sec: 11643.8 | Learning Rate: 1.4e-04\n",
            "Epoch Step:     10 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.28 | Tokens / Sec: 11616.0 | Learning Rate: 1.5e-04\n",
            "Epoch Step:     11 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.18 | Tokens / Sec: 11607.1 | Learning Rate: 1.7e-04\n",
            "Epoch Step:     12 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.11 | Tokens / Sec: 11485.8 | Learning Rate: 1.9e-04\n",
            "Epoch Step:     13 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.01 | Tokens / Sec: 11546.8 | Learning Rate: 2.0e-04\n",
            "Epoch Step:     14 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.91 | Tokens / Sec: 11704.5 | Learning Rate: 2.2e-04\n",
            "Epoch Step:     15 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.81 | Tokens / Sec: 12683.5 | Learning Rate: 2.4e-04\n",
            "Epoch Step:     16 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.74 | Tokens / Sec: 11343.4 | Learning Rate: 2.5e-04\n",
            "Epoch Step:     17 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.64 | Tokens / Sec: 11388.2 | Learning Rate: 2.7e-04\n",
            "Epoch Step:     18 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.56 | Tokens / Sec: 11659.0 | Learning Rate: 2.9e-04\n",
            "Epoch Step:     19 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.51 | Tokens / Sec: 11412.4 | Learning Rate: 3.0e-04\n",
            "Epoch Step:     20 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.47 | Tokens / Sec: 11574.4 | Learning Rate: 3.2e-04\n",
            "Epoch Step:     21 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.39 | Tokens / Sec: 11499.5 | Learning Rate: 3.4e-04\n",
            "Epoch Step:     22 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.35 | Tokens / Sec: 11526.4 | Learning Rate: 3.5e-04\n",
            "Epoch Step:     23 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.34 | Tokens / Sec: 12522.0 | Learning Rate: 3.7e-04\n",
            "Epoch Step:     24 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.25 | Tokens / Sec: 11707.7 | Learning Rate: 3.9e-04\n",
            "Epoch Step:     25 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.26 | Tokens / Sec: 11671.9 | Learning Rate: 4.0e-04\n",
            "Epoch Step:     26 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.20 | Tokens / Sec: 11563.5 | Learning Rate: 4.2e-04\n",
            "Epoch Step:     27 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.19 | Tokens / Sec: 11509.4 | Learning Rate: 4.4e-04\n",
            "Epoch Step:     28 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.10 | Tokens / Sec: 11362.0 | Learning Rate: 4.5e-04\n",
            "Epoch Step:     29 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.12 | Tokens / Sec: 11750.2 | Learning Rate: 4.7e-04\n",
            "Epoch Step:     30 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.10 | Tokens / Sec: 11532.5 | Learning Rate: 4.9e-04\n",
            "Epoch Step:     31 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.11 | Tokens / Sec: 11400.9 | Learning Rate: 5.0e-04\n",
            "Epoch Step:     32 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 11566.7 | Learning Rate: 5.2e-04\n",
            "Epoch Step:     33 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11738.9 | Learning Rate: 5.4e-04\n",
            "Epoch Step:     34 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11621.1 | Learning Rate: 5.5e-04\n",
            "Epoch Step:     35 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11567.0 | Learning Rate: 5.7e-04\n",
            "Epoch Step:     36 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11564.7 | Learning Rate: 5.9e-04\n",
            "Epoch Step:     37 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11784.0 | Learning Rate: 6.0e-04\n",
            "Epoch Step:     38 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11433.7 | Learning Rate: 6.2e-04\n",
            "Epoch Step:     39 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.40 | Tokens / Sec: 11719.8 | Learning Rate: 6.4e-04\n",
            "Epoch Step:     40 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.11 | Tokens / Sec: 11761.3 | Learning Rate: 6.5e-04\n",
            "Epoch Step:     41 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11395.8 | Learning Rate: 6.7e-04\n",
            "Epoch Step:     42 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.11 | Tokens / Sec: 10963.0 | Learning Rate: 6.9e-04\n",
            "Epoch Step:     43 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11726.0 | Learning Rate: 7.0e-04\n",
            "Epoch Step:     44 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11578.2 | Learning Rate: 7.2e-04\n",
            "Epoch Step:     45 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11617.3 | Learning Rate: 7.3e-04\n",
            "Epoch Step:     46 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11711.7 | Learning Rate: 7.5e-04\n",
            "Epoch Step:     47 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11349.0 | Learning Rate: 7.7e-04\n",
            "Epoch Step:     48 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11618.3 | Learning Rate: 7.8e-04\n",
            "Epoch Step:     49 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11524.0 | Learning Rate: 8.0e-04\n",
            "Epoch Step:     50 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11259.0 | Learning Rate: 8.2e-04\n",
            "Epoch Step:     51 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11469.0 | Learning Rate: 8.3e-04\n",
            "Epoch Step:     52 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 12086.2 | Learning Rate: 8.5e-04\n",
            "Epoch Step:     53 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 12041.3 | Learning Rate: 8.7e-04\n",
            "Epoch Step:     54 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11619.4 | Learning Rate: 8.8e-04\n",
            "Epoch Step:     55 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11631.8 | Learning Rate: 9.0e-04\n",
            "Epoch Step:     56 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11612.3 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     57 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11547.4 | Learning Rate: 9.3e-04\n",
            "Epoch Step:     58 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11998.7 | Learning Rate: 9.5e-04\n",
            "Epoch Step:     59 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 11818.6 | Learning Rate: 9.7e-04\n",
            "Epoch Step:     60 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 11581.3 | Learning Rate: 9.8e-04\n",
            "Epoch Step:     61 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11660.4 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     62 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11660.6 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     63 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11512.6 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     64 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11480.3 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     65 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11622.1 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     66 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11585.0 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     67 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11568.0 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     68 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 11623.7 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     69 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 11733.8 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     70 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.10 | Tokens / Sec: 11424.4 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     71 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 11131.6 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     72 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.11 | Tokens / Sec: 11697.5 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     73 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.11 | Tokens / Sec: 11554.5 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     74 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11665.6 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     75 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11509.2 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     76 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11616.6 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     77 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11534.1 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     78 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 11588.8 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     79 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 11908.9 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     80 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11798.7 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     81 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 12079.3 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     82 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11713.7 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     83 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 11349.5 | Learning Rate: 9.9e-04\n",
            "Epoch Step:     84 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11655.6 | Learning Rate: 9.9e-04\n",
            "Epoch Step:     85 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 11720.0 | Learning Rate: 9.8e-04\n",
            "Epoch Step:     86 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11586.8 | Learning Rate: 9.8e-04\n",
            "Epoch Step:     87 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 11656.2 | Learning Rate: 9.7e-04\n",
            "Epoch Step:     88 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11464.8 | Learning Rate: 9.7e-04\n",
            "Epoch Step:     89 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11766.3 | Learning Rate: 9.6e-04\n",
            "Epoch Step:     90 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 11939.3 | Learning Rate: 9.5e-04\n",
            "Epoch Step:     91 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 11717.9 | Learning Rate: 9.5e-04\n",
            "Epoch Step:     92 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 11611.2 | Learning Rate: 9.4e-04\n",
            "Epoch Step:     93 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 11566.1 | Learning Rate: 9.4e-04\n",
            "Epoch Step:     94 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 11282.3 | Learning Rate: 9.3e-04\n",
            "Epoch Step:     95 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 11818.5 | Learning Rate: 9.3e-04\n",
            "Epoch Step:     96 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11498.3 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     97 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.07 | Tokens / Sec: 11703.1 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     98 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 11702.5 | Learning Rate: 9.1e-04\n",
            "Epoch Step:     99 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 11663.0 | Learning Rate: 9.1e-04\n",
            "Epoch Step:    100 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 10998.3 | Learning Rate: 9.1e-04\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.eval()\n",
        "max_len = 20\n",
        "src_mask = torch.ones(1, 1, max_len).to(device)\n",
        "TP = 0\n",
        "\n",
        "total = len(noisyAlphabetDataSet)\n",
        "correctlyPredicted = []\n",
        "wronglyPredicted = []\n",
        "\n",
        "for i in range(total):\n",
        "  src = torch.tensor(noisyAlphabetDataSet[i])[1].long().view(1, -1)\n",
        "\n",
        "  predicted = greedy_decode(model, src, src_mask, max_len=max_len, start_symbol=0)\n",
        "  original = torch.tensor(alphabetDataSet[i])[1].long().view(1, -1)\n",
        "\n",
        "  if torch.all(original == predicted):\n",
        "    correctlyPredicted.append(predicted)\n",
        "  else:\n",
        "    wronglyPredicted.append(predicted)\n",
        "  \n",
        "\n",
        "\n",
        "  TP += torch.all(original == predicted)\n",
        "\n",
        "print(TP/total)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_UP6fsi3454C",
        "outputId": "1d7b11dc-e8cd-4550-c7dd-eb2f4f2a9252"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(0.0364, device='cuda:0')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "wordsbyID = list(word2id.keys())\n",
        "for i in correctlyPredicted:\n",
        "  word = []\n",
        "  for j in i[0]:\n",
        "    if j == 0:\n",
        "      continue\n",
        "    if j == 2:\n",
        "      break\n",
        "    word.extend([wordsbyID[j]])\n",
        "  print(''.join(word))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q7iS7Zp1btxx",
        "outputId": "629cf8ba-0c4f-47a8-d5a6-fd6882ae6d36"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ideology\n",
            "laying\n",
            "anchors\n",
            "1896\n",
            "singing\n",
            "bob\n",
            "cashew\n",
            "beating\n",
            "railing\n",
            "ingenious\n",
            "thing\n",
            "teamed\n",
            "imaginative\n",
            "prophet\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Problem 2.3** *(3 points)* \n",
        ">Extend this word-level spelling correction model to sentence-level. You can assume that the number of characters of each sentence is 100 or less. You do not have to report accuracy, but find one example where the word-level model fails and sentence-level model correctly predicts."
      ],
      "metadata": {
        "id": "AQohpPVDZetk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "maxSenLen = 100\n",
        "batchSize = 64"
      ],
      "metadata": {
        "id": "me5W9gCX7K0m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tokenizers\n",
        "!pip install -q datasets"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MDp8nBCW4EuK",
        "outputId": "a679bb8f-a632-4b28-833b-f787e821b762"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tokenizers\n",
            "  Downloading tokenizers-0.12.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.6 MB 4.1 MB/s \n",
            "\u001b[?25hInstalling collected packages: tokenizers\n",
            "Successfully installed tokenizers-0.12.1\n",
            "\u001b[K     |████████████████████████████████| 342 kB 4.1 MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 45.3 MB/s \n",
            "\u001b[K     |████████████████████████████████| 212 kB 49.3 MB/s \n",
            "\u001b[K     |████████████████████████████████| 136 kB 51.7 MB/s \n",
            "\u001b[K     |████████████████████████████████| 84 kB 3.4 MB/s \n",
            "\u001b[K     |████████████████████████████████| 127 kB 48.4 MB/s \n",
            "\u001b[K     |████████████████████████████████| 94 kB 3.2 MB/s \n",
            "\u001b[K     |████████████████████████████████| 271 kB 51.5 MB/s \n",
            "\u001b[K     |████████████████████████████████| 144 kB 50.4 MB/s \n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "import torchtext\n",
        "from torchtext.data import get_tokenizer\n",
        "from tokenizers import normalizers\n",
        "\n",
        "normalize = normalizers.BertNormalizer().normalize_str\n",
        "squad_dataset = load_dataset('squad')\n",
        "dataSet = squad_dataset['validation']\n",
        "\n",
        "string = ' '.join([str(elem).strip('\\n') for elem in dataSet['context']])\n",
        "\n",
        "\n",
        "\n",
        "sentences = normalize(string).split('.')\n",
        "sentences = [i for i in sentences if len(i) > 10 and len(i) <= 98]\n",
        "\n",
        "alphabatedSentences = [list(i) for i in sentences]\n",
        "\n",
        "vocab = ['SOW', 'EOW', 'PAD', 'UNK'] + list(set(''.join([str(elem).strip('\\n') for elem in sentences])))\n",
        "word2id = {word: id_ for id_, word in enumerate(vocab)}\n",
        "\n",
        "noisySentences = []\n",
        "noiseAmount = 1\n",
        "for i in alphabatedSentences:\n",
        "  for _ in range(noiseAmount):\n",
        "    replaceID = torch.randint(1, len(i), [1])[0]\n",
        "    replaingID = torch.randint(4, len(vocab), [1])[0]\n",
        "    temp = i.copy()\n",
        "    temp[replaceID] = vocab[replaingID]\n",
        "  noisySentences.append(temp)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296,
          "referenced_widgets": [
            "ff71ddd10743434996fb13a8b9677a18",
            "c3a71ee4bde04c50a0d5cf2ae1fff363",
            "8278b27cbd354d22a37701e4d6023307",
            "9489adc142dc4ad783240d8c94a73222",
            "21641a620e714aaaab2ed31cf142ed17",
            "88a75ee4e10b40f1a92fb5cd9a9cbdea",
            "be399348f68b423a99bd826e0f6084c6",
            "642a79519bbc49989b4341262ce092e0",
            "d982cab25b814a549e58f054f1a62419",
            "f843310cad7f47a69bcb57ee446e6f0c",
            "2468cd3cd35f4a7fa0ab033b51fc6050",
            "2096a94ea9c94aa38c88212467544283",
            "ffbe472b7f2341f787d91ddf8c83e116",
            "af8a5bf634214746b5a751dbdd2c0e3e",
            "bddefd5c146b494d99a53127cb730d73",
            "c873724f461549f3a478cccf1166fc14",
            "37588b53bba649efb365e8a96e5c570a",
            "3b5d374900c34ff2a94d015b64d5be20",
            "08f6e271c19846148925c82edb5165b0",
            "b35c676d4ffa4aa981467b8648c96a7d",
            "a2ade48952b740a8a9eea0aadb074cc8",
            "c2a752cd2dd84aadad0c4c1d5b783300",
            "ddb9a123cdaf4b8e903bbe78287d9770",
            "706912ebfea84f588d3dd5371bb96b51",
            "182079f7e57b45c2a9b00ffe4952f501",
            "edbe462be8124949935cb7b7788bdbf8",
            "3c5d968bb5e54084bb4d175943dc7a9a",
            "bda8369113874827a0d98309f5cceee6",
            "d223fd25fb054844880e3771dae6c464",
            "93227e2bb3814a2585cd2e4123e2ac2a",
            "e8ae324a04af468196faf0c4938b5afe",
            "a298570d5b284a74b229a4c1aa29f750",
            "65c6b2b6683749919589c83b05ec099e",
            "d4cfd77d93794a7e824f77791eae77fb",
            "9bb8e839f5fa4cf1b7e5b22fb3ef2ff4",
            "329cf6925aa14c2a947dacde130ee4c4",
            "4971ce65fd6346ff9262ac2f62a1411e",
            "ca96bf0a918c43f782a8471978464994",
            "be1a2aee094b40b395d752d886df52c0",
            "7dc55d8cd9094e7288e43aac9ef53f71",
            "2d694a5560684f3aae134bc2b67f2fda",
            "0518797c416a425da61416fe09960b2c",
            "46f95355e57448709198f0a8802d4972",
            "e69411bdd38045feaab7dd381c6e62d2",
            "547c86dc57b549399bfefbf50df1a098",
            "23a5d51abc69425ebb1b648a121a4857",
            "e5b78781c71f4549b6230ce22a171d85",
            "4d9942b4c4ef41f396e6216c2da3de88",
            "41a231e839354c939f98b29950d2cf9b",
            "887a5aafd1624c0ba7d98f1ee6633d4c",
            "ee0875b69ead4aacacb8d332a734a148",
            "aabeea230e174b1987f28f0aafb01c6f",
            "d7b70a89374b46c28066b764c5d2d395",
            "7a59f8fe9c3c4279b62bd989a9fdec18",
            "ee020deea53a4adba31065f32a290d37",
            "60008e73a66e4cc7a594e4601e71e5da",
            "b8fda90b244642a0a690c946d06fb09a",
            "3c69aa1607c74a389bca4f4dc0c75485",
            "22175274de004ca888b042243c675017",
            "fc4a52f89a0d4a6693c36e00ae6e8251",
            "8dae1cb7046c49af9d2a42643313a721",
            "118eb58e0d184dadb6d582f65f07f68d",
            "ae3c2bbd7c974904925a1897018a9f6e",
            "b93c1c2e186d4589ae237f1da02239a3",
            "550f5fb0bfbd488d9f0271b74abd63d6",
            "0fa2a256983242649b0962d50e5bb001",
            "14c28af50391410baee55177373c18c2",
            "3583279816fd44a19b0ad4c0bda2f72a",
            "b90a991aa3dd4f7bb7606a1722c7de2b",
            "651fc1a59de84ccdb9c7af06e7cd6a3e",
            "4bc12c00da0e4190ad06841884ceaf4f",
            "dc1b8851f6a7459db141a54356cd2ee0",
            "81774770d14a465b8f0e8f91e7771a79",
            "d5592d2afbac4b358b005f1b27426799",
            "237fa7839e584dba9931e88926318af8",
            "733a2f07439544239ca8931b753e632e",
            "44d0fcc7a5934d20b0dd4035a2dc2ba0",
            "a5e0818eb54c4958983b0fe85c9feb39",
            "1530c4615501412fa02918fb7f63ff2a",
            "70d7b75f333a4d5995baef95659a32ac",
            "a702df381c2147e18a49fa6b62ae1966",
            "f413a767f48842548c227888734457c5",
            "fe8264020f364d86b6c9b86152145bbd",
            "7326255bdf204b7a989552e26aa539d7",
            "f7be8d4f66fb42e3812d7bbf41ffb86f",
            "b1dd9dd9ecf7472f84f342ebb4f513ba",
            "e2caa6406bd041fbb4dfcb9b0ab127fe",
            "b6a1ecac13e749a3ab5b1699aa0cbd57",
            "395860a06482436c8ce971f76998c12e",
            "550774b7251941d9aafc23e64726c8fe",
            "a8aa36aa80874fa0b0898b53b36639c7",
            "b450cf4492c24591a1261a1c6d1a7556",
            "6edbf331c6554ac883f9110aff31b583",
            "b52daff01c3d4c72a2c8dd3d54a052fa",
            "70d2357f5e7049788d1715ef8c2a8110",
            "6a0d992ca00c40d9972d6f0b0e46e4ac",
            "75e2f4342f2c40d384091e720ca6c76d",
            "1a0ca7744e064ea7a961903408288453",
            "ccfdafb04cda4aca8beaa3115e17e587"
          ]
        },
        "id": "wFKCNPQnBqWk",
        "outputId": "50979221-d4c7-44ab-b0d9-64a8a21fd2a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/1.97k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ff71ddd10743434996fb13a8b9677a18"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/1.02k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2096a94ea9c94aa38c88212467544283"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset squad/plain_text (download: 33.51 MiB, generated: 85.63 MiB, post-processed: Unknown size, total: 119.14 MiB) to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ddb9a123cdaf4b8e903bbe78287d9770"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/8.12M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d4cfd77d93794a7e824f77791eae77fb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/1.05M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "547c86dc57b549399bfefbf50df1a098"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Extracting data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "60008e73a66e4cc7a594e4601e71e5da"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/87599 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "14c28af50391410baee55177373c18c2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating validation split:   0%|          | 0/10570 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a5e0818eb54c4958983b0fe85c9feb39"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset squad downloaded and prepared to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "395860a06482436c8ce971f76998c12e"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "numberOfWords = (len(alphabatedSentences)//batchSize)*batchSize\n",
        "alphabetDataSet = torch.zeros([numberOfWords, maxSenLen], device=device).fill_(2)\n",
        "alphabetDataSet[:, 0] = 0\n",
        "alphabetDataSet[:, -1] = 1\n",
        "\n",
        "for i in range(numberOfWords):\n",
        "  for j, alphabet in enumerate(alphabatedSentences[i]):\n",
        "    if alphabet not in word2id:\n",
        "      alphabetDataSet[i][j+1] = 3\n",
        "    else:\n",
        "      alphabetDataSet[i][j+1] = word2id[alphabet]\n",
        "\n",
        "alphabetDataSet = alphabetDataSet.reshape(len(alphabatedSentences)//batchSize, batchSize, maxSenLen)\n",
        "\n",
        "\n",
        "noisyAlphabetDataSet = torch.zeros([numberOfWords, maxSenLen], device=device).fill_(2)\n",
        "noisyAlphabetDataSet[:, 0] = 0\n",
        "noisyAlphabetDataSet[:, -1] = 1\n",
        "\n",
        "for i in range(numberOfWords):\n",
        "  for j, alphabet in enumerate(noisySentences[i]):\n",
        "    if alphabet not in word2id:\n",
        "      noisyAlphabetDataSet[i][j+1] = 3\n",
        "    else:\n",
        "      noisyAlphabetDataSet[i][j+1] = word2id[alphabet]\n",
        "\n",
        "noisyAlphabetDataSet = noisyAlphabetDataSet.reshape(len(alphabatedSentences)//batchSize, batchSize, maxSenLen)"
      ],
      "metadata": {
        "id": "Dz3weBrE7CuO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def originalTraining():\n",
        "  for i in range(len(alphabatedWords)//batchSize):\n",
        "      data = alphabetDataSet[i, :, :].long()\n",
        "      src = data.requires_grad_(False).clone().detach()\n",
        "      tgt = data.requires_grad_(False).clone().detach()\n",
        "      yield Batch(src, tgt, 0)\n",
        "\n",
        "def noisyFineTuning():\n",
        "  for i in range(len(noisyAlphabetDataSet)//batchSize):\n",
        "      sData = noisyAlphabetDataSet[i, :, :].long()\n",
        "      tData = alphabetDataSet[i, :, :].long()\n",
        "      src = sData.requires_grad_(False).clone().detach()\n",
        "      tgt = tData.requires_grad_(False).clone().detach()\n",
        "      yield Batch(src, tgt, 0)"
      ],
      "metadata": {
        "id": "6IeItcYw8k0F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the simple copy task.\n",
        "V = len(vocab)\n",
        "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
        "model = make_model(V, V, N=2)\n",
        "\n",
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(), lr=0.5, betas=(0.9, 0.98), eps=1e-9\n",
        ")\n",
        "lr_scheduler = LambdaLR(\n",
        "    optimizer=optimizer,\n",
        "    lr_lambda=lambda step: rate(\n",
        "        step, model_size=model.src_embed[0].d_model, factor=1.0, warmup=400\n",
        "    ),\n",
        ")\n",
        "\n",
        "model = model.to(device)\n",
        "\n",
        "#for epoch in range(10):\n",
        "#    model.train()\n",
        "#    run_epoch(\n",
        "#        originalTraining(),\n",
        "#        model,\n",
        "#        SimpleLossCompute(model.generator, criterion),\n",
        "#        optimizer,\n",
        "#        lr_scheduler,\n",
        "#        mode=\"train\",\n",
        "#        epochID=epoch+1\n",
        "#    )\n",
        "\n",
        "    \n",
        "for epoch in range(100):\n",
        "    model.train()\n",
        "    run_epoch(\n",
        "        noisyFineTuning(),\n",
        "        model,\n",
        "        SimpleLossCompute(model.generator, criterion),\n",
        "        optimizer,\n",
        "        lr_scheduler,\n",
        "        mode=\"train\",\n",
        "        epochID=epoch+1\n",
        "    )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a5wYLXtN8q4y",
        "outputId": "5777b7d0-8b6d-45cd-a5ac-573da53a5a1b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch Step:      1 | Batch Number:      1 | Accumulation Step:   2 | Loss:   6.63 | Tokens / Sec: 11488.1 | Learning Rate: 5.5e-06\n",
            "Epoch Step:      2 | Batch Number:      1 | Accumulation Step:   2 | Loss:   5.40 | Tokens / Sec: 13354.5 | Learning Rate: 1.7e-05\n",
            "Epoch Step:      3 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.69 | Tokens / Sec: 13287.6 | Learning Rate: 2.8e-05\n",
            "Epoch Step:      4 | Batch Number:      1 | Accumulation Step:   2 | Loss:   2.31 | Tokens / Sec: 13422.5 | Learning Rate: 3.9e-05\n",
            "Epoch Step:      5 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.94 | Tokens / Sec: 13418.8 | Learning Rate: 5.0e-05\n",
            "Epoch Step:      6 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.64 | Tokens / Sec: 13444.2 | Learning Rate: 6.1e-05\n",
            "Epoch Step:      7 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.55 | Tokens / Sec: 13260.7 | Learning Rate: 7.2e-05\n",
            "Epoch Step:      8 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.44 | Tokens / Sec: 13465.5 | Learning Rate: 8.3e-05\n",
            "Epoch Step:      9 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.31 | Tokens / Sec: 13357.7 | Learning Rate: 9.4e-05\n",
            "Epoch Step:     10 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.23 | Tokens / Sec: 13476.6 | Learning Rate: 1.0e-04\n",
            "Epoch Step:     11 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.17 | Tokens / Sec: 13488.5 | Learning Rate: 1.2e-04\n",
            "Epoch Step:     12 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.11 | Tokens / Sec: 13381.9 | Learning Rate: 1.3e-04\n",
            "Epoch Step:     13 | Batch Number:      1 | Accumulation Step:   2 | Loss:   1.05 | Tokens / Sec: 13457.5 | Learning Rate: 1.4e-04\n",
            "Epoch Step:     14 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.99 | Tokens / Sec: 13534.6 | Learning Rate: 1.5e-04\n",
            "Epoch Step:     15 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.94 | Tokens / Sec: 13509.4 | Learning Rate: 1.6e-04\n",
            "Epoch Step:     16 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.88 | Tokens / Sec: 13477.7 | Learning Rate: 1.7e-04\n",
            "Epoch Step:     17 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.81 | Tokens / Sec: 13437.3 | Learning Rate: 1.8e-04\n",
            "Epoch Step:     18 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.76 | Tokens / Sec: 13355.8 | Learning Rate: 1.9e-04\n",
            "Epoch Step:     19 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.69 | Tokens / Sec: 13485.6 | Learning Rate: 2.0e-04\n",
            "Epoch Step:     20 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.63 | Tokens / Sec: 13467.1 | Learning Rate: 2.2e-04\n",
            "Epoch Step:     21 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.56 | Tokens / Sec: 13415.8 | Learning Rate: 2.3e-04\n",
            "Epoch Step:     22 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.50 | Tokens / Sec: 13604.9 | Learning Rate: 2.4e-04\n",
            "Epoch Step:     23 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.43 | Tokens / Sec: 13551.1 | Learning Rate: 2.5e-04\n",
            "Epoch Step:     24 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.38 | Tokens / Sec: 13531.8 | Learning Rate: 2.6e-04\n",
            "Epoch Step:     25 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.33 | Tokens / Sec: 13479.0 | Learning Rate: 2.7e-04\n",
            "Epoch Step:     26 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.27 | Tokens / Sec: 13444.3 | Learning Rate: 2.8e-04\n",
            "Epoch Step:     27 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.28 | Tokens / Sec: 13506.9 | Learning Rate: 2.9e-04\n",
            "Epoch Step:     28 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.27 | Tokens / Sec: 13615.8 | Learning Rate: 3.0e-04\n",
            "Epoch Step:     29 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.35 | Tokens / Sec: 13631.1 | Learning Rate: 3.1e-04\n",
            "Epoch Step:     30 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.27 | Tokens / Sec: 13601.7 | Learning Rate: 3.3e-04\n",
            "Epoch Step:     31 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.20 | Tokens / Sec: 13431.6 | Learning Rate: 3.4e-04\n",
            "Epoch Step:     32 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.24 | Tokens / Sec: 13514.9 | Learning Rate: 3.5e-04\n",
            "Epoch Step:     33 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.16 | Tokens / Sec: 13619.5 | Learning Rate: 3.6e-04\n",
            "Epoch Step:     34 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.14 | Tokens / Sec: 13554.4 | Learning Rate: 3.7e-04\n",
            "Epoch Step:     35 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.13 | Tokens / Sec: 13422.7 | Learning Rate: 3.8e-04\n",
            "Epoch Step:     36 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.10 | Tokens / Sec: 13523.2 | Learning Rate: 3.9e-04\n",
            "Epoch Step:     37 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 13683.4 | Learning Rate: 4.0e-04\n",
            "Epoch Step:     38 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 13655.2 | Learning Rate: 4.1e-04\n",
            "Epoch Step:     39 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 13474.5 | Learning Rate: 4.3e-04\n",
            "Epoch Step:     40 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13519.4 | Learning Rate: 4.4e-04\n",
            "Epoch Step:     41 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13493.8 | Learning Rate: 4.5e-04\n",
            "Epoch Step:     42 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13401.6 | Learning Rate: 4.6e-04\n",
            "Epoch Step:     43 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13521.8 | Learning Rate: 4.7e-04\n",
            "Epoch Step:     44 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13499.4 | Learning Rate: 4.8e-04\n",
            "Epoch Step:     45 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13431.4 | Learning Rate: 4.9e-04\n",
            "Epoch Step:     46 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13393.6 | Learning Rate: 5.0e-04\n",
            "Epoch Step:     47 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13431.9 | Learning Rate: 5.1e-04\n",
            "Epoch Step:     48 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 13632.6 | Learning Rate: 5.2e-04\n",
            "Epoch Step:     49 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 13464.2 | Learning Rate: 5.4e-04\n",
            "Epoch Step:     50 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13601.2 | Learning Rate: 5.5e-04\n",
            "Epoch Step:     51 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13521.6 | Learning Rate: 5.6e-04\n",
            "Epoch Step:     52 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13393.1 | Learning Rate: 5.7e-04\n",
            "Epoch Step:     53 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13465.5 | Learning Rate: 5.8e-04\n",
            "Epoch Step:     54 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13557.0 | Learning Rate: 5.9e-04\n",
            "Epoch Step:     55 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13485.3 | Learning Rate: 6.0e-04\n",
            "Epoch Step:     56 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13429.7 | Learning Rate: 6.1e-04\n",
            "Epoch Step:     57 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13563.6 | Learning Rate: 6.2e-04\n",
            "Epoch Step:     58 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 13587.6 | Learning Rate: 6.4e-04\n",
            "Epoch Step:     59 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 13463.2 | Learning Rate: 6.5e-04\n",
            "Epoch Step:     60 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.18 | Tokens / Sec: 13589.3 | Learning Rate: 6.6e-04\n",
            "Epoch Step:     61 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 13480.5 | Learning Rate: 6.7e-04\n",
            "Epoch Step:     62 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.05 | Tokens / Sec: 13495.4 | Learning Rate: 6.8e-04\n",
            "Epoch Step:     63 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.04 | Tokens / Sec: 13591.5 | Learning Rate: 6.9e-04\n",
            "Epoch Step:     64 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13464.8 | Learning Rate: 7.0e-04\n",
            "Epoch Step:     65 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13602.7 | Learning Rate: 7.1e-04\n",
            "Epoch Step:     66 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13500.9 | Learning Rate: 7.2e-04\n",
            "Epoch Step:     67 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13457.4 | Learning Rate: 7.3e-04\n",
            "Epoch Step:     68 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13461.8 | Learning Rate: 7.5e-04\n",
            "Epoch Step:     69 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13628.2 | Learning Rate: 7.6e-04\n",
            "Epoch Step:     70 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13574.5 | Learning Rate: 7.7e-04\n",
            "Epoch Step:     71 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13513.1 | Learning Rate: 7.8e-04\n",
            "Epoch Step:     72 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13562.7 | Learning Rate: 7.9e-04\n",
            "Epoch Step:     73 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13459.9 | Learning Rate: 8.0e-04\n",
            "Epoch Step:     74 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13481.0 | Learning Rate: 8.1e-04\n",
            "Epoch Step:     75 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.09 | Tokens / Sec: 13618.7 | Learning Rate: 8.2e-04\n",
            "Epoch Step:     76 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13547.5 | Learning Rate: 8.3e-04\n",
            "Epoch Step:     77 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 13608.9 | Learning Rate: 8.5e-04\n",
            "Epoch Step:     78 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.14 | Tokens / Sec: 13517.8 | Learning Rate: 8.6e-04\n",
            "Epoch Step:     79 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.12 | Tokens / Sec: 13457.6 | Learning Rate: 8.7e-04\n",
            "Epoch Step:     80 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.15 | Tokens / Sec: 13514.5 | Learning Rate: 8.8e-04\n",
            "Epoch Step:     81 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.19 | Tokens / Sec: 13707.1 | Learning Rate: 8.9e-04\n",
            "Epoch Step:     82 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.12 | Tokens / Sec: 13508.6 | Learning Rate: 9.0e-04\n",
            "Epoch Step:     83 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.08 | Tokens / Sec: 13554.0 | Learning Rate: 9.1e-04\n",
            "Epoch Step:     84 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.06 | Tokens / Sec: 13493.2 | Learning Rate: 9.2e-04\n",
            "Epoch Step:     85 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13495.9 | Learning Rate: 9.3e-04\n",
            "Epoch Step:     86 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.02 | Tokens / Sec: 13521.5 | Learning Rate: 9.4e-04\n",
            "Epoch Step:     87 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13613.2 | Learning Rate: 9.6e-04\n",
            "Epoch Step:     88 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13614.0 | Learning Rate: 9.7e-04\n",
            "Epoch Step:     89 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13624.7 | Learning Rate: 9.8e-04\n",
            "Epoch Step:     90 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13640.9 | Learning Rate: 9.9e-04\n",
            "Epoch Step:     91 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.01 | Tokens / Sec: 13596.9 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     92 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13563.9 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     93 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13448.0 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     94 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13677.8 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     95 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13526.4 | Learning Rate: 1.0e-03\n",
            "Epoch Step:     96 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13414.5 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     97 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13620.5 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     98 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13503.7 | Learning Rate: 1.1e-03\n",
            "Epoch Step:     99 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.00 | Tokens / Sec: 13469.6 | Learning Rate: 1.1e-03\n",
            "Epoch Step:    100 | Batch Number:      1 | Accumulation Step:   2 | Loss:   0.03 | Tokens / Sec: 13587.0 | Learning Rate: 1.1e-03\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_len = 100\n",
        "\n",
        "model.eval()\n",
        "src_mask = torch.ones(1, 1, max_len).to(device)\n",
        "TP = 0\n",
        "\n",
        "total = len(noisyAlphabetDataSet)\n",
        "correctlyPredicted = []\n",
        "\n",
        "for i in range(total):\n",
        "  src = torch.tensor(noisyAlphabetDataSet[i])[1].long().view(1, -1)\n",
        "  src = src\n",
        "\n",
        "  predicted = greedy_decode(model, src, src_mask, max_len=max_len, start_symbol=0)\n",
        "  original = torch.tensor(alphabetDataSet[i])[1].long().view(1, -1)\n",
        "\n",
        "  if torch.all(original == predicted):\n",
        "    correctlyPredicted.append(predicted)\n",
        "  \n",
        "\n",
        "\n",
        "  TP += torch.all(original == predicted)\n",
        "\n",
        "print(TP/total)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JSFNfKit8zNe",
        "outputId": "e0e6f68e-f6fb-4aa3-b92e-bfa05d2ffd14"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(0.0075, device='cuda:0')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "wordsbyID = list(word2id.keys())\n",
        "for i in correctlyPredicted:\n",
        "  word = []\n",
        "  for j in i[0]:\n",
        "    if j == 0:\n",
        "      continue\n",
        "    if j == 2:\n",
        "      break\n",
        "    word.extend([wordsbyID[j]])\n",
        "  print(''.join(word))\n",
        "  "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iqq4Eu7uf4xT",
        "outputId": "0a72c3ac-ce80-4636-88b1-646087b9bca3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " cbs broadcast super bowl 50 in the u\n",
            " the miami bid depended on whether the stadium underwent renovations\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "None of the word here is found in word word level correction, hence this model is solving atleast one example which is not solveable by uppar model"
      ],
      "metadata": {
        "id": "XCQwni2g4OQJ"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "oDaIrTN1FtSb"
      ],
      "name": "KAIST AI605 Assignment 3.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "37a1c3f646f34a69a351789eb5058154": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f6a4cf97a61c4589a16dd2484690e042",
              "IPY_MODEL_f40f0e17a44d46d8aea607479fe19ca9",
              "IPY_MODEL_de809c06234742af8cb09a122abc7793"
            ],
            "layout": "IPY_MODEL_d9a132afc1d64534bd8a2af52ed68ebc"
          }
        },
        "f6a4cf97a61c4589a16dd2484690e042": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e2ad502ac00f42478d8ec91292477c0b",
            "placeholder": "​",
            "style": "IPY_MODEL_bd5a726b8115452ebdc95ec055156709",
            "value": "Downloading builder script: "
          }
        },
        "f40f0e17a44d46d8aea607479fe19ca9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_be0372fe2e354da683de2fc44cbdf53a",
            "max": 1967,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_011e721dd0474dadb22d4f2542702565",
            "value": 1967
          }
        },
        "de809c06234742af8cb09a122abc7793": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7a0b6ff42a634d8385547a90b67d8478",
            "placeholder": "​",
            "style": "IPY_MODEL_2150053d072540f498852f02c8ad2ce0",
            "value": " 5.27k/? [00:00&lt;00:00, 129kB/s]"
          }
        },
        "d9a132afc1d64534bd8a2af52ed68ebc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2ad502ac00f42478d8ec91292477c0b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd5a726b8115452ebdc95ec055156709": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "be0372fe2e354da683de2fc44cbdf53a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "011e721dd0474dadb22d4f2542702565": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7a0b6ff42a634d8385547a90b67d8478": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2150053d072540f498852f02c8ad2ce0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "90e52321f2f042739987e5e4689b9584": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3631fc5e601f4f949926142280103f7b",
              "IPY_MODEL_8bb5f3908e25416d92e60b760c09a985",
              "IPY_MODEL_cb7c358c324a4f48be877dd8f1a366d4"
            ],
            "layout": "IPY_MODEL_7fda162eb92b4ec8af5858ebaaa0b530"
          }
        },
        "3631fc5e601f4f949926142280103f7b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_85d42260600646c29f651906d07b0614",
            "placeholder": "​",
            "style": "IPY_MODEL_56c12d049c674101be93d2d58e365194",
            "value": "Downloading metadata: "
          }
        },
        "8bb5f3908e25416d92e60b760c09a985": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_145c60d16a6d43c397a5aec8bbbb88b1",
            "max": 1021,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e341996a7fa84bf998cd724691e8f4a1",
            "value": 1021
          }
        },
        "cb7c358c324a4f48be877dd8f1a366d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_18d173973a9a471eb4214c0bcaa05f67",
            "placeholder": "​",
            "style": "IPY_MODEL_cfe4aae8f0cf4cccbe9a3381838e6737",
            "value": " 2.36k/? [00:00&lt;00:00, 54.3kB/s]"
          }
        },
        "7fda162eb92b4ec8af5858ebaaa0b530": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85d42260600646c29f651906d07b0614": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "56c12d049c674101be93d2d58e365194": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "145c60d16a6d43c397a5aec8bbbb88b1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e341996a7fa84bf998cd724691e8f4a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "18d173973a9a471eb4214c0bcaa05f67": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cfe4aae8f0cf4cccbe9a3381838e6737": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1ed385fe271c426aa3ebe78fabd818fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_97cc0d91c6aa4214992de1e9a4422cb9",
              "IPY_MODEL_8f38e1ba93b844cdab487fe952a50e91",
              "IPY_MODEL_dc979228c3cf42229c921f9ebb544051"
            ],
            "layout": "IPY_MODEL_312266c9c3e14c50a02ff96b7c71ceaf"
          }
        },
        "97cc0d91c6aa4214992de1e9a4422cb9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0e04f3664880435198f1a50fa8c1ee92",
            "placeholder": "​",
            "style": "IPY_MODEL_1003465b490a46a5a4a3a74993d93ed5",
            "value": "Downloading data files: 100%"
          }
        },
        "8f38e1ba93b844cdab487fe952a50e91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a2a23cbdb65c46d6baf033d87e688622",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5f784fc250d04dec93928b2c23b98530",
            "value": 2
          }
        },
        "dc979228c3cf42229c921f9ebb544051": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0be2c167de3242199af03b341cb5e87e",
            "placeholder": "​",
            "style": "IPY_MODEL_aa64c77fa6654dacbe2c59b77d3931dc",
            "value": " 2/2 [00:01&lt;00:00,  1.84it/s]"
          }
        },
        "312266c9c3e14c50a02ff96b7c71ceaf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0e04f3664880435198f1a50fa8c1ee92": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1003465b490a46a5a4a3a74993d93ed5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a2a23cbdb65c46d6baf033d87e688622": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f784fc250d04dec93928b2c23b98530": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0be2c167de3242199af03b341cb5e87e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "aa64c77fa6654dacbe2c59b77d3931dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "636eb2f3d0ee4dfba757f5ab3f82dfba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_21c861a5de814336934aba3a7f5633cb",
              "IPY_MODEL_5cc3ab4ac7ff43d5a5916a0152bf241e",
              "IPY_MODEL_82de3dd9c58640a3b8454ac5a1ef3cf2"
            ],
            "layout": "IPY_MODEL_60f0bc965af84127a6caff61f60eb29c"
          }
        },
        "21c861a5de814336934aba3a7f5633cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3a8a88035be34d67b46e8e1f6eaa65f6",
            "placeholder": "​",
            "style": "IPY_MODEL_94c12462d0784a46b573f58bd0ac809f",
            "value": "Downloading data: "
          }
        },
        "5cc3ab4ac7ff43d5a5916a0152bf241e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fc9b554fb3bb4fbe9c678d27199963a4",
            "max": 8116577,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_38286e8e71fc46e585feba5535d6c56a",
            "value": 8116577
          }
        },
        "82de3dd9c58640a3b8454ac5a1ef3cf2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c9fb158b66e943a799c9a84f2339c045",
            "placeholder": "​",
            "style": "IPY_MODEL_790cdfe161714e9295cd46b1fa6e7765",
            "value": " 30.3M/? [00:00&lt;00:00, 51.6MB/s]"
          }
        },
        "60f0bc965af84127a6caff61f60eb29c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3a8a88035be34d67b46e8e1f6eaa65f6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "94c12462d0784a46b573f58bd0ac809f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fc9b554fb3bb4fbe9c678d27199963a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "38286e8e71fc46e585feba5535d6c56a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c9fb158b66e943a799c9a84f2339c045": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "790cdfe161714e9295cd46b1fa6e7765": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c4fdfcfb6a36475cbbaebb09f5572231": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3fa66a14af0c4bce9da4241a662de37c",
              "IPY_MODEL_3154bc23f2bf4f60ab1249478880fd2f",
              "IPY_MODEL_ad81a932fa2c450897896ef774a380a8"
            ],
            "layout": "IPY_MODEL_3eb3debc46e34bc0a0d186d4d38f047b"
          }
        },
        "3fa66a14af0c4bce9da4241a662de37c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3fa79e95bc2c49dbac7d90f77e81399a",
            "placeholder": "​",
            "style": "IPY_MODEL_5fc89844e65a4dd9b4122376876e023b",
            "value": "Downloading data: "
          }
        },
        "3154bc23f2bf4f60ab1249478880fd2f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f1e1f121a7e448c28a7671b28b5fbf9b",
            "max": 1054280,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9c4aafd33c0c483cbb7367eaa38f1ec2",
            "value": 1054280
          }
        },
        "ad81a932fa2c450897896ef774a380a8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e01e5e853b7b4fbc883edc12254bac30",
            "placeholder": "​",
            "style": "IPY_MODEL_6f289e2a8cba495394b6ed58b4b9fa13",
            "value": " 4.85M/? [00:00&lt;00:00, 12.5MB/s]"
          }
        },
        "3eb3debc46e34bc0a0d186d4d38f047b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3fa79e95bc2c49dbac7d90f77e81399a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5fc89844e65a4dd9b4122376876e023b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f1e1f121a7e448c28a7671b28b5fbf9b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9c4aafd33c0c483cbb7367eaa38f1ec2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e01e5e853b7b4fbc883edc12254bac30": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6f289e2a8cba495394b6ed58b4b9fa13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "32fd852d8f21436f910a2afc600273e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_47cf2dfd8e7b43f295246bb1a31068ae",
              "IPY_MODEL_7ae1101a8b17496ba0fc7665910684db",
              "IPY_MODEL_bd866563605d4b9795e156f7058d33c8"
            ],
            "layout": "IPY_MODEL_0537e872515a441a87c1e61944b283b7"
          }
        },
        "47cf2dfd8e7b43f295246bb1a31068ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e030e39f46a44d8499bf1cf970bd5c14",
            "placeholder": "​",
            "style": "IPY_MODEL_b41c1351af7349d48849fa871e898690",
            "value": "Extracting data files: 100%"
          }
        },
        "7ae1101a8b17496ba0fc7665910684db": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_57f23c622fde491a90a9e4f27c4f0d46",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_921c15f474474996891162e5ed4b769c",
            "value": 2
          }
        },
        "bd866563605d4b9795e156f7058d33c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_57d403f7b7ac40a595c79d09b1099711",
            "placeholder": "​",
            "style": "IPY_MODEL_c781a4e8a56c46068ac82e6c29a7d256",
            "value": " 2/2 [00:00&lt;00:00, 51.62it/s]"
          }
        },
        "0537e872515a441a87c1e61944b283b7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e030e39f46a44d8499bf1cf970bd5c14": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b41c1351af7349d48849fa871e898690": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "57f23c622fde491a90a9e4f27c4f0d46": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "921c15f474474996891162e5ed4b769c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "57d403f7b7ac40a595c79d09b1099711": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c781a4e8a56c46068ac82e6c29a7d256": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1de15df05d1d4286985253db63f0e8e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ff0ea9831b7c458bb01ea9b185b65679",
              "IPY_MODEL_f9927354daf34230b8108ca46e54c25c",
              "IPY_MODEL_bef24215718748d5980285299e4c7d97"
            ],
            "layout": "IPY_MODEL_a72a7b1fc8e84bec8542dc0a003186bb"
          }
        },
        "ff0ea9831b7c458bb01ea9b185b65679": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b5607a20871c471cbcb4d7dc74fa3415",
            "placeholder": "​",
            "style": "IPY_MODEL_5e2373d2396445fbac09bbb8db774aa5",
            "value": "Generating train split: 100%"
          }
        },
        "f9927354daf34230b8108ca46e54c25c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_73f83e5fdc854821bca5b96a4c43fb7d",
            "max": 87599,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8cdcd0b034744325a98686b6a4990a39",
            "value": 87599
          }
        },
        "bef24215718748d5980285299e4c7d97": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5a8d6b909a2346869c6f13fb6244cdd3",
            "placeholder": "​",
            "style": "IPY_MODEL_90fb40158daa4eb58a1f90ef27c47b23",
            "value": " 87490/87599 [00:20&lt;00:00, 8761.99 examples/s]"
          }
        },
        "a72a7b1fc8e84bec8542dc0a003186bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b5607a20871c471cbcb4d7dc74fa3415": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5e2373d2396445fbac09bbb8db774aa5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73f83e5fdc854821bca5b96a4c43fb7d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8cdcd0b034744325a98686b6a4990a39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5a8d6b909a2346869c6f13fb6244cdd3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90fb40158daa4eb58a1f90ef27c47b23": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "00eb8bc6166c42069c261c2789b48b20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_21bbecdeae51471ba2fd4ea49a72f94f",
              "IPY_MODEL_84912be0495945c4810113cd4b78eebc",
              "IPY_MODEL_34ac6894b4124ec3b086f4b126c5f6f0"
            ],
            "layout": "IPY_MODEL_a241157c8bb14cdea1b8d93542e12d62"
          }
        },
        "21bbecdeae51471ba2fd4ea49a72f94f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dbd3c6a402754be98995540d76cb4bb6",
            "placeholder": "​",
            "style": "IPY_MODEL_e0652c0246cb4ce1aa4c7050f9b98762",
            "value": "Generating validation split:  98%"
          }
        },
        "84912be0495945c4810113cd4b78eebc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1af0b0bebc424cb6858c0d7bf1683b1f",
            "max": 10570,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fb84f9f4ce5a499c9adcacf1cb6bee91",
            "value": 10570
          }
        },
        "34ac6894b4124ec3b086f4b126c5f6f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_60609b0928dd4b7b89423a5caaf5a1a5",
            "placeholder": "​",
            "style": "IPY_MODEL_cbdb79ae1478465d94c55bbeeeef6a8f",
            "value": " 10382/10570 [00:01&lt;00:00, 5527.54 examples/s]"
          }
        },
        "a241157c8bb14cdea1b8d93542e12d62": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dbd3c6a402754be98995540d76cb4bb6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0652c0246cb4ce1aa4c7050f9b98762": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1af0b0bebc424cb6858c0d7bf1683b1f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fb84f9f4ce5a499c9adcacf1cb6bee91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "60609b0928dd4b7b89423a5caaf5a1a5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cbdb79ae1478465d94c55bbeeeef6a8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "79b99b9a258f49e09ad27b402854240d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_daa82bc51780498b9d3a381a936e2d7e",
              "IPY_MODEL_b8d51a6cceac462da3af8dc6d45d5982",
              "IPY_MODEL_3891cd26ee414eb380ce38aca5884fdc"
            ],
            "layout": "IPY_MODEL_5739fedfcd414558be05f7673c9e0556"
          }
        },
        "daa82bc51780498b9d3a381a936e2d7e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_14916a09e75748abaa413a6633926aa7",
            "placeholder": "​",
            "style": "IPY_MODEL_0b43f0a6e13d4ba8b63bd04799ef93da",
            "value": "100%"
          }
        },
        "b8d51a6cceac462da3af8dc6d45d5982": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d8bab05b445f4211bf4a442acec8a09a",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_74bc092eb82a48eaadfb8515b490f6ab",
            "value": 2
          }
        },
        "3891cd26ee414eb380ce38aca5884fdc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e97f623a655946d0a67abd3900f2d4bf",
            "placeholder": "​",
            "style": "IPY_MODEL_68669a5c4aa94cc69ce80b442cbfc0b7",
            "value": " 2/2 [00:00&lt;00:00, 41.85it/s]"
          }
        },
        "5739fedfcd414558be05f7673c9e0556": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "14916a09e75748abaa413a6633926aa7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0b43f0a6e13d4ba8b63bd04799ef93da": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d8bab05b445f4211bf4a442acec8a09a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74bc092eb82a48eaadfb8515b490f6ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e97f623a655946d0a67abd3900f2d4bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "68669a5c4aa94cc69ce80b442cbfc0b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ff71ddd10743434996fb13a8b9677a18": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c3a71ee4bde04c50a0d5cf2ae1fff363",
              "IPY_MODEL_8278b27cbd354d22a37701e4d6023307",
              "IPY_MODEL_9489adc142dc4ad783240d8c94a73222"
            ],
            "layout": "IPY_MODEL_21641a620e714aaaab2ed31cf142ed17"
          }
        },
        "c3a71ee4bde04c50a0d5cf2ae1fff363": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_88a75ee4e10b40f1a92fb5cd9a9cbdea",
            "placeholder": "​",
            "style": "IPY_MODEL_be399348f68b423a99bd826e0f6084c6",
            "value": "Downloading builder script: "
          }
        },
        "8278b27cbd354d22a37701e4d6023307": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_642a79519bbc49989b4341262ce092e0",
            "max": 1967,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d982cab25b814a549e58f054f1a62419",
            "value": 1967
          }
        },
        "9489adc142dc4ad783240d8c94a73222": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f843310cad7f47a69bcb57ee446e6f0c",
            "placeholder": "​",
            "style": "IPY_MODEL_2468cd3cd35f4a7fa0ab033b51fc6050",
            "value": " 5.27k/? [00:00&lt;00:00, 110kB/s]"
          }
        },
        "21641a620e714aaaab2ed31cf142ed17": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "88a75ee4e10b40f1a92fb5cd9a9cbdea": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "be399348f68b423a99bd826e0f6084c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "642a79519bbc49989b4341262ce092e0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d982cab25b814a549e58f054f1a62419": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f843310cad7f47a69bcb57ee446e6f0c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2468cd3cd35f4a7fa0ab033b51fc6050": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2096a94ea9c94aa38c88212467544283": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ffbe472b7f2341f787d91ddf8c83e116",
              "IPY_MODEL_af8a5bf634214746b5a751dbdd2c0e3e",
              "IPY_MODEL_bddefd5c146b494d99a53127cb730d73"
            ],
            "layout": "IPY_MODEL_c873724f461549f3a478cccf1166fc14"
          }
        },
        "ffbe472b7f2341f787d91ddf8c83e116": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_37588b53bba649efb365e8a96e5c570a",
            "placeholder": "​",
            "style": "IPY_MODEL_3b5d374900c34ff2a94d015b64d5be20",
            "value": "Downloading metadata: "
          }
        },
        "af8a5bf634214746b5a751dbdd2c0e3e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_08f6e271c19846148925c82edb5165b0",
            "max": 1021,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b35c676d4ffa4aa981467b8648c96a7d",
            "value": 1021
          }
        },
        "bddefd5c146b494d99a53127cb730d73": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a2ade48952b740a8a9eea0aadb074cc8",
            "placeholder": "​",
            "style": "IPY_MODEL_c2a752cd2dd84aadad0c4c1d5b783300",
            "value": " 2.36k/? [00:00&lt;00:00, 47.5kB/s]"
          }
        },
        "c873724f461549f3a478cccf1166fc14": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "37588b53bba649efb365e8a96e5c570a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3b5d374900c34ff2a94d015b64d5be20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "08f6e271c19846148925c82edb5165b0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b35c676d4ffa4aa981467b8648c96a7d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a2ade48952b740a8a9eea0aadb074cc8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c2a752cd2dd84aadad0c4c1d5b783300": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ddb9a123cdaf4b8e903bbe78287d9770": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_706912ebfea84f588d3dd5371bb96b51",
              "IPY_MODEL_182079f7e57b45c2a9b00ffe4952f501",
              "IPY_MODEL_edbe462be8124949935cb7b7788bdbf8"
            ],
            "layout": "IPY_MODEL_3c5d968bb5e54084bb4d175943dc7a9a"
          }
        },
        "706912ebfea84f588d3dd5371bb96b51": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bda8369113874827a0d98309f5cceee6",
            "placeholder": "​",
            "style": "IPY_MODEL_d223fd25fb054844880e3771dae6c464",
            "value": "Downloading data files: 100%"
          }
        },
        "182079f7e57b45c2a9b00ffe4952f501": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_93227e2bb3814a2585cd2e4123e2ac2a",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e8ae324a04af468196faf0c4938b5afe",
            "value": 2
          }
        },
        "edbe462be8124949935cb7b7788bdbf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a298570d5b284a74b229a4c1aa29f750",
            "placeholder": "​",
            "style": "IPY_MODEL_65c6b2b6683749919589c83b05ec099e",
            "value": " 2/2 [00:03&lt;00:00,  1.60s/it]"
          }
        },
        "3c5d968bb5e54084bb4d175943dc7a9a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bda8369113874827a0d98309f5cceee6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d223fd25fb054844880e3771dae6c464": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "93227e2bb3814a2585cd2e4123e2ac2a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8ae324a04af468196faf0c4938b5afe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a298570d5b284a74b229a4c1aa29f750": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "65c6b2b6683749919589c83b05ec099e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d4cfd77d93794a7e824f77791eae77fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9bb8e839f5fa4cf1b7e5b22fb3ef2ff4",
              "IPY_MODEL_329cf6925aa14c2a947dacde130ee4c4",
              "IPY_MODEL_4971ce65fd6346ff9262ac2f62a1411e"
            ],
            "layout": "IPY_MODEL_ca96bf0a918c43f782a8471978464994"
          }
        },
        "9bb8e839f5fa4cf1b7e5b22fb3ef2ff4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_be1a2aee094b40b395d752d886df52c0",
            "placeholder": "​",
            "style": "IPY_MODEL_7dc55d8cd9094e7288e43aac9ef53f71",
            "value": "Downloading data: "
          }
        },
        "329cf6925aa14c2a947dacde130ee4c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2d694a5560684f3aae134bc2b67f2fda",
            "max": 8116577,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0518797c416a425da61416fe09960b2c",
            "value": 8116577
          }
        },
        "4971ce65fd6346ff9262ac2f62a1411e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_46f95355e57448709198f0a8802d4972",
            "placeholder": "​",
            "style": "IPY_MODEL_e69411bdd38045feaab7dd381c6e62d2",
            "value": " 30.3M/? [00:00&lt;00:00, 52.8MB/s]"
          }
        },
        "ca96bf0a918c43f782a8471978464994": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "be1a2aee094b40b395d752d886df52c0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7dc55d8cd9094e7288e43aac9ef53f71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2d694a5560684f3aae134bc2b67f2fda": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0518797c416a425da61416fe09960b2c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "46f95355e57448709198f0a8802d4972": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e69411bdd38045feaab7dd381c6e62d2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "547c86dc57b549399bfefbf50df1a098": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_23a5d51abc69425ebb1b648a121a4857",
              "IPY_MODEL_e5b78781c71f4549b6230ce22a171d85",
              "IPY_MODEL_4d9942b4c4ef41f396e6216c2da3de88"
            ],
            "layout": "IPY_MODEL_41a231e839354c939f98b29950d2cf9b"
          }
        },
        "23a5d51abc69425ebb1b648a121a4857": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_887a5aafd1624c0ba7d98f1ee6633d4c",
            "placeholder": "​",
            "style": "IPY_MODEL_ee0875b69ead4aacacb8d332a734a148",
            "value": "Downloading data: "
          }
        },
        "e5b78781c71f4549b6230ce22a171d85": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aabeea230e174b1987f28f0aafb01c6f",
            "max": 1054280,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d7b70a89374b46c28066b764c5d2d395",
            "value": 1054280
          }
        },
        "4d9942b4c4ef41f396e6216c2da3de88": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7a59f8fe9c3c4279b62bd989a9fdec18",
            "placeholder": "​",
            "style": "IPY_MODEL_ee020deea53a4adba31065f32a290d37",
            "value": " 4.85M/? [00:00&lt;00:00, 24.3MB/s]"
          }
        },
        "41a231e839354c939f98b29950d2cf9b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "887a5aafd1624c0ba7d98f1ee6633d4c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee0875b69ead4aacacb8d332a734a148": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "aabeea230e174b1987f28f0aafb01c6f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d7b70a89374b46c28066b764c5d2d395": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7a59f8fe9c3c4279b62bd989a9fdec18": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee020deea53a4adba31065f32a290d37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "60008e73a66e4cc7a594e4601e71e5da": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b8fda90b244642a0a690c946d06fb09a",
              "IPY_MODEL_3c69aa1607c74a389bca4f4dc0c75485",
              "IPY_MODEL_22175274de004ca888b042243c675017"
            ],
            "layout": "IPY_MODEL_fc4a52f89a0d4a6693c36e00ae6e8251"
          }
        },
        "b8fda90b244642a0a690c946d06fb09a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8dae1cb7046c49af9d2a42643313a721",
            "placeholder": "​",
            "style": "IPY_MODEL_118eb58e0d184dadb6d582f65f07f68d",
            "value": "Extracting data files: 100%"
          }
        },
        "3c69aa1607c74a389bca4f4dc0c75485": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ae3c2bbd7c974904925a1897018a9f6e",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b93c1c2e186d4589ae237f1da02239a3",
            "value": 2
          }
        },
        "22175274de004ca888b042243c675017": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_550f5fb0bfbd488d9f0271b74abd63d6",
            "placeholder": "​",
            "style": "IPY_MODEL_0fa2a256983242649b0962d50e5bb001",
            "value": " 2/2 [00:00&lt;00:00, 44.98it/s]"
          }
        },
        "fc4a52f89a0d4a6693c36e00ae6e8251": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8dae1cb7046c49af9d2a42643313a721": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "118eb58e0d184dadb6d582f65f07f68d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ae3c2bbd7c974904925a1897018a9f6e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b93c1c2e186d4589ae237f1da02239a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "550f5fb0bfbd488d9f0271b74abd63d6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0fa2a256983242649b0962d50e5bb001": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "14c28af50391410baee55177373c18c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3583279816fd44a19b0ad4c0bda2f72a",
              "IPY_MODEL_b90a991aa3dd4f7bb7606a1722c7de2b",
              "IPY_MODEL_651fc1a59de84ccdb9c7af06e7cd6a3e"
            ],
            "layout": "IPY_MODEL_4bc12c00da0e4190ad06841884ceaf4f"
          }
        },
        "3583279816fd44a19b0ad4c0bda2f72a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dc1b8851f6a7459db141a54356cd2ee0",
            "placeholder": "​",
            "style": "IPY_MODEL_81774770d14a465b8f0e8f91e7771a79",
            "value": "Generating train split:  99%"
          }
        },
        "b90a991aa3dd4f7bb7606a1722c7de2b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d5592d2afbac4b358b005f1b27426799",
            "max": 87599,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_237fa7839e584dba9931e88926318af8",
            "value": 87599
          }
        },
        "651fc1a59de84ccdb9c7af06e7cd6a3e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_733a2f07439544239ca8931b753e632e",
            "placeholder": "​",
            "style": "IPY_MODEL_44d0fcc7a5934d20b0dd4035a2dc2ba0",
            "value": " 87066/87599 [00:20&lt;00:00, 8899.54 examples/s]"
          }
        },
        "4bc12c00da0e4190ad06841884ceaf4f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc1b8851f6a7459db141a54356cd2ee0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81774770d14a465b8f0e8f91e7771a79": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d5592d2afbac4b358b005f1b27426799": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "237fa7839e584dba9931e88926318af8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "733a2f07439544239ca8931b753e632e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "44d0fcc7a5934d20b0dd4035a2dc2ba0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a5e0818eb54c4958983b0fe85c9feb39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1530c4615501412fa02918fb7f63ff2a",
              "IPY_MODEL_70d7b75f333a4d5995baef95659a32ac",
              "IPY_MODEL_a702df381c2147e18a49fa6b62ae1966"
            ],
            "layout": "IPY_MODEL_f413a767f48842548c227888734457c5"
          }
        },
        "1530c4615501412fa02918fb7f63ff2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fe8264020f364d86b6c9b86152145bbd",
            "placeholder": "​",
            "style": "IPY_MODEL_7326255bdf204b7a989552e26aa539d7",
            "value": "Generating validation split:  98%"
          }
        },
        "70d7b75f333a4d5995baef95659a32ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f7be8d4f66fb42e3812d7bbf41ffb86f",
            "max": 10570,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b1dd9dd9ecf7472f84f342ebb4f513ba",
            "value": 10570
          }
        },
        "a702df381c2147e18a49fa6b62ae1966": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e2caa6406bd041fbb4dfcb9b0ab127fe",
            "placeholder": "​",
            "style": "IPY_MODEL_b6a1ecac13e749a3ab5b1699aa0cbd57",
            "value": " 10324/10570 [00:01&lt;00:00, 4448.54 examples/s]"
          }
        },
        "f413a767f48842548c227888734457c5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fe8264020f364d86b6c9b86152145bbd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7326255bdf204b7a989552e26aa539d7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f7be8d4f66fb42e3812d7bbf41ffb86f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b1dd9dd9ecf7472f84f342ebb4f513ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e2caa6406bd041fbb4dfcb9b0ab127fe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b6a1ecac13e749a3ab5b1699aa0cbd57": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "395860a06482436c8ce971f76998c12e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_550774b7251941d9aafc23e64726c8fe",
              "IPY_MODEL_a8aa36aa80874fa0b0898b53b36639c7",
              "IPY_MODEL_b450cf4492c24591a1261a1c6d1a7556"
            ],
            "layout": "IPY_MODEL_6edbf331c6554ac883f9110aff31b583"
          }
        },
        "550774b7251941d9aafc23e64726c8fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b52daff01c3d4c72a2c8dd3d54a052fa",
            "placeholder": "​",
            "style": "IPY_MODEL_70d2357f5e7049788d1715ef8c2a8110",
            "value": "100%"
          }
        },
        "a8aa36aa80874fa0b0898b53b36639c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6a0d992ca00c40d9972d6f0b0e46e4ac",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_75e2f4342f2c40d384091e720ca6c76d",
            "value": 2
          }
        },
        "b450cf4492c24591a1261a1c6d1a7556": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1a0ca7744e064ea7a961903408288453",
            "placeholder": "​",
            "style": "IPY_MODEL_ccfdafb04cda4aca8beaa3115e17e587",
            "value": " 2/2 [00:00&lt;00:00, 42.25it/s]"
          }
        },
        "6edbf331c6554ac883f9110aff31b583": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b52daff01c3d4c72a2c8dd3d54a052fa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "70d2357f5e7049788d1715ef8c2a8110": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6a0d992ca00c40d9972d6f0b0e46e4ac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "75e2f4342f2c40d384091e720ca6c76d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1a0ca7744e064ea7a961903408288453": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ccfdafb04cda4aca8beaa3115e17e587": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}