OxML2020_Sequence_Models.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "OxML 2020: Sequence Models Workshop",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WIshKEvfssy4",
        "colab_type": "text"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1z-VzwrzVfh8ZFG6p3pXFJHxXF_cOtMC6)\n",
        "# **Sequence Models Workshop for OxML 2020** \n",
        "\n",
        "---\n",
        "\n",
        "### **Prepared by *Asmita Poddar* & *Piotr Kozakowski***\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BT2uPkuyJDu2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1c7e3ffb-6403-4acc-885a-54d9b429a41e"
      },
      "source": [
        "#!pip install torch torchtext\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CvH8K1HMOIyE",
        "colab_type": "text"
      },
      "source": [
        "## **Language Modeling**\n",
        "\n",
        "Language Modelling is the core problem for a number of of Natural Language Processing (NLP) tasks such as speech to text, conversational system, and text summarization.\n",
        "\n",
        "In this tutorial, you will learn how to create a language model for ***natural language text generation*** by implementing and training **Long Short Term Memory (LSTM)** networks and **Transformers**. \n",
        "\n",
        "A trained language model learns the likelihood of occurrence of a word based on the previous sequence of words used in the text. Language models can be operated at character level, n-gram level, sentence level or even paragraph level.\n",
        "\n",
        "After completing this tutorial, you will know:\n",
        "\n",
        "\n",
        "*   The challenge of developing a good framing of a word-based language model for a corpus of text using *PyTorch*\n",
        "*   How to generate sequences using a fit language model.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jIosEPO9JSCF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "import pandas as pd\n",
        "import re\n",
        "import numpy as np\n",
        "import random\n",
        "import time\n",
        "import tqdm\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "from torchtext import data\n",
        "from torchtext import datasets"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bJMCYh6-2661",
        "colab_type": "text"
      },
      "source": [
        "## Data\n",
        "\n",
        "The [WikiText language modeling dataset](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) is a collection of over 100 million tokens extracted from the set of verified `Good` and `Featured` articles on Wikipedia.\n",
        "\n",
        "PyTorch provides many inbuilt datasets for many applications as well as tools for data processing. \n",
        "\n",
        "In this tutorial, we will use the Pytorch Dataset - [wikitext2](https://torchtext.readthedocs.io/en/latest/datasets.html#wikitext-2). \n",
        "\n",
        "The texts in this dataset have already been tokenized to word level, and split into train, validation, test sets. We further split the data into batches for model training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ce_-GAZBKHBM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "71d12662-4437-48e1-8db8-2b8b319fdf89"
      },
      "source": [
        "field = data.Field()\n",
        "(train, val, test) = datasets.language_modeling.WikiText2.splits(text_field=field)\n",
        "\n",
        "train_data = next(iter(train))\n",
        "val_data = next(iter(val))\n",
        "test_data = next(iter(test))\n",
        "\n",
        "print('# Train tokens: ', len(train_data.text))  \n",
        "print('# Val tokens: ', len(val_data.text))      \n",
        "print('# Test tokens: ', len(test_data.text))    "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "# Train tokens:  2088628\n",
            "# Val tokens:  217646\n",
            "# Test tokens:  245569\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xt9uBlz0CNkE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8fda519d-cc13-4a36-9737-73826bc0249c"
      },
      "source": [
        "# We will need to know the size of the vocabulary later for\n",
        "#  both defining the word embedding layer in the model,\n",
        "# and for encoding output words.\n",
        "field.build_vocab(train, vectors=None)\n",
        "print('Vocabulary size:', len(field.vocab))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary size: 33279\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dynvpvh2AbER",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BATCH_SIZE = 128\n",
        "SEQ_LEN = 40\n",
        "# Split the data into batches of size BATCH_SIZE \n",
        "(train, val, test) = data.BPTTIterator.splits((train, val, test), batch_size=BATCH_SIZE, bptt_len=SEQ_LEN)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EVeqgdiH9ROX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 318
        },
        "outputId": "6f2b5feb-34b5-4a3a-ac9d-781f00b4211c"
      },
      "source": [
        "batch = next(iter(train))\n",
        "print('Batch text shape: ', batch.text.shape)     # [seq_len x batch_size]\n",
        "print('Batch text: \\n', batch.text)               #  consecutive words are in rows of the tensor\n",
        "print('Batch target: \\n', batch.target)           # target is the same as text, but shifted one position up"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Batch text shape:  torch.Size([40, 128])\n",
            "Batch text: \n",
            " tensor([[    9,  1080,    38,  ..., 20496,     5,    28],\n",
            "        [   11,   343, 24830,  ...,    21,     2,     2],\n",
            "        [ 3932,     5,     3,  ...,  7154,  4917,   508],\n",
            "        ...,\n",
            "        [  955,  2531,    22,  ...,     6,    42,    27],\n",
            "        [    3,  1500,    27,  ...,  2661,   247,  1530],\n",
            "        [   25,   637, 22950,  ...,  1689,  1509,     0]])\n",
            "Batch target: \n",
            " tensor([[   11,   343, 24830,  ...,    21,     2,     2],\n",
            "        [ 3932,     5,     3,  ...,  7154,  4917,   508],\n",
            "        [ 4429,     2,    56,  ...,     0,  7129,     5],\n",
            "        ...,\n",
            "        [    3,  1500,    27,  ...,  2661,   247,  1530],\n",
            "        [   25,   637, 22950,  ...,  1689,  1509,     0],\n",
            "        [   10,   586,     3,  ...,    45,    11,    13]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mCiDsYLpsog-",
        "colab_type": "text"
      },
      "source": [
        "# Long Short Term Memory (LSTM) Networks\n",
        "Recurrent Neural Networks suffer from two problems: \n",
        "* vanishing gradient and exploding gradient\n",
        "* inability to capture long-term dependencies\n",
        "\n",
        "LSTM solve this issue by explicitly introducing a memory unit, called the cell into the network. Read more at: https://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
        "\n",
        "Therefore, this single unit makes decision by considering the current input, previous output and previous memory. And it generates a new output and alters its memory.\n",
        "\n",
        "Layers in the LSTM model:\n",
        "1. **Input Layer**: Takes the sequence of words as input\n",
        "2. **Embedding Layer**: Creates embeddings from the input sequence using a simple lookup table that stores embeddings of a fixed dictionary and size\n",
        "3. **LSTM Layer**: Computes the output using LSTM units. I have added 128 units in the layer, but this number can be fine tuned later.\n",
        "4. **Dropout Layer**: A regularisation layer which randomly turns-off the activations of some neurons in the LSTM layer. It helps in preventing over fitting.\n",
        "5. **Output Layer** : Computes the probability of the best possible next word as output\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7aIdSeqp872h",
        "colab_type": "text"
      },
      "source": [
        "### Model\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Hg15tkYTDYe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "VOCAB_SIZE = len(field.vocab)\n",
        "\n",
        "## TODO: Experiment with for fine-tuning the model\n",
        "EMB_SIZE = 256\n",
        "NHIDDEN = 128\n",
        "NLAYERS = 3\n",
        "DROPOUT = 0.5\n",
        "EPOCHS = 3\n",
        "LR = 0.001"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgUEsO1f2Ayq",
        "colab_type": "text"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1JMR622gBqx5U5jZPQLxM1FxAKmscKeM-)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjzsPrHalLsw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTMModel(nn.Module):\n",
        "    \"\"\"Container module with an encoder, a recurrent module, and a decoder.\"\"\"\n",
        "\n",
        "    def __init__(self, vocab_size, emb_size, nhidden, nlayers, dropout=0.5):\n",
        "        super(LSTMModel, self).__init__()\n",
        "        \n",
        "        self.vocab_size = vocab_size\n",
        "        self.drop = nn.Dropout(dropout)\n",
        "        self.encoder = nn.Embedding(vocab_size, emb_size)\n",
        "        self.lstm = nn.LSTM(emb_size, nhidden, nlayers, dropout=dropout)\n",
        "        self.decoder = nn.Linear(nhidden, vocab_size)\n",
        "\n",
        "        self.init_weights()\n",
        "        self.nhidden = nhidden\n",
        "        self.nlayers = nlayers\n",
        "\n",
        "    # Initialise weights for encoder and decoder (uniform initialisation)\n",
        "    def init_weights(self):\n",
        "        initrange = 0.1\n",
        "        nn.init.uniform_(self.encoder.weight, -initrange, initrange)\n",
        "        nn.init.zeros_(self.decoder.weight)\n",
        "        nn.init.uniform_(self.decoder.weight, -initrange, initrange)\n",
        "\n",
        "     # Initialise weights for hidden $ cell units with zeros\n",
        "    def init_hidden(self, bsz):\n",
        "        weight = next(self.parameters())\n",
        "        return (weight.new_zeros(self.nlayers, bsz, self.nhidden),\n",
        "                    weight.new_zeros(self.nlayers, bsz, self.nhidden))\n",
        "\n",
        "    def forward(self, input, hidden):\n",
        "        emb = self.drop(self.encoder(input))\n",
        "        output, hidden = self.lstm(emb, hidden)\n",
        "        output = self.drop(output)\n",
        "        decoded = self.decoder(output)\n",
        "        decoded = decoded.view(-1, self.vocab_size)\n",
        "        return F.log_softmax(decoded, dim=1), hidden\n",
        "\n",
        "   "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2UTTPxHcS4pH",
        "colab_type": "text"
      },
      "source": [
        "### Model Training\n",
        "\n",
        "##### **Model:** LSTM\n",
        "\n",
        "##### **Loss Function:** Negative Log-Likelihood (NLL):  $L(y) = −log(y)$\n",
        "\n",
        "[](https://ljvmiranda921.github.io/assets/png/cs231n-ann/neg_log.png)\n",
        "\n",
        "##### **Optimizer:** Adam Optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aIjFltCzkajA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "outputId": "f9d83380-d3f3-429f-b5e3-0ed6fc063b38"
      },
      "source": [
        "model = LSTMModel(VOCAB_SIZE, EMB_SIZE, NHIDDEN, NLAYERS, DROPOUT).to(device)\n",
        "loss_fn = nn.NLLLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=LR)\n",
        "\n",
        "print(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "LSTMModel(\n",
            "  (drop): Dropout(p=0.5, inplace=False)\n",
            "  (encoder): Embedding(33279, 256)\n",
            "  (lstm): LSTM(256, 128, num_layers=3, dropout=0.5)\n",
            "  (decoder): Linear(in_features=128, out_features=33279, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h1v6mRMqg6Vv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 230
        },
        "outputId": "9cd76b6b-98ad-4ced-9c52-2ba1dab09dde"
      },
      "source": [
        "eval_it = iter(val)\n",
        "\n",
        "\n",
        "# Run training on one batch\n",
        "def run_on_batch(batch):\n",
        "  text = batch.text.cuda()\n",
        "  target = batch.target.cuda()\n",
        "  target = target.view(-1)\n",
        "  hidden = model.init_hidden(BATCH_SIZE)\n",
        "\n",
        "  # Starting each batch, we detach the hidden state from how it was previously produced.\n",
        "  # If we didn't, the model would try backpropagating all the way to start of the dataset.\n",
        "  hidden = (hidden[0].detach(), hidden[1].detach())\n",
        "  pred, hidden = model(text, hidden)\n",
        "  return loss_fn(pred, target)\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  print('**** Running Epoch {} ****'.format(epoch))\n",
        "\n",
        "  for (i, batch) in enumerate(train):\n",
        "    \n",
        "    if batch.text.shape[1] != BATCH_SIZE:\n",
        "      #Skip an incomplete batch.\n",
        "      print('Skipping incomplete batch {} ...'.format(i))\n",
        "      continue\n",
        "\n",
        "    # Turn on training mode which enables dropout.\n",
        "    model.train()\n",
        "    model.zero_grad()\n",
        "\n",
        "    train_loss = run_on_batch(batch)\n",
        "    train_loss.backward()\n",
        "\n",
        "    # `clip_grad_norm` helps prevent the exploding gradient problem in LSTMs.\n",
        "    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # max norm of the gradients = 1.0\n",
        "    optimizer.step()\n",
        "    \n",
        "    if i % 400 == 0:\n",
        "      print('Batch {} ----'.format(i) )\n",
        "      print('Train loss: ', float(train_loss) )\n",
        "\n",
        "      # Turn on evaluation mode which disables dropout.\n",
        "      model.eval()\n",
        "      eval_loss = float(run_on_batch(next(eval_it)))\n",
        "      print('Eval loss: ',eval_loss )\n",
        "\n",
        "torch.save(model, 'model.pt')\n",
        "\n",
        "# Download model from Colab\n",
        "from google.colab import files\n",
        "files.download(\"model.pt\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "**** Running Epoch 0 ****\n",
            "Batch 0 ----\n",
            "Train loss:  6.0173773765563965\n",
            "Eval loss:  5.884695529937744\n",
            "**** Running Epoch 1 ****\n",
            "Batch 0 ----\n",
            "Train loss:  5.95809268951416\n",
            "Eval loss:  5.77343225479126\n",
            "**** Running Epoch 2 ****\n",
            "Batch 0 ----\n",
            "Train loss:  5.909502983093262\n",
            "Eval loss:  5.8311262130737305\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_358f80b3-b825-4a2c-a796-5fb86dac088d\", \"model.pt\", 53101627)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9WnaEmwSnyR",
        "colab_type": "text"
      },
      "source": [
        "### Text Generation \n",
        " \n",
        "\n",
        "Now, we are ready to generate text. \n",
        "\n",
        "We begin with some input (seed) word and use the trained model to predict the next word based on the input words. Then, we append the predicted word into the input, and have the model predict the next word and so on. We continue the process until we obtain a sequence with the length we want (here we generate 20 words). \n",
        "\n",
        "The multiple generated words can be appended together to get the generated sequence!\n",
        "\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "33Ghp1EzlRKG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "example = next(eval_it).text.cuda()\n",
        "buffer = example.clone()\n",
        "input = torch.tensor([[example[0][0]]]).to(device)\n",
        "\n",
        "### TODO: Your own input\n",
        "#input = torch.randint(VOCAB_SIZE, (1, 1), dtype=torch.long).to(device). #generating a random starting word\n",
        "### TODO: Specify no. of words to be generated \n",
        "NO_WORDS = 20"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jfR4rVku-Y4W",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "4e167f82-d478-472a-f847-d485732ea4f8"
      },
      "source": [
        "# Load saved trained model\n",
        "with open('model.pt', 'rb') as f:\n",
        "    model = torch.load(f).to(device)\n",
        "    \n",
        "generated_text = ''\n",
        "\n",
        "# Turn on evaluation mode which disables dropout.\n",
        "model.eval()\n",
        "hidden = model.init_hidden(1)\n",
        "\n",
        "for i in range(NO_WORDS):\n",
        "  output, hidden = model(input, hidden)\n",
        "  word_weights = output.squeeze().exp().cpu()\n",
        "  word_idx = torch.multinomial(word_weights, 1)[0]\n",
        "  input.fill_(word_idx)   # current word is input to next hidden unit\n",
        "  generated_text = generated_text + field.vocab.itos[word_idx]+ ' '\n",
        "\n",
        "dataset_text = [field.vocab.itos[idx] for idx in buffer[:, 0]]\n",
        "print('INPUT WORD: ', field.vocab.itos[buffer[0][0]])\n",
        "print('DATASET PORTION:', dataset_text[: NO_WORDS])\n",
        "print('GENERATED TEXT: ', generated_text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INPUT WORD:  before\n",
            "DATASET PORTION: ['before', 'hatching', 'into', '<unk>', 'larvae', '.', 'Homarus', 'gammarus', 'is', 'a', 'highly', 'esteemed', 'food', ',', 'and', 'is', 'widely', 'caught', 'using', 'lobster']\n",
            "GENERATED TEXT:  that meanwhile and earn FOCA after d , the line started that the tenure , <unk> of overseas and several \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iQ_rCsR9s-rY",
        "colab_type": "text"
      },
      "source": [
        "# Transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMkAaJ93tvUd",
        "colab_type": "text"
      },
      "source": [
        "Define the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BrOzmm-Dvmc2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# from https://github.com/pytorch/examples/blob/master/word_language_model/model.py\n",
        "\n",
        "class PositionalEncoding(nn.Module):\n",
        "\n",
        "    def __init__(self, d_model, dropout=0.1, max_len=5000):\n",
        "        super().__init__()\n",
        "        self.dropout = nn.Dropout(p=dropout)\n",
        "\n",
        "        pe = torch.zeros(max_len, d_model)\n",
        "        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
        "        pe[:, 0::2] = torch.sin(position * div_term)\n",
        "        pe[:, 1::2] = torch.cos(position * div_term)\n",
        "        pe = pe.unsqueeze(0).transpose(0, 1)\n",
        "        self.register_buffer('pe', pe)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x + self.pe[:x.size(0), :]\n",
        "        return self.dropout(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VzVnYLVaeYQ_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TransformerLM(nn.Module):\n",
        "\n",
        "  def __init__(\n",
        "      self,\n",
        "      vocab_size,\n",
        "      n_layers=3,\n",
        "      n_heads=4,\n",
        "      d_model=256,\n",
        "      d_ff=512,\n",
        "      dropout=0.5,\n",
        "      activation='relu',\n",
        "  ):\n",
        "    super().__init__()\n",
        "    \n",
        "    self.embedding = nn.Embedding(vocab_size, d_model)\n",
        "    self.pos_encoder = PositionalEncoding(d_model, dropout)\n",
        "    encoder_layer = nn.TransformerEncoderLayer(d_model, n_heads, d_ff, dropout, activation)\n",
        "    encoder_norm = nn.LayerNorm(d_model)\n",
        "    self.encoder = nn.TransformerEncoder(encoder_layer, n_layers, encoder_norm)\n",
        "    self.output = nn.Linear(d_model, vocab_size)\n",
        "    self.output.weight = self.embedding.weight\n",
        "    self.log_softmax = nn.LogSoftmax(dim=-1)\n",
        "\n",
        "    self.init_weights()\n",
        "\n",
        "    self._d_model = d_model\n",
        "    self._mask = None\n",
        "    self._memory = None\n",
        "\n",
        "  def init_weights(self):\n",
        "    initrange = 0.1\n",
        "    nn.init.uniform_(self.embedding.weight, -initrange, initrange)\n",
        "    nn.init.zeros_(self.output.weight)\n",
        "    nn.init.uniform_(self.output.weight, -initrange, initrange)\n",
        "\n",
        "  def _generate_square_subsequent_mask(self, sz):\n",
        "    mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)\n",
        "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
        "    return mask\n",
        "\n",
        "  def forward(self, x):\n",
        "    if self._mask is None:\n",
        "      self._mask = self._generate_square_subsequent_mask(x.size(0)).to(x.device)\n",
        "\n",
        "    x = self.embedding(x) * math.sqrt(self._d_model)\n",
        "    x = self.pos_encoder(x)\n",
        "    x = self.encoder(x, self._mask)\n",
        "    x = self.output(x)\n",
        "    return self.log_softmax(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_An-Z_R0tpJx",
        "colab_type": "text"
      },
      "source": [
        "Prepare the data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NQpFpQkIeKi9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 128\n",
        "seq_len = 32\n",
        "\n",
        "field = data.Field()\n",
        "(train, val, test) = datasets.language_modeling.WikiText2.splits(text_field=field)\n",
        "field.build_vocab(train, max_size=None)\n",
        "(train, val, test) = data.BPTTIterator.splits(\n",
        "    (train, val, test), batch_size=batch_size, bptt_len=seq_len, repeat=True, shuffle=True\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FsEl28kBttNE",
        "colab_type": "text"
      },
      "source": [
        "Train the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mvn_cOjzuPSY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8b912780-776d-448a-ecc0-3564805e880b"
      },
      "source": [
        "model = TransformerLM(vocab_size=len(field.vocab)).cuda()\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "eval_it = iter(val)\n",
        "\n",
        "def run_on_batch(batch):\n",
        "  text = batch.text.cuda()\n",
        "  target = batch.target.cuda()\n",
        "  pred = model(text).permute(0, 2, 1)\n",
        "  return loss_fn(pred, target)\n",
        "\n",
        "for (i, batch) in tqdm.tqdm(enumerate(train)):\n",
        "  if batch.text.size() != (seq_len, batch_size):\n",
        "    # Skip an incomplete batch.\n",
        "    continue\n",
        "\n",
        "  optimizer.zero_grad()\n",
        "  loss = run_on_batch(batch)\n",
        "  loss.backward()\n",
        "  torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "  optimizer.step()\n",
        "\n",
        "  if i % (len(train) // 10) == 0:\n",
        "    print('train loss:', float(loss))\n",
        "    model.eval()\n",
        "    print('eval loss:', float(run_on_batch(next(eval_it))))\n",
        "    model.train()\n",
        "\n",
        "  if i == 3000:\n",
        "    break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "0it [00:00, ?it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 10.744112014770508\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1it [00:01,  1.19s/it]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "eval loss: 9.901606559753418\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "2it [00:01,  1.05it/s]\u001b[A\n",
            "3it [00:01,  1.27it/s]\u001b[A\n",
            "4it [00:02,  1.49it/s]\u001b[A\n",
            "5it [00:02,  1.70it/s]\u001b[A\n",
            "6it [00:03,  1.88it/s]\u001b[A\n",
            "7it [00:03,  2.03it/s]\u001b[A\n",
            "8it [00:03,  2.15it/s]\u001b[A\n",
            "9it [00:04,  2.25it/s]\u001b[A\n",
            "10it [00:04,  2.32it/s]\u001b[A\n",
            "11it [00:05,  2.38it/s]\u001b[A\n",
            "12it [00:05,  2.42it/s]\u001b[A\n",
            "13it [00:05,  2.46it/s]\u001b[A\n",
            "14it [00:06,  2.48it/s]\u001b[A\n",
            "15it [00:06,  2.51it/s]\u001b[A\n",
            "16it [00:07,  2.53it/s]\u001b[A\n",
            "17it [00:07,  2.54it/s]\u001b[A\n",
            "18it [00:07,  2.54it/s]\u001b[A\n",
            "19it [00:08,  2.54it/s]\u001b[A\n",
            "20it [00:08,  2.54it/s]\u001b[A\n",
            "21it [00:09,  2.54it/s]\u001b[A\n",
            "22it [00:09,  2.54it/s]\u001b[A\n",
            "23it [00:09,  2.53it/s]\u001b[A\n",
            "24it [00:10,  2.52it/s]\u001b[A\n",
            "25it [00:10,  2.52it/s]\u001b[A\n",
            "26it [00:11,  2.52it/s]\u001b[A\n",
            "27it [00:11,  2.54it/s]\u001b[A\n",
            "28it [00:11,  2.55it/s]\u001b[A\n",
            "29it [00:12,  2.55it/s]\u001b[A\n",
            "30it [00:12,  2.55it/s]\u001b[A\n",
            "31it [00:13,  2.55it/s]\u001b[A\n",
            "32it [00:13,  2.53it/s]\u001b[A\n",
            "33it [00:13,  2.53it/s]\u001b[A\n",
            "34it [00:14,  2.53it/s]\u001b[A\n",
            "35it [00:14,  2.52it/s]\u001b[A\n",
            "36it [00:15,  2.53it/s]\u001b[A\n",
            "37it [00:15,  2.53it/s]\u001b[A\n",
            "38it [00:15,  2.53it/s]\u001b[A\n",
            "39it [00:16,  2.54it/s]\u001b[A\n",
            "40it [00:16,  2.55it/s]\u001b[A\n",
            "41it [00:16,  2.54it/s]\u001b[A\n",
            "42it [00:17,  2.55it/s]\u001b[A\n",
            "43it [00:17,  2.55it/s]\u001b[A\n",
            "44it [00:18,  2.56it/s]\u001b[A\n",
            "45it [00:18,  2.55it/s]\u001b[A\n",
            "46it [00:18,  2.54it/s]\u001b[A\n",
            "47it [00:19,  2.54it/s]\u001b[A\n",
            "48it [00:19,  2.53it/s]\u001b[A\n",
            "49it [00:20,  2.53it/s]\u001b[A\n",
            "50it [00:20,  2.53it/s]\u001b[A\n",
            "51it [00:20,  2.54it/s]\u001b[A\n",
            "52it [00:21,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 7.180344104766846\n",
            "eval loss: 6.867320537567139\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "53it [00:21,  2.34it/s]\u001b[A\n",
            "54it [00:22,  2.39it/s]\u001b[A\n",
            "55it [00:22,  2.43it/s]\u001b[A\n",
            "56it [00:23,  2.46it/s]\u001b[A\n",
            "57it [00:23,  2.48it/s]\u001b[A\n",
            "58it [00:23,  2.49it/s]\u001b[A\n",
            "59it [00:24,  2.50it/s]\u001b[A\n",
            "60it [00:24,  2.50it/s]\u001b[A\n",
            "61it [00:25,  2.51it/s]\u001b[A\n",
            "62it [00:25,  2.52it/s]\u001b[A\n",
            "63it [00:25,  2.53it/s]\u001b[A\n",
            "64it [00:26,  2.54it/s]\u001b[A\n",
            "65it [00:26,  2.55it/s]\u001b[A\n",
            "66it [00:27,  2.55it/s]\u001b[A\n",
            "67it [00:27,  2.54it/s]\u001b[A\n",
            "68it [00:27,  2.53it/s]\u001b[A\n",
            "69it [00:28,  2.53it/s]\u001b[A\n",
            "70it [00:28,  2.53it/s]\u001b[A\n",
            "71it [00:28,  2.53it/s]\u001b[A\n",
            "72it [00:29,  2.52it/s]\u001b[A\n",
            "73it [00:29,  2.52it/s]\u001b[A\n",
            "74it [00:30,  2.51it/s]\u001b[A\n",
            "75it [00:30,  2.51it/s]\u001b[A\n",
            "76it [00:30,  2.51it/s]\u001b[A\n",
            "77it [00:31,  2.52it/s]\u001b[A\n",
            "78it [00:31,  2.53it/s]\u001b[A\n",
            "79it [00:32,  2.54it/s]\u001b[A\n",
            "80it [00:32,  2.55it/s]\u001b[A\n",
            "81it [00:32,  2.55it/s]\u001b[A\n",
            "82it [00:33,  2.55it/s]\u001b[A\n",
            "83it [00:33,  2.55it/s]\u001b[A\n",
            "84it [00:34,  2.54it/s]\u001b[A\n",
            "85it [00:34,  2.54it/s]\u001b[A\n",
            "86it [00:34,  2.55it/s]\u001b[A\n",
            "87it [00:35,  2.53it/s]\u001b[A\n",
            "88it [00:35,  2.52it/s]\u001b[A\n",
            "89it [00:36,  2.51it/s]\u001b[A\n",
            "90it [00:36,  2.51it/s]\u001b[A\n",
            "91it [00:36,  2.51it/s]\u001b[A\n",
            "92it [00:37,  2.51it/s]\u001b[A\n",
            "93it [00:37,  2.52it/s]\u001b[A\n",
            "94it [00:38,  2.52it/s]\u001b[A\n",
            "95it [00:38,  2.53it/s]\u001b[A\n",
            "96it [00:38,  2.54it/s]\u001b[A\n",
            "97it [00:39,  2.54it/s]\u001b[A\n",
            "98it [00:39,  2.54it/s]\u001b[A\n",
            "99it [00:40,  2.55it/s]\u001b[A\n",
            "100it [00:40,  2.53it/s]\u001b[A\n",
            "101it [00:40,  2.52it/s]\u001b[A\n",
            "102it [00:41,  2.52it/s]\u001b[A\n",
            "103it [00:41,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.884403228759766\n",
            "eval loss: 6.732758045196533\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "104it [00:42,  2.32it/s]\u001b[A\n",
            "105it [00:42,  2.39it/s]\u001b[A\n",
            "106it [00:42,  2.44it/s]\u001b[A\n",
            "107it [00:43,  2.47it/s]\u001b[A\n",
            "108it [00:43,  2.48it/s]\u001b[A\n",
            "109it [00:44,  2.49it/s]\u001b[A\n",
            "110it [00:44,  2.50it/s]\u001b[A\n",
            "111it [00:44,  2.50it/s]\u001b[A\n",
            "112it [00:45,  2.51it/s]\u001b[A\n",
            "113it [00:45,  2.52it/s]\u001b[A\n",
            "114it [00:46,  2.52it/s]\u001b[A\n",
            "115it [00:46,  2.53it/s]\u001b[A\n",
            "116it [00:46,  2.52it/s]\u001b[A\n",
            "117it [00:47,  2.52it/s]\u001b[A\n",
            "118it [00:47,  2.52it/s]\u001b[A\n",
            "119it [00:48,  2.53it/s]\u001b[A\n",
            "120it [00:48,  2.54it/s]\u001b[A\n",
            "121it [00:48,  2.55it/s]\u001b[A\n",
            "122it [00:49,  2.55it/s]\u001b[A\n",
            "123it [00:49,  2.55it/s]\u001b[A\n",
            "124it [00:50,  2.55it/s]\u001b[A\n",
            "125it [00:50,  2.54it/s]\u001b[A\n",
            "126it [00:50,  2.54it/s]\u001b[A\n",
            "127it [00:51,  2.53it/s]\u001b[A\n",
            "128it [00:51,  2.52it/s]\u001b[A\n",
            "129it [00:52,  2.51it/s]\u001b[A\n",
            "130it [00:52,  2.52it/s]\u001b[A\n",
            "131it [00:52,  2.52it/s]\u001b[A\n",
            "132it [00:53,  2.52it/s]\u001b[A\n",
            "133it [00:53,  2.52it/s]\u001b[A\n",
            "134it [00:54,  2.53it/s]\u001b[A\n",
            "135it [00:54,  2.54it/s]\u001b[A\n",
            "136it [00:54,  2.54it/s]\u001b[A\n",
            "137it [00:55,  2.55it/s]\u001b[A\n",
            "138it [00:55,  2.55it/s]\u001b[A\n",
            "139it [00:56,  2.53it/s]\u001b[A\n",
            "140it [00:56,  2.53it/s]\u001b[A\n",
            "141it [00:56,  2.53it/s]\u001b[A\n",
            "142it [00:57,  2.53it/s]\u001b[A\n",
            "143it [00:57,  2.53it/s]\u001b[A\n",
            "144it [00:58,  2.53it/s]\u001b[A\n",
            "145it [00:58,  2.52it/s]\u001b[A\n",
            "146it [00:58,  2.52it/s]\u001b[A\n",
            "147it [00:59,  2.53it/s]\u001b[A\n",
            "148it [00:59,  2.52it/s]\u001b[A\n",
            "149it [00:59,  2.53it/s]\u001b[A\n",
            "150it [01:00,  2.54it/s]\u001b[A\n",
            "151it [01:00,  2.55it/s]\u001b[A\n",
            "152it [01:01,  2.55it/s]\u001b[A\n",
            "153it [01:01,  2.55it/s]\u001b[A\n",
            "154it [01:02,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.691217422485352\n",
            "eval loss: 6.485075950622559\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "155it [01:02,  2.33it/s]\u001b[A\n",
            "156it [01:02,  2.39it/s]\u001b[A\n",
            "157it [01:03,  2.44it/s]\u001b[A\n",
            "158it [01:03,  2.48it/s]\u001b[A\n",
            "159it [01:04,  2.50it/s]\u001b[A\n",
            "160it [01:04,  2.51it/s]\u001b[A\n",
            "161it [01:04,  2.52it/s]\u001b[A\n",
            "162it [01:05,  2.52it/s]\u001b[A\n",
            "163it [01:05,  2.52it/s]\u001b[A\n",
            "164it [01:06,  2.52it/s]\u001b[A\n",
            "165it [01:06,  2.52it/s]\u001b[A\n",
            "166it [01:06,  2.52it/s]\u001b[A\n",
            "167it [01:07,  2.52it/s]\u001b[A\n",
            "168it [01:07,  2.51it/s]\u001b[A\n",
            "169it [01:08,  2.52it/s]\u001b[A\n",
            "170it [01:08,  2.53it/s]\u001b[A\n",
            "171it [01:08,  2.54it/s]\u001b[A\n",
            "172it [01:09,  2.54it/s]\u001b[A\n",
            "173it [01:09,  2.55it/s]\u001b[A\n",
            "174it [01:10,  2.54it/s]\u001b[A\n",
            "175it [01:10,  2.54it/s]\u001b[A\n",
            "176it [01:10,  2.53it/s]\u001b[A\n",
            "177it [01:11,  2.52it/s]\u001b[A\n",
            "178it [01:11,  2.51it/s]\u001b[A\n",
            "179it [01:11,  2.51it/s]\u001b[A\n",
            "180it [01:12,  2.51it/s]\u001b[A\n",
            "181it [01:12,  2.51it/s]\u001b[A\n",
            "182it [01:13,  2.49it/s]\u001b[A\n",
            "183it [01:13,  2.50it/s]\u001b[A\n",
            "184it [01:13,  2.51it/s]\u001b[A\n",
            "185it [01:14,  2.52it/s]\u001b[A\n",
            "186it [01:14,  2.52it/s]\u001b[A\n",
            "187it [01:15,  2.52it/s]\u001b[A\n",
            "188it [01:15,  2.53it/s]\u001b[A\n",
            "189it [01:15,  2.53it/s]\u001b[A\n",
            "190it [01:16,  2.53it/s]\u001b[A\n",
            "191it [01:16,  2.54it/s]\u001b[A\n",
            "192it [01:17,  2.55it/s]\u001b[A\n",
            "193it [01:17,  2.54it/s]\u001b[A\n",
            "194it [01:17,  2.54it/s]\u001b[A\n",
            "195it [01:18,  2.53it/s]\u001b[A\n",
            "196it [01:18,  2.52it/s]\u001b[A\n",
            "197it [01:19,  2.53it/s]\u001b[A\n",
            "198it [01:19,  2.52it/s]\u001b[A\n",
            "199it [01:19,  2.52it/s]\u001b[A\n",
            "200it [01:20,  2.52it/s]\u001b[A\n",
            "201it [01:20,  2.52it/s]\u001b[A\n",
            "202it [01:21,  2.51it/s]\u001b[A\n",
            "203it [01:21,  2.50it/s]\u001b[A\n",
            "204it [01:21,  2.51it/s]\u001b[A\n",
            "205it [01:22,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.525059700012207\n",
            "eval loss: 6.398266315460205\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "206it [01:22,  2.31it/s]\u001b[A\n",
            "207it [01:23,  2.38it/s]\u001b[A\n",
            "208it [01:23,  2.43it/s]\u001b[A\n",
            "209it [01:24,  2.45it/s]\u001b[A\n",
            "210it [01:24,  2.46it/s]\u001b[A\n",
            "211it [01:24,  2.48it/s]\u001b[A\n",
            "212it [01:25,  2.50it/s]\u001b[A\n",
            "213it [01:25,  2.51it/s]\u001b[A\n",
            "214it [01:26,  2.51it/s]\u001b[A\n",
            "215it [01:26,  2.51it/s]\u001b[A\n",
            "216it [01:26,  2.51it/s]\u001b[A\n",
            "217it [01:27,  2.51it/s]\u001b[A\n",
            "218it [01:27,  2.51it/s]\u001b[A\n",
            "219it [01:28,  2.51it/s]\u001b[A\n",
            "220it [01:28,  2.51it/s]\u001b[A\n",
            "221it [01:28,  2.50it/s]\u001b[A\n",
            "222it [01:29,  2.49it/s]\u001b[A\n",
            "223it [01:29,  2.51it/s]\u001b[A\n",
            "224it [01:30,  2.52it/s]\u001b[A\n",
            "225it [01:30,  2.53it/s]\u001b[A\n",
            "226it [01:30,  2.54it/s]\u001b[A\n",
            "227it [01:31,  2.54it/s]\u001b[A\n",
            "228it [01:31,  2.55it/s]\u001b[A\n",
            "229it [01:31,  2.54it/s]\u001b[A\n",
            "230it [01:32,  2.53it/s]\u001b[A\n",
            "231it [01:32,  2.52it/s]\u001b[A\n",
            "232it [01:33,  2.51it/s]\u001b[A\n",
            "233it [01:33,  2.50it/s]\u001b[A\n",
            "234it [01:33,  2.50it/s]\u001b[A\n",
            "235it [01:34,  2.51it/s]\u001b[A\n",
            "236it [01:34,  2.52it/s]\u001b[A\n",
            "237it [01:35,  2.52it/s]\u001b[A\n",
            "238it [01:35,  2.52it/s]\u001b[A\n",
            "239it [01:35,  2.52it/s]\u001b[A\n",
            "240it [01:36,  2.52it/s]\u001b[A\n",
            "241it [01:36,  2.52it/s]\u001b[A\n",
            "242it [01:37,  2.53it/s]\u001b[A\n",
            "243it [01:37,  2.53it/s]\u001b[A\n",
            "244it [01:37,  2.55it/s]\u001b[A\n",
            "245it [01:38,  2.54it/s]\u001b[A\n",
            "246it [01:38,  2.54it/s]\u001b[A\n",
            "247it [01:39,  2.54it/s]\u001b[A\n",
            "248it [01:39,  2.54it/s]\u001b[A\n",
            "249it [01:39,  2.54it/s]\u001b[A\n",
            "250it [01:40,  2.52it/s]\u001b[A\n",
            "251it [01:40,  2.52it/s]\u001b[A\n",
            "252it [01:41,  2.53it/s]\u001b[A\n",
            "253it [01:41,  2.52it/s]\u001b[A\n",
            "254it [01:41,  2.52it/s]\u001b[A\n",
            "255it [01:42,  2.51it/s]\u001b[A\n",
            "256it [01:42,  2.23it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.504654407501221\n",
            "eval loss: 6.262438774108887\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "257it [01:43,  2.31it/s]\u001b[A\n",
            "258it [01:43,  2.38it/s]\u001b[A\n",
            "259it [01:44,  2.43it/s]\u001b[A\n",
            "260it [01:44,  2.44it/s]\u001b[A\n",
            "261it [01:44,  2.47it/s]\u001b[A\n",
            "262it [01:45,  2.48it/s]\u001b[A\n",
            "263it [01:45,  2.50it/s]\u001b[A\n",
            "264it [01:46,  2.51it/s]\u001b[A\n",
            "265it [01:46,  2.51it/s]\u001b[A\n",
            "266it [01:46,  2.52it/s]\u001b[A\n",
            "267it [01:47,  2.52it/s]\u001b[A\n",
            "268it [01:47,  2.53it/s]\u001b[A\n",
            "269it [01:47,  2.53it/s]\u001b[A\n",
            "270it [01:48,  2.54it/s]\u001b[A\n",
            "271it [01:48,  2.54it/s]\u001b[A\n",
            "272it [01:49,  2.54it/s]\u001b[A\n",
            "273it [01:49,  2.54it/s]\u001b[A\n",
            "274it [01:49,  2.54it/s]\u001b[A\n",
            "275it [01:50,  2.54it/s]\u001b[A\n",
            "276it [01:50,  2.52it/s]\u001b[A\n",
            "277it [01:51,  2.52it/s]\u001b[A\n",
            "278it [01:51,  2.52it/s]\u001b[A\n",
            "279it [01:51,  2.53it/s]\u001b[A\n",
            "280it [01:52,  2.53it/s]\u001b[A\n",
            "281it [01:52,  2.52it/s]\u001b[A\n",
            "282it [01:53,  2.52it/s]\u001b[A\n",
            "283it [01:53,  2.54it/s]\u001b[A\n",
            "284it [01:53,  2.54it/s]\u001b[A\n",
            "285it [01:54,  2.55it/s]\u001b[A\n",
            "286it [01:54,  2.55it/s]\u001b[A\n",
            "287it [01:55,  2.56it/s]\u001b[A\n",
            "288it [01:55,  2.54it/s]\u001b[A\n",
            "289it [01:55,  2.53it/s]\u001b[A\n",
            "290it [01:56,  2.52it/s]\u001b[A\n",
            "291it [01:56,  2.52it/s]\u001b[A\n",
            "292it [01:57,  2.52it/s]\u001b[A\n",
            "293it [01:57,  2.52it/s]\u001b[A\n",
            "294it [01:57,  2.51it/s]\u001b[A\n",
            "295it [01:58,  2.49it/s]\u001b[A\n",
            "296it [01:58,  2.49it/s]\u001b[A\n",
            "297it [01:59,  2.50it/s]\u001b[A\n",
            "298it [01:59,  2.51it/s]\u001b[A\n",
            "299it [01:59,  2.52it/s]\u001b[A\n",
            "300it [02:00,  2.52it/s]\u001b[A\n",
            "301it [02:00,  2.52it/s]\u001b[A\n",
            "302it [02:01,  2.53it/s]\u001b[A\n",
            "303it [02:01,  2.54it/s]\u001b[A\n",
            "304it [02:01,  2.54it/s]\u001b[A\n",
            "305it [02:02,  2.55it/s]\u001b[A\n",
            "306it [02:02,  2.54it/s]\u001b[A\n",
            "307it [02:03,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.48200798034668\n",
            "eval loss: 6.059976100921631\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "308it [02:03,  2.32it/s]\u001b[A\n",
            "309it [02:03,  2.37it/s]\u001b[A\n",
            "310it [02:04,  2.41it/s]\u001b[A\n",
            "311it [02:04,  2.44it/s]\u001b[A\n",
            "312it [02:05,  2.45it/s]\u001b[A\n",
            "313it [02:05,  2.46it/s]\u001b[A\n",
            "314it [02:05,  2.49it/s]\u001b[A\n",
            "315it [02:06,  2.51it/s]\u001b[A\n",
            "316it [02:06,  2.51it/s]\u001b[A\n",
            "317it [02:07,  2.53it/s]\u001b[A\n",
            "318it [02:07,  2.54it/s]\u001b[A\n",
            "319it [02:07,  2.54it/s]\u001b[A\n",
            "320it [02:08,  2.53it/s]\u001b[A\n",
            "321it [02:08,  2.52it/s]\u001b[A\n",
            "322it [02:09,  2.52it/s]\u001b[A\n",
            "323it [02:09,  2.52it/s]\u001b[A\n",
            "324it [02:09,  2.52it/s]\u001b[A\n",
            "325it [02:10,  2.52it/s]\u001b[A\n",
            "326it [02:10,  2.53it/s]\u001b[A\n",
            "327it [02:11,  2.53it/s]\u001b[A\n",
            "328it [02:11,  2.52it/s]\u001b[A\n",
            "329it [02:11,  2.53it/s]\u001b[A\n",
            "330it [02:12,  2.53it/s]\u001b[A\n",
            "331it [02:12,  2.54it/s]\u001b[A\n",
            "332it [02:13,  2.54it/s]\u001b[A\n",
            "333it [02:13,  2.54it/s]\u001b[A\n",
            "334it [02:13,  2.55it/s]\u001b[A\n",
            "335it [02:14,  2.53it/s]\u001b[A\n",
            "336it [02:14,  2.52it/s]\u001b[A\n",
            "337it [02:15,  2.52it/s]\u001b[A\n",
            "338it [02:15,  2.53it/s]\u001b[A\n",
            "339it [02:15,  2.52it/s]\u001b[A\n",
            "340it [02:16,  2.51it/s]\u001b[A\n",
            "341it [02:16,  2.52it/s]\u001b[A\n",
            "342it [02:17,  2.53it/s]\u001b[A\n",
            "343it [02:17,  2.52it/s]\u001b[A\n",
            "344it [02:17,  2.52it/s]\u001b[A\n",
            "345it [02:18,  2.52it/s]\u001b[A\n",
            "346it [02:18,  2.53it/s]\u001b[A\n",
            "347it [02:19,  2.53it/s]\u001b[A\n",
            "348it [02:19,  2.54it/s]\u001b[A\n",
            "349it [02:19,  2.55it/s]\u001b[A\n",
            "350it [02:20,  2.56it/s]\u001b[A\n",
            "351it [02:20,  2.55it/s]\u001b[A\n",
            "352it [02:20,  2.54it/s]\u001b[A\n",
            "353it [02:21,  2.54it/s]\u001b[A\n",
            "354it [02:21,  2.54it/s]\u001b[A\n",
            "355it [02:22,  2.54it/s]\u001b[A\n",
            "356it [02:22,  2.54it/s]\u001b[A\n",
            "357it [02:22,  2.55it/s]\u001b[A\n",
            "358it [02:23,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.279682159423828\n",
            "eval loss: 5.93257474899292\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "359it [02:23,  2.33it/s]\u001b[A\n",
            "360it [02:24,  2.39it/s]\u001b[A\n",
            "361it [02:24,  2.43it/s]\u001b[A\n",
            "362it [02:25,  2.46it/s]\u001b[A\n",
            "363it [02:25,  2.47it/s]\u001b[A\n",
            "364it [02:25,  2.49it/s]\u001b[A\n",
            "365it [02:26,  2.49it/s]\u001b[A\n",
            "366it [02:26,  2.51it/s]\u001b[A\n",
            "367it [02:27,  2.51it/s]\u001b[A\n",
            "368it [02:27,  2.50it/s]\u001b[A\n",
            "369it [02:27,  2.51it/s]\u001b[A\n",
            "370it [02:28,  2.50it/s]\u001b[A\n",
            "371it [02:28,  2.52it/s]\u001b[A\n",
            "372it [02:29,  2.53it/s]\u001b[A\n",
            "373it [02:29,  2.54it/s]\u001b[A\n",
            "374it [02:29,  2.54it/s]\u001b[A\n",
            "375it [02:30,  2.54it/s]\u001b[A\n",
            "376it [02:30,  2.54it/s]\u001b[A\n",
            "377it [02:31,  2.52it/s]\u001b[A\n",
            "378it [02:31,  2.51it/s]\u001b[A\n",
            "379it [02:31,  2.51it/s]\u001b[A\n",
            "380it [02:32,  2.51it/s]\u001b[A\n",
            "381it [02:32,  2.51it/s]\u001b[A\n",
            "382it [02:33,  2.51it/s]\u001b[A\n",
            "383it [02:33,  2.51it/s]\u001b[A\n",
            "384it [02:33,  2.50it/s]\u001b[A\n",
            "385it [02:34,  2.51it/s]\u001b[A\n",
            "386it [02:34,  2.52it/s]\u001b[A\n",
            "387it [02:35,  2.52it/s]\u001b[A\n",
            "388it [02:35,  2.53it/s]\u001b[A\n",
            "389it [02:35,  2.54it/s]\u001b[A\n",
            "390it [02:36,  2.54it/s]\u001b[A\n",
            "391it [02:36,  2.55it/s]\u001b[A\n",
            "392it [02:36,  2.55it/s]\u001b[A\n",
            "393it [02:37,  2.55it/s]\u001b[A\n",
            "394it [02:37,  2.53it/s]\u001b[A\n",
            "395it [02:38,  2.51it/s]\u001b[A\n",
            "396it [02:38,  2.52it/s]\u001b[A\n",
            "397it [02:38,  2.52it/s]\u001b[A\n",
            "398it [02:39,  2.52it/s]\u001b[A\n",
            "399it [02:39,  2.52it/s]\u001b[A\n",
            "400it [02:40,  2.52it/s]\u001b[A\n",
            "401it [02:40,  2.52it/s]\u001b[A\n",
            "402it [02:40,  2.52it/s]\u001b[A\n",
            "403it [02:41,  2.52it/s]\u001b[A\n",
            "404it [02:41,  2.52it/s]\u001b[A\n",
            "405it [02:42,  2.53it/s]\u001b[A\n",
            "406it [02:42,  2.53it/s]\u001b[A\n",
            "407it [02:42,  2.54it/s]\u001b[A\n",
            "408it [02:43,  2.54it/s]\u001b[A\n",
            "409it [02:43,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.24365234375\n",
            "eval loss: 6.002635955810547\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "410it [02:44,  2.32it/s]\u001b[A\n",
            "411it [02:44,  2.38it/s]\u001b[A\n",
            "412it [02:45,  2.41it/s]\u001b[A\n",
            "413it [02:45,  2.44it/s]\u001b[A\n",
            "414it [02:45,  2.46it/s]\u001b[A\n",
            "415it [02:46,  2.47it/s]\u001b[A\n",
            "416it [02:46,  2.50it/s]\u001b[A\n",
            "417it [02:47,  2.51it/s]\u001b[A\n",
            "418it [02:47,  2.52it/s]\u001b[A\n",
            "419it [02:47,  2.53it/s]\u001b[A\n",
            "420it [02:48,  2.53it/s]\u001b[A\n",
            "421it [02:48,  2.53it/s]\u001b[A\n",
            "422it [02:49,  2.54it/s]\u001b[A\n",
            "423it [02:49,  2.54it/s]\u001b[A\n",
            "424it [02:49,  2.54it/s]\u001b[A\n",
            "425it [02:50,  2.54it/s]\u001b[A\n",
            "426it [02:50,  2.53it/s]\u001b[A\n",
            "427it [02:50,  2.53it/s]\u001b[A\n",
            "428it [02:51,  2.53it/s]\u001b[A\n",
            "429it [02:51,  2.52it/s]\u001b[A\n",
            "430it [02:52,  2.51it/s]\u001b[A\n",
            "431it [02:52,  2.50it/s]\u001b[A\n",
            "432it [02:52,  2.51it/s]\u001b[A\n",
            "433it [02:53,  2.51it/s]\u001b[A\n",
            "434it [02:53,  2.52it/s]\u001b[A\n",
            "435it [02:54,  2.53it/s]\u001b[A\n",
            "436it [02:54,  2.55it/s]\u001b[A\n",
            "437it [02:54,  2.55it/s]\u001b[A\n",
            "438it [02:55,  2.54it/s]\u001b[A\n",
            "439it [02:55,  2.52it/s]\u001b[A\n",
            "440it [02:56,  2.53it/s]\u001b[A\n",
            "441it [02:56,  2.53it/s]\u001b[A\n",
            "442it [02:56,  2.52it/s]\u001b[A\n",
            "443it [02:57,  2.53it/s]\u001b[A\n",
            "444it [02:57,  2.52it/s]\u001b[A\n",
            "445it [02:58,  2.53it/s]\u001b[A\n",
            "446it [02:58,  2.52it/s]\u001b[A\n",
            "447it [02:58,  2.53it/s]\u001b[A\n",
            "448it [02:59,  2.54it/s]\u001b[A\n",
            "449it [02:59,  2.54it/s]\u001b[A\n",
            "450it [03:00,  2.55it/s]\u001b[A\n",
            "451it [03:00,  2.56it/s]\u001b[A\n",
            "452it [03:00,  2.55it/s]\u001b[A\n",
            "453it [03:01,  2.54it/s]\u001b[A\n",
            "454it [03:01,  2.54it/s]\u001b[A\n",
            "455it [03:02,  2.53it/s]\u001b[A\n",
            "456it [03:02,  2.53it/s]\u001b[A\n",
            "457it [03:02,  2.52it/s]\u001b[A\n",
            "458it [03:03,  2.53it/s]\u001b[A\n",
            "459it [03:03,  2.53it/s]\u001b[A\n",
            "460it [03:04,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.204072952270508\n",
            "eval loss: 5.843438625335693\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "461it [03:04,  2.33it/s]\u001b[A\n",
            "462it [03:04,  2.39it/s]\u001b[A\n",
            "463it [03:05,  2.42it/s]\u001b[A\n",
            "464it [03:05,  2.45it/s]\u001b[A\n",
            "465it [03:06,  2.47it/s]\u001b[A\n",
            "466it [03:06,  2.48it/s]\u001b[A\n",
            "467it [03:06,  2.50it/s]\u001b[A\n",
            "468it [03:07,  2.50it/s]\u001b[A\n",
            "469it [03:07,  2.50it/s]\u001b[A\n",
            "470it [03:08,  2.51it/s]\u001b[A\n",
            "471it [03:08,  2.52it/s]\u001b[A\n",
            "472it [03:08,  2.53it/s]\u001b[A\n",
            "473it [03:09,  2.53it/s]\u001b[A\n",
            "474it [03:09,  2.54it/s]\u001b[A\n",
            "475it [03:10,  2.55it/s]\u001b[A\n",
            "476it [03:10,  2.54it/s]\u001b[A\n",
            "477it [03:10,  2.54it/s]\u001b[A\n",
            "478it [03:11,  2.53it/s]\u001b[A\n",
            "479it [03:11,  2.53it/s]\u001b[A\n",
            "480it [03:12,  2.52it/s]\u001b[A\n",
            "481it [03:12,  2.52it/s]\u001b[A\n",
            "482it [03:12,  2.52it/s]\u001b[A\n",
            "483it [03:13,  2.53it/s]\u001b[A\n",
            "484it [03:13,  2.52it/s]\u001b[A\n",
            "485it [03:14,  2.52it/s]\u001b[A\n",
            "486it [03:14,  2.53it/s]\u001b[A\n",
            "487it [03:14,  2.53it/s]\u001b[A\n",
            "488it [03:15,  2.52it/s]\u001b[A\n",
            "489it [03:15,  2.53it/s]\u001b[A\n",
            "490it [03:16,  2.54it/s]\u001b[A\n",
            "491it [03:16,  2.54it/s]\u001b[A\n",
            "492it [03:16,  2.54it/s]\u001b[A\n",
            "493it [03:17,  2.54it/s]\u001b[A\n",
            "494it [03:17,  2.54it/s]\u001b[A\n",
            "495it [03:18,  2.54it/s]\u001b[A\n",
            "496it [03:18,  2.53it/s]\u001b[A\n",
            "497it [03:18,  2.53it/s]\u001b[A\n",
            "498it [03:19,  2.52it/s]\u001b[A\n",
            "499it [03:19,  2.52it/s]\u001b[A\n",
            "500it [03:20,  2.52it/s]\u001b[A\n",
            "501it [03:20,  2.52it/s]\u001b[A\n",
            "502it [03:20,  2.52it/s]\u001b[A\n",
            "503it [03:21,  2.52it/s]\u001b[A\n",
            "504it [03:21,  2.52it/s]\u001b[A\n",
            "505it [03:22,  2.52it/s]\u001b[A\n",
            "506it [03:22,  2.53it/s]\u001b[A\n",
            "507it [03:22,  2.54it/s]\u001b[A\n",
            "508it [03:23,  2.53it/s]\u001b[A\n",
            "509it [03:23,  2.53it/s]\u001b[A\n",
            "511it [03:24,  2.77it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.03971004486084\n",
            "eval loss: 5.802977085113525\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "512it [03:24,  2.68it/s]\u001b[A\n",
            "513it [03:24,  2.63it/s]\u001b[A\n",
            "514it [03:25,  2.59it/s]\u001b[A\n",
            "515it [03:25,  2.58it/s]\u001b[A\n",
            "516it [03:26,  2.57it/s]\u001b[A\n",
            "517it [03:26,  2.56it/s]\u001b[A\n",
            "518it [03:26,  2.55it/s]\u001b[A\n",
            "519it [03:27,  2.55it/s]\u001b[A\n",
            "520it [03:27,  2.55it/s]\u001b[A\n",
            "521it [03:28,  2.53it/s]\u001b[A\n",
            "522it [03:28,  2.53it/s]\u001b[A\n",
            "523it [03:28,  2.53it/s]\u001b[A\n",
            "524it [03:29,  2.52it/s]\u001b[A\n",
            "525it [03:29,  2.52it/s]\u001b[A\n",
            "526it [03:30,  2.52it/s]\u001b[A\n",
            "527it [03:30,  2.53it/s]\u001b[A\n",
            "528it [03:30,  2.53it/s]\u001b[A\n",
            "529it [03:31,  2.52it/s]\u001b[A\n",
            "530it [03:31,  2.52it/s]\u001b[A\n",
            "531it [03:32,  2.51it/s]\u001b[A\n",
            "532it [03:32,  2.52it/s]\u001b[A\n",
            "533it [03:32,  2.52it/s]\u001b[A\n",
            "534it [03:33,  2.53it/s]\u001b[A\n",
            "535it [03:33,  2.54it/s]\u001b[A\n",
            "536it [03:34,  2.54it/s]\u001b[A\n",
            "537it [03:34,  2.55it/s]\u001b[A\n",
            "538it [03:34,  2.54it/s]\u001b[A\n",
            "539it [03:35,  2.54it/s]\u001b[A\n",
            "540it [03:35,  2.54it/s]\u001b[A\n",
            "541it [03:36,  2.53it/s]\u001b[A\n",
            "542it [03:36,  2.53it/s]\u001b[A\n",
            "543it [03:36,  2.53it/s]\u001b[A\n",
            "544it [03:37,  2.53it/s]\u001b[A\n",
            "545it [03:37,  2.53it/s]\u001b[A\n",
            "546it [03:37,  2.53it/s]\u001b[A\n",
            "547it [03:38,  2.54it/s]\u001b[A\n",
            "548it [03:38,  2.55it/s]\u001b[A\n",
            "549it [03:39,  2.55it/s]\u001b[A\n",
            "550it [03:39,  2.55it/s]\u001b[A\n",
            "551it [03:39,  2.53it/s]\u001b[A\n",
            "552it [03:40,  2.53it/s]\u001b[A\n",
            "553it [03:40,  2.53it/s]\u001b[A\n",
            "554it [03:41,  2.52it/s]\u001b[A\n",
            "555it [03:41,  2.53it/s]\u001b[A\n",
            "556it [03:41,  2.53it/s]\u001b[A\n",
            "557it [03:42,  2.53it/s]\u001b[A\n",
            "558it [03:42,  2.52it/s]\u001b[A\n",
            "559it [03:43,  2.53it/s]\u001b[A\n",
            "560it [03:43,  2.52it/s]\u001b[A\n",
            "561it [03:43,  2.51it/s]\u001b[A\n",
            "562it [03:44,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.087571620941162\n",
            "eval loss: 5.901940822601318\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "563it [03:44,  2.32it/s]\u001b[A\n",
            "564it [03:45,  2.38it/s]\u001b[A\n",
            "565it [03:45,  2.41it/s]\u001b[A\n",
            "566it [03:46,  2.45it/s]\u001b[A\n",
            "567it [03:46,  2.46it/s]\u001b[A\n",
            "568it [03:46,  2.48it/s]\u001b[A\n",
            "569it [03:47,  2.49it/s]\u001b[A\n",
            "570it [03:47,  2.50it/s]\u001b[A\n",
            "571it [03:48,  2.51it/s]\u001b[A\n",
            "572it [03:48,  2.51it/s]\u001b[A\n",
            "573it [03:48,  2.51it/s]\u001b[A\n",
            "574it [03:49,  2.53it/s]\u001b[A\n",
            "575it [03:49,  2.53it/s]\u001b[A\n",
            "576it [03:50,  2.54it/s]\u001b[A\n",
            "577it [03:50,  2.54it/s]\u001b[A\n",
            "578it [03:50,  2.54it/s]\u001b[A\n",
            "579it [03:51,  2.54it/s]\u001b[A\n",
            "580it [03:51,  2.54it/s]\u001b[A\n",
            "581it [03:51,  2.54it/s]\u001b[A\n",
            "582it [03:52,  2.53it/s]\u001b[A\n",
            "583it [03:52,  2.53it/s]\u001b[A\n",
            "584it [03:53,  2.53it/s]\u001b[A\n",
            "585it [03:53,  2.52it/s]\u001b[A\n",
            "586it [03:53,  2.52it/s]\u001b[A\n",
            "587it [03:54,  2.51it/s]\u001b[A\n",
            "588it [03:54,  2.51it/s]\u001b[A\n",
            "589it [03:55,  2.51it/s]\u001b[A\n",
            "590it [03:55,  2.52it/s]\u001b[A\n",
            "591it [03:55,  2.53it/s]\u001b[A\n",
            "592it [03:56,  2.52it/s]\u001b[A\n",
            "593it [03:56,  2.54it/s]\u001b[A\n",
            "594it [03:57,  2.54it/s]\u001b[A\n",
            "595it [03:57,  2.54it/s]\u001b[A\n",
            "596it [03:57,  2.54it/s]\u001b[A\n",
            "597it [03:58,  2.54it/s]\u001b[A\n",
            "598it [03:58,  2.53it/s]\u001b[A\n",
            "599it [03:59,  2.52it/s]\u001b[A\n",
            "600it [03:59,  2.52it/s]\u001b[A\n",
            "601it [03:59,  2.52it/s]\u001b[A\n",
            "602it [04:00,  2.53it/s]\u001b[A\n",
            "603it [04:00,  2.53it/s]\u001b[A\n",
            "604it [04:01,  2.52it/s]\u001b[A\n",
            "605it [04:01,  2.53it/s]\u001b[A\n",
            "606it [04:01,  2.54it/s]\u001b[A\n",
            "607it [04:02,  2.54it/s]\u001b[A\n",
            "608it [04:02,  2.55it/s]\u001b[A\n",
            "609it [04:03,  2.54it/s]\u001b[A\n",
            "610it [04:03,  2.55it/s]\u001b[A\n",
            "611it [04:03,  2.54it/s]\u001b[A\n",
            "612it [04:04,  2.53it/s]\u001b[A\n",
            "613it [04:04,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.049144268035889\n",
            "eval loss: 5.858266830444336\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "614it [04:05,  2.32it/s]\u001b[A\n",
            "615it [04:05,  2.38it/s]\u001b[A\n",
            "616it [04:05,  2.43it/s]\u001b[A\n",
            "617it [04:06,  2.47it/s]\u001b[A\n",
            "618it [04:06,  2.48it/s]\u001b[A\n",
            "619it [04:07,  2.49it/s]\u001b[A\n",
            "620it [04:07,  2.51it/s]\u001b[A\n",
            "621it [04:07,  2.51it/s]\u001b[A\n",
            "622it [04:08,  2.51it/s]\u001b[A\n",
            "623it [04:08,  2.51it/s]\u001b[A\n",
            "624it [04:09,  2.52it/s]\u001b[A\n",
            "625it [04:09,  2.52it/s]\u001b[A\n",
            "626it [04:09,  2.52it/s]\u001b[A\n",
            "627it [04:10,  2.52it/s]\u001b[A\n",
            "628it [04:10,  2.50it/s]\u001b[A\n",
            "629it [04:11,  2.50it/s]\u001b[A\n",
            "630it [04:11,  2.51it/s]\u001b[A\n",
            "631it [04:11,  2.51it/s]\u001b[A\n",
            "632it [04:12,  2.53it/s]\u001b[A\n",
            "633it [04:12,  2.52it/s]\u001b[A\n",
            "634it [04:13,  2.53it/s]\u001b[A\n",
            "635it [04:13,  2.53it/s]\u001b[A\n",
            "636it [04:13,  2.54it/s]\u001b[A\n",
            "637it [04:14,  2.55it/s]\u001b[A\n",
            "638it [04:14,  2.54it/s]\u001b[A\n",
            "639it [04:15,  2.54it/s]\u001b[A\n",
            "640it [04:15,  2.53it/s]\u001b[A\n",
            "641it [04:15,  2.52it/s]\u001b[A\n",
            "642it [04:16,  2.51it/s]\u001b[A\n",
            "643it [04:16,  2.51it/s]\u001b[A\n",
            "644it [04:17,  2.52it/s]\u001b[A\n",
            "645it [04:17,  2.52it/s]\u001b[A\n",
            "646it [04:17,  2.52it/s]\u001b[A\n",
            "647it [04:18,  2.51it/s]\u001b[A\n",
            "648it [04:18,  2.51it/s]\u001b[A\n",
            "649it [04:19,  2.52it/s]\u001b[A\n",
            "650it [04:19,  2.51it/s]\u001b[A\n",
            "651it [04:19,  2.52it/s]\u001b[A\n",
            "652it [04:20,  2.53it/s]\u001b[A\n",
            "653it [04:20,  2.53it/s]\u001b[A\n",
            "654it [04:21,  2.54it/s]\u001b[A\n",
            "655it [04:21,  2.54it/s]\u001b[A\n",
            "656it [04:21,  2.54it/s]\u001b[A\n",
            "657it [04:22,  2.53it/s]\u001b[A\n",
            "658it [04:22,  2.52it/s]\u001b[A\n",
            "659it [04:23,  2.52it/s]\u001b[A\n",
            "660it [04:23,  2.52it/s]\u001b[A\n",
            "661it [04:23,  2.53it/s]\u001b[A\n",
            "662it [04:24,  2.52it/s]\u001b[A\n",
            "663it [04:24,  2.53it/s]\u001b[A\n",
            "664it [04:25,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.964940071105957\n",
            "eval loss: 5.697671890258789\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "665it [04:25,  2.32it/s]\u001b[A\n",
            "666it [04:25,  2.37it/s]\u001b[A\n",
            "667it [04:26,  2.42it/s]\u001b[A\n",
            "668it [04:26,  2.46it/s]\u001b[A\n",
            "669it [04:27,  2.48it/s]\u001b[A\n",
            "670it [04:27,  2.49it/s]\u001b[A\n",
            "671it [04:27,  2.50it/s]\u001b[A\n",
            "672it [04:28,  2.52it/s]\u001b[A\n",
            "673it [04:28,  2.51it/s]\u001b[A\n",
            "674it [04:29,  2.52it/s]\u001b[A\n",
            "675it [04:29,  2.51it/s]\u001b[A\n",
            "676it [04:29,  2.52it/s]\u001b[A\n",
            "677it [04:30,  2.52it/s]\u001b[A\n",
            "678it [04:30,  2.51it/s]\u001b[A\n",
            "679it [04:31,  2.52it/s]\u001b[A\n",
            "680it [04:31,  2.54it/s]\u001b[A\n",
            "681it [04:31,  2.54it/s]\u001b[A\n",
            "682it [04:32,  2.53it/s]\u001b[A\n",
            "683it [04:32,  2.54it/s]\u001b[A\n",
            "684it [04:33,  2.53it/s]\u001b[A\n",
            "685it [04:33,  2.53it/s]\u001b[A\n",
            "686it [04:33,  2.53it/s]\u001b[A\n",
            "687it [04:34,  2.53it/s]\u001b[A\n",
            "688it [04:34,  2.53it/s]\u001b[A\n",
            "689it [04:35,  2.53it/s]\u001b[A\n",
            "690it [04:35,  2.53it/s]\u001b[A\n",
            "691it [04:35,  2.54it/s]\u001b[A\n",
            "692it [04:36,  2.54it/s]\u001b[A\n",
            "693it [04:36,  2.53it/s]\u001b[A\n",
            "694it [04:37,  2.54it/s]\u001b[A\n",
            "695it [04:37,  2.54it/s]\u001b[A\n",
            "696it [04:37,  2.55it/s]\u001b[A\n",
            "697it [04:38,  2.52it/s]\u001b[A\n",
            "698it [04:38,  2.52it/s]\u001b[A\n",
            "699it [04:39,  2.53it/s]\u001b[A\n",
            "700it [04:39,  2.52it/s]\u001b[A\n",
            "701it [04:39,  2.52it/s]\u001b[A\n",
            "702it [04:40,  2.52it/s]\u001b[A\n",
            "703it [04:40,  2.52it/s]\u001b[A\n",
            "704it [04:40,  2.52it/s]\u001b[A\n",
            "705it [04:41,  2.51it/s]\u001b[A\n",
            "706it [04:41,  2.51it/s]\u001b[A\n",
            "707it [04:42,  2.51it/s]\u001b[A\n",
            "708it [04:42,  2.51it/s]\u001b[A\n",
            "709it [04:42,  2.51it/s]\u001b[A\n",
            "710it [04:43,  2.51it/s]\u001b[A\n",
            "711it [04:43,  2.51it/s]\u001b[A\n",
            "712it [04:44,  2.52it/s]\u001b[A\n",
            "713it [04:44,  2.51it/s]\u001b[A\n",
            "714it [04:44,  2.52it/s]\u001b[A\n",
            "715it [04:45,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.901734352111816\n",
            "eval loss: 5.778346061706543\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "716it [04:45,  2.33it/s]\u001b[A\n",
            "717it [04:46,  2.38it/s]\u001b[A\n",
            "718it [04:46,  2.41it/s]\u001b[A\n",
            "719it [04:47,  2.44it/s]\u001b[A\n",
            "720it [04:47,  2.46it/s]\u001b[A\n",
            "721it [04:47,  2.48it/s]\u001b[A\n",
            "722it [04:48,  2.49it/s]\u001b[A\n",
            "723it [04:48,  2.49it/s]\u001b[A\n",
            "724it [04:49,  2.49it/s]\u001b[A\n",
            "725it [04:49,  2.50it/s]\u001b[A\n",
            "726it [04:49,  2.52it/s]\u001b[A\n",
            "727it [04:50,  2.53it/s]\u001b[A\n",
            "728it [04:50,  2.52it/s]\u001b[A\n",
            "729it [04:51,  2.52it/s]\u001b[A\n",
            "730it [04:51,  2.53it/s]\u001b[A\n",
            "731it [04:51,  2.54it/s]\u001b[A\n",
            "732it [04:52,  2.54it/s]\u001b[A\n",
            "733it [04:52,  2.54it/s]\u001b[A\n",
            "734it [04:53,  2.54it/s]\u001b[A\n",
            "735it [04:53,  2.54it/s]\u001b[A\n",
            "736it [04:53,  2.53it/s]\u001b[A\n",
            "737it [04:54,  2.53it/s]\u001b[A\n",
            "738it [04:54,  2.53it/s]\u001b[A\n",
            "739it [04:55,  2.53it/s]\u001b[A\n",
            "740it [04:55,  2.53it/s]\u001b[A\n",
            "741it [04:55,  2.53it/s]\u001b[A\n",
            "742it [04:56,  2.54it/s]\u001b[A\n",
            "743it [04:56,  2.55it/s]\u001b[A\n",
            "744it [04:56,  2.55it/s]\u001b[A\n",
            "745it [04:57,  2.55it/s]\u001b[A\n",
            "746it [04:57,  2.54it/s]\u001b[A\n",
            "747it [04:58,  2.54it/s]\u001b[A\n",
            "748it [04:58,  2.52it/s]\u001b[A\n",
            "749it [04:58,  2.51it/s]\u001b[A\n",
            "750it [04:59,  2.52it/s]\u001b[A\n",
            "751it [04:59,  2.50it/s]\u001b[A\n",
            "752it [05:00,  2.50it/s]\u001b[A\n",
            "753it [05:00,  2.51it/s]\u001b[A\n",
            "754it [05:00,  2.50it/s]\u001b[A\n",
            "755it [05:01,  2.50it/s]\u001b[A\n",
            "756it [05:01,  2.50it/s]\u001b[A\n",
            "757it [05:02,  2.52it/s]\u001b[A\n",
            "758it [05:02,  2.52it/s]\u001b[A\n",
            "759it [05:02,  2.52it/s]\u001b[A\n",
            "760it [05:03,  2.53it/s]\u001b[A\n",
            "761it [05:03,  2.52it/s]\u001b[A\n",
            "762it [05:04,  2.53it/s]\u001b[A\n",
            "763it [05:04,  2.55it/s]\u001b[A\n",
            "764it [05:04,  2.55it/s]\u001b[A\n",
            "765it [05:05,  2.55it/s]\u001b[A\n",
            "766it [05:05,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.963535308837891\n",
            "eval loss: 5.748756408691406\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "767it [05:06,  2.32it/s]\u001b[A\n",
            "768it [05:06,  2.37it/s]\u001b[A\n",
            "769it [05:07,  2.41it/s]\u001b[A\n",
            "770it [05:07,  2.44it/s]\u001b[A\n",
            "771it [05:07,  2.45it/s]\u001b[A\n",
            "772it [05:08,  2.47it/s]\u001b[A\n",
            "773it [05:08,  2.49it/s]\u001b[A\n",
            "774it [05:09,  2.51it/s]\u001b[A\n",
            "775it [05:09,  2.53it/s]\u001b[A\n",
            "776it [05:09,  2.53it/s]\u001b[A\n",
            "777it [05:10,  2.53it/s]\u001b[A\n",
            "778it [05:10,  2.54it/s]\u001b[A\n",
            "779it [05:11,  2.53it/s]\u001b[A\n",
            "780it [05:11,  2.53it/s]\u001b[A\n",
            "781it [05:11,  2.53it/s]\u001b[A\n",
            "782it [05:12,  2.53it/s]\u001b[A\n",
            "783it [05:12,  2.51it/s]\u001b[A\n",
            "784it [05:13,  2.51it/s]\u001b[A\n",
            "785it [05:13,  2.52it/s]\u001b[A\n",
            "786it [05:13,  2.53it/s]\u001b[A\n",
            "787it [05:14,  2.53it/s]\u001b[A\n",
            "788it [05:14,  2.53it/s]\u001b[A\n",
            "789it [05:14,  2.53it/s]\u001b[A\n",
            "790it [05:15,  2.53it/s]\u001b[A\n",
            "791it [05:15,  2.53it/s]\u001b[A\n",
            "792it [05:16,  2.54it/s]\u001b[A\n",
            "793it [05:16,  2.54it/s]\u001b[A\n",
            "794it [05:16,  2.54it/s]\u001b[A\n",
            "795it [05:17,  2.52it/s]\u001b[A\n",
            "796it [05:17,  2.51it/s]\u001b[A\n",
            "797it [05:18,  2.52it/s]\u001b[A\n",
            "798it [05:18,  2.52it/s]\u001b[A\n",
            "799it [05:18,  2.52it/s]\u001b[A\n",
            "800it [05:19,  2.51it/s]\u001b[A\n",
            "801it [05:19,  2.52it/s]\u001b[A\n",
            "802it [05:20,  2.53it/s]\u001b[A\n",
            "803it [05:20,  2.51it/s]\u001b[A\n",
            "804it [05:20,  2.51it/s]\u001b[A\n",
            "805it [05:21,  2.51it/s]\u001b[A\n",
            "806it [05:21,  2.52it/s]\u001b[A\n",
            "807it [05:22,  2.53it/s]\u001b[A\n",
            "808it [05:22,  2.54it/s]\u001b[A\n",
            "809it [05:22,  2.54it/s]\u001b[A\n",
            "810it [05:23,  2.54it/s]\u001b[A\n",
            "811it [05:23,  2.54it/s]\u001b[A\n",
            "812it [05:24,  2.54it/s]\u001b[A\n",
            "813it [05:24,  2.53it/s]\u001b[A\n",
            "814it [05:24,  2.53it/s]\u001b[A\n",
            "815it [05:25,  2.52it/s]\u001b[A\n",
            "816it [05:25,  2.52it/s]\u001b[A\n",
            "817it [05:26,  2.23it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 6.009275436401367\n",
            "eval loss: 5.692862510681152\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "818it [05:26,  2.30it/s]\u001b[A\n",
            "819it [05:27,  2.38it/s]\u001b[A\n",
            "820it [05:27,  2.42it/s]\u001b[A\n",
            "821it [05:27,  2.45it/s]\u001b[A\n",
            "822it [05:28,  2.49it/s]\u001b[A\n",
            "823it [05:28,  2.51it/s]\u001b[A\n",
            "824it [05:29,  2.51it/s]\u001b[A\n",
            "825it [05:29,  2.52it/s]\u001b[A\n",
            "826it [05:29,  2.52it/s]\u001b[A\n",
            "827it [05:30,  2.52it/s]\u001b[A\n",
            "828it [05:30,  2.52it/s]\u001b[A\n",
            "829it [05:30,  2.52it/s]\u001b[A\n",
            "830it [05:31,  2.51it/s]\u001b[A\n",
            "831it [05:31,  2.50it/s]\u001b[A\n",
            "832it [05:32,  2.51it/s]\u001b[A\n",
            "833it [05:32,  2.51it/s]\u001b[A\n",
            "834it [05:32,  2.50it/s]\u001b[A\n",
            "835it [05:33,  2.51it/s]\u001b[A\n",
            "836it [05:33,  2.52it/s]\u001b[A\n",
            "837it [05:34,  2.53it/s]\u001b[A\n",
            "838it [05:34,  2.54it/s]\u001b[A\n",
            "839it [05:34,  2.54it/s]\u001b[A\n",
            "840it [05:35,  2.54it/s]\u001b[A\n",
            "841it [05:35,  2.54it/s]\u001b[A\n",
            "842it [05:36,  2.54it/s]\u001b[A\n",
            "843it [05:36,  2.53it/s]\u001b[A\n",
            "844it [05:36,  2.53it/s]\u001b[A\n",
            "845it [05:37,  2.52it/s]\u001b[A\n",
            "846it [05:37,  2.53it/s]\u001b[A\n",
            "847it [05:38,  2.53it/s]\u001b[A\n",
            "848it [05:38,  2.52it/s]\u001b[A\n",
            "849it [05:38,  2.53it/s]\u001b[A\n",
            "850it [05:39,  2.53it/s]\u001b[A\n",
            "851it [05:39,  2.54it/s]\u001b[A\n",
            "852it [05:40,  2.55it/s]\u001b[A\n",
            "853it [05:40,  2.55it/s]\u001b[A\n",
            "854it [05:40,  2.53it/s]\u001b[A\n",
            "855it [05:41,  2.52it/s]\u001b[A\n",
            "856it [05:41,  2.53it/s]\u001b[A\n",
            "857it [05:42,  2.53it/s]\u001b[A\n",
            "858it [05:42,  2.53it/s]\u001b[A\n",
            "859it [05:42,  2.52it/s]\u001b[A\n",
            "860it [05:43,  2.53it/s]\u001b[A\n",
            "861it [05:43,  2.53it/s]\u001b[A\n",
            "862it [05:44,  2.52it/s]\u001b[A\n",
            "863it [05:44,  2.52it/s]\u001b[A\n",
            "864it [05:44,  2.53it/s]\u001b[A\n",
            "865it [05:45,  2.53it/s]\u001b[A\n",
            "866it [05:45,  2.54it/s]\u001b[A\n",
            "867it [05:46,  2.54it/s]\u001b[A\n",
            "868it [05:46,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.835292339324951\n",
            "eval loss: 5.6659464836120605\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "869it [05:46,  2.32it/s]\u001b[A\n",
            "870it [05:47,  2.38it/s]\u001b[A\n",
            "871it [05:47,  2.43it/s]\u001b[A\n",
            "872it [05:48,  2.46it/s]\u001b[A\n",
            "873it [05:48,  2.48it/s]\u001b[A\n",
            "874it [05:48,  2.50it/s]\u001b[A\n",
            "875it [05:49,  2.52it/s]\u001b[A\n",
            "876it [05:49,  2.54it/s]\u001b[A\n",
            "877it [05:50,  2.54it/s]\u001b[A\n",
            "878it [05:50,  2.53it/s]\u001b[A\n",
            "879it [05:50,  2.53it/s]\u001b[A\n",
            "880it [05:51,  2.53it/s]\u001b[A\n",
            "881it [05:51,  2.53it/s]\u001b[A\n",
            "882it [05:52,  2.52it/s]\u001b[A\n",
            "883it [05:52,  2.53it/s]\u001b[A\n",
            "884it [05:52,  2.53it/s]\u001b[A\n",
            "885it [05:53,  2.52it/s]\u001b[A\n",
            "886it [05:53,  2.51it/s]\u001b[A\n",
            "887it [05:54,  2.51it/s]\u001b[A\n",
            "888it [05:54,  2.51it/s]\u001b[A\n",
            "889it [05:54,  2.52it/s]\u001b[A\n",
            "890it [05:55,  2.53it/s]\u001b[A\n",
            "891it [05:55,  2.54it/s]\u001b[A\n",
            "892it [05:56,  2.55it/s]\u001b[A\n",
            "893it [05:56,  2.54it/s]\u001b[A\n",
            "894it [05:56,  2.53it/s]\u001b[A\n",
            "895it [05:57,  2.52it/s]\u001b[A\n",
            "896it [05:57,  2.52it/s]\u001b[A\n",
            "897it [05:58,  2.52it/s]\u001b[A\n",
            "898it [05:58,  2.52it/s]\u001b[A\n",
            "899it [05:58,  2.52it/s]\u001b[A\n",
            "900it [05:59,  2.52it/s]\u001b[A\n",
            "901it [05:59,  2.52it/s]\u001b[A\n",
            "902it [06:00,  2.52it/s]\u001b[A\n",
            "903it [06:00,  2.52it/s]\u001b[A\n",
            "904it [06:00,  2.52it/s]\u001b[A\n",
            "905it [06:01,  2.51it/s]\u001b[A\n",
            "906it [06:01,  2.50it/s]\u001b[A\n",
            "907it [06:02,  2.51it/s]\u001b[A\n",
            "908it [06:02,  2.53it/s]\u001b[A\n",
            "909it [06:02,  2.53it/s]\u001b[A\n",
            "910it [06:03,  2.53it/s]\u001b[A\n",
            "911it [06:03,  2.54it/s]\u001b[A\n",
            "912it [06:03,  2.54it/s]\u001b[A\n",
            "913it [06:04,  2.52it/s]\u001b[A\n",
            "914it [06:04,  2.53it/s]\u001b[A\n",
            "915it [06:05,  2.53it/s]\u001b[A\n",
            "916it [06:05,  2.53it/s]\u001b[A\n",
            "917it [06:05,  2.52it/s]\u001b[A\n",
            "918it [06:06,  2.51it/s]\u001b[A\n",
            "919it [06:06,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.820431232452393\n",
            "eval loss: 5.605086326599121\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "920it [06:07,  2.32it/s]\u001b[A\n",
            "921it [06:07,  2.39it/s]\u001b[A\n",
            "922it [06:08,  2.44it/s]\u001b[A\n",
            "923it [06:08,  2.47it/s]\u001b[A\n",
            "924it [06:08,  2.48it/s]\u001b[A\n",
            "925it [06:09,  2.49it/s]\u001b[A\n",
            "926it [06:09,  2.50it/s]\u001b[A\n",
            "927it [06:10,  2.51it/s]\u001b[A\n",
            "928it [06:10,  2.51it/s]\u001b[A\n",
            "929it [06:10,  2.49it/s]\u001b[A\n",
            "930it [06:11,  2.51it/s]\u001b[A\n",
            "931it [06:11,  2.52it/s]\u001b[A\n",
            "932it [06:12,  2.52it/s]\u001b[A\n",
            "933it [06:12,  2.52it/s]\u001b[A\n",
            "934it [06:12,  2.53it/s]\u001b[A\n",
            "935it [06:13,  2.53it/s]\u001b[A\n",
            "936it [06:13,  2.52it/s]\u001b[A\n",
            "937it [06:14,  2.52it/s]\u001b[A\n",
            "938it [06:14,  2.54it/s]\u001b[A\n",
            "939it [06:14,  2.54it/s]\u001b[A\n",
            "940it [06:15,  2.54it/s]\u001b[A\n",
            "941it [06:15,  2.54it/s]\u001b[A\n",
            "942it [06:16,  2.53it/s]\u001b[A\n",
            "943it [06:16,  2.53it/s]\u001b[A\n",
            "944it [06:16,  2.53it/s]\u001b[A\n",
            "945it [06:17,  2.53it/s]\u001b[A\n",
            "946it [06:17,  2.53it/s]\u001b[A\n",
            "947it [06:17,  2.53it/s]\u001b[A\n",
            "948it [06:18,  2.51it/s]\u001b[A\n",
            "949it [06:18,  2.51it/s]\u001b[A\n",
            "950it [06:19,  2.52it/s]\u001b[A\n",
            "951it [06:19,  2.52it/s]\u001b[A\n",
            "952it [06:19,  2.53it/s]\u001b[A\n",
            "953it [06:20,  2.52it/s]\u001b[A\n",
            "954it [06:20,  2.53it/s]\u001b[A\n",
            "955it [06:21,  2.53it/s]\u001b[A\n",
            "956it [06:21,  2.54it/s]\u001b[A\n",
            "957it [06:21,  2.54it/s]\u001b[A\n",
            "958it [06:22,  2.53it/s]\u001b[A\n",
            "959it [06:22,  2.53it/s]\u001b[A\n",
            "960it [06:23,  2.53it/s]\u001b[A\n",
            "961it [06:23,  2.53it/s]\u001b[A\n",
            "962it [06:23,  2.54it/s]\u001b[A\n",
            "963it [06:24,  2.53it/s]\u001b[A\n",
            "964it [06:24,  2.53it/s]\u001b[A\n",
            "965it [06:25,  2.53it/s]\u001b[A\n",
            "966it [06:25,  2.53it/s]\u001b[A\n",
            "967it [06:25,  2.52it/s]\u001b[A\n",
            "968it [06:26,  2.53it/s]\u001b[A\n",
            "969it [06:26,  2.54it/s]\u001b[A\n",
            "970it [06:27,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.8187408447265625\n",
            "eval loss: 5.55510950088501\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "971it [06:27,  2.34it/s]\u001b[A\n",
            "972it [06:28,  2.39it/s]\u001b[A\n",
            "973it [06:28,  2.42it/s]\u001b[A\n",
            "974it [06:28,  2.46it/s]\u001b[A\n",
            "975it [06:29,  2.49it/s]\u001b[A\n",
            "976it [06:29,  2.50it/s]\u001b[A\n",
            "977it [06:30,  2.52it/s]\u001b[A\n",
            "978it [06:30,  2.53it/s]\u001b[A\n",
            "979it [06:30,  2.52it/s]\u001b[A\n",
            "980it [06:31,  2.53it/s]\u001b[A\n",
            "981it [06:31,  2.53it/s]\u001b[A\n",
            "982it [06:31,  2.53it/s]\u001b[A\n",
            "983it [06:32,  2.52it/s]\u001b[A\n",
            "984it [06:32,  2.52it/s]\u001b[A\n",
            "985it [06:33,  2.52it/s]\u001b[A\n",
            "986it [06:33,  2.52it/s]\u001b[A\n",
            "987it [06:33,  2.52it/s]\u001b[A\n",
            "988it [06:34,  2.52it/s]\u001b[A\n",
            "989it [06:34,  2.53it/s]\u001b[A\n",
            "990it [06:35,  2.54it/s]\u001b[A\n",
            "991it [06:35,  2.54it/s]\u001b[A\n",
            "992it [06:35,  2.55it/s]\u001b[A\n",
            "993it [06:36,  2.55it/s]\u001b[A\n",
            "994it [06:36,  2.55it/s]\u001b[A\n",
            "995it [06:37,  2.53it/s]\u001b[A\n",
            "996it [06:37,  2.53it/s]\u001b[A\n",
            "997it [06:37,  2.52it/s]\u001b[A\n",
            "998it [06:38,  2.53it/s]\u001b[A\n",
            "999it [06:38,  2.52it/s]\u001b[A\n",
            "1000it [06:39,  2.51it/s]\u001b[A\n",
            "1001it [06:39,  2.52it/s]\u001b[A\n",
            "1002it [06:39,  2.52it/s]\u001b[A\n",
            "1003it [06:40,  2.53it/s]\u001b[A\n",
            "1004it [06:40,  2.52it/s]\u001b[A\n",
            "1005it [06:41,  2.52it/s]\u001b[A\n",
            "1006it [06:41,  2.53it/s]\u001b[A\n",
            "1007it [06:41,  2.53it/s]\u001b[A\n",
            "1008it [06:42,  2.52it/s]\u001b[A\n",
            "1009it [06:42,  2.52it/s]\u001b[A\n",
            "1010it [06:43,  2.54it/s]\u001b[A\n",
            "1011it [06:43,  2.53it/s]\u001b[A\n",
            "1012it [06:43,  2.53it/s]\u001b[A\n",
            "1013it [06:44,  2.53it/s]\u001b[A\n",
            "1014it [06:44,  2.53it/s]\u001b[A\n",
            "1015it [06:45,  2.53it/s]\u001b[A\n",
            "1016it [06:45,  2.52it/s]\u001b[A\n",
            "1017it [06:45,  2.53it/s]\u001b[A\n",
            "1018it [06:46,  2.53it/s]\u001b[A\n",
            "1019it [06:46,  2.53it/s]\u001b[A\n",
            "1021it [06:47,  2.76it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.717184543609619\n",
            "eval loss: 5.708514213562012\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1022it [06:47,  2.68it/s]\u001b[A\n",
            "1023it [06:47,  2.64it/s]\u001b[A\n",
            "1024it [06:48,  2.60it/s]\u001b[A\n",
            "1025it [06:48,  2.59it/s]\u001b[A\n",
            "1026it [06:49,  2.57it/s]\u001b[A\n",
            "1027it [06:49,  2.54it/s]\u001b[A\n",
            "1028it [06:49,  2.53it/s]\u001b[A\n",
            "1029it [06:50,  2.54it/s]\u001b[A\n",
            "1030it [06:50,  2.54it/s]\u001b[A\n",
            "1031it [06:51,  2.53it/s]\u001b[A\n",
            "1032it [06:51,  2.53it/s]\u001b[A\n",
            "1033it [06:51,  2.53it/s]\u001b[A\n",
            "1034it [06:52,  2.51it/s]\u001b[A\n",
            "1035it [06:52,  2.50it/s]\u001b[A\n",
            "1036it [06:53,  2.50it/s]\u001b[A\n",
            "1037it [06:53,  2.51it/s]\u001b[A\n",
            "1038it [06:53,  2.52it/s]\u001b[A\n",
            "1039it [06:54,  2.52it/s]\u001b[A\n",
            "1040it [06:54,  2.53it/s]\u001b[A\n",
            "1041it [06:55,  2.55it/s]\u001b[A\n",
            "1042it [06:55,  2.55it/s]\u001b[A\n",
            "1043it [06:55,  2.54it/s]\u001b[A\n",
            "1044it [06:56,  2.53it/s]\u001b[A\n",
            "1045it [06:56,  2.53it/s]\u001b[A\n",
            "1046it [06:57,  2.53it/s]\u001b[A\n",
            "1047it [06:57,  2.52it/s]\u001b[A\n",
            "1048it [06:57,  2.52it/s]\u001b[A\n",
            "1049it [06:58,  2.51it/s]\u001b[A\n",
            "1050it [06:58,  2.50it/s]\u001b[A\n",
            "1051it [06:59,  2.50it/s]\u001b[A\n",
            "1052it [06:59,  2.51it/s]\u001b[A\n",
            "1053it [06:59,  2.52it/s]\u001b[A\n",
            "1054it [07:00,  2.51it/s]\u001b[A\n",
            "1055it [07:00,  2.51it/s]\u001b[A\n",
            "1056it [07:01,  2.52it/s]\u001b[A\n",
            "1057it [07:01,  2.53it/s]\u001b[A\n",
            "1058it [07:01,  2.54it/s]\u001b[A\n",
            "1059it [07:02,  2.54it/s]\u001b[A\n",
            "1060it [07:02,  2.54it/s]\u001b[A\n",
            "1061it [07:03,  2.55it/s]\u001b[A\n",
            "1062it [07:03,  2.54it/s]\u001b[A\n",
            "1063it [07:03,  2.53it/s]\u001b[A\n",
            "1064it [07:04,  2.52it/s]\u001b[A\n",
            "1065it [07:04,  2.52it/s]\u001b[A\n",
            "1066it [07:05,  2.53it/s]\u001b[A\n",
            "1067it [07:05,  2.52it/s]\u001b[A\n",
            "1068it [07:05,  2.51it/s]\u001b[A\n",
            "1069it [07:06,  2.51it/s]\u001b[A\n",
            "1070it [07:06,  2.52it/s]\u001b[A\n",
            "1071it [07:07,  2.51it/s]\u001b[A\n",
            "1072it [07:07,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.770957946777344\n",
            "eval loss: 5.588698387145996\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1073it [07:07,  2.33it/s]\u001b[A\n",
            "1074it [07:08,  2.38it/s]\u001b[A\n",
            "1075it [07:08,  2.42it/s]\u001b[A\n",
            "1076it [07:09,  2.45it/s]\u001b[A\n",
            "1077it [07:09,  2.47it/s]\u001b[A\n",
            "1078it [07:09,  2.48it/s]\u001b[A\n",
            "1079it [07:10,  2.49it/s]\u001b[A\n",
            "1080it [07:10,  2.50it/s]\u001b[A\n",
            "1081it [07:11,  2.51it/s]\u001b[A\n",
            "1082it [07:11,  2.51it/s]\u001b[A\n",
            "1083it [07:11,  2.51it/s]\u001b[A\n",
            "1084it [07:12,  2.53it/s]\u001b[A\n",
            "1085it [07:12,  2.54it/s]\u001b[A\n",
            "1086it [07:13,  2.55it/s]\u001b[A\n",
            "1087it [07:13,  2.54it/s]\u001b[A\n",
            "1088it [07:13,  2.54it/s]\u001b[A\n",
            "1089it [07:14,  2.54it/s]\u001b[A\n",
            "1090it [07:14,  2.53it/s]\u001b[A\n",
            "1091it [07:15,  2.53it/s]\u001b[A\n",
            "1092it [07:15,  2.53it/s]\u001b[A\n",
            "1093it [07:15,  2.53it/s]\u001b[A\n",
            "1094it [07:16,  2.53it/s]\u001b[A\n",
            "1095it [07:16,  2.53it/s]\u001b[A\n",
            "1096it [07:17,  2.53it/s]\u001b[A\n",
            "1097it [07:17,  2.53it/s]\u001b[A\n",
            "1098it [07:17,  2.54it/s]\u001b[A\n",
            "1099it [07:18,  2.54it/s]\u001b[A\n",
            "1100it [07:18,  2.55it/s]\u001b[A\n",
            "1101it [07:19,  2.55it/s]\u001b[A\n",
            "1102it [07:19,  2.55it/s]\u001b[A\n",
            "1103it [07:19,  2.54it/s]\u001b[A\n",
            "1104it [07:20,  2.53it/s]\u001b[A\n",
            "1105it [07:20,  2.53it/s]\u001b[A\n",
            "1106it [07:20,  2.53it/s]\u001b[A\n",
            "1107it [07:21,  2.52it/s]\u001b[A\n",
            "1108it [07:21,  2.53it/s]\u001b[A\n",
            "1109it [07:22,  2.53it/s]\u001b[A\n",
            "1110it [07:22,  2.53it/s]\u001b[A\n",
            "1111it [07:22,  2.53it/s]\u001b[A\n",
            "1112it [07:23,  2.54it/s]\u001b[A\n",
            "1113it [07:23,  2.55it/s]\u001b[A\n",
            "1114it [07:24,  2.55it/s]\u001b[A\n",
            "1115it [07:24,  2.55it/s]\u001b[A\n",
            "1116it [07:24,  2.54it/s]\u001b[A\n",
            "1117it [07:25,  2.53it/s]\u001b[A\n",
            "1118it [07:25,  2.52it/s]\u001b[A\n",
            "1119it [07:26,  2.52it/s]\u001b[A\n",
            "1120it [07:26,  2.52it/s]\u001b[A\n",
            "1121it [07:26,  2.52it/s]\u001b[A\n",
            "1122it [07:27,  2.51it/s]\u001b[A\n",
            "1123it [07:27,  2.23it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.714615345001221\n",
            "eval loss: 5.644379138946533\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1124it [07:28,  2.31it/s]\u001b[A\n",
            "1125it [07:28,  2.37it/s]\u001b[A\n",
            "1126it [07:29,  2.42it/s]\u001b[A\n",
            "1127it [07:29,  2.46it/s]\u001b[A\n",
            "1128it [07:29,  2.49it/s]\u001b[A\n",
            "1129it [07:30,  2.50it/s]\u001b[A\n",
            "1130it [07:30,  2.51it/s]\u001b[A\n",
            "1131it [07:31,  2.52it/s]\u001b[A\n",
            "1132it [07:31,  2.52it/s]\u001b[A\n",
            "1133it [07:31,  2.53it/s]\u001b[A\n",
            "1134it [07:32,  2.52it/s]\u001b[A\n",
            "1135it [07:32,  2.53it/s]\u001b[A\n",
            "1136it [07:33,  2.53it/s]\u001b[A\n",
            "1137it [07:33,  2.53it/s]\u001b[A\n",
            "1138it [07:33,  2.54it/s]\u001b[A\n",
            "1139it [07:34,  2.55it/s]\u001b[A\n",
            "1140it [07:34,  2.56it/s]\u001b[A\n",
            "1141it [07:34,  2.55it/s]\u001b[A\n",
            "1142it [07:35,  2.55it/s]\u001b[A\n",
            "1143it [07:35,  2.54it/s]\u001b[A\n",
            "1144it [07:36,  2.54it/s]\u001b[A\n",
            "1145it [07:36,  2.54it/s]\u001b[A\n",
            "1146it [07:36,  2.53it/s]\u001b[A\n",
            "1147it [07:37,  2.53it/s]\u001b[A\n",
            "1148it [07:37,  2.53it/s]\u001b[A\n",
            "1149it [07:38,  2.53it/s]\u001b[A\n",
            "1150it [07:38,  2.54it/s]\u001b[A\n",
            "1151it [07:38,  2.55it/s]\u001b[A\n",
            "1152it [07:39,  2.55it/s]\u001b[A\n",
            "1153it [07:39,  2.55it/s]\u001b[A\n",
            "1154it [07:40,  2.55it/s]\u001b[A\n",
            "1155it [07:40,  2.54it/s]\u001b[A\n",
            "1156it [07:40,  2.53it/s]\u001b[A\n",
            "1157it [07:41,  2.51it/s]\u001b[A\n",
            "1158it [07:41,  2.52it/s]\u001b[A\n",
            "1159it [07:42,  2.51it/s]\u001b[A\n",
            "1160it [07:42,  2.51it/s]\u001b[A\n",
            "1161it [07:42,  2.51it/s]\u001b[A\n",
            "1162it [07:43,  2.50it/s]\u001b[A\n",
            "1163it [07:43,  2.50it/s]\u001b[A\n",
            "1164it [07:44,  2.51it/s]\u001b[A\n",
            "1165it [07:44,  2.50it/s]\u001b[A\n",
            "1166it [07:44,  2.51it/s]\u001b[A\n",
            "1167it [07:45,  2.50it/s]\u001b[A\n",
            "1168it [07:45,  2.50it/s]\u001b[A\n",
            "1169it [07:46,  2.50it/s]\u001b[A\n",
            "1170it [07:46,  2.49it/s]\u001b[A\n",
            "1171it [07:46,  2.49it/s]\u001b[A\n",
            "1172it [07:47,  2.50it/s]\u001b[A\n",
            "1173it [07:47,  2.52it/s]\u001b[A\n",
            "1174it [07:48,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.656882286071777\n",
            "eval loss: 5.538461208343506\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1175it [07:48,  2.33it/s]\u001b[A\n",
            "1176it [07:49,  2.38it/s]\u001b[A\n",
            "1177it [07:49,  2.42it/s]\u001b[A\n",
            "1178it [07:49,  2.44it/s]\u001b[A\n",
            "1179it [07:50,  2.47it/s]\u001b[A\n",
            "1180it [07:50,  2.49it/s]\u001b[A\n",
            "1181it [07:51,  2.50it/s]\u001b[A\n",
            "1182it [07:51,  2.51it/s]\u001b[A\n",
            "1183it [07:51,  2.53it/s]\u001b[A\n",
            "1184it [07:52,  2.53it/s]\u001b[A\n",
            "1185it [07:52,  2.54it/s]\u001b[A\n",
            "1186it [07:52,  2.54it/s]\u001b[A\n",
            "1187it [07:53,  2.54it/s]\u001b[A\n",
            "1188it [07:53,  2.54it/s]\u001b[A\n",
            "1189it [07:54,  2.54it/s]\u001b[A\n",
            "1190it [07:54,  2.53it/s]\u001b[A\n",
            "1191it [07:54,  2.53it/s]\u001b[A\n",
            "1192it [07:55,  2.53it/s]\u001b[A\n",
            "1193it [07:55,  2.52it/s]\u001b[A\n",
            "1194it [07:56,  2.53it/s]\u001b[A\n",
            "1195it [07:56,  2.53it/s]\u001b[A\n",
            "1196it [07:56,  2.53it/s]\u001b[A\n",
            "1197it [07:57,  2.52it/s]\u001b[A\n",
            "1198it [07:57,  2.52it/s]\u001b[A\n",
            "1199it [07:58,  2.53it/s]\u001b[A\n",
            "1200it [07:58,  2.52it/s]\u001b[A\n",
            "1201it [07:58,  2.53it/s]\u001b[A\n",
            "1202it [07:59,  2.54it/s]\u001b[A\n",
            "1203it [07:59,  2.54it/s]\u001b[A\n",
            "1204it [08:00,  2.54it/s]\u001b[A\n",
            "1205it [08:00,  2.53it/s]\u001b[A\n",
            "1206it [08:00,  2.53it/s]\u001b[A\n",
            "1207it [08:01,  2.53it/s]\u001b[A\n",
            "1208it [08:01,  2.53it/s]\u001b[A\n",
            "1209it [08:02,  2.53it/s]\u001b[A\n",
            "1210it [08:02,  2.52it/s]\u001b[A\n",
            "1211it [08:02,  2.52it/s]\u001b[A\n",
            "1212it [08:03,  2.51it/s]\u001b[A\n",
            "1213it [08:03,  2.51it/s]\u001b[A\n",
            "1214it [08:04,  2.50it/s]\u001b[A\n",
            "1215it [08:04,  2.52it/s]\u001b[A\n",
            "1216it [08:04,  2.51it/s]\u001b[A\n",
            "1217it [08:05,  2.52it/s]\u001b[A\n",
            "1218it [08:05,  2.53it/s]\u001b[A\n",
            "1219it [08:06,  2.55it/s]\u001b[A\n",
            "1220it [08:06,  2.53it/s]\u001b[A\n",
            "1221it [08:06,  2.53it/s]\u001b[A\n",
            "1222it [08:07,  2.53it/s]\u001b[A\n",
            "1223it [08:07,  2.53it/s]\u001b[A\n",
            "1224it [08:08,  2.52it/s]\u001b[A\n",
            "1225it [08:08,  2.23it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.6443257331848145\n",
            "eval loss: 5.55673885345459\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1226it [08:08,  2.30it/s]\u001b[A\n",
            "1227it [08:09,  2.35it/s]\u001b[A\n",
            "1228it [08:09,  2.39it/s]\u001b[A\n",
            "1229it [08:10,  2.43it/s]\u001b[A\n",
            "1230it [08:10,  2.46it/s]\u001b[A\n",
            "1231it [08:10,  2.49it/s]\u001b[A\n",
            "1232it [08:11,  2.50it/s]\u001b[A\n",
            "1233it [08:11,  2.52it/s]\u001b[A\n",
            "1234it [08:12,  2.53it/s]\u001b[A\n",
            "1235it [08:12,  2.53it/s]\u001b[A\n",
            "1236it [08:12,  2.53it/s]\u001b[A\n",
            "1237it [08:13,  2.52it/s]\u001b[A\n",
            "1238it [08:13,  2.53it/s]\u001b[A\n",
            "1239it [08:14,  2.53it/s]\u001b[A\n",
            "1240it [08:14,  2.52it/s]\u001b[A\n",
            "1241it [08:14,  2.53it/s]\u001b[A\n",
            "1242it [08:15,  2.53it/s]\u001b[A\n",
            "1243it [08:15,  2.53it/s]\u001b[A\n",
            "1244it [08:16,  2.53it/s]\u001b[A\n",
            "1245it [08:16,  2.54it/s]\u001b[A\n",
            "1246it [08:16,  2.54it/s]\u001b[A\n",
            "1247it [08:17,  2.55it/s]\u001b[A\n",
            "1248it [08:17,  2.55it/s]\u001b[A\n",
            "1249it [08:18,  2.55it/s]\u001b[A\n",
            "1250it [08:18,  2.54it/s]\u001b[A\n",
            "1251it [08:18,  2.54it/s]\u001b[A\n",
            "1252it [08:19,  2.53it/s]\u001b[A\n",
            "1253it [08:19,  2.53it/s]\u001b[A\n",
            "1254it [08:20,  2.53it/s]\u001b[A\n",
            "1255it [08:20,  2.53it/s]\u001b[A\n",
            "1256it [08:20,  2.52it/s]\u001b[A\n",
            "1257it [08:21,  2.51it/s]\u001b[A\n",
            "1258it [08:21,  2.52it/s]\u001b[A\n",
            "1259it [08:22,  2.53it/s]\u001b[A\n",
            "1260it [08:22,  2.54it/s]\u001b[A\n",
            "1261it [08:22,  2.55it/s]\u001b[A\n",
            "1262it [08:23,  2.55it/s]\u001b[A\n",
            "1263it [08:23,  2.54it/s]\u001b[A\n",
            "1264it [08:23,  2.53it/s]\u001b[A\n",
            "1265it [08:24,  2.53it/s]\u001b[A\n",
            "1266it [08:24,  2.53it/s]\u001b[A\n",
            "1267it [08:25,  2.53it/s]\u001b[A\n",
            "1268it [08:25,  2.53it/s]\u001b[A\n",
            "1269it [08:25,  2.53it/s]\u001b[A\n",
            "1270it [08:26,  2.52it/s]\u001b[A\n",
            "1271it [08:26,  2.51it/s]\u001b[A\n",
            "1272it [08:27,  2.51it/s]\u001b[A\n",
            "1273it [08:27,  2.52it/s]\u001b[A\n",
            "1274it [08:27,  2.53it/s]\u001b[A\n",
            "1275it [08:28,  2.54it/s]\u001b[A\n",
            "1276it [08:28,  2.27it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.702193260192871\n",
            "eval loss: 5.554844379425049\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1277it [08:29,  2.34it/s]\u001b[A\n",
            "1278it [08:29,  2.39it/s]\u001b[A\n",
            "1279it [08:30,  2.42it/s]\u001b[A\n",
            "1280it [08:30,  2.45it/s]\u001b[A\n",
            "1281it [08:30,  2.48it/s]\u001b[A\n",
            "1282it [08:31,  2.49it/s]\u001b[A\n",
            "1283it [08:31,  2.51it/s]\u001b[A\n",
            "1284it [08:32,  2.51it/s]\u001b[A\n",
            "1285it [08:32,  2.53it/s]\u001b[A\n",
            "1286it [08:32,  2.54it/s]\u001b[A\n",
            "1287it [08:33,  2.53it/s]\u001b[A\n",
            "1288it [08:33,  2.53it/s]\u001b[A\n",
            "1289it [08:34,  2.53it/s]\u001b[A\n",
            "1290it [08:34,  2.53it/s]\u001b[A\n",
            "1291it [08:34,  2.52it/s]\u001b[A\n",
            "1292it [08:35,  2.52it/s]\u001b[A\n",
            "1293it [08:35,  2.52it/s]\u001b[A\n",
            "1294it [08:36,  2.52it/s]\u001b[A\n",
            "1295it [08:36,  2.51it/s]\u001b[A\n",
            "1296it [08:36,  2.51it/s]\u001b[A\n",
            "1297it [08:37,  2.52it/s]\u001b[A\n",
            "1298it [08:37,  2.52it/s]\u001b[A\n",
            "1299it [08:37,  2.53it/s]\u001b[A\n",
            "1300it [08:38,  2.53it/s]\u001b[A\n",
            "1301it [08:38,  2.54it/s]\u001b[A\n",
            "1302it [08:39,  2.55it/s]\u001b[A\n",
            "1303it [08:39,  2.54it/s]\u001b[A\n",
            "1304it [08:39,  2.54it/s]\u001b[A\n",
            "1305it [08:40,  2.53it/s]\u001b[A\n",
            "1306it [08:40,  2.53it/s]\u001b[A\n",
            "1307it [08:41,  2.53it/s]\u001b[A\n",
            "1308it [08:41,  2.53it/s]\u001b[A\n",
            "1309it [08:41,  2.53it/s]\u001b[A\n",
            "1310it [08:42,  2.52it/s]\u001b[A\n",
            "1311it [08:42,  2.52it/s]\u001b[A\n",
            "1312it [08:43,  2.52it/s]\u001b[A\n",
            "1313it [08:43,  2.53it/s]\u001b[A\n",
            "1314it [08:43,  2.52it/s]\u001b[A\n",
            "1315it [08:44,  2.52it/s]\u001b[A\n",
            "1316it [08:44,  2.53it/s]\u001b[A\n",
            "1317it [08:45,  2.54it/s]\u001b[A\n",
            "1318it [08:45,  2.53it/s]\u001b[A\n",
            "1319it [08:45,  2.53it/s]\u001b[A\n",
            "1320it [08:46,  2.53it/s]\u001b[A\n",
            "1321it [08:46,  2.53it/s]\u001b[A\n",
            "1322it [08:47,  2.52it/s]\u001b[A\n",
            "1323it [08:47,  2.51it/s]\u001b[A\n",
            "1324it [08:47,  2.51it/s]\u001b[A\n",
            "1325it [08:48,  2.52it/s]\u001b[A\n",
            "1326it [08:48,  2.52it/s]\u001b[A\n",
            "1327it [08:49,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.752259254455566\n",
            "eval loss: 5.394573211669922\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1328it [08:49,  2.32it/s]\u001b[A\n",
            "1329it [08:50,  2.39it/s]\u001b[A\n",
            "1330it [08:50,  2.43it/s]\u001b[A\n",
            "1331it [08:50,  2.47it/s]\u001b[A\n",
            "1332it [08:51,  2.48it/s]\u001b[A\n",
            "1333it [08:51,  2.49it/s]\u001b[A\n",
            "1334it [08:51,  2.50it/s]\u001b[A\n",
            "1335it [08:52,  2.51it/s]\u001b[A\n",
            "1336it [08:52,  2.52it/s]\u001b[A\n",
            "1337it [08:53,  2.52it/s]\u001b[A\n",
            "1338it [08:53,  2.52it/s]\u001b[A\n",
            "1339it [08:53,  2.53it/s]\u001b[A\n",
            "1340it [08:54,  2.53it/s]\u001b[A\n",
            "1341it [08:54,  2.54it/s]\u001b[A\n",
            "1342it [08:55,  2.54it/s]\u001b[A\n",
            "1343it [08:55,  2.55it/s]\u001b[A\n",
            "1344it [08:55,  2.55it/s]\u001b[A\n",
            "1345it [08:56,  2.53it/s]\u001b[A\n",
            "1346it [08:56,  2.53it/s]\u001b[A\n",
            "1347it [08:57,  2.52it/s]\u001b[A\n",
            "1348it [08:57,  2.52it/s]\u001b[A\n",
            "1349it [08:57,  2.52it/s]\u001b[A\n",
            "1350it [08:58,  2.53it/s]\u001b[A\n",
            "1351it [08:58,  2.53it/s]\u001b[A\n",
            "1352it [08:59,  2.52it/s]\u001b[A\n",
            "1353it [08:59,  2.52it/s]\u001b[A\n",
            "1354it [08:59,  2.52it/s]\u001b[A\n",
            "1355it [09:00,  2.52it/s]\u001b[A\n",
            "1356it [09:00,  2.52it/s]\u001b[A\n",
            "1357it [09:01,  2.52it/s]\u001b[A\n",
            "1358it [09:01,  2.52it/s]\u001b[A\n",
            "1359it [09:01,  2.53it/s]\u001b[A\n",
            "1360it [09:02,  2.53it/s]\u001b[A\n",
            "1361it [09:02,  2.54it/s]\u001b[A\n",
            "1362it [09:03,  2.54it/s]\u001b[A\n",
            "1363it [09:03,  2.54it/s]\u001b[A\n",
            "1364it [09:03,  2.54it/s]\u001b[A\n",
            "1365it [09:04,  2.53it/s]\u001b[A\n",
            "1366it [09:04,  2.52it/s]\u001b[A\n",
            "1367it [09:05,  2.53it/s]\u001b[A\n",
            "1368it [09:05,  2.53it/s]\u001b[A\n",
            "1369it [09:05,  2.52it/s]\u001b[A\n",
            "1370it [09:06,  2.53it/s]\u001b[A\n",
            "1371it [09:06,  2.53it/s]\u001b[A\n",
            "1372it [09:07,  2.55it/s]\u001b[A\n",
            "1373it [09:07,  2.55it/s]\u001b[A\n",
            "1374it [09:07,  2.55it/s]\u001b[A\n",
            "1375it [09:08,  2.55it/s]\u001b[A\n",
            "1376it [09:08,  2.53it/s]\u001b[A\n",
            "1377it [09:08,  2.53it/s]\u001b[A\n",
            "1378it [09:09,  2.23it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.612976551055908\n",
            "eval loss: 5.4909844398498535\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1379it [09:09,  2.31it/s]\u001b[A\n",
            "1380it [09:10,  2.36it/s]\u001b[A\n",
            "1381it [09:10,  2.40it/s]\u001b[A\n",
            "1382it [09:11,  2.44it/s]\u001b[A\n",
            "1383it [09:11,  2.47it/s]\u001b[A\n",
            "1384it [09:11,  2.49it/s]\u001b[A\n",
            "1385it [09:12,  2.51it/s]\u001b[A\n",
            "1386it [09:12,  2.52it/s]\u001b[A\n",
            "1387it [09:13,  2.53it/s]\u001b[A\n",
            "1388it [09:13,  2.53it/s]\u001b[A\n",
            "1389it [09:13,  2.53it/s]\u001b[A\n",
            "1390it [09:14,  2.53it/s]\u001b[A\n",
            "1391it [09:14,  2.53it/s]\u001b[A\n",
            "1392it [09:15,  2.53it/s]\u001b[A\n",
            "1393it [09:15,  2.53it/s]\u001b[A\n",
            "1394it [09:15,  2.53it/s]\u001b[A\n",
            "1395it [09:16,  2.53it/s]\u001b[A\n",
            "1396it [09:16,  2.53it/s]\u001b[A\n",
            "1397it [09:17,  2.54it/s]\u001b[A\n",
            "1398it [09:17,  2.55it/s]\u001b[A\n",
            "1399it [09:17,  2.55it/s]\u001b[A\n",
            "1400it [09:18,  2.55it/s]\u001b[A\n",
            "1401it [09:18,  2.54it/s]\u001b[A\n",
            "1402it [09:19,  2.53it/s]\u001b[A\n",
            "1403it [09:19,  2.53it/s]\u001b[A\n",
            "1404it [09:19,  2.53it/s]\u001b[A\n",
            "1405it [09:20,  2.52it/s]\u001b[A\n",
            "1406it [09:20,  2.52it/s]\u001b[A\n",
            "1407it [09:21,  2.52it/s]\u001b[A\n",
            "1408it [09:21,  2.52it/s]\u001b[A\n",
            "1409it [09:21,  2.52it/s]\u001b[A\n",
            "1410it [09:22,  2.51it/s]\u001b[A\n",
            "1411it [09:22,  2.52it/s]\u001b[A\n",
            "1412it [09:22,  2.52it/s]\u001b[A\n",
            "1413it [09:23,  2.52it/s]\u001b[A\n",
            "1414it [09:23,  2.53it/s]\u001b[A\n",
            "1415it [09:24,  2.53it/s]\u001b[A\n",
            "1416it [09:24,  2.54it/s]\u001b[A\n",
            "1417it [09:24,  2.54it/s]\u001b[A\n",
            "1418it [09:25,  2.54it/s]\u001b[A\n",
            "1419it [09:25,  2.53it/s]\u001b[A\n",
            "1420it [09:26,  2.52it/s]\u001b[A\n",
            "1421it [09:26,  2.52it/s]\u001b[A\n",
            "1422it [09:26,  2.52it/s]\u001b[A\n",
            "1423it [09:27,  2.53it/s]\u001b[A\n",
            "1424it [09:27,  2.52it/s]\u001b[A\n",
            "1425it [09:28,  2.52it/s]\u001b[A\n",
            "1426it [09:28,  2.53it/s]\u001b[A\n",
            "1427it [09:28,  2.52it/s]\u001b[A\n",
            "1428it [09:29,  2.53it/s]\u001b[A\n",
            "1429it [09:29,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.606976509094238\n",
            "eval loss: 5.51396369934082\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1430it [09:30,  2.33it/s]\u001b[A\n",
            "1431it [09:30,  2.38it/s]\u001b[A\n",
            "1432it [09:31,  2.41it/s]\u001b[A\n",
            "1433it [09:31,  2.45it/s]\u001b[A\n",
            "1434it [09:31,  2.47it/s]\u001b[A\n",
            "1435it [09:32,  2.49it/s]\u001b[A\n",
            "1436it [09:32,  2.50it/s]\u001b[A\n",
            "1437it [09:33,  2.50it/s]\u001b[A\n",
            "1438it [09:33,  2.51it/s]\u001b[A\n",
            "1439it [09:33,  2.51it/s]\u001b[A\n",
            "1440it [09:34,  2.51it/s]\u001b[A\n",
            "1441it [09:34,  2.51it/s]\u001b[A\n",
            "1442it [09:35,  2.53it/s]\u001b[A\n",
            "1443it [09:35,  2.54it/s]\u001b[A\n",
            "1444it [09:35,  2.54it/s]\u001b[A\n",
            "1445it [09:36,  2.55it/s]\u001b[A\n",
            "1446it [09:36,  2.53it/s]\u001b[A\n",
            "1447it [09:37,  2.53it/s]\u001b[A\n",
            "1448it [09:37,  2.52it/s]\u001b[A\n",
            "1449it [09:37,  2.53it/s]\u001b[A\n",
            "1450it [09:38,  2.52it/s]\u001b[A\n",
            "1451it [09:38,  2.52it/s]\u001b[A\n",
            "1452it [09:38,  2.52it/s]\u001b[A\n",
            "1453it [09:39,  2.52it/s]\u001b[A\n",
            "1454it [09:39,  2.53it/s]\u001b[A\n",
            "1455it [09:40,  2.51it/s]\u001b[A\n",
            "1456it [09:40,  2.51it/s]\u001b[A\n",
            "1457it [09:40,  2.52it/s]\u001b[A\n",
            "1458it [09:41,  2.53it/s]\u001b[A\n",
            "1459it [09:41,  2.54it/s]\u001b[A\n",
            "1460it [09:42,  2.55it/s]\u001b[A\n",
            "1461it [09:42,  2.55it/s]\u001b[A\n",
            "1462it [09:42,  2.54it/s]\u001b[A\n",
            "1463it [09:43,  2.53it/s]\u001b[A\n",
            "1464it [09:43,  2.53it/s]\u001b[A\n",
            "1465it [09:44,  2.53it/s]\u001b[A\n",
            "1466it [09:44,  2.52it/s]\u001b[A\n",
            "1467it [09:44,  2.52it/s]\u001b[A\n",
            "1468it [09:45,  2.52it/s]\u001b[A\n",
            "1469it [09:45,  2.52it/s]\u001b[A\n",
            "1470it [09:46,  2.52it/s]\u001b[A\n",
            "1471it [09:46,  2.52it/s]\u001b[A\n",
            "1472it [09:46,  2.52it/s]\u001b[A\n",
            "1473it [09:47,  2.53it/s]\u001b[A\n",
            "1474it [09:47,  2.54it/s]\u001b[A\n",
            "1475it [09:48,  2.54it/s]\u001b[A\n",
            "1476it [09:48,  2.52it/s]\u001b[A\n",
            "1477it [09:48,  2.53it/s]\u001b[A\n",
            "1478it [09:49,  2.52it/s]\u001b[A\n",
            "1479it [09:49,  2.53it/s]\u001b[A\n",
            "1480it [09:50,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.608867645263672\n",
            "eval loss: 5.621983051300049\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1481it [09:50,  2.32it/s]\u001b[A\n",
            "1482it [09:51,  2.37it/s]\u001b[A\n",
            "1483it [09:51,  2.40it/s]\u001b[A\n",
            "1484it [09:51,  2.44it/s]\u001b[A\n",
            "1485it [09:52,  2.47it/s]\u001b[A\n",
            "1486it [09:52,  2.50it/s]\u001b[A\n",
            "1487it [09:53,  2.52it/s]\u001b[A\n",
            "1488it [09:53,  2.53it/s]\u001b[A\n",
            "1489it [09:53,  2.53it/s]\u001b[A\n",
            "1490it [09:54,  2.53it/s]\u001b[A\n",
            "1491it [09:54,  2.52it/s]\u001b[A\n",
            "1492it [09:54,  2.52it/s]\u001b[A\n",
            "1493it [09:55,  2.52it/s]\u001b[A\n",
            "1494it [09:55,  2.52it/s]\u001b[A\n",
            "1495it [09:56,  2.51it/s]\u001b[A\n",
            "1496it [09:56,  2.51it/s]\u001b[A\n",
            "1497it [09:56,  2.52it/s]\u001b[A\n",
            "1498it [09:57,  2.52it/s]\u001b[A\n",
            "1499it [09:57,  2.52it/s]\u001b[A\n",
            "1500it [09:58,  2.52it/s]\u001b[A\n",
            "1501it [09:58,  2.52it/s]\u001b[A\n",
            "1502it [09:58,  2.53it/s]\u001b[A\n",
            "1503it [09:59,  2.53it/s]\u001b[A\n",
            "1504it [09:59,  2.52it/s]\u001b[A\n",
            "1505it [10:00,  2.54it/s]\u001b[A\n",
            "1506it [10:00,  2.54it/s]\u001b[A\n",
            "1507it [10:00,  2.54it/s]\u001b[A\n",
            "1508it [10:01,  2.54it/s]\u001b[A\n",
            "1509it [10:01,  2.54it/s]\u001b[A\n",
            "1510it [10:02,  2.53it/s]\u001b[A\n",
            "1511it [10:02,  2.53it/s]\u001b[A\n",
            "1512it [10:02,  2.53it/s]\u001b[A\n",
            "1513it [10:03,  2.53it/s]\u001b[A\n",
            "1514it [10:03,  2.52it/s]\u001b[A\n",
            "1515it [10:04,  2.52it/s]\u001b[A\n",
            "1516it [10:04,  2.52it/s]\u001b[A\n",
            "1517it [10:04,  2.53it/s]\u001b[A\n",
            "1518it [10:05,  2.54it/s]\u001b[A\n",
            "1519it [10:05,  2.54it/s]\u001b[A\n",
            "1520it [10:06,  2.55it/s]\u001b[A\n",
            "1521it [10:06,  2.55it/s]\u001b[A\n",
            "1522it [10:06,  2.53it/s]\u001b[A\n",
            "1523it [10:07,  2.52it/s]\u001b[A\n",
            "1524it [10:07,  2.52it/s]\u001b[A\n",
            "1525it [10:08,  2.53it/s]\u001b[A\n",
            "1526it [10:08,  2.53it/s]\u001b[A\n",
            "1527it [10:08,  2.53it/s]\u001b[A\n",
            "1528it [10:09,  2.53it/s]\u001b[A\n",
            "1529it [10:09,  2.53it/s]\u001b[A\n",
            "1531it [10:10,  2.77it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.515800952911377\n",
            "eval loss: 5.511759281158447\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1532it [10:10,  2.69it/s]\u001b[A\n",
            "1533it [10:10,  2.64it/s]\u001b[A\n",
            "1534it [10:11,  2.61it/s]\u001b[A\n",
            "1535it [10:11,  2.58it/s]\u001b[A\n",
            "1536it [10:12,  2.57it/s]\u001b[A\n",
            "1537it [10:12,  2.56it/s]\u001b[A\n",
            "1538it [10:12,  2.55it/s]\u001b[A\n",
            "1539it [10:13,  2.53it/s]\u001b[A\n",
            "1540it [10:13,  2.52it/s]\u001b[A\n",
            "1541it [10:14,  2.53it/s]\u001b[A\n",
            "1542it [10:14,  2.52it/s]\u001b[A\n",
            "1543it [10:14,  2.53it/s]\u001b[A\n",
            "1544it [10:15,  2.54it/s]\u001b[A\n",
            "1545it [10:15,  2.55it/s]\u001b[A\n",
            "1546it [10:16,  2.55it/s]\u001b[A\n",
            "1547it [10:16,  2.55it/s]\u001b[A\n",
            "1548it [10:16,  2.55it/s]\u001b[A\n",
            "1549it [10:17,  2.54it/s]\u001b[A\n",
            "1550it [10:17,  2.53it/s]\u001b[A\n",
            "1551it [10:18,  2.53it/s]\u001b[A\n",
            "1552it [10:18,  2.53it/s]\u001b[A\n",
            "1553it [10:18,  2.54it/s]\u001b[A\n",
            "1554it [10:19,  2.53it/s]\u001b[A\n",
            "1555it [10:19,  2.53it/s]\u001b[A\n",
            "1556it [10:20,  2.53it/s]\u001b[A\n",
            "1557it [10:20,  2.54it/s]\u001b[A\n",
            "1558it [10:20,  2.54it/s]\u001b[A\n",
            "1559it [10:21,  2.55it/s]\u001b[A\n",
            "1560it [10:21,  2.55it/s]\u001b[A\n",
            "1561it [10:22,  2.55it/s]\u001b[A\n",
            "1562it [10:22,  2.54it/s]\u001b[A\n",
            "1563it [10:22,  2.54it/s]\u001b[A\n",
            "1564it [10:23,  2.54it/s]\u001b[A\n",
            "1565it [10:23,  2.54it/s]\u001b[A\n",
            "1566it [10:23,  2.52it/s]\u001b[A\n",
            "1567it [10:24,  2.53it/s]\u001b[A\n",
            "1568it [10:24,  2.53it/s]\u001b[A\n",
            "1569it [10:25,  2.51it/s]\u001b[A\n",
            "1570it [10:25,  2.51it/s]\u001b[A\n",
            "1571it [10:25,  2.51it/s]\u001b[A\n",
            "1572it [10:26,  2.52it/s]\u001b[A\n",
            "1573it [10:26,  2.53it/s]\u001b[A\n",
            "1574it [10:27,  2.54it/s]\u001b[A\n",
            "1575it [10:27,  2.54it/s]\u001b[A\n",
            "1576it [10:27,  2.55it/s]\u001b[A\n",
            "1577it [10:28,  2.54it/s]\u001b[A\n",
            "1578it [10:28,  2.53it/s]\u001b[A\n",
            "1579it [10:29,  2.53it/s]\u001b[A\n",
            "1580it [10:29,  2.52it/s]\u001b[A\n",
            "1581it [10:29,  2.53it/s]\u001b[A\n",
            "1582it [10:30,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.570403099060059\n",
            "eval loss: 5.483953475952148\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1583it [10:30,  2.32it/s]\u001b[A\n",
            "1584it [10:31,  2.38it/s]\u001b[A\n",
            "1585it [10:31,  2.43it/s]\u001b[A\n",
            "1586it [10:32,  2.46it/s]\u001b[A\n",
            "1587it [10:32,  2.48it/s]\u001b[A\n",
            "1588it [10:32,  2.49it/s]\u001b[A\n",
            "1589it [10:33,  2.50it/s]\u001b[A\n",
            "1590it [10:33,  2.51it/s]\u001b[A\n",
            "1591it [10:34,  2.51it/s]\u001b[A\n",
            "1592it [10:34,  2.52it/s]\u001b[A\n",
            "1593it [10:34,  2.52it/s]\u001b[A\n",
            "1594it [10:35,  2.52it/s]\u001b[A\n",
            "1595it [10:35,  2.53it/s]\u001b[A\n",
            "1596it [10:36,  2.53it/s]\u001b[A\n",
            "1597it [10:36,  2.53it/s]\u001b[A\n",
            "1598it [10:36,  2.53it/s]\u001b[A\n",
            "1599it [10:37,  2.54it/s]\u001b[A\n",
            "1600it [10:37,  2.55it/s]\u001b[A\n",
            "1601it [10:37,  2.54it/s]\u001b[A\n",
            "1602it [10:38,  2.54it/s]\u001b[A\n",
            "1603it [10:38,  2.53it/s]\u001b[A\n",
            "1604it [10:39,  2.53it/s]\u001b[A\n",
            "1605it [10:39,  2.52it/s]\u001b[A\n",
            "1606it [10:39,  2.51it/s]\u001b[A\n",
            "1607it [10:40,  2.52it/s]\u001b[A\n",
            "1608it [10:40,  2.52it/s]\u001b[A\n",
            "1609it [10:41,  2.52it/s]\u001b[A\n",
            "1610it [10:41,  2.52it/s]\u001b[A\n",
            "1611it [10:41,  2.52it/s]\u001b[A\n",
            "1612it [10:42,  2.52it/s]\u001b[A\n",
            "1613it [10:42,  2.53it/s]\u001b[A\n",
            "1614it [10:43,  2.54it/s]\u001b[A\n",
            "1615it [10:43,  2.55it/s]\u001b[A\n",
            "1616it [10:43,  2.55it/s]\u001b[A\n",
            "1617it [10:44,  2.55it/s]\u001b[A\n",
            "1618it [10:44,  2.53it/s]\u001b[A\n",
            "1619it [10:45,  2.54it/s]\u001b[A\n",
            "1620it [10:45,  2.53it/s]\u001b[A\n",
            "1621it [10:45,  2.53it/s]\u001b[A\n",
            "1622it [10:46,  2.52it/s]\u001b[A\n",
            "1623it [10:46,  2.53it/s]\u001b[A\n",
            "1624it [10:47,  2.53it/s]\u001b[A\n",
            "1625it [10:47,  2.52it/s]\u001b[A\n",
            "1626it [10:47,  2.52it/s]\u001b[A\n",
            "1627it [10:48,  2.52it/s]\u001b[A\n",
            "1628it [10:48,  2.53it/s]\u001b[A\n",
            "1629it [10:49,  2.54it/s]\u001b[A\n",
            "1630it [10:49,  2.53it/s]\u001b[A\n",
            "1631it [10:49,  2.53it/s]\u001b[A\n",
            "1632it [10:50,  2.54it/s]\u001b[A\n",
            "1633it [10:50,  2.25it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.497532844543457\n",
            "eval loss: 5.497312068939209\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1634it [10:51,  2.32it/s]\u001b[A\n",
            "1635it [10:51,  2.38it/s]\u001b[A\n",
            "1636it [10:51,  2.43it/s]\u001b[A\n",
            "1637it [10:52,  2.46it/s]\u001b[A\n",
            "1638it [10:52,  2.49it/s]\u001b[A\n",
            "1639it [10:53,  2.52it/s]\u001b[A\n",
            "1640it [10:53,  2.53it/s]\u001b[A\n",
            "1641it [10:53,  2.52it/s]\u001b[A\n",
            "1642it [10:54,  2.53it/s]\u001b[A\n",
            "1643it [10:54,  2.53it/s]\u001b[A\n",
            "1644it [10:55,  2.53it/s]\u001b[A\n",
            "1645it [10:55,  2.53it/s]\u001b[A\n",
            "1646it [10:55,  2.53it/s]\u001b[A\n",
            "1647it [10:56,  2.53it/s]\u001b[A\n",
            "1648it [10:56,  2.53it/s]\u001b[A\n",
            "1649it [10:57,  2.53it/s]\u001b[A\n",
            "1650it [10:57,  2.54it/s]\u001b[A\n",
            "1651it [10:57,  2.55it/s]\u001b[A\n",
            "1652it [10:58,  2.54it/s]\u001b[A\n",
            "1653it [10:58,  2.54it/s]\u001b[A\n",
            "1654it [10:59,  2.54it/s]\u001b[A\n",
            "1655it [10:59,  2.54it/s]\u001b[A\n",
            "1656it [10:59,  2.53it/s]\u001b[A\n",
            "1657it [11:00,  2.52it/s]\u001b[A\n",
            "1658it [11:00,  2.53it/s]\u001b[A\n",
            "1659it [11:01,  2.53it/s]\u001b[A\n",
            "1660it [11:01,  2.53it/s]\u001b[A\n",
            "1661it [11:01,  2.52it/s]\u001b[A\n",
            "1662it [11:02,  2.52it/s]\u001b[A\n",
            "1663it [11:02,  2.53it/s]\u001b[A\n",
            "1664it [11:03,  2.53it/s]\u001b[A\n",
            "1665it [11:03,  2.54it/s]\u001b[A\n",
            "1666it [11:03,  2.54it/s]\u001b[A\n",
            "1667it [11:04,  2.55it/s]\u001b[A\n",
            "1668it [11:04,  2.54it/s]\u001b[A\n",
            "1669it [11:04,  2.54it/s]\u001b[A\n",
            "1670it [11:05,  2.53it/s]\u001b[A\n",
            "1671it [11:05,  2.52it/s]\u001b[A\n",
            "1672it [11:06,  2.51it/s]\u001b[A\n",
            "1673it [11:06,  2.51it/s]\u001b[A\n",
            "1674it [11:06,  2.52it/s]\u001b[A\n",
            "1675it [11:07,  2.52it/s]\u001b[A\n",
            "1676it [11:07,  2.51it/s]\u001b[A\n",
            "1677it [11:08,  2.52it/s]\u001b[A\n",
            "1678it [11:08,  2.51it/s]\u001b[A\n",
            "1679it [11:08,  2.53it/s]\u001b[A\n",
            "1680it [11:09,  2.53it/s]\u001b[A\n",
            "1681it [11:09,  2.53it/s]\u001b[A\n",
            "1682it [11:10,  2.54it/s]\u001b[A\n",
            "1683it [11:10,  2.55it/s]\u001b[A\n",
            "1684it [11:11,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.473233699798584\n",
            "eval loss: 5.531450271606445\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1685it [11:11,  2.32it/s]\u001b[A\n",
            "1686it [11:11,  2.38it/s]\u001b[A\n",
            "1687it [11:12,  2.41it/s]\u001b[A\n",
            "1688it [11:12,  2.45it/s]\u001b[A\n",
            "1689it [11:13,  2.48it/s]\u001b[A\n",
            "1690it [11:13,  2.50it/s]\u001b[A\n",
            "1691it [11:13,  2.52it/s]\u001b[A\n",
            "1692it [11:14,  2.53it/s]\u001b[A\n",
            "1693it [11:14,  2.54it/s]\u001b[A\n",
            "1694it [11:15,  2.54it/s]\u001b[A\n",
            "1695it [11:15,  2.53it/s]\u001b[A\n",
            "1696it [11:15,  2.53it/s]\u001b[A\n",
            "1697it [11:16,  2.53it/s]\u001b[A\n",
            "1698it [11:16,  2.53it/s]\u001b[A\n",
            "1699it [11:17,  2.53it/s]\u001b[A\n",
            "1700it [11:17,  2.53it/s]\u001b[A\n",
            "1701it [11:17,  2.52it/s]\u001b[A\n",
            "1702it [11:18,  2.52it/s]\u001b[A\n",
            "1703it [11:18,  2.52it/s]\u001b[A\n",
            "1704it [11:18,  2.53it/s]\u001b[A\n",
            "1705it [11:19,  2.54it/s]\u001b[A\n",
            "1706it [11:19,  2.55it/s]\u001b[A\n",
            "1707it [11:20,  2.55it/s]\u001b[A\n",
            "1708it [11:20,  2.55it/s]\u001b[A\n",
            "1709it [11:20,  2.53it/s]\u001b[A\n",
            "1710it [11:21,  2.53it/s]\u001b[A\n",
            "1711it [11:21,  2.53it/s]\u001b[A\n",
            "1712it [11:22,  2.53it/s]\u001b[A\n",
            "1713it [11:22,  2.52it/s]\u001b[A\n",
            "1714it [11:22,  2.53it/s]\u001b[A\n",
            "1715it [11:23,  2.51it/s]\u001b[A\n",
            "1716it [11:23,  2.52it/s]\u001b[A\n",
            "1717it [11:24,  2.53it/s]\u001b[A\n",
            "1718it [11:24,  2.53it/s]\u001b[A\n",
            "1719it [11:24,  2.52it/s]\u001b[A\n",
            "1720it [11:25,  2.53it/s]\u001b[A\n",
            "1721it [11:25,  2.54it/s]\u001b[A\n",
            "1722it [11:26,  2.55it/s]\u001b[A\n",
            "1723it [11:26,  2.55it/s]\u001b[A\n",
            "1724it [11:26,  2.55it/s]\u001b[A\n",
            "1725it [11:27,  2.53it/s]\u001b[A\n",
            "1726it [11:27,  2.52it/s]\u001b[A\n",
            "1727it [11:28,  2.52it/s]\u001b[A\n",
            "1728it [11:28,  2.51it/s]\u001b[A\n",
            "1729it [11:28,  2.52it/s]\u001b[A\n",
            "1730it [11:29,  2.52it/s]\u001b[A\n",
            "1731it [11:29,  2.53it/s]\u001b[A\n",
            "1732it [11:30,  2.53it/s]\u001b[A\n",
            "1733it [11:30,  2.54it/s]\u001b[A\n",
            "1734it [11:30,  2.55it/s]\u001b[A\n",
            "1735it [11:31,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.515231132507324\n",
            "eval loss: 5.629790306091309\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1736it [11:31,  2.33it/s]\u001b[A\n",
            "1737it [11:32,  2.38it/s]\u001b[A\n",
            "1738it [11:32,  2.42it/s]\u001b[A\n",
            "1739it [11:32,  2.45it/s]\u001b[A\n",
            "1740it [11:33,  2.47it/s]\u001b[A\n",
            "1741it [11:33,  2.49it/s]\u001b[A\n",
            "1742it [11:34,  2.50it/s]\u001b[A\n",
            "1743it [11:34,  2.51it/s]\u001b[A\n",
            "1744it [11:34,  2.52it/s]\u001b[A\n",
            "1745it [11:35,  2.52it/s]\u001b[A\n",
            "1746it [11:35,  2.53it/s]\u001b[A\n",
            "1747it [11:36,  2.54it/s]\u001b[A\n",
            "1748it [11:36,  2.54it/s]\u001b[A\n",
            "1749it [11:36,  2.54it/s]\u001b[A\n",
            "1750it [11:37,  2.53it/s]\u001b[A\n",
            "1751it [11:37,  2.53it/s]\u001b[A\n",
            "1752it [11:38,  2.52it/s]\u001b[A\n",
            "1753it [11:38,  2.53it/s]\u001b[A\n",
            "1754it [11:38,  2.51it/s]\u001b[A\n",
            "1755it [11:39,  2.50it/s]\u001b[A\n",
            "1756it [11:39,  2.51it/s]\u001b[A\n",
            "1757it [11:40,  2.51it/s]\u001b[A\n",
            "1758it [11:40,  2.52it/s]\u001b[A\n",
            "1759it [11:40,  2.52it/s]\u001b[A\n",
            "1760it [11:41,  2.52it/s]\u001b[A\n",
            "1761it [11:41,  2.53it/s]\u001b[A\n",
            "1762it [11:42,  2.53it/s]\u001b[A\n",
            "1763it [11:42,  2.54it/s]\u001b[A\n",
            "1764it [11:42,  2.55it/s]\u001b[A\n",
            "1765it [11:43,  2.55it/s]\u001b[A\n",
            "1766it [11:43,  2.54it/s]\u001b[A\n",
            "1767it [11:44,  2.54it/s]\u001b[A\n",
            "1768it [11:44,  2.54it/s]\u001b[A\n",
            "1769it [11:44,  2.53it/s]\u001b[A\n",
            "1770it [11:45,  2.52it/s]\u001b[A\n",
            "1771it [11:45,  2.52it/s]\u001b[A\n",
            "1772it [11:46,  2.52it/s]\u001b[A\n",
            "1773it [11:46,  2.53it/s]\u001b[A\n",
            "1774it [11:46,  2.52it/s]\u001b[A\n",
            "1775it [11:47,  2.52it/s]\u001b[A\n",
            "1776it [11:47,  2.51it/s]\u001b[A\n",
            "1777it [11:48,  2.52it/s]\u001b[A\n",
            "1778it [11:48,  2.52it/s]\u001b[A\n",
            "1779it [11:48,  2.52it/s]\u001b[A\n",
            "1780it [11:49,  2.54it/s]\u001b[A\n",
            "1781it [11:49,  2.55it/s]\u001b[A\n",
            "1782it [11:49,  2.54it/s]\u001b[A\n",
            "1783it [11:50,  2.54it/s]\u001b[A\n",
            "1784it [11:50,  2.54it/s]\u001b[A\n",
            "1785it [11:51,  2.53it/s]\u001b[A\n",
            "1786it [11:51,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.523017883300781\n",
            "eval loss: 5.502058506011963\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1787it [11:52,  2.32it/s]\u001b[A\n",
            "1788it [11:52,  2.38it/s]\u001b[A\n",
            "1789it [11:52,  2.43it/s]\u001b[A\n",
            "1790it [11:53,  2.47it/s]\u001b[A\n",
            "1791it [11:53,  2.49it/s]\u001b[A\n",
            "1792it [11:54,  2.50it/s]\u001b[A\n",
            "1793it [11:54,  2.51it/s]\u001b[A\n",
            "1794it [11:54,  2.51it/s]\u001b[A\n",
            "1795it [11:55,  2.51it/s]\u001b[A\n",
            "1796it [11:55,  2.52it/s]\u001b[A\n",
            "1797it [11:56,  2.52it/s]\u001b[A\n",
            "1798it [11:56,  2.52it/s]\u001b[A\n",
            "1799it [11:56,  2.52it/s]\u001b[A\n",
            "1800it [11:57,  2.52it/s]\u001b[A\n",
            "1801it [11:57,  2.52it/s]\u001b[A\n",
            "1802it [11:58,  2.51it/s]\u001b[A\n",
            "1803it [11:58,  2.51it/s]\u001b[A\n",
            "1804it [11:58,  2.52it/s]\u001b[A\n",
            "1805it [11:59,  2.52it/s]\u001b[A\n",
            "1806it [11:59,  2.53it/s]\u001b[A\n",
            "1807it [12:00,  2.55it/s]\u001b[A\n",
            "1808it [12:00,  2.55it/s]\u001b[A\n",
            "1809it [12:00,  2.54it/s]\u001b[A\n",
            "1810it [12:01,  2.54it/s]\u001b[A\n",
            "1811it [12:01,  2.53it/s]\u001b[A\n",
            "1812it [12:02,  2.54it/s]\u001b[A\n",
            "1813it [12:02,  2.52it/s]\u001b[A\n",
            "1814it [12:02,  2.51it/s]\u001b[A\n",
            "1815it [12:03,  2.52it/s]\u001b[A\n",
            "1816it [12:03,  2.51it/s]\u001b[A\n",
            "1817it [12:04,  2.52it/s]\u001b[A\n",
            "1818it [12:04,  2.52it/s]\u001b[A\n",
            "1819it [12:04,  2.52it/s]\u001b[A\n",
            "1820it [12:05,  2.53it/s]\u001b[A\n",
            "1821it [12:05,  2.53it/s]\u001b[A\n",
            "1822it [12:05,  2.54it/s]\u001b[A\n",
            "1823it [12:06,  2.55it/s]\u001b[A\n",
            "1824it [12:06,  2.56it/s]\u001b[A\n",
            "1825it [12:07,  2.55it/s]\u001b[A\n",
            "1826it [12:07,  2.54it/s]\u001b[A\n",
            "1827it [12:07,  2.54it/s]\u001b[A\n",
            "1828it [12:08,  2.54it/s]\u001b[A\n",
            "1829it [12:08,  2.53it/s]\u001b[A\n",
            "1830it [12:09,  2.52it/s]\u001b[A\n",
            "1831it [12:09,  2.53it/s]\u001b[A\n",
            "1832it [12:09,  2.53it/s]\u001b[A\n",
            "1833it [12:10,  2.53it/s]\u001b[A\n",
            "1834it [12:10,  2.53it/s]\u001b[A\n",
            "1835it [12:11,  2.54it/s]\u001b[A\n",
            "1836it [12:11,  2.55it/s]\u001b[A\n",
            "1837it [12:12,  2.26it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.60330057144165\n",
            "eval loss: 5.472036838531494\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1838it [12:12,  2.34it/s]\u001b[A\n",
            "1839it [12:12,  2.39it/s]\u001b[A\n",
            "1840it [12:13,  2.43it/s]\u001b[A\n",
            "1841it [12:13,  2.46it/s]\u001b[A\n",
            "1842it [12:14,  2.49it/s]\u001b[A\n",
            "1843it [12:14,  2.51it/s]\u001b[A\n",
            "1844it [12:14,  2.52it/s]\u001b[A\n",
            "1845it [12:15,  2.53it/s]\u001b[A\n",
            "1846it [12:15,  2.53it/s]\u001b[A\n",
            "1847it [12:15,  2.52it/s]\u001b[A\n",
            "1848it [12:16,  2.51it/s]\u001b[A\n",
            "1849it [12:16,  2.52it/s]\u001b[A\n",
            "1850it [12:17,  2.51it/s]\u001b[A\n",
            "1851it [12:17,  2.50it/s]\u001b[A\n",
            "1852it [12:17,  2.50it/s]\u001b[A\n",
            "1853it [12:18,  2.51it/s]\u001b[A\n",
            "1854it [12:18,  2.51it/s]\u001b[A\n",
            "1855it [12:19,  2.52it/s]\u001b[A\n",
            "1856it [12:19,  2.52it/s]\u001b[A\n",
            "1857it [12:19,  2.52it/s]\u001b[A\n",
            "1858it [12:20,  2.51it/s]\u001b[A\n",
            "1859it [12:20,  2.51it/s]\u001b[A\n",
            "1860it [12:21,  2.52it/s]\u001b[A\n",
            "1861it [12:21,  2.53it/s]\u001b[A\n",
            "1862it [12:21,  2.54it/s]\u001b[A\n",
            "1863it [12:22,  2.55it/s]\u001b[A\n",
            "1864it [12:22,  2.55it/s]\u001b[A\n",
            "1865it [12:23,  2.54it/s]\u001b[A\n",
            "1866it [12:23,  2.54it/s]\u001b[A\n",
            "1867it [12:23,  2.54it/s]\u001b[A\n",
            "1868it [12:24,  2.53it/s]\u001b[A\n",
            "1869it [12:24,  2.52it/s]\u001b[A\n",
            "1870it [12:25,  2.51it/s]\u001b[A\n",
            "1871it [12:25,  2.52it/s]\u001b[A\n",
            "1872it [12:25,  2.52it/s]\u001b[A\n",
            "1873it [12:26,  2.52it/s]\u001b[A\n",
            "1874it [12:26,  2.52it/s]\u001b[A\n",
            "1875it [12:27,  2.53it/s]\u001b[A\n",
            "1876it [12:27,  2.54it/s]\u001b[A\n",
            "1877it [12:27,  2.54it/s]\u001b[A\n",
            "1878it [12:28,  2.56it/s]\u001b[A\n",
            "1879it [12:28,  2.55it/s]\u001b[A\n",
            "1880it [12:29,  2.54it/s]\u001b[A\n",
            "1881it [12:29,  2.53it/s]\u001b[A\n",
            "1882it [12:29,  2.53it/s]\u001b[A\n",
            "1883it [12:30,  2.53it/s]\u001b[A\n",
            "1884it [12:30,  2.53it/s]\u001b[A\n",
            "1885it [12:31,  2.52it/s]\u001b[A\n",
            "1886it [12:31,  2.51it/s]\u001b[A\n",
            "1887it [12:31,  2.52it/s]\u001b[A\n",
            "1888it [12:32,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.465083599090576\n",
            "eval loss: 5.492844581604004\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1889it [12:32,  2.31it/s]\u001b[A\n",
            "1890it [12:33,  2.38it/s]\u001b[A\n",
            "1891it [12:33,  2.43it/s]\u001b[A\n",
            "1892it [12:33,  2.45it/s]\u001b[A\n",
            "1893it [12:34,  2.47it/s]\u001b[A\n",
            "1894it [12:34,  2.49it/s]\u001b[A\n",
            "1895it [12:35,  2.49it/s]\u001b[A\n",
            "1896it [12:35,  2.50it/s]\u001b[A\n",
            "1897it [12:35,  2.51it/s]\u001b[A\n",
            "1898it [12:36,  2.52it/s]\u001b[A\n",
            "1899it [12:36,  2.52it/s]\u001b[A\n",
            "1900it [12:37,  2.52it/s]\u001b[A\n",
            "1901it [12:37,  2.53it/s]\u001b[A\n",
            "1902it [12:37,  2.54it/s]\u001b[A\n",
            "1903it [12:38,  2.54it/s]\u001b[A\n",
            "1904it [12:38,  2.55it/s]\u001b[A\n",
            "1905it [12:39,  2.55it/s]\u001b[A\n",
            "1906it [12:39,  2.55it/s]\u001b[A\n",
            "1907it [12:39,  2.54it/s]\u001b[A\n",
            "1908it [12:40,  2.54it/s]\u001b[A\n",
            "1909it [12:40,  2.54it/s]\u001b[A\n",
            "1910it [12:41,  2.54it/s]\u001b[A\n",
            "1911it [12:41,  2.53it/s]\u001b[A\n",
            "1912it [12:41,  2.52it/s]\u001b[A\n",
            "1913it [12:42,  2.52it/s]\u001b[A\n",
            "1914it [12:42,  2.51it/s]\u001b[A\n",
            "1915it [12:43,  2.52it/s]\u001b[A\n",
            "1916it [12:43,  2.53it/s]\u001b[A\n",
            "1917it [12:43,  2.54it/s]\u001b[A\n",
            "1918it [12:44,  2.55it/s]\u001b[A\n",
            "1919it [12:44,  2.55it/s]\u001b[A\n",
            "1920it [12:45,  2.54it/s]\u001b[A\n",
            "1921it [12:45,  2.54it/s]\u001b[A\n",
            "1922it [12:45,  2.54it/s]\u001b[A\n",
            "1923it [12:46,  2.53it/s]\u001b[A\n",
            "1924it [12:46,  2.53it/s]\u001b[A\n",
            "1925it [12:47,  2.53it/s]\u001b[A\n",
            "1926it [12:47,  2.52it/s]\u001b[A\n",
            "1927it [12:47,  2.52it/s]\u001b[A\n",
            "1928it [12:48,  2.51it/s]\u001b[A\n",
            "1929it [12:48,  2.53it/s]\u001b[A\n",
            "1930it [12:48,  2.53it/s]\u001b[A\n",
            "1931it [12:49,  2.54it/s]\u001b[A\n",
            "1932it [12:49,  2.55it/s]\u001b[A\n",
            "1933it [12:50,  2.55it/s]\u001b[A\n",
            "1934it [12:50,  2.55it/s]\u001b[A\n",
            "1935it [12:50,  2.54it/s]\u001b[A\n",
            "1936it [12:51,  2.53it/s]\u001b[A\n",
            "1937it [12:51,  2.53it/s]\u001b[A\n",
            "1938it [12:52,  2.54it/s]\u001b[A\n",
            "1939it [12:52,  2.24it/s]\u001b[A"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "train loss: 5.454954147338867\n",
            "eval loss: 5.440124034881592\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "1940it [12:53,  2.31it/s]\u001b[A\n",
            "1941it [12:53,  2.38it/s]\u001b[A\n",
            "1942it [12:53,  2.43it/s]\u001b[A\n",
            "1943it [12:54,  2.46it/s]\u001b[A\n",
            "1944it [12:54,  2.48it/s]\u001b[A\n",
            "1945it [12:55,  2.50it/s]\u001b[A\n",
            "1946it [12:55,  2.50it/s]\u001b[A\n",
            "1947it [12:55,  2.51it/s]\u001b[A\n",
            "1948it [12:56,  2.52it/s]\u001b[A\n",
            "1949it [12:56,  2.52it/s]\u001b[A\n",
            "1950it [12:57,  2.52it/s]\u001b[A\n",
            "1951it [12:57,  2.52it/s]\u001b[A\n",
            "1952it [12:57,  2.53it/s]\u001b[A\n",
            "1953it [12:58,  2.54it/s]\u001b[A\n",
            "1954it [12:58,  2.55it/s]\u001b[A\n",
            "1955it [12:59,  2.55it/s]\u001b[A\n",
            "1956it [12:59,  2.55it/s]\u001b[A\n",
            "1957it [12:59,  2.55it/s]\u001b[A\n",
            "1958it [13:00,  2.54it/s]\u001b[A\n",
            "1959it [13:00,  2.53it/s]\u001b[A\n",
            "1960it [13:00,  2.53it/s]\u001b[A\n",
            "1961it [13:01,  2.53it/s]\u001b[A\n",
            "1962it [13:01,  2.51it/s]\u001b[A\n",
            "1963it [13:02,  2.51it/s]\u001b[A\n",
            "1964it [13:02,  2.51it/s]\u001b[A\n",
            "1965it [13:02,  2.51it/s]\u001b[A\n",
            "1966it [13:03,  2.51it/s]\u001b[A\n",
            "1967it [13:03,  2.51it/s]\u001b[A\n",
            "1968it [13:04,  2.52it/s]\u001b[A\n",
            "1969it [13:04,  2.53it/s]\u001b[A\n",
            "1970it [13:04,  2.53it/s]\u001b[A\n",
            "1971it [13:05,  2.54it/s]\u001b[A\n",
            "1972it [13:05,  2.55it/s]\u001b[A\n",
            "1973it [13:06,  2.55it/s]\u001b[A\n",
            "1974it [13:06,  2.54it/s]\u001b[A\n",
            "1975it [13:06,  2.54it/s]\u001b[A\n",
            "1976it [13:07,  2.54it/s]\u001b[A\n",
            "1977it [13:07,  2.52it/s]\u001b[A\n",
            "1978it [13:08,  2.52it/s]\u001b[A\n",
            "1979it [13:08,  2.52it/s]\u001b[A\n",
            "1980it [13:08,  2.53it/s]\u001b[A"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ASd09piCICMC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e9cc2fa7-5f02-41b4-faf7-fe998c76824d"
      },
      "source": [
        "example = next(eval_it).text.cuda()\n",
        "buffer = example.clone()\n",
        "model.eval()\n",
        "for i in range(example.size(0) // 2, example.size(0)):\n",
        "  pred = model(buffer)\n",
        "  buffer[i] = torch.multinomial(torch.exp(pred[i - 1, 0]), 1)\n",
        "\n",
        "text = [field.vocab.itos[idx] for idx in buffer[:, 0]]\n",
        "print('dataset portion:', text[: example.size(0) // 2])\n",
        "print('predicted portion:', text[example.size(0) // 2 :])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dataset portion: ['is', 'a', 'highly', '<unk>', 'food', ',', 'and', 'is', 'widely', 'caught', 'using', 'lobster', '<unk>', ',', 'mostly', 'around']\n",
            "predicted portion: ['any', '<unk>', 'and', 'other', 'methods', 'of', 'poor', 'and', 'hold', 'that', 'are', 'treated', 'to', 'a', 'person', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iU7sDnuxG3DO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 612
        },
        "outputId": "f7eb8cd3-9c40-4b93-a7d4-38d76843cb3d"
      },
      "source": [
        "|"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Hutchings',\n",
              " '(',\n",
              " 'Aniston',\n",
              " 'of',\n",
              " 'University',\n",
              " 'with',\n",
              " 'in',\n",
              " 'of',\n",
              " 'urgent',\n",
              " 'signal',\n",
              " '<unk>',\n",
              " 'seller',\n",
              " ',',\n",
              " ',',\n",
              " 'force',\n",
              " \"'s\",\n",
              " 'she',\n",
              " 'television',\n",
              " '!',\n",
              " 'could',\n",
              " '.',\n",
              " ',',\n",
              " 'package',\n",
              " 'and',\n",
              " 'Disciplina',\n",
              " ')',\n",
              " ',',\n",
              " '\"',\n",
              " 'had',\n",
              " ',',\n",
              " ',',\n",
              " '.',\n",
              " 'founded',\n",
              " 'in',\n",
              " '<unk>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCzcurten8cD",
        "colab_type": "text"
      },
      "source": [
        "Adapted from [PyTorch](https://github.com/pytorch/examples/tree/master/word_language_model)"
      ]
    }
  ]
}