Week07 - Language Models Part 01.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OE7fXh-OSJYF"
   },
   "outputs": [],
   "source": [
    "!pip3 -qq install torch==0.4.1\n",
    "!pip -qq install torchtext==0.3.1\n",
    "!wget -qq --no-check-certificate 'https://drive.google.com/uc?export=download&id=1Pq4aklVdj-sOnQw68e1ZZ_ImMiC8IR1V' -O tweets.csv.zip\n",
    "!wget -qq --no-check-certificate \"https://drive.google.com/uc?export=download&id=1ji7dhr9FojPeV51dDlKRERIqr3vdZfhu\" -O surnames.txt\n",
    "!unzip tweets.csv.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uhvfH55PUJ8K"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    from torch.cuda import FloatTensor, LongTensor\n",
    "    DEVICE = torch.device('cuda')\n",
    "else:\n",
    "    from torch import FloatTensor, LongTensor\n",
    "    DEVICE = torch.device('cpu')\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jVcnkGDgxfNx"
   },
   "source": [
    "# Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Kjg1Z3xxmEP"
   },
   "source": [
    "\n",
    "*The language model* is a piece that is able to estimate the probabilities of meeting the sequence of words $ w_1, \\ldots, w_n $:\n",
    "\n",
    "$$ \\mathbf {P} (w_1, \\ldots, w_n) = \\prod_k \\mathbf {P} (w_k | w_ {k-1}, \\ldots, w_ {1}). $$\n",
    "\n",
    "Interpretable and interesting here are conditional probabilities - what word does the language model expect following the data. We all have such a language model, that's it. For example, in this context\n",
    "\n",
    "<center>\n",
    "<img src=\"https://hsto.org/web/956/239/601/95623960157b4e15a1b3f599aed62ed2.png\" width=\"20%\">\n",
    "</center>\n",
    "\n",
    "my language model says - after * honest * hardly * my * will go. But * and * or, of course, * the rules * - very much so.\n",
    "\n",
    "And the task is to learn how to generate political tweets in the image and likeness of `Russian Troll Tweets`. Dataset taken from here: https://www.kaggle.com/vikasg/russian-troll-tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JpjfUoN4_WY7"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('tweets.csv')\n",
    "\n",
    "data.text.sample(15).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WAQ4d__2_sAz"
   },
   "source": [
    "Yes, the results will be persistent, I warn you immediately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Qvqidof7Fsi"
   },
   "source": [
    "## Reading data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OSu56oDX-KY5"
   },
   "source": [
    "Has anyone already gotten enough of writing all these builds, dictionaries - is that all? Personally, me - yes!\n",
    "\n",
    "In pytorch there is a special class for generating batches - `Dataset`. Instead of writing a function like `iterate_batches`, you can inherit from it and override the methods` __len__` and `__getitem__` ... and implement almost everything that was in ʻiterate_batches in them. Not impressive yet, is it?\n",
    "\n",
    "There is also a `DataLoader` that can work with dataset. It allows you to make shuffle batches and generate them in separate processes - this is especially important when the generation of a batch is a long operation. For example, in pictures. You can read about all this here: [Data Loading and Processing Tutorial] (https://pytorch.org/tutorials/beginner/data_loading_tutorial.html).\n",
    "\n",
    "But so far it is still not very cool, it seems to me. Another thing is interesting - pytorch has a separate library in the repository - [torchtext] (https://github.com/pytorch/text). Here it already gives us special implementations of `Dataset` for working with text and all sorts of tools that make life a little easier.\n",
    "\n",
    "The library, in my opinion, lacks tutorials that show how to work with it - but you can read the source code, it is nice.\n",
    "\n",
    "The plan is to build a class `torchtext.data.Dataset`, create an iterator for it, and learn the model.\n",
    "\n",
    "This data is initialized with two parameters:\n",
    "```\n",
    "            examples: List of Examples.\n",
    "            fields (List (tuple (str, Field))): The Fields to use in this tuple. The\n",
    "                The field is the field name.\n",
    "```\n",
    "We will understand first with the second.\n",
    "\n",
    "`Field` is such a meta-information for dataset + sample handler.\n",
    "It has a bunch of options that are easier to look at [here] (https://github.com/pytorch/text/blob/master/torchtext/data/field.py). In short, he can preprocess (for example, tokenize) sentences, build a dictionary (mapping from word to index), build batchy — add paddings and convert to tensors. What else is needed in life?\n",
    "\n",
    "We will do a character-level language model, so tokenization for us is the transformation of a string into a set of characters. We also ask you to add the special characters `<s>` and `</ s>` to the beginning and end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ilAMVxA8Xy4L"
   },
   "outputs": [],
   "source": [
    "from torchtext.data import Field\n",
    "\n",
    "text_field = Field(init_token='<s>', eos_token='</s>', lower=True, tokenize=lambda line: list(line))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L_i0Z6JhF0rA"
   },
   "source": [
    "\n",
    "Preprocessing will look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B-8IPlPHFyKa"
   },
   "outputs": [],
   "source": [
    "text_field.preprocess(data.text.iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "19NFhTSNF1_1"
   },
   "source": [
    "\n",
    "Convert everything and look at the length distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wz1QnivMBmU3"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "data['text'] = data['text'].fillna('')\n",
    "lines = data.apply(lambda row: text_field.preprocess(row['text']), axis=1).tolist()\n",
    "\n",
    "lengths = [len(line) for line in lines]\n",
    "\n",
    "plt.hist(lengths, bins=30)[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dE9rPW9UHE7d"
   },
   "source": [
    "Cut off too short lines and convert the remaining ones to `Example`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jfTlpBxODBg8"
   },
   "outputs": [],
   "source": [
    "from torchtext.data import Example\n",
    "\n",
    "lines = [line for line in lines if len(line) >= 50]\n",
    "\n",
    "fields = [('text', text_field)]\n",
    "examples = [Example.fromlist([line], fields) for line in lines]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7z1wPlz_HeEP"
   },
   "source": [
    "By `Example` you can get back all the fields that we shoved there. For example, now we have created one `text` field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iGMRSuk_HYCm"
   },
   "outputs": [],
   "source": [
    "examples[0].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Yef1bv2MQcEA"
   },
   "source": [
    "Let's build, at last,:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gSccEmVIHAaQ"
   },
   "outputs": [],
   "source": [
    "from torchtext.data import Dataset\n",
    "\n",
    "dataset = Dataset(examples, fields)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vEe5YXIpRCYD"
   },
   "source": [
    "Dataset can be divided into parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "21whmJDFRBV1"
   },
   "outputs": [],
   "source": [
    "train_dataset, test_dataset = dataset.split(split_ratio=0.75)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "14CyhugSQsOf"
   },
   "source": [
    "On it you can build a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NQs3jbhyQkJD"
   },
   "outputs": [],
   "source": [
    "text_field.build_vocab(train_dataset, min_freq=30)\n",
    "\n",
    "print('Vocab size =', len(text_field.vocab))\n",
    "print(text_field.vocab.itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-_EAdgsWRTzj"
   },
   "source": [
    "Finally, it can be iterated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qaEMoxdVG98p"
   },
   "outputs": [],
   "source": [
    "from torchtext.data import BucketIterator\n",
    "\n",
    "train_iter, test_iter = BucketIterator.splits(datasets=(train_dataset, test_dataset), batch_sizes=(32, 128), \n",
    "                                              shuffle=True, device=DEVICE, sort=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RMG4L1-5RXnb"
   },
   "outputs": [],
   "source": [
    "batch = next(iter(train_iter))\n",
    "\n",
    "batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7ZZgplOkReeq"
   },
   "outputs": [],
   "source": [
    "batch.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oTrSUkqEhZzh"
   },
   "source": [
    "## Perplexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gqc9HpTM-FwD"
   },
   "source": [
    "Our task, as always, needs to start with two questions - which metric is optimized and which baseline.\n",
    "\n",
    "With a metric, everything is simple - we want the model to know how best to approximate the distribution of words in a language. We don’t have a whole language, so let's do a test sample.\n",
    "\n",
    "It is possible to calculate cross-entropy losses on it:\n",
    "\n",
    "$$H(w_1, \\ldots, w_n) = - \\frac 1n \\sum_k \\log\\mathbf{P}(w_k | w_{k-1}, \\ldots, w_1).$$\n",
    "\n",
    "Here the probability $ \\mathbf {P} $ is the probability estimated by our language model. The ideal model would give a probability equal to 1 for words in the text and the losses would be zero - although this is of course impossible, even you cannot predict the next word, what to say about a soulless machine.\n",
    "\n",
    "Thus, as always, we optimize cross-entropy and strive to make it as low as possible.\n",
    "\n",
    "Well, almost everything. There is also a separate metric for language models - * perplexion *. These are simply exponential cross-entropy losses:\n",
    "\n",
    "$$PP(w_1, \\ldots, w_n) = e^{H(w_1, \\ldots, w_n)} = e^{- \\frac 1n \\sum_k \\log\\mathbf{P}(w_k | w_{k-1}, \\ldots, w_1)} = \\left(\\mathbf{P}(w_1, \\ldots, w_n) \\right)^{-\\frac 1n}.$$\n",
    "\n",
    "Its measurement has some sacred meaning besides banal interpretability: we will present a model that predicts words from the dictionary equally likely, regardless of context. For her, $ \\mathbf {P} (w) = \\frac 1 N $, where $ N $ is the size of the dictionary, and perplexion will be equal to the size of the dictionary - $ N $. Of course, this is a completely stupid model, but looking at it, one can interpret the perplexion of real models as the level of ambiguity of word generation.\n",
    "\n",
    "For example, in the model with perplexia 100, the choice of the next word is also ambiguous, as the choice of a uniform distribution among 100 words. And if such perplexion was achieved on a dictionary of 100,000, it turns out that we managed to reduce this ambiguity by three orders of magnitude compared with the blunt randomness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xW8I0lKv9y1H"
   },
   "source": [
    "## Бейзлайн"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7wInBuBn-DIf"
   },
   "source": [
    "In general, baseline is also very simple here. We, in fact, even looked at it on the course of concepts: [N-gram language model] (https://colab.research.google.com/drive/1lz9vO6Ue5zOiowEx0-koXNiejBrrnbj0). It is possible to calculate the probabilities of N-grams of words by the frequency of their occurrence in the learning package. Then use the approximation $ \\mathbf {P} (w_k | w_1, \\ldots, w_ {k-1}) \\approx \\mathbf {P} (w_k | w_ {k-1}, \\ldots, w_ {kN + 1 }) $.\n",
    "\n",
    "Apply better mesh to implement the same.\n",
    "\n",
    "<center>\n",
    "<img src=\"https://image.ibb.co/buMnLf/2018-10-22-00-22-56.png\" width=\"20%\">\n",
    "</center>\n",
    "\n",
    "*From cs224n, Lecture 8 [pdf](http://web.stanford.edu/class/cs224n/lectures/lecture8.pdf)*\n",
    "\n",
    "A sequence of words comes to the input, they are inserted, and then with the help of the output layer the next word is considered the most likely.\n",
    "\n",
    "Stop ... But we have already implemented this! In the Word2vec CBoW model, we contextually predicted the central word - the only difference is that now we have only the left context. So, everything, we go to the next model?\n",
    "\n",
    "Not! There is still something to have fun. In Word2vec, we formed batchy like this:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://image.ibb.co/bs3wgV/training-data.png\" width=\"20%\">\n",
    "</center>\n",
    "    \n",
    "*From [Word2Vec Tutorial - The Skip-Gram Model, Chris McCormic](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)*\n",
    "\n",
    "That is, a set of <context, word> pairs was cut from the text (and somehow they were used depending on the method).\n",
    "\n",
    "It is irrational - every word is repeated many times. But you can use convolutional networks - they will apply the operation of multiplication by $ W $ for each window for us. As a result, the size of the input batch will be much smaller.\n",
    "\n",
    "To process everything correctly, you need to add padding to the beginning of the sequence with the size of `window_size - 1` - then the first word will be predicted by` <pad> ... <pad> <s> `.\n",
    "\n",
    "**Task** Implement a language model with a fixed window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A-tn_Gmi3pU0"
   },
   "outputs": [],
   "source": [
    "class ConvLM(nn.Module):\n",
    "    def __init__(self, vocab_size, window_size=5, emb_dim=16, filters_count=128):\n",
    "        super().__init__()\n",
    "        \n",
    "        self._window_size = window_size\n",
    "        \n",
    "        <init layers>\n",
    "        \n",
    "    def forward(self, inputs):\n",
    "        <apply>\n",
    "        \n",
    "        return output, None  # hacky way to use training cycle for RNN and Conv simultaneously"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bjpOLKBH5yS5"
   },
   "source": [
    "Check that it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ks_RTZ14nMRz"
   },
   "outputs": [],
   "source": [
    "model = ConvLM(vocab_size=len(train_iter.dataset.fields['text'].vocab)).to(DEVICE)\n",
    "\n",
    "model(batch.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lb_2VTBW5v_7"
   },
   "source": [
    "**Task** Implement a function to sample a sequence from a language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0oUg0BjV2JjE"
   },
   "outputs": [],
   "source": [
    "def sample(probs, temp):\n",
    "    probs = F.log_softmax(probs.squeeze(), dim=0)\n",
    "    probs = (probs / temp).exp()\n",
    "    probs /= probs.sum()\n",
    "    probs = probs.cpu().numpy()\n",
    "\n",
    "    return np.random.choice(np.arange(len(probs)), p=probs)\n",
    "\n",
    "\n",
    "def generate(model, temp=0.7):\n",
    "    model.eval()\n",
    "    \n",
    "    history = [train_dataset.fields['text'].vocab.stoi['<s>']]\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for _ in range(150):\n",
    "            <sample next character and print it (use end='' in print function)>\n",
    "\n",
    "generate(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CXuN871a852l"
   },
   "source": [
    "**Task** We still have not set any target. And we will need to predict the following words - that is, just the input tensor shifted by 1. Implement target building and loss calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CGLkcXARjhTM"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "def do_epoch(model, criterion, data_iter, unk_idx, pad_idx, optimizer=None, name=None):\n",
    "    epoch_loss = 0\n",
    "    \n",
    "    is_train = not optimizer is None\n",
    "    name = name or ''\n",
    "    model.train(is_train)\n",
    "    \n",
    "    batches_count = len(data_iter)\n",
    "    \n",
    "    with torch.autograd.set_grad_enabled(is_train):\n",
    "        with tqdm(total=batches_count) as progress_bar:\n",
    "            for i, batch in enumerate(data_iter):                \n",
    "                logits, _ = model(batch.text)\n",
    "\n",
    "                <implement loss calc>\n",
    "                \n",
    "                epoch_loss += loss.item()\n",
    "\n",
    "                if optimizer:\n",
    "                    optimizer.zero_grad()\n",
    "                    loss.backward()\n",
    "                    nn.utils.clip_grad_norm_(model.parameters(), 1.)\n",
    "                    optimizer.step()\n",
    "\n",
    "                progress_bar.update()\n",
    "                progress_bar.set_description('{:>5s} Loss = {:.5f}, PPX = {:.2f}'.format(name, loss.item(), \n",
    "                                                                                         math.exp(loss.item())))\n",
    "                \n",
    "            progress_bar.set_description('{:>5s} Loss = {:.5f}, PPX = {:.2f}'.format(\n",
    "                name, epoch_loss / batches_count, math.exp(epoch_loss / batches_count))\n",
    "            )\n",
    "\n",
    "    return epoch_loss / batches_count\n",
    "\n",
    "\n",
    "def fit(model, criterion, optimizer, train_iter, epochs_count=1, unk_idx=0, pad_idx=1, val_iter=None):\n",
    "    for epoch in range(epochs_count):\n",
    "        name_prefix = '[{} / {}] '.format(epoch + 1, epochs_count)\n",
    "        train_loss = do_epoch(model, criterion, train_iter, unk_idx, pad_idx, optimizer, name_prefix + 'Train:')\n",
    "        \n",
    "        if not val_iter is None:\n",
    "            val_loss = do_epoch(model, criterion, val_iter, unk_idx, pad_idx, None, name_prefix + '  Val:')\n",
    "\n",
    "        generate(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LIj0Lcdh9UJy"
   },
   "outputs": [],
   "source": [
    "model = ConvLM(vocab_size=len(train_iter.dataset.fields['text'].vocab)).to(DEVICE)\n",
    "\n",
    "pad_idx = train_iter.dataset.fields['text'].vocab.stoi['<pad>']\n",
    "unk_idx = train_iter.dataset.fields['text'].vocab.stoi['<unk>']\n",
    "criterion = nn.CrossEntropyLoss(reduction='none').to(DEVICE)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "fit(model, criterion, optimizer, train_iter, epochs_count=30, unk_idx=unk_idx, pad_idx=pad_idx, val_iter=test_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FycAd6MWMvYy"
   },
   "source": [
    "**Task** To wean the model to sample `<unk>` can explicitly forbid it in sepliruyuschey function - but you can not just teach it to them. Implement masking for both padding and unknown words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rQJKn1Uw94_0"
   },
   "source": [
    "## Recurrent language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HeSojPwh_ZSS"
   },
   "source": [
    "Obviously, I want to use not a fixed history window, but all the information about the already generated one. At a minimum, I want to know when we have a limit of characters in a tweet.\n",
    "For this, recurrent language models are used:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://hsto.org/web/dc1/7c2/c4e/dc17c2c4e9ac434eb5346ada2c412c9a.png\" width=\"20%\">\n",
    "</center>\n",
    "    \n",
    "The previous token is transmitted to the network as well as the previous RNN state. About the entire history is coded (should be), and the previous token is needed in order to know what kind of token was sampled from the distribution predicted at the last step.\n",
    "\n",
    "** Assignment ** We have done this several times already - implement again the network that will be engaged in language modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x8ndCRZLl4ZZ"
   },
   "outputs": [],
   "source": [
    "class RnnLM(nn.Module):\n",
    "    def __init__(self, vocab_size, emb_dim=16, lstm_hidden_dim=128, num_layers=1):\n",
    "        super().__init__()\n",
    "\n",
    "        self._emb = nn.Embedding(vocab_size, emb_dim)\n",
    "        self._rnn = nn.LSTM(input_size=emb_dim, hidden_size=lstm_hidden_dim)\n",
    "        self._out_layer = nn.Linear(lstm_hidden_dim, vocab_size)\n",
    "\n",
    "    def forward(self, inputs, hidden=None):\n",
    "        <implement me>\n",
    "        return output, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H3MjLgDKBNsD"
   },
   "source": [
    "**Task** Implement a function to sample sentences from a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZJSXu_Pr_kYL"
   },
   "outputs": [],
   "source": [
    "def generate(model, temp=0.8):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        prev_token = train_iter.dataset.fields['text'].vocab.stoi['<s>']\n",
    "        end_token = train_iter.dataset.fields['text'].vocab.stoi['</s>']\n",
    "        \n",
    "        hidden = None\n",
    "        for _ in range(150):\n",
    "            <print sampled character>\n",
    "\n",
    "generate(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cibfrMxo_Gjg"
   },
   "outputs": [],
   "source": [
    "model = RnnLM(vocab_size=len(train_iter.dataset.fields['text'].vocab)).to(DEVICE)\n",
    "\n",
    "pad_idx = train_iter.dataset.fields['text'].vocab.stoi['<pad>']\n",
    "unk_idx = train_iter.dataset.fields['text'].vocab.stoi['<unk>']\n",
    "criterion = nn.CrossEntropyLoss(reduction='none').to(DEVICE)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "fit(model, criterion, optimizer, train_iter, epochs_count=30, unk_idx=unk_idx, pad_idx=pad_idx, val_iter=test_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8cCcKrWjBzCp"
   },
   "source": [
    "## Model improvement\n",
    "\n",
    "\n",
    "We have only used Adam so far. In general, you can achieve better results with the usual `SGD`, if you really try.\n",
    " \n",
    "** Task ** Replace the optimizer with `optim.SGD (model.parameters (), lr = 20., Weight_decay = 1e-6)`. For example. Or other options to choose from.\n",
    "\n",
    "### Dropout\n",
    "\n",
    "Recall what a dropout is.\n",
    "\n",
    "In essence, this is the multiplication of a randomly generated mask of zeros and ones by the input vector (+ normalization).\n",
    "\n",
    "For example, for the Dropout (p) layer:\n",
    "\n",
    "$$m = \\frac1{1-p} \\cdot \\text{Bernouli}(1 - p)$$\n",
    "$$\\tilde h = m \\odot h $$\n",
    "\n",
    "\n",
    "In recurrent networks for a long time they could not screw the dropout. They tried to do this by generating a random mask:  \n",
    "\n",
    "from [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://arxiv.org/abs/1512.05287)\n",
    "\n",
    "\n",
    "It turned out that it is more correct to make the mask fixed: for each step the same elements should be zero.\n",
    "\n",
    "For pytorch, there is no normal embedded variational dropout in LSTM. But there is [AWD-LSTM] (https://github.com/salesforce/awd-lstm-lm).\n",
    "\n",
    "I advise you to look at the review of different ways of applying dropout in recurrent networks: [Dropout in Recurrent Networks - Part 1] (https://becominghuman.ai/learning-note-dropout-in-recurrent-networks-part-1-57a9c19a2307) ( at the end - links to Part 2 and 3).\n",
    "\n",
    "**Task** Implement a variation dropout. For this you need to sample the mask `(1, batch_size, inp_dim)` for the input tensor of the size `(seq_len, batch_size, inp_dim)` from the distribution $ \\text {Bernouli} (1 - p) $, multiply it by $ \\frac1 {1 -p} $ and multiply the input tensor by it.\n",
    "\n",
    "Thanks to broadcasting, each timestamp from the input tensor is multiplied by the same mask - and there must be happiness.\n",
    "\n",
    "Although it is better to compare with the usual `nn.Dropout`, suddenly the difference will not be noticeable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aDv4nutY-WOw"
   },
   "outputs": [],
   "source": [
    "class LockedDropout(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, inputs, dropout=0.5):\n",
    "        if not self.training or not dropout:\n",
    "            return inputs\n",
    "        \n",
    "        <implement me>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9m-InMeoIiCA"
   },
   "source": [
    "## Conditional generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J7aB2_YxIl-c"
   },
   "source": [
    "We have already classified names by language. We now learn how to generate a surname for a given language.\n",
    "\n",
    "Let's use the heir of `Dataset` -` TabularDataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wa5benKoJMfc"
   },
   "outputs": [],
   "source": [
    "from torchtext.data import TabularDataset\n",
    "\n",
    "name_field = Field(init_token='<s>', eos_token='</s>', lower=True, tokenize=lambda line: list(line))\n",
    "lang_field = Field(sequential=False)\n",
    "\n",
    "dataset = TabularDataset(\n",
    "    path='surnames.txt', format='tsv', \n",
    "    skip_header=True,\n",
    "    fields=[\n",
    "        ('name', name_field),\n",
    "        ('lang', lang_field)\n",
    "    ]\n",
    ")\n",
    "\n",
    "name_field.build_vocab(dataset)\n",
    "lang_field.build_vocab(dataset)\n",
    "\n",
    "print(name_field.vocab.itos)\n",
    "print(lang_field.vocab.itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qp3SZHAsK85C"
   },
   "source": [
    "Let's break the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kh-KKh08J5Oq"
   },
   "outputs": [],
   "source": [
    "train_dataset, val_dataset = dataset.split(split_ratio=0.25, stratified=True, strata_field='lang')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nIzaiUKDK_PG"
   },
   "source": [
    "**Task** Make a language model that accepts both the previous generated character and the index of the language to which this word belongs. Build embeddings for the symbol and for the language, concatenate them - and then everything is the same.\n",
    "\n",
    "It is necessary to train this model and write the function-generator of surnames for a given language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9VfdL29AELhu"
   },
   "source": [
    "# In the wild"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GDqxGVo5EOfb"
   },
   "source": [
    "Let's apply our knowledge to the combat mission: [Kaggle Toxic Comment Classification Challenge] (https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/).\n",
    "\n",
    "It is about the classification of messages in several categories. The network architecture should be as follows: some encoder (for example, LSTM) builds the embedding sequence. Then, the output layer should predict 6 categories - but not with cross-entropy losses, but with `nn.BCEWithLogitsLoss` - because the categories are not mutually exclusive.\n",
    "\n",
    "Tip: Understand the tokenization that `Field` can do. Download the pre-trained vocabulary embeddings, as we did. Build a network and write a learning cycle for it.\n",
    "\n",
    "**Task** Download data from kaggle, train something and make a package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8obdAs_E0zRb"
   },
   "source": [
    "# Referrence\n",
    "\n",
    "[A Friendly Introduction to Cross-Entropy Loss, Rob DiPietro](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/)\n",
    "\n",
    "[A Tutorial on Torchtext, Allen Nie](http://anie.me/On-Torchtext/)\n",
    "\n",
    "[Dropout in Recurrent Networks, Ceshine Lee](https://becominghuman.ai/learning-note-dropout-in-recurrent-networks-part-1-57a9c19a2307)\n",
    "\n",
    "[The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "\n",
    "[The unreasonable effectiveness of Character-level Language Models, Yoav Goldberg](http://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139)\n",
    "\n",
    "[Unsupervised Sentiment Neuron, OpenAI](https://blog.openai.com/unsupervised-sentiment-neuron/)\n",
    "\n",
    "[Как научить свою нейросеть генерировать стихи](https://habr.com/post/334046/)\n",
    "\n",
    "\n",
    "[cs224n, \"Lecture 8: Recurrent Neural Networks and Language Models\"](https://www.youtube.com/watch?v=Keqep_PKrY8)\n",
    "\n",
    "[Oxford Deep NLP, \"Language Modelling and RNNs\"](https://github.com/oxford-cs-deepnlp-2017/lectures#5-lecture-3---language-modelling-and-rnns-part-1-phil-blunsom)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "oTrSUkqEhZzh"
   ],
   "name": "Week 07 - Language Models.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}