Week06 - RNNs Part 02.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8sVtGHmA9aBM"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    from torch.cuda import FloatTensor, LongTensor\n",
    "else:\n",
    "    from torch import FloatTensor, LongTensor\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-6CNKM3b4hT1"
   },
   "source": [
    "# RNN, part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O_XkoGNQUeGm"
   },
   "source": [
    "## POS Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QFEtWrS_4rUs"
   },
   "source": [
    "We have already looked at the use of recurrent networks for classification.\n",
    "\n",
    "<center>\n",
    "<img src=\"http://karpathy.github.io/assets/rnn/diags.jpeg\" width=\"30%\">\n",
    "</center>\n",
    "\n",
    "*From [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)*\n",
    "\n",
    "Let's move on to another option - sequence labeling (the last picture).\n",
    "\n",
    "The most popular examples for such a problem setting are Part-of-Speech Tagging and Named Entity Recognition.\n",
    "\n",
    "We will now solve POS Tagging for English.\n",
    "\n",
    "We will work with the following tags:\n",
    "- ADJ - adjective (new, good, high, ...)\n",
    "- ADP - adposition (on, of, at, ...)\n",
    "- ADV - adverb (really, already, still, ...)\n",
    "- CONJ - conjunction (and, or, but, ...)\n",
    "- DET - determiner, article (the, a, some, ...)\n",
    "- NOUN - noun (year, home, costs, ...)\n",
    "- NUM - numeral (twenty-four, fourth, 1991, ...)\n",
    "- PRT - particle (at, on, out, ...)\n",
    "- PRON - pronoun (he, their, her, ...)\n",
    "- VERB - verb (is, say, told, ...)\n",
    "- . - punctuation marks (. , ;)\n",
    "- X - other (ersatz, esprit, dunno, ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TiA2dGmgF1rW"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "nltk.download('brown')\n",
    "nltk.download('universal_tagset')\n",
    "\n",
    "data = nltk.corpus.brown.tagged_sents(tagset='universal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QstS4NO0L97c"
   },
   "outputs": [],
   "source": [
    "for word, tag in data[0]:\n",
    "    print('{:15}\\t{}'.format(word, tag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "epdW8u_YXcAv"
   },
   "source": [
    "Construct a partitioning into train / val / test - finally, everything is just like normal people.\n",
    "\n",
    "We will study on the train, according to val - we will select the parameters and do all sorts of early stopping, and on test - we will accept the model for its final quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xTai8Ta0lgwL"
   },
   "outputs": [],
   "source": [
    "train_data, test_data = train_test_split(data, test_size=0.25, random_state=42)\n",
    "train_data, val_data = train_test_split(train_data, test_size=0.15, random_state=42)\n",
    "\n",
    "print('Words count in train set:', sum(len(sent) for sent in train_data))\n",
    "print('Words count in val set:', sum(len(sent) for sent in val_data))\n",
    "print('Words count in test set:', sum(len(sent) for sent in test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eChdLNGtXyP0"
   },
   "source": [
    "Construct mappings from words to an index and from a tag to an index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pCjwwDs6Zq9x"
   },
   "outputs": [],
   "source": [
    "words = {word for sample in train_data for word, tag in sample}\n",
    "word2ind = {word: ind + 1 for ind, word in enumerate(words)}\n",
    "word2ind['<pad>'] = 0\n",
    "\n",
    "tags = {tag for sample in train_data for word, tag in sample}\n",
    "tag2ind = {tag: ind + 1 for ind, tag in enumerate(tags)}\n",
    "tag2ind['<pad>'] = 0\n",
    "\n",
    "print('Unique words in train = {}. Tags = {}'.format(len(word2ind), tags))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "URC1B2nvPGFt"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "tag_distribution = Counter(tag for sample in train_data for _, tag in sample)\n",
    "tag_distribution = [tag_distribution[tag] for tag in tags]\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "\n",
    "bar_width = 0.35\n",
    "plt.bar(np.arange(len(tags)), tag_distribution, bar_width, align='center', alpha=0.5)\n",
    "plt.xticks(np.arange(len(tags)), tags)\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gArQwbzWWkgi"
   },
   "source": [
    "What is the easiest tagger you can think of? Let's just memorize which tags are most likely for a word (or for a sequence):\n",
    "\n",
    "<center>\n",
    "<img src=\"https://www.nltk.org/images/tag-context.png\">\n",
    "</center>\n",
    "    \n",
    "*From [Categorizing and Tagging Words, nltk](https://www.nltk.org/book/ch05.html)*\n",
    "\n",
    "The picture shows that two previous predicted tags + current word are used to predict $ t_n $. According to the case, the probability for $ P (t_n | w_n, t_ {n-1}, t_ {n-2}) $ is considered, the tag with the maximum probability is selected.\n",
    "\n",
    "This idea is implemented more accurately in Hidden Markov Models: the probabilities $ P (w_n | t_n), P (t_n | t_ {n-1}, t_ {n-2}) $ are calculated from the training building, and their product is maximized.\n",
    "\n",
    "The simplest option is a unigram model that takes into account only the word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5rWmSToIaeAo"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "default_tagger = nltk.DefaultTagger('NN')\n",
    "\n",
    "unigram_tagger = nltk.UnigramTagger(train_data, backoff=default_tagger)\n",
    "print('Accuracy of unigram tagger = {:.2%}'.format(unigram_tagger.evaluate(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "07Ymb_MkbWsF"
   },
   "source": [
    "Add transition probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vjz_Rk0bbMyH"
   },
   "outputs": [],
   "source": [
    "bigram_tagger = nltk.BigramTagger(train_data, backoff=unigram_tagger)\n",
    "print('Accuracy of bigram tagger = {:.2%}'.format(bigram_tagger.evaluate(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uWMw6QHvbaDd"
   },
   "source": [
    "Note that the backoff is important:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8XCuxEBVbOY_"
   },
   "outputs": [],
   "source": [
    "trigram_tagger = nltk.TrigramTagger(train_data)\n",
    "print('Accuracy of trigram tagger = {:.2%}'.format(trigram_tagger.evaluate(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4t3xyYd__8d-"
   },
   "source": [
    "## We increase context with recurrent networks.\n",
    "\n",
    "The unigram model works surprisingly well, but we are going to learn the nets.\n",
    "\n",
    "Homonymy is the main reason why the unigram model is bad:\n",
    "* “He cashed a check at the ** bank **” *\n",
    "vs\n",
    "* “He sat on the ** bank ** of the river” *\n",
    "\n",
    "Therefore, it is very useful for us to consider the context when predicting the tag.\n",
    "\n",
    "Let's use LSTM - it can work with the context very well:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://image.ibb.co/kgmoff/Baseline-Tagger.png\" width=\"20%\">\n",
    "</center>\n",
    "\n",
    "Blue shows the selection of features from the word, LSTM Orange - it builds word embeddings according to the context, and then the green logistic regression makes tag predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RtRbz1SwgEqc"
   },
   "outputs": [],
   "source": [
    "def convert_data(data, word2ind, tag2ind):\n",
    "    X = [[word2ind.get(word, 0) for word, _ in sample] for sample in data]\n",
    "    y = [[tag2ind[tag] for _, tag in sample] for sample in data]\n",
    "    \n",
    "    return X, y\n",
    "\n",
    "X_train, y_train = convert_data(train_data, word2ind, tag2ind)\n",
    "X_val, y_val = convert_data(val_data, word2ind, tag2ind)\n",
    "X_test, y_test = convert_data(test_data, word2ind, tag2ind)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DhsTKZalfih6"
   },
   "outputs": [],
   "source": [
    "def iterate_batches(data, batch_size):\n",
    "    X, y = data\n",
    "    n_samples = len(X)\n",
    "\n",
    "    indices = np.arange(n_samples)\n",
    "    np.random.shuffle(indices)\n",
    "    \n",
    "    for start in range(0, n_samples, batch_size):\n",
    "        end = min(start + batch_size, n_samples)\n",
    "        \n",
    "        batch_indices = indices[start:end]\n",
    "        \n",
    "        max_sent_len = max(len(X[ind]) for ind in batch_indices)\n",
    "        X_batch = np.zeros((max_sent_len, len(batch_indices)))\n",
    "        y_batch = np.zeros((max_sent_len, len(batch_indices)))\n",
    "        \n",
    "        for batch_ind, sample_ind in enumerate(batch_indices):\n",
    "            X_batch[:len(X[sample_ind]), batch_ind] = X[sample_ind]\n",
    "            y_batch[:len(y[sample_ind]), batch_ind] = y[sample_ind]\n",
    "            \n",
    "        yield X_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l4XsRII5kW5x"
   },
   "outputs": [],
   "source": [
    "X_batch, y_batch = next(iterate_batches((X_train, y_train), 4))\n",
    "\n",
    "X_batch, y_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C5I9E9P6eFYv"
   },
   "source": [
    "**Task** Implement `LSTMTagger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WVEHju54d68T"
   },
   "outputs": [],
   "source": [
    "class LSTMTagger(nn.Module):\n",
    "    def __init__(self, vocab_size, tagset_size, word_emb_dim=100, lstm_hidden_dim=128, lstm_layers_count=1):\n",
    "        super().__init__()\n",
    "        \n",
    "        <create layers>\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        <apply them>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q_HA8zyheYGH"
   },
   "source": [
    "**Task** Learn how to count accuracy and loss (and at the same time check that the model works)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jbrxsZ2mehWB"
   },
   "outputs": [],
   "source": [
    "model = LSTMTagger(\n",
    "    vocab_size=len(word2ind),\n",
    "    tagset_size=len(tag2ind)\n",
    ")\n",
    "\n",
    "X_batch, y_batch = torch.LongTensor(X_batch), torch.LongTensor(y_batch)\n",
    "\n",
    "logits = model(X_batch)\n",
    "\n",
    "<calc accuracy>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GMUyUm1hgpe3"
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "<calc loss>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nSgV3NPUpcjH"
   },
   "source": [
    "**Task** Insert this calculation in the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FprPQ0gllo7b"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "def do_epoch(model, criterion, data, batch_size, optimizer=None, name=None):\n",
    "    epoch_loss = 0\n",
    "    correct_count = 0\n",
    "    sum_count = 0\n",
    "    \n",
    "    is_train = not optimizer is None\n",
    "    name = name or ''\n",
    "    model.train(is_train)\n",
    "    \n",
    "    batches_count = math.ceil(len(data[0]) / batch_size)\n",
    "    \n",
    "    with torch.autograd.set_grad_enabled(is_train):\n",
    "        with tqdm(total=batches_count) as progress_bar:\n",
    "            for i, (X_batch, y_batch) in enumerate(iterate_batches(data, batch_size)):\n",
    "                X_batch, y_batch = LongTensor(X_batch), LongTensor(y_batch)\n",
    "                logits = model(X_batch)\n",
    "\n",
    "                loss = <calc loss>\n",
    "\n",
    "                epoch_loss += loss.item()\n",
    "\n",
    "                if optimizer:\n",
    "                    optimizer.zero_grad()\n",
    "                    loss.backward()\n",
    "                    optimizer.step()\n",
    "\n",
    "                cur_correct_count, cur_sum_count = <calc accuracy>\n",
    "\n",
    "                correct_count += cur_correct_count\n",
    "                sum_count += cur_sum_count\n",
    "\n",
    "                progress_bar.update()\n",
    "                progress_bar.set_description('{:>5s} Loss = {:.5f}, Accuracy = {:.2%}'.format(\n",
    "                    name, loss.item(), cur_correct_count / cur_sum_count)\n",
    "                )\n",
    "                \n",
    "            progress_bar.set_description('{:>5s} Loss = {:.5f}, Accuracy = {:.2%}'.format(\n",
    "                name, epoch_loss / batches_count, correct_count / sum_count)\n",
    "            )\n",
    "\n",
    "    return epoch_loss / batches_count, correct_count / sum_count\n",
    "\n",
    "\n",
    "def fit(model, criterion, optimizer, train_data, epochs_count=1, batch_size=32,\n",
    "        val_data=None, val_batch_size=None):\n",
    "        \n",
    "    if not val_data is None and val_batch_size is None:\n",
    "        val_batch_size = batch_size\n",
    "        \n",
    "    for epoch in range(epochs_count):\n",
    "        name_prefix = '[{} / {}] '.format(epoch + 1, epochs_count)\n",
    "        train_loss, train_acc = do_epoch(model, criterion, train_data, batch_size, optimizer, name_prefix + 'Train:')\n",
    "        \n",
    "        if not val_data is None:\n",
    "            val_loss, val_acc = do_epoch(model, criterion, val_data, val_batch_size, None, name_prefix + '  Val:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pqfbeh1ltEYa"
   },
   "outputs": [],
   "source": [
    "model = LSTMTagger(\n",
    "    vocab_size=len(word2ind),\n",
    "    tagset_size=len(tag2ind)\n",
    ").cuda()\n",
    "\n",
    "criterion = nn.CrossEntropyLoss().cuda()\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "fit(model, criterion, optimizer, train_data=(X_train, y_train), epochs_count=50,\n",
    "    batch_size=64, val_data=(X_val, y_val), val_batch_size=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m0qGetIhfUE5"
   },
   "source": [
    "### Masking\n",
    "\n",
    "**Task** Check Yourself - Do You Consider Losses and Accuracy on Paddings? It is very easy to get high quality due to this.\n",
    "\n",
    "The loss function has a parameter `ignore_index`, for such purposes. For accuracy, you need to use masking - multiplication by a mask of zeros and ones, where zeros are on padding positions (and then averaging over non-zero positions in the mask)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nAfV2dEOfHo5"
   },
   "source": [
    "**Task** Calculate the quality of the model on the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "98wr38_rw55D"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PXUTSFaEHbDG"
   },
   "source": [
    "### Bidirectional LSTM\n",
    "\n",
    "Thanks to BiLSTM, you can use both contests at once in predicting the word tag. Those. for each $ w_i $ forward token, LSTM will issue the view $ \\mathbf {f_i} \\sim (w_1, \\ldots, w_i) $ - built over the entire left context - and $ \\mathbf {b_i} \\sim (w_n, \\ldots, w_i) $ - representation of the right context. Their concatenation will automatically capture the entire accessible context of the word: $ \\ mathbf {h_i} = [\\mathbf {f_i}, \\mathbf {b_i}] \\sim (w_1, \\ldots, w_n) $.\n",
    "\n",
    "<center>\n",
    "<img src=\"https://www.researchgate.net/profile/Wang_Ling/publication/280912217/figure/fig2/AS:391505383575555@1470353565299/Illustration-of-our-neural-network-for-POS-tagging.png\" width=\"20%\">\n",
    "</center>\n",
    "    \n",
    "*From [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://arxiv.org/abs/1508.02096)*\n",
    "\n",
    "**Task** Add Bidirectional LSTM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZTXmYGD_ANhm"
   },
   "source": [
    "### Pre-Learned Embeddings\n",
    "\n",
    "We know what a cool thing - the pre-learning embedding. With the current size of the training sample, it was still possible to teach them from scratch - with a smaller one it would be completely bad.\n",
    "\n",
    "Therefore, the standard pipeline is to download embeds, shove them into the net. Run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uZpY_Q1xZ18h"
   },
   "outputs": [],
   "source": [
    "import gensim.downloader as api\n",
    "\n",
    "w2v_model = api.load('glove-wiki-gigaword-100')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KYogOoKlgtcf"
   },
   "source": [
    "\n",
    "Let's build a submatrix for words from our training sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VsCstxiO03oT"
   },
   "outputs": [],
   "source": [
    "known_count = 0\n",
    "embeddings = np.zeros((len(word2ind), w2v_model.vectors.shape[1]))\n",
    "for word, ind in word2ind.items():\n",
    "    word = word.lower()\n",
    "    if word in w2v_model.vocab:\n",
    "        embeddings[ind] = w2v_model.get_vector(word)\n",
    "        known_count += 1\n",
    "        \n",
    "print('Know {} out of {} word embeddings'.format(known_count, len(word2ind)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HcG7i-R8hbY3"
   },
   "source": [
    "**Task** Make a model with a pre-trained matrix. Use `nn.Embedding.from_pretrained`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LxaRBpQd0pat"
   },
   "outputs": [],
   "source": [
    "class LSTMTaggerWithPretrainedEmbs(nn.Module):\n",
    "    def __init__(self, embeddings, tagset_size, lstm_hidden_dim=64, lstm_layers_count=1):\n",
    "        super().__init__()\n",
    "        \n",
    "        <create me>\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        <use me>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EBtI6BDE-Fc7"
   },
   "outputs": [],
   "source": [
    "model = LSTMTaggerWithPretrainedEmbs(\n",
    "    embeddings=embeddings,\n",
    "    tagset_size=len(tag2ind)\n",
    ").cuda()\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index=0)\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "fit(model, criterion, optimizer, train_data=(X_train, y_train), epochs_count=50,\n",
    "    batch_size=64, val_data=(X_val, y_val), val_batch_size=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2Ne_8f24h8kg"
   },
   "source": [
    "**Task** Estimate the quality of the model on the test sample. Please note that it is not at all necessary to be limited to vectors from the trimmed matrix - there may well be words in the test that were not in the train and for which there are embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HPUuAPGhEGVR"
   },
   "outputs": [],
   "source": [
    "<calc test accuracy>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "enF9GAPAN3RB"
   },
   "source": [
    "### Pre-training of pre-trained vectors\n",
    "\n",
    "** Assignment ** Why not try to teach a vector? To do this, simply replace the `freeze = False` flag in the` from_pretrained` method. Try it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5AdB6olUiyf7"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZVJet3RQix98"
   },
   "source": [
    "** Task ** In fact, it is clear why this is bad - after that, the old pre-trained vectors (which did not fall into the train) cannot be used. Check what quality is obtained on the test with the old vectors.\n",
    "\n",
    "To deal with this, you can use this technique: impose $ l_2 $ -regularization on the pre-trained vectors so that they do not move away from the original vectors, and for words whose embeddings we do not know, build random vectors and teach them as usual.\n",
    "\n",
    "You can read about it a little bit here: [Pseudo-rehearsal: A simple solution to catastrophic forgetting for NLP] (https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting) or in the Goldberg book.\n",
    "\n",
    "** Task ** Try to implement it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "23abeGwPp163"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EfN1olf6RZne"
   },
   "source": [
    "## We need to go deeper, character network\n",
    "\n",
    "\n",
    "\n",
    "Let me remind you that in the last lesson we built an LSTM network that processed character sequences and predicted which language the word belongs to.\n",
    "\n",
    "LSTM acted as a feature extractor that works with an arbitrary size sequence of characters (well, almost arbitrary - we were limited to the maximum word length). Butch for the network had the dimension `(max_word_len, batch_size)`.\n",
    "\n",
    "Now, again, we want to use the same idea to extract features from a sequence of characters — because a sequence of characters should be useful for predicting a part of speech, right?\n",
    "\n",
    "The network will have to remember, for example, that `-ly` is often about an adverb, and` -tion` about a noun.\n",
    "\n",
    "<center>\n",
    "<img src=\"https://image.ibb.co/kzbh6L/Char-Bi-LSTM.png\" width=\"20%\">\n",
    "</center>\n",
    "    \n",
    "The rest of the network will be the same.\n",
    "\n",
    "Find the boundary for the length of the words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SczGwL8Cy0Ws"
   },
   "outputs": [],
   "source": [
    "from collections import Counter \n",
    "    \n",
    "def find_max_len(counter, threshold):\n",
    "    sum_count = sum(counter.values())\n",
    "    cum_count = 0\n",
    "    for i in range(max(counter)):\n",
    "        cum_count += counter[i]\n",
    "        if cum_count > sum_count * threshold:\n",
    "            return i\n",
    "    return max(counter)\n",
    "\n",
    "word_len_counter = Counter()\n",
    "for sent in data:\n",
    "    for word, _ in sent:\n",
    "        word_len_counter[len(word)] += 1\n",
    "    \n",
    "threshold = 0.99\n",
    "MAX_WORD_LEN = find_max_len(word_len_counter, threshold)\n",
    "\n",
    "print('Max word len for {:.0%} of words is {}'.format(threshold, MAX_WORD_LEN))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YlArjEvqkMGk"
   },
   "source": [
    "Построим алфавит:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-LWXHmXGcotd"
   },
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "def get_range(first_symb, last_symb):\n",
    "    return set(chr(c) for c in range(ord(first_symb), ord(last_symb) + 1))\n",
    "\n",
    "chars = get_range('a', 'z') | get_range('A', 'Z') | get_range('0', '9') | set(punctuation)\n",
    "char2ind = {c : i + 1 for i, c in enumerate(chars)}\n",
    "char2ind['<pad>'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v0OS9WQjkO9b"
   },
   "source": [
    "**Task** Convert the data, as in the function above - only now the words should be displayed not in one index, but in a sequence.\n",
    "\n",
    "Trim the words by `MAX_WORD_LEN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k3Q3arGCmgi-"
   },
   "outputs": [],
   "source": [
    "def convert_data(data, char2ind, tag2ind):\n",
    "    X, y = <calc it>\n",
    "    return X, y\n",
    "  \n",
    "X_train, y_train = convert_data(train_data, char2ind, tag2ind)\n",
    "X_val, y_val = convert_data(val_data, char2ind, tag2ind)\n",
    "X_test, y_test = convert_data(test_data, char2ind, tag2ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1SMmXMx5Rr5z"
   },
   "source": [
    "Напишем генератор батчей:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c835LEVERXzl"
   },
   "outputs": [],
   "source": [
    "def iterate_batches(data, batch_size):\n",
    "    X, y = data\n",
    "    n_samples = len(X)\n",
    "\n",
    "    indices = np.arange(n_samples)\n",
    "    np.random.shuffle(indices)\n",
    "    \n",
    "    for start in range(0, n_samples, batch_size):\n",
    "        end = min(start + batch_size, n_samples)\n",
    "        \n",
    "        batch_indices = indices[start: end]\n",
    "        \n",
    "        sent_len = max(len(X[ind]) for ind in batch_indices)\n",
    "        word_len = max(len(word) for ind in batch_indices for word in X[ind])\n",
    "            \n",
    "        X_batch = np.zeros((sent_len, len(batch_indices), word_len))\n",
    "        y_batch = np.zeros((sent_len, len(batch_indices)))\n",
    "        \n",
    "        for batch_ind, sample_ind in enumerate(batch_indices):\n",
    "            for word_ind, word in enumerate(X[sample_ind]):\n",
    "                X_batch[word_ind, batch_ind, :len(word)] = word\n",
    "            y_batch[:len(y[sample_ind]), batch_ind] = y[sample_ind]\n",
    "            \n",
    "        yield X_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zWcRRe11jFI8"
   },
   "outputs": [],
   "source": [
    "X_batch, y_batch = next(iterate_batches((X_train, y_train), 4))\n",
    "\n",
    "X_batch.shape, y_batch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yfY7FcXCknzX"
   },
   "source": [
    "**Task** Implement a network that accepts a batch of size `(seq_len, batch_size, word_len)` and returns `(seq_len, batch_size, word_emb_dim)`. This can be any function that can in a sequence of arbitrary length. We have already looked at convolutional and recurrent networks for such a task — try both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f1qs96uAY3Wd"
   },
   "outputs": [],
   "source": [
    "class CharsEmbedding(nn.Module):\n",
    "    def __init__(self, vocab_size, char_emb_dim=24, word_emb_dim=100):\n",
    "        super().__init__()\n",
    "        \n",
    "        <create Conv or LSTM encoder>\n",
    "        \n",
    "    def forward(self, inputs):\n",
    "        <apply>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ag2R5sIglLhh"
   },
   "source": [
    "**Задание** Реализуйте теггер с эмбеддингами символьного уровня."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TRB8tAOAa_YW"
   },
   "outputs": [],
   "source": [
    "class LSTMTagger(nn.Module):\n",
    "    def __init__(self, char_vocab_size, tagset_size, char_emb_dim=24, \n",
    "                 word_emb_dim=128, lstm_hidden_dim=128, lstm_layers_count=1):\n",
    "        super().__init__()\n",
    "        \n",
    "        <create it>\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        <apply>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cEaWjN0qjFfe"
   },
   "outputs": [],
   "source": [
    "model = LSTMTagger(char_vocab_size=len(char2ind), tagset_size=len(tag2ind)).cuda()\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index=0).cuda()\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "fit(model, criterion, optimizer, train_data=(X_train, y_train), epochs_count=20, \n",
    "    batch_size=24, val_data=(X_val, y_val), val_batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OGDJqyG9lTxV"
   },
   "source": [
    "**Задание** Оцените его качество."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OCUj9_nqjgrL"
   },
   "outputs": [],
   "source": [
    "_, test_accuracy = do_epoch(model, criterion, (X_test, y_test), batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "12HYYmSzlZtm"
   },
   "source": [
    "### Visualization\n",
    "\n",
    "** Task ** Calculate symbol level embeddings (trained inside the model before this) for 1000 random words from `word2ind`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qyXEJ6MUG8PE"
   },
   "outputs": [],
   "source": [
    "embeddings, index2word = <calc me>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K2klT31GSWlR"
   },
   "outputs": [],
   "source": [
    "import bokeh.models as bm, bokeh.plotting as pl\n",
    "from bokeh.io import output_notebook\n",
    "\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.preprocessing import scale\n",
    "\n",
    "\n",
    "def draw_vectors(x, y, radius=10, alpha=0.25, color='blue',\n",
    "                 width=600, height=400, show=True, **kwargs):\n",
    "    \"\"\" draws an interactive plot for data points with auxilirary info on hover \"\"\"\n",
    "    output_notebook()\n",
    "    \n",
    "    if isinstance(color, str): \n",
    "        color = [color] * len(x)\n",
    "    data_source = bm.ColumnDataSource({ 'x' : x, 'y' : y, 'color': color, **kwargs })\n",
    "\n",
    "    fig = pl.figure(active_scroll='wheel_zoom', width=width, height=height)\n",
    "    fig.scatter('x', 'y', size=radius, color='color', alpha=alpha, source=data_source)\n",
    "\n",
    "    fig.add_tools(bm.HoverTool(tooltips=[(key, \"@\" + key) for key in kwargs.keys()]))\n",
    "    if show: \n",
    "        pl.show(fig)\n",
    "    return fig\n",
    "\n",
    "\n",
    "def get_tsne_projection(word_vectors):\n",
    "    tsne = TSNE(n_components=2, verbose=100)\n",
    "    return scale(tsne.fit_transform(word_vectors))\n",
    "    \n",
    "    \n",
    "def visualize_embeddings(embeddings, token):\n",
    "    tsne = get_tsne_projection(embeddings)\n",
    "    draw_vectors(tsne[:, 0], tsne[:, 1], token=token)\n",
    "    \n",
    "\n",
    "visualize_embeddings(embeddings, index2word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TgmDHM9Dl7W7"
   },
   "source": [
    "**Task** Calculate embeddings for all words from the train and for several random words from the test that are not found in the train, find their closest neighbors in their embeddigs of the symbol level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1bctty__mOOz"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WzAozOANpnT1"
   },
   "source": [
    "### Word embeddings\n",
    "\n",
    "** Task ** Only symbolic embeddingings may not be enough. Give another word embeddings. Words should result in lower case — case-related attributes should catch character LSTM.\n",
    "\n",
    "These embeddings can simply be concatenated, folded, or the gate can be used (as in LSTM). For example, by embedding, the words predict $ o = \\sigma (w) $ - how good it is and combine in this proportion with symbolic embedding: $ o \\odot w + (1 - o) \\odot \\tilde w $, where $ \\tilde w $ - embedding of the word, obtained by the symbolic level. Check out the different options.\n",
    "\n",
    "### Communication word embeddingov and embeddingov character level\n",
    "In word embeddings, we build a mapping from a word to an index. As a result, the input batch is quite small - it is good for learning (faster transfer to a video card). With symbolic embeddings, trouble - but it can be fixed.\n",
    "\n",
    "Let's assume for each word in `word2ind` its sequence of character indexes. Get a matrix. This matrix can be transferred with the model to a video card. Then a batch will be needed from the word indices - using it you can make a look (using `F. embedding`) in the matrix and get a three-dimensional matrix with symbols.\n",
    "\n",
    "The advantage is that you can get both word embeddings and symbol level embeds immediately in one batch. It is convenient and energy efficient.\n",
    "\n",
    "Another idea is that after we have trained the model, it is possible to prescribe the embeddingings of words of the symbolic level - lookup in the embeddingings table is much simpler than a convolutional or recurrent network over symbols. Thus, for example, embeddings are obtained in [FastText] (https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md) - they are also initially considered at the symbolic (N-gram) level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9gzxhGe6okls"
   },
   "source": [
    "## Encoder-decoder\n",
    "\n",
    "You can complicate the model - add another recurrent layer. The first layer will serve as a sequence encoder, the second, easier one - to decode the sequence. Decode means that it must take as input the state for the given token from the encoder and the previous predicted tag.\n",
    "\n",
    "Everything will look something like this:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://image.ibb.co/jOrfT0/Encoder-Decoder.png\" width=\"20%\">\n",
    "</center>\n",
    "\n",
    "Green is already `LSTM`, not` Linear`, but it assumes immediately a hidden state from the previous token (green arrow), the previous predicted tag (dotted arrow) and the state from BiLSTM - the contextual representation of the word.\n",
    "\n",
    "This model should be trained with teacher-forcing - passing the correct labels as answers to the dotted arrows. On the prediction, you need to implement a beam search — keep several best paths (tag sequences) for the decoded sequence at once.\n",
    "\n",
    "**Task** Risk to realize it.\n",
    "\n",
    "(In general, we will deal with this in more detail, when we get to the machine translation - you can come back here after it :))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bEJAch5Bj4vQ"
   },
   "source": [
    "# Referrence\n",
    "\n",
    "Speech and Language Processing, Chapter 8, Part-of-speech Tagging. Daniel Jurafsky [[pdf](https://web.stanford.edu/~jurafsky/slp3/8.pdf)]\n",
    "\n",
    "Learning Character-level Representations for Part-of-Speech Tagging, dos Santos et al, 2014 [pdf](http://proceedings.mlr.press/v32/santos14.pdf)  \n",
    "Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation, Wang Ling et al, 2015 [arxiv](https://arxiv.org/abs/1508.02096)  \n",
    "Bidirectional LSTM-CRF Models for Sequence Tagging, Zhiheng Huang et al, 2015 [arxiv](https://arxiv.org/abs/1508.01991)  \n",
    "End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF, Xuezhe Ma et al, 2016 [arxiv](https://arxiv.org/abs/1603.01354)  \n",
    "Improving Part-of-speech Tagging via Multi-task Learning and Character-level Word Representations, Daniil Anastasyev et al, 2018 [pdf](http://www.dialog-21.ru/media/4282/anastasyevdg.pdf) :)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Week 06 - RNNs, part 2.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}