03 - Sequence learning with Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence learning with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In recent years, there has been a considerable increase in the attention for sequence modelling in Deep Learning. In this chapter, we will delve into the exciting topic of recurrent in neural networks. We will build upon the POS-Chunk data which we loaded in the previous chapter. Importantly, we will demonstrate that a keras model (as any Theano or TensorFlow graph) can have multiple inputs and outputs. We will show, for instance, that it is perfectly possible to to train a model that **simultaneously** learns to pos tag and chunk. As always, we first set up our booth:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import codecs\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from keras.utils import np_utils\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the CONLL train data again, but this time, we also load the chunk labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def load_data(path):\n",
    "    data = []\n",
    "    for line in codecs.open(path, 'r', 'utf8'):\n",
    "        line = line.strip()\n",
    "        if line:\n",
    "            try:\n",
    "                token, pos, chunk = line.strip().split()\n",
    "                data.append((token, pos, chunk))\n",
    "            except:\n",
    "                pass\n",
    "    return data\n",
    "        \n",
    "train_data = load_data('data/seq/train.txt')\n",
    "\n",
    "print(len(train_data))\n",
    "for i in train_data[:10]:\n",
    "    print(' - '.join(i))\n",
    "    \n",
    "train_tokens, train_pos, train_chunk = zip(*train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Let us start with the POS labels, which we encode as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tag_encoder = LabelEncoder()\n",
    "tag_encoder.fit(train_pos)\n",
    "print('Total nb POS tags:', len(tag_encoder.classes_))\n",
    "\n",
    "y_train_pos = tag_encoder.transform(train_pos)\n",
    "\n",
    "Y_train_pos = np_utils.to_categorical(y_train_pos,\n",
    "                                  nb_classes=len(tag_encoder.classes_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we vectorize our training instances as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "vocab = Counter(train_tokens)\n",
    "indexer = {'<unk>': 0}\n",
    "\n",
    "for k, v in vocab.most_common():\n",
    "    indexer[k] = len(indexer)\n",
    "\n",
    "nb_left, nb_right = 2, 1\n",
    "\n",
    "def vectorize(tokens):\n",
    "    sequences = []\n",
    "    for curr_idx, token in enumerate(tokens):\n",
    "        left_context = tokens[(curr_idx - 2) : curr_idx]\n",
    "        while len(left_context) < nb_left:\n",
    "            left_context = ['<unk>'] + left_context\n",
    "\n",
    "        right_context = tokens[curr_idx + 1 : curr_idx + 2]\n",
    "        while len(right_context) < nb_right:\n",
    "            right_context += ['<unk>']\n",
    "\n",
    "        seq = left_context + [token] + right_context\n",
    "\n",
    "        ints = [indexer[t] if t in indexer else 0 for t in seq]\n",
    "\n",
    "        sequences.append(ints)\n",
    "    \n",
    "    return np.array(sequences, dtype='int32')\n",
    "\n",
    "X_train = vectorize(list(train_tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start building our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Dense, Activation\n",
    "\n",
    "from keras.layers.recurrent import LSTM\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=len(indexer), output_dim=150,\n",
    "                    input_length=nb_left + 1 + nb_right))\n",
    "model.add(LSTM(100, return_sequences=False, activation='tanh'))\n",
    "model.add(Dense(len(tag_encoder.classes_)))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, if you compare the following model to the one we had before, you see that only a single line has changed: instead of collapsing our 4 embedding vectors into a single, flat vector, we now have a **recurrent layer** loop over the embeddings and produce a single vector representation of the sequence at the end (hence `return_sequences=False`). The recurent layer we use is a [Long-Short Term Memory](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)) layer: such a layer will loop through our embedding vectors and model them *as a sequence* from left to tight. While the exact working of an LSTM is well out of the scope of this tutorial, the main advantage is that such a layer can remember information from previous timesteps for a pretty long time and have this information affect the way it processes the vectors it sees along the way. Let us test the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(X_train, Y_train_pos, batch_size=50, nb_epoch=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The functional API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have been working with keras's 'vanilla' model, i.e. the simple Sequential model where we plainly stack a series of layers on top of each other. There are many cases, however, where we would to like to have a more flexible way of construing a layer graph. Below, we will first work with a simple, yet highly relevant example: we will combine a left-to-right LSTM, with a right-to-left LSTM. To this end, we will make use of keras's extremely powerful `Model`, which is part of its so-called 'functional' API. When working with the `Model`, we first need to tell keras what our input data will look like. In our case, it primarily need to know the length of the integer sequences we will be passing it later. Note by the way, that we never have to specify anything about the size of the batches itself, because the model is largely agnostic of this, because we start actually fitting it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.layers import Input\n",
    "context_input = Input(shape=(nb_left + 1 + nb_right,), dtype='int32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assigned our `Input()` object to a variable called `context_input`. If we would like to feed this layer into our embeddings layer, we can now use the following, highly functional syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "embedding = Embedding(input_dim=len(indexer), output_dim=150)(context_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of a single layer can be fed to multiple new layers. This is exactly what we do in the next code block. After retrieving the relevant series of embedding vectors, we pass both to a left-to-right LSTM, as well as a right-to-left LSTM (cf. `go_backwards=True`). Both recurrent layers will produce a different result, but take the same input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "left_to_right = LSTM(100, return_sequences=False, activation='tanh')(embedding)\n",
    "right_to_left = LSTM(100, return_sequences=False, activation='tanh', go_backwards=True)(embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the functional API, it is also no problem to have multiple incoming layers. In the following code block, we combina the result of our two LSTM into a single layer, through summing them. This is where the `merge` layers comes in handy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.layers import merge\n",
    "merged = merge([left_to_right, right_to_left], mode='sum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now top of the model with a plain output layer to predict our class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "output = Dense(len(tag_encoder.classes_), activation='softmax')(merged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our full graph, we can instantiate our actual model, by specifying the start point and point of our graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "model = Model(input=context_input, output=output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model architecture neatly reflects the branching of our graph tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let us compile and test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(X_train, Y_train_pos, batch_size=50, nb_epoch=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you will have seen, each time we add more candy to our network, it becomes slower to train. (Luckily, we have GPU to take of that, but more about that later.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple inputs and outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have seen that is easy to branch and merge mayers using the functional API, it also becomes easy to understand that models can have multiple inputs and outputs. One additional input which we might want is the following: right now, our model using the same embeddings matrix for both the focus word and the context words. Intuitively however, it would make sense to reserve a second weight matrix for the focus token only. This is easy to achieve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we engineer a new input feature for our data outside keras. We separately encode the focus token again, using our vocabulary index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train_focus = [indexer[focus] if focus in indexer else 0 for focus in train_tokens]\n",
    "X_train_focus = np.array(X_train_focus, dtype='int32')\n",
    "print(len(X_train_focus))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now, we start building our graph, specifying *two*, instead of a single `Input` -- pay attention to the difference in the `shape` argument specified. To be able to distentangle both inputs later, it is useful to provide them with a name in the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "context_input = Input(shape=(4,), dtype='int32', name='context')\n",
    "focus_input = Input(shape=(1,), dtype='int32', name='focus')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can create two separate embedding layers, which will have two completely independent weight matrices, so that we can train different representations for the focus and context tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "context_embedding = Embedding(input_dim=len(indexer), output_dim=150)(context_input)\n",
    "focus_embedding = Embedding(input_dim=len(indexer), output_dim=150)(focus_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the `context_embedding`, we can proceed as we did before, with two LSTM branches, that get merged afterwards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "left_to_right = LSTM(100, return_sequences=False, activation='tanh')(context_embedding)\n",
    "right_to_left = LSTM(100, return_sequences=False, activation='tanh', go_backwards=True)(context_embedding)\n",
    "merged1 = merge([left_to_right, right_to_left], mode='sum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Silly enough, our embedding matrix is a 'matrix' consisting of a single vector, which means that we can safely flatten it into a plain vector, without losing any relevant spatio-temporal information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.layers.core import Flatten\n",
    "flat_context = Flatten()(focus_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the `merge` trick again to combine our two 'heads'. However, instead of summing them, we now simple **concatenate** the vectors for the layer pairs into a single long vector, using the `mode` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "merged2 = merge([merged1, flat_context], mode='concat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On top of this result, we can place our output layer again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pos_output = Dense(len(tag_encoder.classes_), activation='softmax', name='pos')(merged2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we now instantiate our `Model`, make sure that you specify a list of inputs (but a single output):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "model = Model(input=[context_input, focus_input], output=pos_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's fitting time! Note that, because we gave our layers names above, we can feed in the data using "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.summary()\n",
    "model.fit({'context':X_train,\n",
    "           'focus': X_train_focus},\n",
    "          {'pos': Y_train_pos},\n",
    "          batch_size=50, nb_epoch=3, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this is embarrasingly slow... Notice however that separating the embedding matrices for the input and context features leads to a quite a dramatic improvement in the fitting capacity of our model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us go *completely* nuts, and also add multiple outputs to our model. Interestingly, keras makes it really easy to learn several task simultaneously. In our case, we could try to predict both POS tags and chunking labels at the same time. Let us encode the chunk labels first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "chunk_encoder = LabelEncoder()\n",
    "chunk_encoder.fit(train_chunk)\n",
    "print('Total nb chunk labels:', len(chunk_encoder.classes_))\n",
    "\n",
    "y_train_chunk = chunk_encoder.transform(train_chunk)\n",
    "\n",
    "Y_train_chunk = np_utils.to_categorical(y_train_chunk,\n",
    "                                  nb_classes=len(chunk_encoder.classes_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding a second output is as simple as this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "context_input = Input(shape=(4,), dtype='int32', name='context')\n",
    "focus_input = Input(shape=(1,), dtype='int32', name='focus')\n",
    "\n",
    "context_embedding = Embedding(input_dim=len(indexer), output_dim=150)(context_input)\n",
    "left_to_right = LSTM(100, return_sequences=False, activation='tanh')(context_embedding)\n",
    "right_to_left = LSTM(100, return_sequences=False, activation='tanh', go_backwards=True)(context_embedding)\n",
    "merged1 = merge([left_to_right, right_to_left], mode='sum')\n",
    "\n",
    "focus_embedding = Embedding(input_dim=len(indexer), output_dim=150)(focus_input)\n",
    "flat_context = Flatten()(focus_embedding)\n",
    "\n",
    "merged2 = merge([merged1, flat_context], mode='concat')\n",
    "\n",
    "pos_output = Dense(len(tag_encoder.classes_), activation='softmax', name='pos')(merged2)\n",
    "chunk_output = Dense(len(chunk_encoder.classes_), activation='softmax', name='chunk')(merged2)\n",
    "\n",
    "model = Model(input=[context_input, focus_input], output=[pos_output, chunk_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only thing we add is another 'outgoing' softmax layer. Apart from that, when instantiating our model, we also use a list of outputs. Let us train this horse:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit({'context': X_train,\n",
    "           'focus': X_train_focus},\n",
    "          {'pos': Y_train_pos,\n",
    "           'chunk': Y_train_chunk},\n",
    "          batch_size=100, nb_epoch=10, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}