dert_transformer_based_object_detection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## In this presentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Sequence2sequence models\n",
    "- Attention mechanism\n",
    "- Transformers for NLP\n",
    "- Transformers for images\n",
    "- Set loss\n",
    "- Transformer based object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequence2sequence models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sequence to sequence model was developed for machine translation with RNN (LSTM or GRU) encoder and decoder\n",
    "<br>\n",
    "The architecture was first RNN, the encoder to encode the sentence in the one language, for instance French into the fixed size vector: $300$, $400$, $1024$, $2048$, etc\n",
    "<br>\n",
    "and second RNN, the decoder, to decode this vector to the sentence in other language, for instance in English "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of sequence2sequence:\n",
    "<img src=\"images/detr/seq2seq_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The RNN models are used to deal with sequences remembering the previous states information:\n",
    "<img src=\"images/detr/rnn_1.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Visualization of the recurrent neural network\n",
    "<img src=\"images/detr/rnn_anim_1.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Each cell is the neural network with recurrent connections, connection of the hidden layer:\n",
    "<img src=\"images/detr/rnn_anim_2.gif\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of translation:\n",
    "<img src=\"images/detr/seq2seq_2.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Turns out, that this model works quite well if we choose \"good\" pair of sentences and train it for long time\n",
    "<br>\n",
    "As I remember in the original paper, they sad that, when we reverse the input sentences it significantly improved the accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This model works not only for machine translation, but for other tasks like image captioning:\n",
    "<img src=\"images/detr/imcap_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main idea here, instead of using RNN as encoder, use ConvNet and use RNN only as decoder\n",
    "<br>\n",
    "Use pre-trained model without last layer to encoder image to the vector and feed this vector to the decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/imcap_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Turns out that when sentence becomes longer, RNN models are not good enough to forward the information about first words at the end or even in the middle and translation performance is suffering\n",
    "<br>\n",
    "To deal with this, we need somehow pay an attention on the parts of the sentence which are most influential for the part (for the particular word) in the output sentence\n",
    "<br>\n",
    "Naturally, when we try to translate long text, we first pay an attention on particular part of it, on particular words and part by part assemble the result "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's create an probabilistic mask for hidden layers $h^1, h^2, \\dots, h^n$ which are vectors, with weights:\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "\\alpha^{<1, 1>} & \\alpha^{<1, 2>} & \\cdots & \\alpha^{<1, n>} \\\\\n",
    "\\alpha^{<2, 1>} & \\alpha^{<2, 2>} & \\cdots & \\alpha^{<2, n>} \\\\\n",
    "\\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "\\alpha^{<m, 1>} & \\alpha^{<m, 2>} & \\cdots & \\alpha^{<m, n>} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "<br>\n",
    "where\n",
    "$\\sum_{j=1}^n{\\alpha^{<i, j>}} = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And for each hidden state (which is vector by itself) of decoder $s^t$ we generate the vector \n",
    "$$\n",
    "c^t = \\sum_{i=1}^n{\\alpha^{<t, i>} \\cdot h^i}\n",
    "$$\n",
    "<br>\n",
    "and concatenate $c^t$ with the original hidden state for output and input in the next recurrent unit $<c^t, s^t>$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Visualization of the attention:\n",
    "<img src=\"images/detr/attnt_1.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The question is how we generate this $\\alpha$, we use softmax:\n",
    "$$\n",
    "\\alpha^{<i, t>} = \\frac{exp(z^{<t, i>})}{\\sum_{j=1}^{n}exp(z^{<t, j>})}\n",
    "$$\n",
    "but how we calculate $z^{<t, i>}$ values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to generate function $f$ which consumes $h^i$ and $s^{t-1}$ and outputs the $z^{<t, i>}$ (because, $s^t$ is $s^{t-1}$ influenced) one function for all data\n",
    "<br>\n",
    "$$\n",
    "z^{<t, i>} = f(<h^i, s^{t-1}>)\n",
    "$$\n",
    "<br>\n",
    "Here $<.,.>$ means concatenation\n",
    "<br>\n",
    "So we can use a \"small\" neural network with maybe $1$ or $2$ hidden layer as our function $f$\n",
    "<br>\n",
    "The encoder RNN, decoder RNN and $f$ are all trained together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Attention is trained probabilities for each decoder's hidden state how it should influence encoders hidden state.\n",
    "<br>\n",
    "The same might be applied to any tensor to any other tensor transition like images or graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Attention improved performance of translation of long sequences and even image captioning with longer and more precise descriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Attention heatmaps:\n",
    "<img src=\"images/detr/attnt_2.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformers for NLP (<a href=\"http://jalammar.github.io/illustrated-transformer/\">source</a>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://arxiv.org/abs/1706.03762\"> Attention is all you need</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"http://jalammar.github.io/illustrated-transformer/\">The Illustrated Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://kazemnejad.com/blog/transformer_architecture_positional_encoding/\">Transformer Architecture: The Positional Encoding</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://medium.com/lsc-psd/introduction-of-self-attention-layer-in-transformer-fc7bff63f3bc\">Introduction of Self-Attention Layer in Transformer</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Transformer architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transformer consists with encoder and decoder and first was introduced as an alternative of sequence2sequence models for machine translation.\n",
    "<br>\n",
    "\n",
    "The key concept is that input sentence is fed as matrix with fixed dimensional word embedding at each row:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "x_{1, 1} & x_{1, 2} & \\cdots & x_{1, d} \\\\\n",
    "x_{2 1} & x_{2, 2} & \\cdots & x_{2, d} \\\\\n",
    "\\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "x_{m, 1} & x_{m, 2} & \\cdots & x_{m, d} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "and is multiplied on weights matrix with fixed dimensional columns (the same as word embeddings) and rows (hyper-parameter, $64$ in original paper):\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "w_{1, 1} & w_{1, 2} & \\cdots & w_{1, s} \\\\\n",
    "w_{2 1} & w_{2, 2} & \\cdots & w_{2, s} \\\\\n",
    "\\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "w_{d, 1} & w_{d, 2} & \\cdots & w_{d, s} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "So output is always fixed size $\\mathbb{R}^{d \\times s}$ and theoretically model can consume any length sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Model architecture:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The encoder and decoder networks:\n",
    "<img src=\"images/detr/transf_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Encoder consists with several different networks stacked together as well as decoder ($6$ in paper):\n",
    "<img src=\"images/detr/transf_2.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The architecture of each layer are similar: (multi-head) self-attention and then feed-forward layers for encoder and decoder, plus decoder has encoder-decoder attention layer as sequence2sequence models:\n",
    "<img src=\"images/detr/transf_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Self-attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First step: create three different vectors from each input vector (word embedding) using three different weight matrices: query, key and value\n",
    "\n",
    "Notice that these new vectors are smaller in dimension than the embedding vector. Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512. They don’t HAVE to be smaller, this is an architecture choice to make the computation of multiheaded attention (mostly) constant\n",
    "<img src=\"images/detr/selfatt_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The second step: calculate the score for each query with different keys:\n",
    "\n",
    "<br>\n",
    "The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring. So if we’re processing the self-attention for the word in position #1, the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2\n",
    "<img src=\"images/detr/selfatt_2.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Third step: Divide the scores by 8 (the square root of the dimension of the key vectors (in the paper 64 and divide on 8 respectively)\n",
    "<br>\n",
    "Fourth step: SoftMax the the results:\n",
    "<img src=\"images/detr/selfatt_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fifth step: Multiply each value vector on this scores, this will generate probability masked values as before\n",
    "<br>\n",
    "Sixth step: Sum all value vectors as output for first embedding:\n",
    "<img src=\"images/detr/selfatt_4.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The resulting vector is one we can send along to the feed-forward neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "All above steps might be done in matrix calculation:\n",
    "<img src=\"images/detr/selfatt_5.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then calculate attention outputs\n",
    "<img src=\"images/detr/selfatt_6.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of single self attention, multi-head attention is applied with different weights ($8$ in original paper):\n",
    "<img src=\"images/detr/selfatt_7.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Per embedding, different outputs are generated:\n",
    "<img src=\"images/detr/selfatt_8.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then outputs are concatenated horizontally and additional weights matrix is used to produce single matrix:\n",
    "<img src=\"images/detr/selfatt_9.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the big picture, performance is improved with multi-head attention (compare to features map):\n",
    "<img src=\"images/detr/selfatt_10.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Positional encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There is no notion of word order (1st word, 2nd word, ..) in the transformer architecture, thus positional encoding is applied, for $d$ dimensional embeddings:\n",
    "$$\n",
    "\\text{PE}(pos,2i)=sin\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right),\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "\\text{PE}(pos,2i+1)=cos\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right).\n",
    "$$\n",
    "generate the $d$ dimensional $\\mathbb{R}^d$ vectors with encoded positional information\n",
    "<br>\n",
    "$d_{model}=512$ model $i \\in [0, 255]$ in paper\n",
    "\n",
    "<br>\n",
    "This is not the only possible method for positional encoding. It, however, gives the advantage of being able to scale to unseen lengths of sequences (e.g. if our trained model is asked to translate a sentence longer than any of those in our training set).\n",
    "<br>\n",
    "\n",
    "The main thing here is, positional encoders should be distinguishable and periodic in order to encode sentences with length which was not seen during the training.\n",
    "<br>\n",
    "\n",
    "From the paper: \"We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PEpos+k can be represented as a linear function of PEpos.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence. The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention:\n",
    "<img src=\"images/detr/selfatt_11.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For example:\n",
    "<img src=\"images/detr/selfatt_12.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In the following figure, each row corresponds the a positional encoding of a vector. So the first row would be the vector we’d add to the embedding of the first word in an input sequence. Each row contains 512 values – each with a value between 1 and -1. We’ve color-coded them so the pattern is visible:\n",
    "<img src=\"images/detr/selfatt_13.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual connections:\n",
    "<img src=\"images/detr/selfatt_14.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Different type of normalizations (not only batch normalization exists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of batch normalization, lets learn mean and standard deviation for instance, layer, group, etc:\n",
    "<img src=\"images/detr/norm_1.png\" height=\"1000\" width=\"1000\">\n",
    "<br>\n",
    "For images anything beside batch normalization does not give any improvement and sometimes deteriorates performance, because of channel structure, but for transformer architecture, according to the inter-text context and (multi-head) attention it significantly improves performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "More detailed illustration of layer normalization:\n",
    "<img src=\"images/detr/norm_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Layer normalization in transformer associated with self attention, here input embeddings $X$ and output of the layer are summed to preserve the original information:\n",
    "<img src=\"images/detr/selfatt_15.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This goes for the sub-layers of the decoder as well. If we’re to think of a Transformer of 2 stacked encoders and decoders, it would look something like this:\n",
    "<img src=\"images/detr/selfatt_16.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V. These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence:\n",
    "<img src=\"images/detr/selfatt_17.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The following steps repeat the process until a special symbol is reached indicating the transformer decoder has completed its output. The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. And just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.\n",
    "\n",
    "<img src=\"images/detr/selfatt_18.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The “Encoder-Decoder Attention” layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Output of transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transformers has fully connected last layers with SoftMax activation of the vocabulary length vector, encoding by the probability, one-hot encoded word:\n",
    "<img src=\"images/detr/transf_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformers for images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://arxiv.org/abs/1802.05751\">Image Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/1904.09925\">Attention Augmented Convolutional Networks</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi-head self-attention technique was used for image decoding as well. The experiments show that mixed approach along with ConvNet layers increases accuracy and performs better than fully attention based model:\n",
    "<br>\n",
    "\"We test our method on the CIFAR-100 and ImageNet classification [22, 9] and the COCO object detection [27] tasks, across a wide range of architectures at different com- putational budgets, including a state-of-the art resource constrained architecture [42]. Attention Augmentation yields systematic improvements with minimal additional computational burden and notably outperforms the popu- lar Squeeze-and-Excitation [17] channelwise attention ap- proach in all experiments. In particular, Attention Augmen- tation achieves a 1.3% top-1 accuracy ImageNet on top of a ResNet50 baseline and 1.4 mAP increase in COCO ob- ject detection on top of a RetinaNet baseline. Suprisingly, experiments also reveal that fully self-attentional models, a special case of Attention Augmentation, only perform slightly worse than their fully convolutional counterparts on ImageNet, indicating that self-attention is a powerful stand- alone computational primitive for image classification.\"\n",
    "<br>\n",
    "\n",
    "In my opinion ConvNet layer have the property to forget inactive features, remove noise and extract word level features from continuous data where multi-head self attention shines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given the input tensor of shape\n",
    "$$\n",
    "(H, W, F_{in})\n",
    "$$\n",
    "multi-head self-attention is defined as:\n",
    "<br>\n",
    "\n",
    "$$\n",
    "O_h=\\text{SoftMax}(\\frac{(X \\cdot W_q)(X \\cdot W_k)}{\\sqrt{d_k^h}}) \\cdot (X \\cdot W_v)\n",
    "$$\n",
    "<br>\n",
    "\n",
    "Where $W_q, W_k \\in \\mathbb{R}^{F_{in} \\times d_k^h}$ and $W_v \\in \\mathbb{R}^{F_{in} \\times d_v^h}$ are learned liner transformations which map $X$ to queries $Q = X \\cdot W_q$, keys $K = X \\cdot W_k$ and values $X \\cdot W_v$\n",
    "<br>\n",
    "\n",
    "Output of multi-head self-attention layer is \\:\n",
    "$$\n",
    "\\text{MHA}(X) = \\text{concat}(Q_1, O_2, \\dots, O_{N_{h}}) \\cdot W^O\n",
    "$$\n",
    "<br>\n",
    "\n",
    "$$\n",
    "W^O = \\mathbb{R}^{d_u \\times d_u}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "Then the output is reshaped in $(H, W, d_u)$ to match the original input dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At the end attention augmented convolution is:\n",
    "$$\n",
    "\\text{AAConv}(X) = \\text{concat}( \\text{Conv}(X),\\text{MHA}(X))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Performance of attention augmented models:\n",
    "<img src=\"images/detr/aaconv_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Set prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Set prediction is applied to match two sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/dert_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer based object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers\">End-to-end Object Detection with Transformers</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://medium.com/lsc-psd/detr-object-detection-with-transformer-a97104ea1723\">DETR, Object detection with Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://www.youtube.com/watch?v=T35ba_VXkMY\">Paper explained in video</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://www.youtube.com/watch?v=LfUsGv-ESbc\">[Code] How to use Facebook's DETR object detection algorithm in Python (Full Tutorial)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The attention layers, the transformer architecture will encode spatial information:\n",
    "<img src=\"images/detr/detr_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_5.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_6.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_7.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_8.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_9.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_10.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_11.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_12.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/questions_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}