Hw2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import numpy as np\n",
    "from math import ceil\n",
    "import xml.etree.ElementTree as ET\n",
    "from util import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Data\n",
    "Given a word in a sentence, we transform it in a feature vector which includes the embedding vector and the POS tag. Thus, a sentence is a sequence of features vectors of the same size. The input space of our network will then be all the possibile features vector.\n",
    "\n",
    "The output space will be the union of the lemmas of unambiguous words and the possibile senses FOUND in the training data. \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The output space size is: 49219\n",
      "The features vectors have size: 301\n"
     ]
    }
   ],
   "source": [
    "embedding_dict, hidden_size = load_embeddings('glove.6B.300d.txt')\n",
    "\n",
    "senses_dict, truesenses_dict, wf_lemmas_dict,senses_per_lemma_dict, out_size = compute_output_space('semcor.data.xml','semcor.gold.key.bnids.txt')\n",
    "\n",
    "print('The output space size is: ' + str(out_size),flush=True)\n",
    "\n",
    "n_features = hidden_size + 1 \n",
    "print('The features vectors have size: ' + str(n_features))\n",
    "\n",
    "#Window size for the padding\n",
    "window_size = 30\n",
    "window_overlap = 5\n",
    "\n",
    "keep_prob = 0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Data\n",
    "We load the training data with a xml_parser function which uses the previously generated data structures to get a list of sentences, labels, sentences' lengths and possible choices for ambiguous lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = ET.parse('semcor.data.xml')\n",
    "root = tree.getroot()\n",
    "\n",
    "tr_sentences,tr_labels,tr_lens,tr_amb_lemmas = xml_parser(root.getchildren(),window_size,window_overlap,embedding_dict,senses_dict,truesenses_dict,wf_lemmas_dict,senses_per_lemma_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation Data\n",
    "Being the instance_id in the validation data different from the ones found in training data, truesenese_dict needs to be updated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('ALL.gold.key.bnids.txt','r') as f:\n",
    "    for line in f.readlines():\n",
    "        instance_id, sense = line.split()\n",
    "        truesenses_dict[instance_id] = sense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then load the validation data but splitting by dataset. The first part just reads the names of dataset in the data. The second part loads the validation data creating dev_data which is organized as follow:\n",
    "\n",
    "dev_data = [D_1,D_2,..] where D_x is a dataset\n",
    "\n",
    "D_x = [sentences,labels,lens,amb_lemmas] for that dataset\n",
    "\n",
    "sentences,labels,lens,amb_lemmas are structed in the same way as before\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = ET.parse('ALL.data.xml')\n",
    "root = tree.getroot()\n",
    "\n",
    "dev_texts = []                                      #list of lists: text split by datasets\n",
    "datasets_names = []                                 #list: names of the found datasets\n",
    "\n",
    "dev_data = []      \n",
    "\n",
    "#First part\n",
    "for text in root.getchildren():\n",
    "    dataset_name = text.attrib['id'].split('.')[0]\n",
    "    \n",
    "    if dataset_name not in datasets_names:  # add the new dataset\n",
    "        datasets_names.append(dataset_name)\n",
    "        dev_texts.append([])\n",
    "        \n",
    "    dev_texts[datasets_names.index(dataset_name)].append(text) #assign the (text) node to the right dataset\n",
    "\n",
    "#Second part\n",
    "for dataset in dev_texts:\n",
    "    \n",
    "    dv_sentences,dv_labels,dv_lens,dv_amb = xml_parser(dataset,window_size,window_overlap,embedding_dict,senses_dict,truesenses_dict,wf_lemmas_dict,senses_per_lemma_dict)\n",
    "    \n",
    "    dev_data.append([dv_sentences,dv_labels,dv_lens,dv_amb])\n",
    "\n",
    "del dev_texts,dv_sentences,dv_labels,dv_lens, dv_amb,root, tree #removing useless data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network\n",
    "The network is structured as following:\n",
    "\n",
    "Sentences, labels and lengths are read trought placeholder.There is one layer of BiLSTM. Lastly there are two output layer, one used for the label prediction and the other for pos tag prediction.\n",
    "\n",
    "    inputs (sentences) has shape (batch_size,window_size,n_features)\n",
    "    targets (labels) has shape (batch_size,window_size,2) (2 = word_label + pos_label)\n",
    "    seq_lens (lens) has shape (batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.placeholder(tf.float32, shape=[None,window_size,n_features])\n",
    "targets = tf.placeholder(tf.int32, shape=[None,None,2])\n",
    "seq_lens = tf.placeholder(tf.int32, shape=[None])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_units = 500\n",
    "\n",
    "cell_fw = tf.contrib.rnn.LSTMCell(lstm_units)\n",
    "cell_bw = tf.contrib.rnn.LSTMCell(lstm_units)\n",
    "\n",
    "#Dropout\n",
    "cell_fw = tf.nn.rnn_cell.DropoutWrapper(cell_fw, output_keep_prob=keep_prob)\n",
    "cell_bw = tf.nn.rnn_cell.DropoutWrapper(cell_bw, output_keep_prob=keep_prob)\n",
    "\n",
    "outputs,_ = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, inputs,sequence_length=seq_lens, dtype=tf.float32)\n",
    "\n",
    "#Concatenation of the two hidden states\n",
    "full_output = tf.concat([outputs[0],outputs[1]],axis=-1)\n",
    "#full_output has shape (batch_size,window_size,lstm_units*2)\n",
    "\n",
    "#Generating a mask to ignore padding\n",
    "mask = tf.sequence_mask(seq_lens)\n",
    "#mask has shape (batch_size,window_size)\n",
    "\n",
    "#Mask repated in time for the attention layer\n",
    "mask_repeated = tf.reshape(tf.tile(mask,[1,window_size]),(-1,window_size,window_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionWithContext:\n",
    "    def __init__(self,input_shape, **kwargs):\n",
    "        super(AttentionWithContext,self).__init__(**kwargs)\n",
    "        self.build(input_shape)\n",
    "        \n",
    "    def build(self,input_shape):\n",
    "        dense = tf.keras.layers.Dense(input_shape[1],activation=tf.nn.tanh,use_bias=True)\n",
    "        self.td = tf.keras.layers.TimeDistributed(dense)\n",
    "        \n",
    "    def __call__(self,inputs,mask):\n",
    "        focused_a = self.td(inputs)\n",
    "        focused_a = tf.exp(focused_a)\n",
    "        \n",
    "        if mask is not None:\n",
    "            mask = tf.cast(mask, tf.float32)\n",
    "            foucsed_a = mask * focused_a\n",
    "            \n",
    "        focused_a /= tf.cast(tf.reduce_sum(focused_a,axis=1,keepdims=True) + tf.keras.backend.epsilon(),tf.float32)\n",
    "        \n",
    "        focused_features = tf.keras.backend.batch_dot(focused_a,inputs)\n",
    "        return [focused_features,focused_a]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "att_output, _ = AttentionWithContext(full_output.get_shape().as_list())(full_output,mask_repeated)\n",
    "\n",
    "output_concat = tf.concat([full_output,att_output],-1)\n",
    "\n",
    "#Size of the last dimension after the concatenation\n",
    "aug_features = output_concat.get_shape()[-1]\n",
    "\n",
    "#Flattening\n",
    "output_flat = tf.reshape(output_concat,[-1,aug_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fully connected layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc_w = 1024\n",
    "\n",
    "W_fc = tf.get_variable('W_fc',shape=[aug_features,fc_w],dtype=tf.float32)\n",
    "b_fc = tf.get_variable('b_fc',shape=[fc_w],dtype=tf.float32) \n",
    "\n",
    "h_fc = tf.nn.relu(tf.matmul(output_flat,W_fc) + b_fc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slicing the targets placeholder to extract the label for the word and the pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_label = tf.squeeze(tf.slice(targets,(0,0,0),(tf.shape(targets)[0],tf.shape(targets)[1],1)),-1)\n",
    "target_pos = tf.squeeze(tf.slice(targets,(0,0,1),(tf.shape(targets)[0],tf.shape(targets)[1],1)),-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss the label for the word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_label = tf.get_variable('W_label',shape=[fc_w,out_size],dtype=tf.float32)\n",
    "b_label = tf.get_variable('b_label',shape=[out_size],dtype=tf.float32) \n",
    "\n",
    "logits_label = tf.matmul(h_fc,W_label) + b_label\n",
    "logits_label = tf.reshape(logits_label,[-1,window_size,out_size])\n",
    "\n",
    "losses_label = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_label,labels=target_label) #UNSCALED LOGITS!\n",
    "#losses_label has shape (batch_size,window_size). Computes the loss for each word\n",
    "\n",
    "#Ignore the losses from padding\n",
    "losses_label = tf.boolean_mask(losses_label,mask)\n",
    "#losses_label has shape (n) where n is the sum of the lens\n",
    "\n",
    "#Summing the losses\n",
    "loss_label = tf.reduce_sum(losses_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss the POS for the word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_pos = tf.get_variable('W_pos',shape=[aug_features,n_tags],dtype=tf.float32)\n",
    "b_pos = tf.get_variable('b_pos',shape=[n_tags],dtype=tf.float32) \n",
    "\n",
    "logits_pos = tf.matmul(output_flat,W_pos) + b_pos\n",
    "logits_pos = tf.reshape(logits_pos,[-1,window_size,n_tags])\n",
    "\n",
    "losses_pos = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_pos,labels=target_pos)\n",
    "losses_pos = tf.boolean_mask(losses_pos,mask)\n",
    "loss_pos = tf.reduce_sum(losses_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Total loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:98: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    }
   ],
   "source": [
    "total_loss = loss_pos + loss_label\n",
    "\n",
    "optimizer = tf.train.AdadeltaOptimizer(learning_rate=0.5).minimize(total_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions and Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_label_values = tf.nn.softmax(logits_label)\n",
    "\n",
    "preds_pos = tf.nn.softmax(logits_pos)\n",
    "preds_pos = tf.argmax(preds_pos,axis=-1)\n",
    "\n",
    "preds_pos = tf.boolean_mask(preds_pos,mask)\n",
    "true_pos = tf.boolean_mask(target_pos,mask)\n",
    "\n",
    "#Accuracy on POS learning\n",
    "correct_pos = tf.cast(tf.equal(tf.cast(preds_pos,tf.int32), true_pos),tf.float32)\n",
    "accuracy_pos = tf.reduce_mean(correct_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary scalars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.summary.scalar('loss_label',loss_label)\n",
    "tf.summary.scalar('loss_pos',loss_pos)\n",
    "tf.summary.scalar('total_loss',total_loss)\n",
    "\n",
    "tf.summary.scalar('accuracy_pos',accuracy_pos)\n",
    "\n",
    "summary = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "sess.run(tf.local_variables_initializer())\n",
    "\n",
    "summary_writer = tf.summary.FileWriter('./summary', sess.graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "saver = tf.train.Saver()\n",
    "\n",
    "if not os.path.exists(\"model\"):\n",
    "    os.makedirs(\"model\")\n",
    "    \n",
    "if tf.train.checkpoint_exists('./model/model.ckpt'):\n",
    "    saver.restore(sess, './model/model.ckpt')\n",
    "    print(\"Previous model restored.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 0\n",
      "Iteration: 10\n",
      "Iteration: 20\n",
      "Iteration: 30\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-27-bb6f1c6d30ca>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     33\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'{} : precision on senses {} recall on senses {} f1_score on senses {} accuracy on pos {}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mprecision\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mrecall\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf1_score\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0macc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m     \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m \u001b[0minputs\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mbatch_input\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtargets\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mbatch_label\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mseq_lens\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mbatch_lens\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    903\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    904\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 905\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    906\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    907\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1135\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1136\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1137\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1138\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1139\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1353\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1354\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1355\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1356\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1357\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1359\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1360\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1361\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1362\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1363\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1338\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1339\u001b[0m           return tf_session.TF_Run(session, options, feed_dict, fetch_list,\n\u001b[0;32m-> 1340\u001b[0;31m                                    target_list, status, run_metadata)\n\u001b[0m\u001b[1;32m   1341\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "epochs = 4\n",
    "\n",
    "batch_size = 32\n",
    "batch_index = 0\n",
    "num_batches_per_epoch = ceil(len(tr_lens)/batch_size)\n",
    "\n",
    "n_iterations = num_batches_per_epoch * epochs\n",
    "\n",
    "for ite in range(n_iterations):\n",
    "    \n",
    "    if ite % 10 == 0:\n",
    "        print('Iteration: ' + str(ite) )\n",
    "        \n",
    "    #Batch\n",
    "    batch_input = tr_sentences[batch_index*batch_size:(batch_index+1)*batch_size]\n",
    "    batch_label = tr_labels[batch_index*batch_size:(batch_index+1)*batch_size]\n",
    "    batch_lens = tr_lens[batch_index*batch_size:(batch_index+1)*batch_size]\n",
    "    batch_dv = tr_amb_lemmas[batch_index*batch_size:(batch_index+1)*batch_size]\n",
    "    \n",
    "    batch_index = (batch_index + 1 ) % num_batches_per_epoch\n",
    "    \n",
    "    if batch_index % 50 == 0:\n",
    "        saver.save(sess, './model/model.ckpt')\n",
    "        print('Batch scores:')\n",
    "        predicted_values,target_values,acc = sess.run([preds_label_values,target_label,accuracy_pos],feed_dict={ inputs : batch_input, targets : batch_label, seq_lens : batch_lens})\n",
    "        precision,_,f1_score = scores_senses(predicted_values,target_values,batch_dv)\n",
    "        print('Training: precision on senses {} f1_score on senses {} accuracy on pos {}'.format(precision,f1_score,acc),flush=True)\n",
    "        \n",
    "        print('Development set scores:')\n",
    "        for dataset,name in zip(dev_data,datasets_names):\n",
    "            predicted_values,target_values, acc = sess.run([preds_label_values,target_label,accuracy_pos],feed_dict={ inputs : dataset[0],targets : dataset[1], seq_lens : dataset[2]})\n",
    "            precision,recall,f1_score = scores_senses(predicted_values,target_values,dataset[3])\n",
    "            print('{} : precision on senses {} recall on senses {} f1_score on senses {} accuracy on pos {}'.format(name,precision,recall,f1_score,acc),flush=True)\n",
    "        \n",
    "    sess.run(optimizer,feed_dict={ inputs : batch_input, targets : batch_label, seq_lens : batch_lens})\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_sentences = []\n",
    "\n",
    "with open('test_data.txt','r') as f:\n",
    "    for line_sentence in f.readlines():\n",
    "        words = line_sentence.split()\n",
    "        test_sentence = []\n",
    "        for word in words:\n",
    "            word_parts = word.split('|')\n",
    "            \n",
    "            word_vector = generate_word_vector_pos(word_parts[1],word_parts[2],embedding_dict)\n",
    "            test_sentence.append(word_vector)\n",
    "        test_sentences.append(test_sentence)\n",
    "\n",
    "        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}