tutorials/intro/Intro_Tutorial_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Snorkel: Extracting Spouse Relations from the News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part II: Generating _and modeling_ noisy training labels\n",
    "\n",
    "In this part of the tutorial, we will write **labeling functions** which express various heuristics, patterns, and [_weak supervision_](http://hazyresearch.github.io/snorkel/blog/weak_supervision.html) strategies to label our data.\n",
    "\n",
    "In most real-world settings, hand-labeled training data is prohibitively expensive and slow to collect. A common scenario, though, is to have access to tons of _unlabeled_ training data, and have some idea of how to label it programmatically. For example:\n",
    "\n",
    "* We may be able to think of text patterns that would indicate two people mentioned in a sentence are married, such as seeing the word \"spouse\" between the mentions.\n",
    "* We may have access to an external _knowledge base (KB)_ that lists some known pairs of married people, and can use these to heuristically label some subset of our data.\n",
    "\n",
    "Our labeling functions will capture these types of strategies. We know that these labeling functions will not be perfect, and some may be quite low-quality, so we will _model_ their accuracies with a generative model, which Snorkel will help us easily apply.\n",
    "\n",
    "This will ultimately produce a single set of **noise-aware training labels**, which we will then use to train an end extraction model in the next notebook.  For more technical details of this overall approach, see our [NIPS 2016 paper](https://arxiv.org/abs/1605.07723)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "import numpy as np\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass from Parts II and III."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using a labeled _development set_\n",
    "\n",
    "In our setting here, we will use the phrase \"development set\" to refer to a _small_ set of examples (here, a subset of our training set) which we label by hand and use to help us develop and refine labeling functions. Unlike the _test set_, which we do not look at and use for final evaluation, we can inspect the development set while writing labeling functions.\n",
    "\n",
    "In our case, we already loaded existing labels for a development set (`split` 1), so we can load them again now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Modeling a Noisy Training Set\n",
    "\n",
    "Our biggest step in the data programming pipeline is the creation - _and modeling_ - of a noisy training set.  We'll approach this in three main steps:\n",
    "\n",
    "1. **Creating labeling functions (LFs):** This is where most of our development time would actually go into if this were a real application. Labeling functions encode our heuristics and weak supervision signals to generate (noisy) labels for our training candidates.\n",
    "\n",
    "2. **Applying the LFs:** Here, we actually use them to label our candidates!\n",
    "\n",
    "3. **Training a generative model of our training set:** Here we learn a model over our LFs, learning their respective accuracies automatically. This will allow us to combine them into a single, higher-quality label set.\n",
    "\n",
    "We'll also add some detail on how to go about _developing labeling functions_ and then _debugging our model_ of them to improve performance.\n",
    "\n",
    "## 1. Creating Labeling Functions\n",
    "\n",
    "In Snorkel, our primary interface through which we provide training signal to the end extraction model we are training is by writing **labeling functions (LFs)** (as opposed to hand-labeling massive training sets).  We'll go through some examples for our spouse extraction task below.\n",
    "\n",
    "A labeling function is just a Python function that accepts a `Candidate` and returns `1` to mark the `Candidate` as true, `-1` to mark the `Candidate` as false, and `0` to abstain from labeling the `Candidate` (note that the non-binary classification setting is covered in the advanced tutorials!).\n",
    "\n",
    "In the next stages of the Snorkel pipeline, we'll train a model to learn the accuracies of the labeling functions and reweight them accordingly, and then use them to train a downstream model. It turns out by doing this, we can get high-quality models even with lower-quality labeling functions. So they don't need to be perfect! Now on to writing some:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from snorkel.lf_helpers import (\n",
    "    get_left_tokens, get_right_tokens, get_between_tokens,\n",
    "    get_text_between, get_tagged_text,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pattern-based LFs\n",
    "These LFs express some common sense text patterns which indicate that a person pair might be married. For example, `LF_husband_wife` looks for words in `spouses` between the person mentions, and `LF_same_last_name` checks to see if the two people have the same last name (but aren't the same whole name)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "spouses = {'spouse', 'wife', 'husband', 'ex-wife', 'ex-husband'}\n",
    "family = {'father', 'mother', 'sister', 'brother', 'son', 'daughter',\n",
    "              'grandfather', 'grandmother', 'uncle', 'aunt', 'cousin'}\n",
    "family = family | {f + '-in-law' for f in family}\n",
    "other = {'boyfriend', 'girlfriend' 'boss', 'employee', 'secretary', 'co-worker'}\n",
    "\n",
    "# Helper function to get last name\n",
    "def last_name(s):\n",
    "    name_parts = s.split(' ')\n",
    "    return name_parts[-1] if len(name_parts) > 1 else None    \n",
    "\n",
    "def LF_husband_wife(c):\n",
    "    return 1 if len(spouses.intersection(get_between_tokens(c))) > 0 else 0\n",
    "\n",
    "def LF_husband_wife_left_window(c):\n",
    "    if len(spouses.intersection(get_left_tokens(c[0], window=2))) > 0:\n",
    "        return 1\n",
    "    elif len(spouses.intersection(get_left_tokens(c[1], window=2))) > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "def LF_same_last_name(c):\n",
    "    p1_last_name = last_name(c.person1.get_span())\n",
    "    p2_last_name = last_name(c.person2.get_span())\n",
    "    if p1_last_name and p2_last_name and p1_last_name == p2_last_name:\n",
    "        if c.person1.get_span() != c.person2.get_span():\n",
    "            return 1\n",
    "    return 0\n",
    "\n",
    "def LF_no_spouse_in_sentence(c):\n",
    "    return -1 if np.random.rand() < 0.75 and len(spouses.intersection(c.get_parent().words)) == 0 else 0\n",
    "\n",
    "def LF_and_married(c):\n",
    "    return 1 if 'and' in get_between_tokens(c) and 'married' in get_right_tokens(c) else 0\n",
    "    \n",
    "def LF_familial_relationship(c):\n",
    "    return -1 if len(family.intersection(get_between_tokens(c))) > 0 else 0\n",
    "\n",
    "def LF_family_left_window(c):\n",
    "    if len(family.intersection(get_left_tokens(c[0], window=2))) > 0:\n",
    "        return -1\n",
    "    elif len(family.intersection(get_left_tokens(c[1], window=2))) > 0:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "def LF_other_relationship(c):\n",
    "    return -1 if len(other.intersection(get_between_tokens(c))) > 0 else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distant Supervision LFs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to writing labeling functions that describe text pattern-based heuristics for labeling training examples, we can also write labeling functions that distantly supervise examples. Here, we'll load in a list of known spouse pairs and check to see if the candidate pair matches one of these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bz2\n",
    "\n",
    "# Function to remove special characters from text\n",
    "def strip_special(s):\n",
    "    return ''.join(c for c in s if ord(c) < 128)\n",
    "\n",
    "# Read in known spouse pairs and save as set of tuples\n",
    "with bz2.BZ2File('data/spouses_dbpedia.csv.bz2', 'rb') as f:\n",
    "    known_spouses = set(\n",
    "        tuple(strip_special(x.decode('utf-8')).strip().split(',')) for x in f.readlines()\n",
    "    )\n",
    "# Last name pairs for known spouses\n",
    "last_names = set([(last_name(x), last_name(y)) for x, y in known_spouses if last_name(x) and last_name(y)])\n",
    "    \n",
    "def LF_distant_supervision(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    return 1 if (p1, p2) in known_spouses or (p2, p1) in known_spouses else 0\n",
    "\n",
    "def LF_distant_supervision_last_names(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    p1n, p2n = last_name(p1), last_name(p2)\n",
    "    return 1 if (p1 != p2) and ((p1n, p2n) in last_names or (p2n, p1n) in last_names) else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For later convenience we group the labeling functions into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "LFs = [\n",
    "    LF_distant_supervision, LF_distant_supervision_last_names, \n",
    "    LF_husband_wife, LF_husband_wife_left_window, LF_same_last_name,\n",
    "    LF_no_spouse_in_sentence, LF_and_married, LF_familial_relationship, \n",
    "    LF_family_left_window, LF_other_relationship\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Developing Labeling Functions\n",
    "\n",
    "Above, we've written a bunch of labeling functions already, which should give you some sense about how to go about it. While writing them, we probably want to check to make sure that they at least work as intended before adding to our set. Suppose we're thinking about writing a simple LF: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LF_wife_in_sentence(c):\n",
    "    \"\"\"A simple example of a labeling function\"\"\"\n",
    "    return 1 if 'wife' in c.get_parent().words else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One simple thing we can do is quickly test it on our development set (or any other set), without saving it to the database.  This is simple to do. For example, we can easily get every candidate that this LF labels as true:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number labeled:', 278)\n"
     ]
    }
   ],
   "source": [
    "labeled = []\n",
    "for c in session.query(Spouse).filter(Spouse.split == 1).all():\n",
    "    if LF_wife_in_sentence(c) != 0:\n",
    "        labeled.append(c)\n",
    "print(\"Number labeled:\", len(labeled))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then easily put this into the Viewer as usual (try it out!):\n",
    "```\n",
    "SentenceNgramViewer(labeled, session)\n",
    "```\n",
    "\n",
    "We also have a simple helper function for getting the empirical accuracy of a single LF with respect to the development set labels for example. This function also returns the evaluation buckets of the candidates (true positive, false positive, true negative, false negative):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.235\n",
      "Recall               1.0\n",
      "F1                   0.38\n",
      "----------------------------------------\n",
      "TP: 62 | FP: 202 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from snorkel.lf_helpers import test_LF\n",
    "tp, fp, tn, fn = test_LF(session, LF_wife_in_sentence, split=1, annotator_name='gold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database.  We'll do this using the `LabelAnnotator` class, a UDF which we will again run with `UDFRunner`.  **Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.**  We start by setting up the class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "labeler = LabelAnnotator(lfs=LFs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we run the `labeler`. Note that we set a random seed for reproducibility, since some of the LFs involve random number generators. Again, this can be run in parallel, given an appropriate database like Postgres is being used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In labeling, found 22276 cand IDs for split 0!\n",
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 22276/22276 [01:54<00:00, 194.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 49s, sys: 1.45 s, total: 1min 50s\n",
      "Wall time: 1min 55s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<22276x10 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 22267 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(1701)\n",
    "%time L_train = labeler.apply(split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we've already created the labels (saved in the database), we can load them in as a sparse matrix here too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 234 ms, sys: 4.58 ms, total: 239 ms\n",
      "Wall time: 240 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<22276x10 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 22267 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time L_train = labeler.load_matrix(session, split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the returned matrix is a special subclass of the `scipy.sparse.csr_matrix` class, with some special features which we demonstrate below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Spouse(Span(\"Anna Paquin\", sentence=36995, chars=[313,323], words=[63,64]), Span(\"Nancy Holt\", sentence=36995, chars=[340,349], words=[69,70]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_candidate(session, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelKey (LF_distant_supervision)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_key(session, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also view statistics about the resulting label matrix.\n",
    "\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a *conflicting* non-zero label for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>0</td>\n",
       "      <td>0.001481</td>\n",
       "      <td>0.001481</td>\n",
       "      <td>0.000628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>1</td>\n",
       "      <td>0.008080</td>\n",
       "      <td>0.007856</td>\n",
       "      <td>0.004758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife</th>\n",
       "      <td>2</td>\n",
       "      <td>0.104642</td>\n",
       "      <td>0.066798</td>\n",
       "      <td>0.017867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>3</td>\n",
       "      <td>0.078021</td>\n",
       "      <td>0.057910</td>\n",
       "      <td>0.010774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_same_last_name</th>\n",
       "      <td>4</td>\n",
       "      <td>0.016700</td>\n",
       "      <td>0.014994</td>\n",
       "      <td>0.010011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>5</td>\n",
       "      <td>0.603026</td>\n",
       "      <td>0.081657</td>\n",
       "      <td>0.009472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_and_married</th>\n",
       "      <td>6</td>\n",
       "      <td>0.000673</td>\n",
       "      <td>0.000539</td>\n",
       "      <td>0.000404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>7</td>\n",
       "      <td>0.104283</td>\n",
       "      <td>0.091489</td>\n",
       "      <td>0.021413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.073352</td>\n",
       "      <td>0.067651</td>\n",
       "      <td>0.012076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_other_relationship</th>\n",
       "      <td>9</td>\n",
       "      <td>0.009337</td>\n",
       "      <td>0.006868</td>\n",
       "      <td>0.001122</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   j  Coverage  Overlaps  Conflicts\n",
       "LF_distant_supervision             0  0.001481  0.001481   0.000628\n",
       "LF_distant_supervision_last_names  1  0.008080  0.007856   0.004758\n",
       "LF_husband_wife                    2  0.104642  0.066798   0.017867\n",
       "LF_husband_wife_left_window        3  0.078021  0.057910   0.010774\n",
       "LF_same_last_name                  4  0.016700  0.014994   0.010011\n",
       "LF_no_spouse_in_sentence           5  0.603026  0.081657   0.009472\n",
       "LF_and_married                     6  0.000673  0.000539   0.000404\n",
       "LF_familial_relationship           7  0.104283  0.091489   0.021413\n",
       "LF_family_left_window              8  0.073352  0.067651   0.012076\n",
       "LF_other_relationship              9  0.009337  0.006868   0.001122"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.lf_stats(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Fitting the Generative Model\n",
    "Now, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inferred cardinality: 2\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning import GenerativeModel\n",
    "\n",
    "gen_model = GenerativeModel()\n",
    "gen_model.train(L_train, epochs=100, decay=0.95, step_size=0.1 / L_train.shape[0], reg_param=1e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.07592901, 0.07395425, 0.11954169, 0.11397737, 0.07065144,\n",
       "       0.6901572 , 0.07358515, 0.15698341, 0.13658573, 0.08221857])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.weights.lf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now apply the generative model to the training candidates to get the noise-aware training label set. We'll refer to these as the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_marginals = gen_model.marginals(L_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at the distribution of the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEmNJREFUeJzt3X+s3Xd93/Hnq/FCC6M4kLuI2dbsDbcoICbSu5AJqepIlxjocKqmyKgbhrq1tqU/tlYqyZiUCYqWrFMzUIHKIx5OxTBR2ileCWReCEKt6pAbkgacNOSShMZWILexE7ZGQE3f++N8vB78uc6995zre66vnw/p6n6/7+/ne877k3NvXvf74xynqpAkadgPTLoBSdLqYzhIkjqGgySpYzhIkjqGgySpYzhIkjqGgySpYzhIkjqGgySps27SDYzqwgsvrM2bN0+6DUk6q9x3331/UVVTC407a8Nh8+bNzMzMTLoNSTqrJPn6YsYteFopyd4kTyf5ylDtt5L8WZIHk/yPJOuHtl2XZDbJI0muHKpva7XZJNcO1bckuafVP5Xk/MVPU5J0JizmmsPHgW2n1A4Cr62q1wFfBa4DSHIxsAN4TdvnI0nOS3Ie8GHgzcDFwDvaWIAbgZuq6lXAcWDXWDOSJI1twXCoqi8Ax06p/a+qOtFWDwEb2/J2YH9VfaeqHgdmgUvb12xVPVZV3wX2A9uTBHgTcFvbfx9w1ZhzkiSNaTnuVvp54DNteQPw5NC2I612uvorgGeHguZkXZI0QWOFQ5L3AieATyxPOws+3+4kM0lm5ubmVuIpJemcNHI4JHkX8FPAz9Xf/ItBR4FNQ8M2ttrp6s8A65OsO6U+r6raU1XTVTU9NbXgnViSpBGNFA5JtgG/Abytqp4f2nQA2JHkRUm2AFuBLwL3AlvbnUnnM7hofaCFyt3A1W3/ncDto01FkrRcFnMr6yeBPwF+NMmRJLuA3wFeChxM8kCS3wWoqsPArcBDwGeBa6rqe+2awi8BdwIPA7e2sQDvAX4tySyDaxA3L+sMJUlLlrP135Cenp4u3wQnSUuT5L6qml5o3Fn7Dulz0eZrPz3yvk/c8NZl7ETSWucH70mSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOguGQ5K9SZ5O8pWh2suTHEzyaPt+QasnyYeSzCZ5MMklQ/vsbOMfTbJzqP5jSb7c9vlQkiz3JCVJS7OYI4ePA9tOqV0L3FVVW4G72jrAm4Gt7Ws38FEYhAlwPfAG4FLg+pOB0sb84tB+pz6XJGmFLRgOVfUF4Ngp5e3Avra8D7hqqH5LDRwC1id5JXAlcLCqjlXVceAgsK1t++GqOlRVBdwy9FiSpAkZ9ZrDRVX1VFv+BnBRW94APDk07kirvVD9yDx1SdIEjX1Buv3FX8vQy4KS7E4yk2Rmbm5uJZ5Sks5Jo4bDN9spIdr3p1v9KLBpaNzGVnuh+sZ56vOqqj1VNV1V01NTUyO2LklayKjhcAA4ecfRTuD2ofo7211LlwHPtdNPdwJXJLmgXYi+ArizbftWksvaXUrvHHosSdKErFtoQJJPAj8BXJjkCIO7jm4Abk2yC/g68PY2/A7gLcAs8DzwboCqOpbk/cC9bdz7qurkRe5/zeCOqB8CPtO+JEkTtGA4VNU7TrPp8nnGFnDNaR5nL7B3nvoM8NqF+pAkrRzfIS1J6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6owVDkn+bZLDSb6S5JNJfjDJliT3JJlN8qkk57exL2rrs2375qHHua7VH0ly5XhTkiSNa+RwSLIB+BVguqpeC5wH7ABuBG6qqlcBx4FdbZddwPFWv6mNI8nFbb/XANuAjyQ5b9S+JEnjG/e00jrgh5KsA14MPAW8Cbitbd8HXNWWt7d12vbLk6TV91fVd6rqcWAWuHTMviRJYxg5HKrqKPCfgT9nEArPAfcBz1bViTbsCLChLW8Anmz7nmjjXzFcn2ef75Nkd5KZJDNzc3Ojti5JWsA4p5UuYPBX/xbg7wIvYXBa6Iypqj1VNV1V01NTU2fyqSTpnDbOaaWfBB6vqrmq+ivgD4A3AuvbaSaAjcDRtnwU2ATQtr8MeGa4Ps8+kqQJGCcc/hy4LMmL27WDy4GHgLuBq9uYncDtbflAW6dt/1xVVavvaHczbQG2Al8coy9J0pjWLTxkflV1T5LbgC8BJ4D7gT3Ap4H9SX6z1W5uu9wM/F6SWeAYgzuUqKrDSW5lECwngGuq6nuj9iVJGt/I4QBQVdcD159Sfox57jaqqm8DP3uax/kA8IFxepEkLR/fIS1J6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqTOWOGQZH2S25L8WZKHk/zjJC9PcjDJo+37BW1sknwoyWySB5NcMvQ4O9v4R5PsHHdSkqTxjHvk8EHgs1X1auAfAg8D1wJ3VdVW4K62DvBmYGv72g18FCDJy4HrgTcAlwLXnwwUSdJkjBwOSV4G/DhwM0BVfbeqngW2A/vasH3AVW15O3BLDRwC1id5JXAlcLCqjlXVceAgsG3UviRJ4xvnyGELMAf8tyT3J/lYkpcAF1XVU23MN4CL2vIG4Mmh/Y+02unqkqQJGScc1gGXAB+tqtcDf8nfnEICoKoKqDGe4/sk2Z1kJsnM3Nzccj2sJOkU44TDEeBIVd3T1m9jEBbfbKeLaN+fbtuPApuG9t/Yaqerd6pqT1VNV9X01NTUGK1Lkl7IyOFQVd8Ankzyo610OfAQcAA4ecfRTuD2tnwAeGe7a+ky4Ll2+ulO4IokF7QL0Ve0miRpQtaNuf8vA59Icj7wGPBuBoFza5JdwNeBt7exdwBvAWaB59tYqupYkvcD97Zx76uqY2P2JUkaw1jhUFUPANPzbLp8nrEFXHOax9kL7B2nF0nS8vEd0pKkjuEgSeoYDpKkjuEgSeoYDpKkjuEgSeoYDpKkjuEgSeqM+w5pSavI5ms/PfK+T9zw1mXsRGc7jxwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSZ2xwyHJeUnuT/KHbX1LknuSzCb5VJLzW/1FbX22bd889BjXtfojSa4ctydJ0niW48jhV4GHh9ZvBG6qqlcBx4Fdrb4LON7qN7VxJLkY2AG8BtgGfCTJecvQlyRpRGOFQ5KNwFuBj7X1AG8CbmtD9gFXteXtbZ22/fI2fjuwv6q+U1WPA7PApeP0JUkaz7hHDv8F+A3gr9v6K4Bnq+pEWz8CbGjLG4AnAdr259r4/1+fZx9J0gSMHA5Jfgp4uqruW8Z+FnrO3UlmkszMzc2t1NNK0jlnnCOHNwJvS/IEsJ/B6aQPAuuTrGtjNgJH2/JRYBNA2/4y4Jnh+jz7fJ+q2lNV01U1PTU1NUbrkqQXMnI4VNV1VbWxqjYzuKD8uar6OeBu4Oo2bCdwe1s+0NZp2z9XVdXqO9rdTFuArcAXR+1LkjS+dQsPWbL3APuT/CZwP3Bzq98M/F6SWeAYg0Chqg4nuRV4CDgBXFNV3zsDfUmSFmlZwqGqPg98vi0/xjx3G1XVt4GfPc3+HwA+sBy9SJLG5zukJUkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEmdkcMhyaYkdyd5KMnhJL/a6i9PcjDJo+37Ba2eJB9KMpvkwSSXDD3Wzjb+0SQ7x5+WJGkc4xw5nAB+vaouBi4DrklyMXAtcFdVbQXuausAbwa2tq/dwEdhECbA9cAbgEuB608GiiRpMkYOh6p6qqq+1Jb/D/AwsAHYDuxrw/YBV7Xl7cAtNXAIWJ/klcCVwMGqOlZVx4GDwLZR+5IkjW9Zrjkk2Qy8HrgHuKiqnmqbvgFc1JY3AE8O7Xak1U5Xn+95dieZSTIzNze3HK1LkuYxdjgk+dvA7wP/pqq+NbytqgqocZ9j6PH2VNV0VU1PTU0t18NKkk4xVjgk+VsMguETVfUHrfzNdrqI9v3pVj8KbBrafWOrna4uSZqQce5WCnAz8HBV/fbQpgPAyTuOdgK3D9Xf2e5augx4rp1+uhO4IskF7UL0Fa0mSZqQdWPs+0bgXwBfTvJAq/074Abg1iS7gK8Db2/b7gDeAswCzwPvBqiqY0neD9zbxr2vqo6N0ZckaUwjh0NV/RGQ02y+fJ7xBVxzmsfaC+wdtRdJ0vLyHdKSpI7hIEnqGA6SpI7hIEnqjHO3kiQBsPnaT4+87xM3vHUZO9Fy8chBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktTxfQ6SzlrjvL8CfI/FC/HIQZLU8chhiXwnqKRzgUcOkqSO4SBJ6hgOkqSO4SBJ6pyTF6THvf1Nkta6czIcpDPJO9q0FnhaSZLUMRwkSR3DQZLU8ZqD1izP/Uuj88hBktRZNeGQZFuSR5LMJrl20v1I0rlsVZxWSnIe8GHgnwJHgHuTHKiqhybb2fLy/RWSzhar5cjhUmC2qh6rqu8C+4HtE+5Jks5Zq+LIAdgAPDm0fgR4w4R6WZO8OHt2mOTR5aSe+2yd81r/vVgt4bAoSXYDu9vq/03yyBJ2vxD4i+XvasWt+Dxy4xl52FX9eixhzqt6Hku0VuayIvM4Q78Xw87UPP7eYgatlnA4CmwaWt/Yat+nqvYAe0Z5giQzVTU9Wnurh/NYXdbKPGDtzMV5LI/Vcs3hXmBrki1Jzgd2AAcm3JMknbNWxZFDVZ1I8kvAncB5wN6qOjzhtiTpnLUqwgGgqu4A7jiDTzHS6ahVyHmsLmtlHrB25uI8lkGqapLPL0lahVbLNQdJ0iqy5sJhoY/hSPLjSb6U5ESSqyfR42IsYh6/luShJA8muSvJom5PW2mLmMe/TPLlJA8k+aMkF0+iz4Us9uNdkvxMkkqyKu+WWcTr8a4kc+31eCDJL0yiz4Us5vVI8vb2O3I4yX9f6R4XaxGvyU1Dr8dXkzy7Io1V1Zr5YnAx+2vA3wfOB/4UuPiUMZuB1wG3AFdPuucx5vFPgBe35X8FfGrSfY84jx8eWn4b8NlJ9z3KPNq4lwJfAA4B05Pue8TX413A70y612WYx1bgfuCCtv53Jt33OD9bQ+N/mcENO2e8t7V25LDgx3BU1RNV9SDw15NocJEWM4+7q+r5tnqIwXtDVpvFzONbQ6svAVbjRbDFfrzL+4EbgW+vZHNLsFY+pmYx8/hF4MNVdRygqp5e4R4Xa6mvyTuAT65EY2stHOb7GI4NE+plHEudxy7gM2e0o9Esah5JrknyNeA/Ab+yQr0txYLzSHIJsKmqVvOnKy725+pn2unK25Jsmmf7pC1mHj8C/EiSP05yKMm2FetuaRb9u95OHW8BPrcCfa25cDjnJPnnwDTwW5PuZVRV9eGq+gfAe4B/P+l+lirJDwC/Dfz6pHtZBv8T2FxVrwMOAvsm3M+o1jE4tfQTDP7a/q9J1k+0o/HtAG6rqu+txJOttXBY1MdwnAUWNY8kPwm8F3hbVX1nhXpbiqW+HvuBq85oR6NZaB4vBV4LfD7JE8BlwIFVeFF6wdejqp4Z+ln6GPBjK9TbUizm5+oIcKCq/qqqHge+yiAsVpul/I7sYIVOKQFr7oL0OuAxBodeJy/uvOY0Yz/O6r0gveA8gNczuJC1ddL9jjmPrUPL/wyYmXTf4/xctfGfZ3VekF7M6/HKoeWfBg5Nuu8R57EN2NeWL2Rw6uYVk+591J8t4NXAE7T3pq1Ib5P+j3MG/mO/hcFfCV8D3ttq72Pw1zXAP2LwV8VfAs8Ahyfd84jz+N/AN4EH2teBSfc84jw+CBxuc7j7hf6nu5rnccrYVRkOi3w9/mN7Pf60vR6vnnTPI84jDE71PQR8Gdgx6Z7H+dkC/gNww0r25TukJUmdtXbNQZK0DAwHSVLHcJAkdQwHSVLHcJAkdQwHSVLHcJAkdQwHSVLn/wFdDKzD7PDa/QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10d99f810>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.hist(train_marginals, bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view the learned accuracy parameters, and other statistics about the LFs learned by the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.534134</td>\n",
       "      <td>0.6665</td>\n",
       "      <td>0.541630</td>\n",
       "      <td>0.360980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.532118</td>\n",
       "      <td>0.6694</td>\n",
       "      <td>0.539711</td>\n",
       "      <td>0.358431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.565538</td>\n",
       "      <td>0.6645</td>\n",
       "      <td>0.574173</td>\n",
       "      <td>0.380980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.565055</td>\n",
       "      <td>0.6702</td>\n",
       "      <td>0.574157</td>\n",
       "      <td>0.377255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.543329</td>\n",
       "      <td>0.6716</td>\n",
       "      <td>0.553972</td>\n",
       "      <td>0.358235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.796530</td>\n",
       "      <td>0.7146</td>\n",
       "      <td>0.804610</td>\n",
       "      <td>0.574902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.526747</td>\n",
       "      <td>0.6711</td>\n",
       "      <td>0.535660</td>\n",
       "      <td>0.343137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.577701</td>\n",
       "      <td>0.6673</td>\n",
       "      <td>0.588448</td>\n",
       "      <td>0.383529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.568233</td>\n",
       "      <td>0.6661</td>\n",
       "      <td>0.572989</td>\n",
       "      <td>0.370196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.540609</td>\n",
       "      <td>0.6698</td>\n",
       "      <td>0.551551</td>\n",
       "      <td>0.366078</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accuracy  Coverage  Precision    Recall\n",
       "0  0.534134    0.6665   0.541630  0.360980\n",
       "1  0.532118    0.6694   0.539711  0.358431\n",
       "2  0.565538    0.6645   0.574173  0.380980\n",
       "3  0.565055    0.6702   0.574157  0.377255\n",
       "4  0.543329    0.6716   0.553972  0.358235\n",
       "5  0.796530    0.7146   0.804610  0.574902\n",
       "6  0.526747    0.6711   0.535660  0.343137\n",
       "7  0.577701    0.6673   0.588448  0.383529\n",
       "8  0.568233    0.6661   0.572989  0.370196\n",
       "9  0.540609    0.6698   0.551551  0.366078"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.learned_lf_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model to Iterate on Labeling Functions\n",
    "\n",
    "Now that we have learned the generative model, we can stop here and use this to potentially debug and/or improve our labeling function set.  First, we apply the LFs to our development set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In labeling, found 2814 cand IDs for split 1!\n",
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2814/2814 [00:17<00:00, 160.33it/s]\n"
     ]
    }
   ],
   "source": [
    "L_dev = labeler.apply_existing(split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we get the score of the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 0.558\n",
      "Neg. class accuracy: 0.928\n",
      "Precision            0.361\n",
      "Recall               0.558\n",
      "F1                   0.438\n",
      "----------------------------------------\n",
      "TP: 106 | FP: 188 | TN: 2436 | FN: 84\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = gen_model.error_analysis(session, L_dev, L_gold_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Generative Model Performance\n",
    "\n",
    "At this point, we should be getting an F1 score of around 0.4 to 0.5 on the development set, which is pretty good!  However, we should be very careful in interpreting this. Since we developed our labeling functions using this development set as a guide, and our generative model is composed of these labeling functions, we expect it to score very well here!  \n",
    "\n",
    "In fact, it is probably somewhat _overfit_ to this set. However this is fine, since in the next tutorial, we'll train a more powerful end extraction model which will generalize beyond the development set, and which we will evaluate on a _blind_ test set (i.e. one we never looked at during development)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doing Some Error Analysis\n",
    "\n",
    "At this point, we might want to look at some examples in one of the error buckets. For example, one of the false negatives that we did not correctly label as true mentions.  To do this, we can again just use the `Viewer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(fn, session)\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc858ac25d0440b3aff4923684f50f22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>SentenceNgramViewer</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "SentenceNgramViewer(cids=[[[12], [56], [74]], [[45], [49], [18]], [[75, 76], [60], [54, 55]], [[48], [27], [8]], [[0, 25], [9], [59]], [[71], [2], [80]], [[28, 30, 31, 82], [38], [50]], [[73], [69, 77], [44]], [[39], [33], [62]], [[63], [52], [17]], [[53], [32], [1, 67]], [[61], [72], [36]], [[40], [20], [35]], [[3], [5, 24], [81, 83]], [[23, 78], [19], [14]], [[26], [4], [7]], [[13], [29], [47]], [[42], [37, 43], [51]], [[22], [58], [15]], [[64, 65], [79], [6]], [[16], [34], [11]], [[21], [66], [10]], [[46], [70], [41]], [[57], [68]]], html=u'<head>\\n<style>\\nspan.candidate {\\n    background-color: rgba(255,255,0,0.3);\\n}\\n\\nspan.candidate-h {\\n    background-color: rgba(255,255,0,1.0);\\n}\\n\\nspan.true-candidate {\\n    background-color: rgba(92,184,92,0.3);\\n}\\n\\nspan.true-candidate-h {\\n    background-color: rgba(92,184,92,1.0);\\n}\\n\\nspan.false-candidate {\\n    background-color: rgba(169,68,66,0.3);\\n}\\n\\nspan.false-candidate-h {\\n    background-color: rgba(169,68,66,1.0);\\n}\\n\\nspan.highlighted {\\n    font-weight: bolder;\\n    box-shadow: 1px 1px 1px grey;\\n}\\n\\nspan.left-candidate {\\n    color: red;\\n}\\n\\nspan.right-candidate {\\n    color: blue;\\n}\\n\\nspan.gold-annotation {\\n    color: red;\\n}\\n\\nspan.highlighted-gold-annotation {\\n    background-color: rgba(217,83,79,0.5);\\n}\\n\\n.panel-heading {\\n    position: relative;\\n}\\n\\n.panel-footer {\\n    position: relative;\\n}\\n\\n.panel-left-justify {\\n    position: absolute;\\n    right: 15px;\\n    top: 20px;\\n}\\n\\n.viewer-page {\\n    display: none;\\n}\\n\\n.help-text {\\n    color: gray;\\n    font-weight: 300;\\n}\\n</style>\\n</head>\\n\\n<div id=\"viewer\" class=\"viewer-pane panel panel-default\" tabindex=\"0\">\\n    <div class=\"panel-heading\">\\n        <h3 class=\"panel-title\"><i class=\"fa fa-eye fa-lg\" aria-hidden=\"true\"></i> Viewer</h3>\\n        <div id=\"candidate-caption\" class=\"panel-left-justify\"></div>\\n    </div>\\n    <div class=\"panel-body\" style=\"height:225px\">\\n<div class=\"viewer-page\" id=\"viewer-page-0\" style=\"display: block;\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"3586\"><span class=\"candidate 12 12-0\">Ward</span><span class=\"\"> was engaged to morning show producer </span><span class=\"candidate 12 12-1\">Melissa Ott</span><span class=\"\">, who had recently gotten a job in Charlotte, North Carolina.\\xa0   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"3810\"><span class=\"\">Just over a week ago, Hanks officiated the wedding of \"Girls\" actress </span><span class=\"candidate 56 56-0\">Allison Williams</span><span class=\"\"> and her beau, CollegeHumor founder </span><span class=\"candidate 56 56-1\">Ricky Van Veen</span><span class=\"\">.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"5196\"><span class=\"\">The Rowan County courthouse was empty of demonstrators this weekend following rallies there last week      Rowan County deputy clerk Brian Mason (left) shakes hands with </span><span class=\"candidate 74 74-0\">James Yates</span><span class=\"\">, and his partner </span><span class=\"candidate 74 74-1\">William Smith Jr</span><span class=\"\"> after issuing their their marriage license at the Rowan County Judicial Center in Morehead, Kentucky\\xa0   Last fall, he decided to stop performing weddings altogether, aside from one in March that had long been scheduled, Korten said.      </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-1\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"5536\"><span class=\"\">So it\\'s without surprise that </span><span class=\"candidate 45 45-0\">Hugh Grant</span><span class=\"\"> and </span><span class=\"candidate 45 45-1\">Elizabeth Hurley</span><span class=\"\"> almost appeared like an old married couple as they stepped out on a leafy London street together on Thursday.\\xa0   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"6044\"><span class=\"\">She celebrated the marriage of her best friend </span><span class=\"candidate 49 49-0\">Jennifer Aniston</span><span class=\"\"> and </span><span class=\"candidate 49 49-1\">Justin Theroux</span><span class=\"\"> last month, when she was reportedly the Maid Of Honor.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"9225\"><span class=\"candidate 18 18-0\">Alexa PenaVega</span><span class=\"\"> and </span><span class=\"candidate 18 18-1\">Carlos PenaVega</span><span class=\"\"> will be the first married couple to compete on a season of \\'Dancing with the Stars.\\'</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-2\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"9563\"><span class=\"\">Work in progress\\':The Russian-born star broke her silence on the former Strictly partners\\' relationship at the weekend   To this day, she maintains she had no involvement in </span><span class=\"candidate 75 75-0\">Ben</span><span class=\"\"> and </span><span class=\"candidate 76 76-0 75 75-1\">Abby</span><span class=\"\">\\'s marriage breakdown, adding: \\'</span><span class=\"candidate 76 76-1\">Ben</span><span class=\"\">\\'s said it many times and I\\'ll say it again - I wasn\\'t involved in the breaking up of his marriage.\\'   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"10111\"><span class=\"\">Ryan this week took a leave of absence from his Nine Network role after Hi-5 star </span><span class=\"candidate 60 60-0\">Lauren Brant\\'s</span><span class=\"\"> fiance </span><span class=\"candidate 60 60-1\">Warren Riley</span><span class=\"\"> went public claiming she had confessed to cheating on him with Ryan.\\xa0   \\xa0 \\xa0      </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"10113\"><span class=\"\">I\\'ll be totally honest with you - I think it\\'s nobody\\'s business but </span><span class=\"candidate 54 55 54-0 55-0\">Ryan</span><span class=\"\">\\'s, </span><span class=\"candidate 55 55-1\">Beau Ryan\\'s</span><span class=\"\"> (Wife), </span><span class=\"candidate 54 54-1\">Lauren Brant</span><span class=\"\"> and her fiance,\\' Johns said.\\xa0   \\'We don\\'t need to hear from Beau Ryan.</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-3\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"12174\"><span class=\"\">A source told Daily Mail Australia a flummoxed </span><span class=\"candidate 48 48-0\">Gabi</span><span class=\"\"> stepped into the graffiti-daubed office, manned by predominantly Chinese staff, and exclaimed: \\'Why am I here... I\\'m not Chinese?\\'\\xa0      I do: Gabi married </span><span class=\"candidate 48 48-1\">Geoffrey Edelsten</span><span class=\"\"> in June at a ramshackle Chinese registry office on a busy thoroughfare just outside the heart of Melbourne\\xa0   \\xa0\\n</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"12312\"><span class=\"\">It\\'s over: </span><span class=\"candidate 27 27-0\">Foster</span><span class=\"\"> and </span><span class=\"candidate 27 27-1\">Robin Wright</span><span class=\"\">, 49, are reported to have split, ending their engagement after more than three years together   \\xa0\\n</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"13110\"><span class=\"candidate 8 8-0\">David</span><span class=\"\"> said </span><span class=\"candidate 8 8-1\">Victoria</span><span class=\"\"> was disappointed to hear that Harper wants to be a footballer   David retired from playing professional football in 2013, and has admitted that looking after his four children has kept him busy and stopped him from missing the sport too much.   \"</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-4\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"14960\"><span class=\"candidate 0 0-0\">Frank</span><span class=\"\"> and </span><span class=\"candidate 25 25-0 0 0-1\">Kathie Lee Gifford\\'s</span><span class=\"\"> son to lead TV Hall of Fame tribute to his father   NEW YORK DAILY NEWS   Tuesday, September 22, 2015, 6:12 PM   There won\\'t be a dry eye in the house at this year\\'s Broadcasting and Cable Hall of Fame induction ceremony.   Confidenti@l is told that Cody Gifford, the son of TV and football great </span><span class=\"candidate 25 25-1\">Frank Gifford</span><span class=\"\">, will accept a posthumous lifetime achievement award on his father\\'s behalf at the Oct. 20 ceremony.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"16733\"><span class=\"candidate 9 9-0\">Printz</span><span class=\"\"> calls Queensland-based Rachel his \\'bestie\\' and surely the food blogger-turned-reality star would tell her best friends first about her plans to get wed to </span><span class=\"candidate 9 9-1\">Sam</span><span class=\"\">   Thrilled with the fact that she is appearing on the hit Network Ten reality dating show, he first took to his social media page to wish her luck.   \\'Wishing my girl, and bestie, @rachelmoore good luck on @thebacheloraustralia this week!!!\\' he wrote.    </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"18162\"><span class=\"\">SiriusXM today announced that world-renowned spiritual and inspirational figures </span><span class=\"candidate 59 59-0\">Joel</span><span class=\"\"> and </span><span class=\"candidate 59 59-1\">Victoria Osteen</span><span class=\"\">, pastors of Lakewood Church, America\\'s largest Christian congregation, will host an exclusive SiriusXM Town Hall broadcast in New York City on October 5.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-5\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"18269\"><span class=\"\">Saudi...  DNA 2 days ago - Front Page Amrita Rai announces marriage to Digvijaya Singh on Facebook *New Delhi:* Journalist </span><span class=\"candidate 71 71-0\">Amrita Rai</span><span class=\"\"> on Sunday announced that she got married to Congress leader </span><span class=\"candidate 71 71-1\">Digvijaya Singh</span><span class=\"\"> as per Hindu rites and desired none of his...  </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"19795\"><span class=\"candidate 2 2-0\">Ryan</span><span class=\"\"> also said a separate firm owned by </span><span class=\"candidate 2 2-1\">Bitter Smith</span><span class=\"\"> and her husband, Paul Smith, does consulting work for the owner of a golf course who is trying to get Arizona Public Service to move a substation to accommodate a redesign of the course, a move opposed by homeowners who would then be nearer to the substation.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"19895\"><span class=\"\">Ana Maria Chavez died hand-in-hand with her husband, Domingo Chavez They both had health problems and were diagnosed with dementia The two met at a dance in 1958 in Texas and married in 1961 They eventually moved to Idaho and worked on farms They have eight children, 30 grandchildren and 32 great-grandchildren\\xa0   </span><span class=\"candidate 80 80-0\">Ana Maria</span><span class=\"\"> and </span><span class=\"candidate 80 80-1\">Domingo Chavez</span><span class=\"\"> met at a  Blue Moon Dance in Pharr, Texas, in 1958.\\xa0   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-6\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"19899\"><span class=\"\">Soul mates: </span><span class=\"candidate 28 30 31 28-0 30-0 31-0\">Domingo</span><span class=\"\"> and </span><span class=\"candidate 31 31-1\">Ana Maria Chavez</span><span class=\"\"> died within hours of each other on Thursday at their son\\'s home in Caldwell, Idaho, after a life of being inseparable from one another      Together until the end: The two died holding hands on a heart-shaped pillow, with </span><span class=\"candidate 28 28-1\">Ana Maria</span><span class=\"\"> passing first and her husband a few hours later\\xa0      Sweethearts: </span><span class=\"candidate 82 82-0 30 30-1\">Ana Maria</span><span class=\"\"> and </span><span class=\"candidate 82 82-1\">Domingo Chavez</span><span class=\"\"> met at a Blue Moon Dance in Pharr, Texas, in 1958.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"20551\"><span class=\"\">People reported </span><span class=\"candidate 38 38-0\">Williams</span><span class=\"\"> and </span><span class=\"candidate 38 38-1\">Ven Veen</span><span class=\"\"> tied the knot Saturday at Brush Creek Ranch in Saratoga, Wyoming, in front of about 200 guests.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"21064\"><span class=\"\">Intensely private when it comes to her personal life, </span><span class=\"candidate 50 50-0\">Ms Bishop</span><span class=\"\">, who was married to Perth property developer </span><span class=\"candidate 50 50-1\">Neil Gillon</span><span class=\"\"> from 1983 to 1988 revealed Mr Panton is \\'supportive\\'.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-7\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"21432\"><span class=\"\">Making a style statement: Nick, 35, looked keen to impress his 21-year-old rumoured girlfriend in a jazzy ensemble      Twinkle toes: The former husband of </span><span class=\"candidate 73 73-0\">Mariah Carey</span><span class=\"\"> tied his mis-matched look together with a pair of glittering loafers     Perfect gentleman: The father-of-two allowed his supposed love interest to lead the way out of Mr Chow following their dinner   </span><span class=\"candidate 73 73-1\">Nick</span><span class=\"\"> appeared to be in high spirits as the rumoured couple made their way from the fine Chinese eatery to their getaway car.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"22901\"><span class=\"\">The competition was run during the live episode, with backstage coverage presented by Stephen Mulhern - who wrote </span><span class=\"candidate 69 69-0\">Adam</span><span class=\"\">\\'s message on the blackboard\\xa0   Fellow Twitter users were quick to congratulate </span><span class=\"candidate 77 77-0\">Adam</span><span class=\"\"> and </span><span class=\"candidate 69 77 69-1 77-1\">Katie</span><span class=\"\">, with many hailing it \\u2018amazing and cute\\u2019 and the \\u2018best proposal ever\\u2019.\\xa0   One wrote: \\'Congratulations!</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"23743\"><span class=\"\">As we meet Scottish couple </span><span class=\"candidate 44 44-0\">John</span><span class=\"\"> and </span><span class=\"candidate 44 44-1\">Karen</span><span class=\"\">, they are living in a hole in the middle of the French countryside.</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-8\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"24667\"><span class=\"\">Earlier in the day, </span><span class=\"candidate 39 39-0\">Ricki-Lee</span><span class=\"\"> went for a dip in the warm waters of her exclusive pool and made sure hubby </span><span class=\"candidate 39 39-1\">Richard</span><span class=\"\"> captured the moment.   \\u2018</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"24674\"><span class=\"\">So in love\\': </span><span class=\"candidate 33 33-0\">Ricki-Lee</span><span class=\"\"> and </span><span class=\"candidate 33 33-1\">Richard</span><span class=\"\"> were all smiles in a selfie on their extended honeymoon   On Monday, the former Australian Idol contestant shared a selfie of the couple on the Greek Island of Santorini, with the pair grinning from ear to ear.   \\'</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"24801\"><span class=\"\">PHOTOS: Stars and their look-alike kids   \\u201cHe doesn\\u2019t like to clean, but he does like to sing,\\u201d </span><span class=\"candidate 62 62-0\">Mowry</span><span class=\"\">, who married </span><span class=\"candidate 62 62-1\">Cory Hardrict</span><span class=\"\"> in 2008, tells Us Weekly.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-9\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"28086\"><span class=\"\">Source: InsideEdition - 1 week ago  This Couple Got Married and Had a Baby All in One Hour\\xa0 01:38 </span><span class=\"candidate 63 63-0\">Stephanie Tallentt</span><span class=\"\"> and </span><span class=\"candidate 63 63-1\">Jason Neice</span><span class=\"\"> had planned to get married before their daughter was born, but the baby refused to wait.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"28703\"><span class=\"\">Other acts hoping to make the grade include married couple </span><span class=\"candidate 52 52-0\">Chris Kennedy</span><span class=\"\"> and </span><span class=\"candidate 52 52-1\">Gabriel Cabrera</span><span class=\"\"> from Milton Keynes, with their version of Ellie Goulding\\'s Love Me Like You Do, and Ghanaian duo Menn On Point who treat the audience to original composition Turn It Up.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"32394\"><span class=\"candidate 17 17-0\">Ceawlin Thynn</span><span class=\"\">, 41, said the row resulted in him banning his mother, the Marchioness of Bath, from attending his wedding to </span><span class=\"candidate 17 17-1\">Emma McQuiston</span><span class=\"\">, 29, and from seeing the couple\\'s 11-month-old son.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-10\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"32409\"><span class=\"\">Relations between the men were so bad that Lord Bath also missed </span><span class=\"candidate 53 53-0\">Ceawlin</span><span class=\"\"> and </span><span class=\"candidate 53 53-1\">Emma</span><span class=\"\">\\'s wedding in 2013.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"33352\"><span class=\"\">Coordinated: </span><span class=\"candidate 32 32-0\">Sam Burgess</span><span class=\"\"> and fianc\\xe9 </span><span class=\"candidate 32 32-1\">Phoebe Hooke</span><span class=\"\">, who have now moved to England, looked the picture of elegance in matching black and white ensembles      Red rose: Ex Bachelor star Anna Henrich wore an off-the-shoulder red gown and a bold red lip to the annual sporting awards event   The former Summer Bay stunner\\'s frock featured a low dip at the top to show her toned stomach, and a high split at the bottom to show her lean legs.\\xa0\\xa0   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"34320\"><span class=\"candidate 1 1-0\">Josh Brolin</span><span class=\"\"> showed off his abs as he went to wipe off his brow following a gym session on Sunday with\\xa0fianc\\xe9e </span><span class=\"candidate 67 67-0 1 1-1\">Kathryn Boyd</span><span class=\"\">   </span><span class=\"candidate 67 67-1\">Josh</span><span class=\"\"> sported a red tank top, which he could be seen pulling up to wipe the sweat off of his face, which revealed the actor\\'s toned abs.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-11\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"34526\"><span class=\"\">And the number of homes being built does not match the growth in our population.\\u2019   </span><span class=\"candidate 61 61-0\">Pam</span><span class=\"\"> and </span><span class=\"candidate 61 61-1\">John</span><span class=\"\"> bought their first home, a new three-bedroom terrace house in Hampshire for \\xa38,700 in 1972, before they married, in their mid-20s.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"34583\"><span class=\"\">It was never replaced, and when I moved out they tried to charge me for the damage by deducting money from my bond.\\u2019   Lucy\\u2019s parents, </span><span class=\"candidate 72 72-0\">Martin</span><span class=\"\">, 52, who owns an engineering firm, and </span><span class=\"candidate 72 72-1\">Julie</span><span class=\"\">, 48, who has a dance school, live in a four-bedroom bungalow, in Nottinghamshire.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"34959\"><span class=\"\">Ex wives club: </span><span class=\"candidate 36 36-0\">Nicole</span><span class=\"\"> was married to </span><span class=\"candidate 36 36-1\">Tom</span><span class=\"\"> from 1990 to 2001 and Katie was married to the Jerry Maguire star from 2006 to 2\\n</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-12\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"35163\"><span class=\"\">(bit.ly/1PI0liP   ) </span><span class=\"candidate 40 40-0\">  Mayer</span><span class=\"\">, who has been married to start-up investor </span><span class=\"candidate 40 40-1\">Zachary Bogue</span><span class=\"\"> since 2009, was an executive at Google Inc before taking the helm at Yahoo in 2012.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"35189\"><span class=\"candidate 20 20-0\">Usher</span><span class=\"\"> has tied the knot with his longtime girlfriend </span><span class=\"candidate 20 20-1\">Grace Miguel</span><span class=\"\">.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"35192\"><span class=\"\">Off the market:</span><span class=\"candidate 35 35-0\">\\xa0Usher</span><span class=\"\"> has tied the knot with his longtime girlfriend </span><span class=\"candidate 35 35-1\">Grace Miguel</span><span class=\"\">, according to UsWeekly   Grace is Usher\\'s manager.</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-13\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"35198\"><span class=\"candidate 3 3-0\">Usher</span><span class=\"\"> was previously married to </span><span class=\"candidate 3 3-1\">Tameka Foster</span><span class=\"\"> from 2007 to 2009.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"36376\"><span class=\"candidate 5 5-0\">Mark</span><span class=\"\"> and </span><span class=\"candidate 24 24-0 5 5-1\">Debby Constantino</span><span class=\"\"> starred in Travel Channel\\'s Ghost Adventures They pair were both found dead in their daughter\\'s Nevada apartment Police say he fired at officers when they attempted to approach building A police source has said </span><span class=\"candidate 24 24-1\">Mr Constantino</span><span class=\"\"> killed her then shot himself</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"36380\"><span class=\"\">Mark Constantino shot his wife Debby (pictured together) and her boyfriend amid a tense standoff with police, it has been claimed    The couple featured in recent years on the Travel Channel\\'s series, including episodes shot at the Mustang Ranch brothel east of Reno, the Goldfield Hotel and the Market Street Cinema in San Francisco     </span><span class=\"candidate 81 81-0\">Mark</span><span class=\"\"> and </span><span class=\"candidate 83 83-0 81 81-1\">Debby Constantino</span><span class=\"\"> (pictured left and right) had a long history of domestic violence, a police source said   Police today told the New York Post Mr </span><span class=\"candidate 83 83-1\">Constantino</span><span class=\"\">\\xa0had killed the pair of them before turning the gun on himself.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-14\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"36509\"><span class=\"\">Tragic: </span><span class=\"candidate 23 23-0\">Dennis Rodeman</span><span class=\"\">, 35, was killed in a hit-and-run during a charity fundraiser on September 9      Bagpipers perform outside during the funeral service in East Lansing, Michigan, on Wednesday      </span><span class=\"candidate 78 78-0\">Rodeman</span><span class=\"\">\\'s uniform was laid in front of his coffin, covered in the Star Spangled Banner, during the ceremony     Rodeman, 35, was married in June and his widow, </span><span class=\"candidate 23 78 23-1 78-1\">Kate</span><span class=\"\"> (together, left), is pregnant with their first child.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"36514\"><span class=\"candidate 19 19-0\">Rodeman</span><span class=\"\">, 35, was married in June and his widow, </span><span class=\"candidate 19 19-1\">Kate</span><span class=\"\">, is pregnant with their first child.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"39451\"><span class=\"candidate 14 14-0\">Colbert</span><span class=\"\">\\'s wife and children (front row) were on hand to watch his performance, as well as his brother (second row)      The show ended with a performance by the show\\'s band, Jon Batiste and Stay Human, which </span><span class=\"candidate 14 14-1\">Colbert</span><span class=\"\"> briefly joined      The marquee above the Ed Sullivan Theater in New York has replaced former host David Letterman\\'s name with Colbert\\'s as he begins his run\\n</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-15\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"39630\"><span class=\"\">On Sunday, </span><span class=\"candidate 26 26-0\">Anita Lesko</span><span class=\"\"> and </span><span class=\"candidate 26 26-1\">Abraham Talmage Nielsen</span><span class=\"\"> got married surrounded by 185 of their friends, family and supporters.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"40535\"><span class=\"candidate 4 4-0\">Jacob</span><span class=\"\"> and </span><span class=\"candidate 4 4-1\">Darchel Mohler</span><span class=\"\"> (NV), whose 13-year-old daughter, Brooklynn Mae Mohler , was at a friend\\'s home when she was unintentionally shot and killed with an unsecured gun on June 4, 2013 .   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"40545\"><span class=\"candidate 7 7-0\">Thomas</span><span class=\"\"> and </span><span class=\"candidate 7 7-1\">Caren Teves</span><span class=\"\"> (AZ), whose 24 year-old Alex Teves was killed in the Aurora movie theater shooting in July 2012 .</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-16\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"40867\"><span class=\"candidate 13 13-0\">Kim</span><span class=\"\"> and </span><span class=\"candidate 13 13-1\">Kanye</span><span class=\"\"> tied the knot in 2014 and welcomed their first daughter, North West, in June 2013.      </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"40868\"><span class=\"\">Plump pout: </span><span class=\"candidate 29 29-0\">Kim</span><span class=\"\"> has been crediting her fuller lips to her pregnancy      Supportive: Kim joined her beau </span><span class=\"candidate 29 29-1\">Kanye West</span><span class=\"\"> (pictured) at the MTV Video Music Awards in Los Angeles Sunday   \\xa0\\n</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"41211\"><span class=\"\">Exclusive   07 SEPTEMBER 2015   </span><span class=\"candidate 47 47-0\">Steven</span><span class=\"\"> and </span><span class=\"candidate 47 47-1\">Alex Gerrard</span><span class=\"\"> have opened the doors to their fabulous Los Angeles \\u2018home from home\\u2019 in an exclusive interview and photoshoot with HELLO! magazine.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-17\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"44637\"><span class=\"\">The $2 million gift from </span><span class=\"candidate 42 42-0\">Jill</span><span class=\"\"> and </span><span class=\"candidate 42 42-1\">Lee</span><span class=\"\">        Kort will enable CHLA neurosurgeons to continue to provide optimal        outcomes for children with brain and spinal cord tumors \\u2013 by addressing        the underlying cause and genetic profile of each tumor, and through        accurate, non-invasive diagnosis and more effective personalized        treatments.    </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"45561\"><span class=\"\">It\\'s a boy   At </span><span class=\"candidate 37 37-0\">Andre</span><span class=\"\"> and </span><span class=\"candidate 43 43-0 37 37-1\">Rhonda</span><span class=\"\">\\'s apartment, </span><span class=\"candidate 43 43-1\">Dre</span><span class=\"\"> wakes up in a cold sweat from a nightmare involving a flashback to when he and his very pregnant wife buried Vernon.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"47994\"><span class=\"\">The 43-year-old actress and the 69-year-old actor play </span><span class=\"candidate 51 51-0\">Gloria</span><span class=\"\"> and </span><span class=\"candidate 51 51-1\">Jay Pritchett</span><span class=\"\">, a happily married May-December couple in the TV comedy, Modern Family.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-18\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"48429\"><span class=\"\">Widow </span><span class=\"candidate 22 22-0\">Kath Rathband</span><span class=\"\">, who cared for the hero police officer after he was shot and blinded by killer Raoul Moat in Newcastle in July 2010, will marry prison guard </span><span class=\"candidate 22 22-1\">John McGee</span><span class=\"\"> next\\u2026\\n</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"49097\"><span class=\"\">He added that one of the most moving moments for him came when he visited </span><span class=\"candidate 58 58-0\">Hall</span><span class=\"\">\\'s widow </span><span class=\"candidate 58 58-1\">Jan</span><span class=\"\"> in New Zealand and heard the tapes of the conversations she\\'d had with her husband, relayed by telephone and radio from the expedition\\'s base camp, shortly before he froze to death near the summit.    \"</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"49605\"><span class=\"candidate 15 15-0\">Mr Asher</span><span class=\"\">, who described the sham marriage as \\'good for two and half years,\\' has now separated from </span><span class=\"candidate 15 15-1\">Jenkinson</span><span class=\"\"> and the wedding was declared void by Lincoln Family Court on 12 June 2014, the court was told.    </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-19\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"49614\"><span class=\"\">A judge spared her jail   When interviewed by police with a solicitor present in March this year, </span><span class=\"candidate 64 65 64-0 65-0\">Jenkinson</span><span class=\"\"> said she was still married to Mr </span><span class=\"candidate 64 64-1\">Ulatowski</span><span class=\"\"> when she married </span><span class=\"candidate 65 65-1\">Mr Asher</span><span class=\"\"> in 2010.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"52457\"><span class=\"\">And in the months since, the late star\\'s husband Thomas Cohen has sought comfort with her close-knit family   The 25-year-old is currently with the Geldof clan in the south of France, where Peaches\\u2019 father </span><span class=\"candidate 79 79-0\">Sir Bob</span><span class=\"\"> married his long-term partner </span><span class=\"candidate 79 79-1\">Jeanne Marine</span><span class=\"\"> last week.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"52480\"><span class=\"candidate 6 6-0\">Sir Bob</span><span class=\"\"> and </span><span class=\"candidate 6 6-1\">Jeanne Marine</span><span class=\"\"> have been together for 19 years and married in the resort of Rayol-Canadel-sur-Mer.      </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-20\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"52497\"><span class=\"candidate 16 16-0\">Jessica</span><span class=\"\">, who has been married to film producer </span><span class=\"candidate 16 16-1\">Cash Warren</span><span class=\"\">, 36, since 2008, shares: \\'Being healthy and strong helps me feel confident and beautiful.\\'   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"57357\"><span class=\"\">In one email, </span><span class=\"candidate 34 34-0\">Mrs Clinton</span><span class=\"\"> responded to Blumenthal: \\'I shared your emails w </span><span class=\"candidate 34 34-1\">Bill</span><span class=\"\"> who thought they were \\'brilliant\\'!</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"58319\"><span class=\"candidate 11 11-0\">Kate</span><span class=\"\"> has spent the bulk of the summer with her children and </span><span class=\"candidate 11 11-1\">Prince William</span><span class=\"\"> at their country home in Norfolk since giving birth to Charlotte, the fourth-in-line to the British throne, on May 2.   </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-21\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"59806\"><span class=\"\">After </span><span class=\"candidate 21 21-0\">Toni Henthorn</span><span class=\"\">, seen here with her and </span><span class=\"candidate 21 21-1\">Harold</span><span class=\"\">\\'s daughter, died in 2012, her relatives realized she had three life insurance policies totaling $4.5million   Henthorn\\'s defense attorney has argued that the two deaths were unfortunate accidents.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"59906\"><span class=\"\">Dunman said Davis\\' office rejected a request on Tuesday for a marriage license from same-sex couple </span><span class=\"candidate 66 66-0\">April Miller</span><span class=\"\"> and </span><span class=\"candidate 66 66-1\">Karen Ann Roberts</span><span class=\"\">, his clients.    </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"63361\"><span class=\"candidate 10 10-0\">Katie</span><span class=\"\">, who is married to design manager </span><span class=\"candidate 10 10-1\">Mark Cross</span><span class=\"\">, 50, added: \\'I know loads of people who work really hard, who\\'d love more children, but don\\u2019t have them because of the accountability of knowing they can\\u2019t afford them.</span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-22\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"64443\"><span class=\"\">Lady Bird Johnson was born as </span><span class=\"candidate 46 46-0\">Claudia Alta Taylor</span><span class=\"\"> and married </span><span class=\"candidate 46 46-1\">Lyndon B. Johnson</span><span class=\"\"> in 1934.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"65152\"><span class=\"\">Preliminary autopsy results show Westerhuis family died of shotgun wounds in south Dakota September 17\\xa0 Deaths of </span><span class=\"candidate 70 70-0\">Nicole Westerhuis</span><span class=\"\"> and children Kailey, Jaeci, Connor and Michael have been ruled homicides </span><span class=\"candidate 70 70-1\">Scott Westerhui\\'s</span><span class=\"\"> manner of death is suspected suicide\\xa0 Firefighters found the family members\\' bodies inside their burned home in Platte, South Dakota\\xa0 \\xa0   The six members of a South Dakota family found dead in the ruins of their fire-ravaged home had been shot in an apparent murder-suicide carried out by the father, authorities revealed Monday.   </span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"65155\"><span class=\"\">The manner of death for </span><span class=\"candidate 41 41-0\">Scott Westerhuis</span><span class=\"\">, </span><span class=\"candidate 41 41-1\">Nicole Westerhuis\\'</span><span class=\"\"> husband, is suspected suicide, it said.   \\xa0\\xa0      </span></li>\\n</ul>\\n</div>\\n\\n<div class=\"viewer-page\" id=\"viewer-page-23\">\\n    <ul class=\"list-group\">\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"65159\"><span class=\"\">Firefighters were called around 5.30am on September 17 to the home of </span><span class=\"candidate 57 57-0\">Scott</span><span class=\"\"> and </span><span class=\"candidate 57 57-1\">Nicole Westerhuis</span><span class=\"\">, about 3 miles south of Platte, about 110 miles west of Sioux Falls.</span></li>\\n\\n<li class=\"list-group-item\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"65162\"><span class=\"\">Charles Mix County Sheriff Randy Thaler said firefighters were called around 5.30am September 17 to the home of </span><span class=\"candidate 68 68-0\">Scott</span><span class=\"\"> and </span><span class=\"candidate 68 68-1\">Nicole Westerhuis</span><span class=\"\">, about 3 miles south of Platte      The family was known for their athleticism and they contributed to the health and well-being of those around them. here</span></li>\\n</ul>\\n</div>\\n</div>\\n    <div class=\"panel-footer\">\\n        <div class=\"btn-group\" role=\"group\" aria-label=\"...\">\\n            \\n            <!-- Previous page --!>\\n            <button id=\"prev-page\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"i\">\\n                <i class=\"fa fa-arrow-up\" aria-hidden=\"true\"></i>\\n            </button>\\n\\n            <!-- Previous context --!>\\n            <button id=\"prev-context\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"i\">\\n                <i class=\"fa fa-step-backward\" aria-hidden=\"true\"></i>\\n            </button>\\n\\n            <!-- Previous candidate --!>\\n            <button id=\"prev-cand\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"j\">\\n                <i class=\"fa fa-arrow-left\" aria-hidden=\"true\"></i>\\n            </button>\\n\\n            <!-- Next candidate --!>\\n            <button id=\"next-cand\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"l\">\\n                <i class=\"fa fa-arrow-right\" aria-hidden=\"true\"></i>\\n            </button>\\n\\n            <!-- Next context --!>\\n            <button id=\"next-context\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"k\">\\n                <i class=\"fa fa-step-forward\" aria-hidden=\"true\"></i>\\n            </button>\\n\\n            <!-- Next page --!>\\n            <button id=\"next-page\" type=\"button\" class=\"btn btn-default\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"k\">\\n                <i class=\"fa fa-arrow-down\" aria-hidden=\"true\"></i>\\n            </button>\\n        </div>\\n        <div class=\"btn-group\" role=\"group\" aria-label=\"...\">\\n            <button id=\"label-true\" type=\"button\" class=\"btn btn-success\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"t\">\\n                <i class=\"fa fa-check\" aria-hidden=\"true\"></i>\\n            </button>\\n            <button id=\"label-false\" type=\"button\" class=\"btn btn-danger\" data-toggle=\"tooltip\" data-placement=\"top\" title=\"f\">\\n                <i class=\"fa fa-times\" aria-hidden=\"true\"></i>\\n            </button>\\n        </div>\\n        &nbsp;&nbsp;&nbsp;\\n        <span class=\"help-text\">Viewer must be focused for keyboard control</span>\\n        <div class=\"page-num panel-left-justify\">Page <span id=\"page\">0</span></div>\\n    </div>\\n</div>\\n')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Spouse(Span(\"Frank\", sentence=14960, chars=[0,4], words=[0,0]), Span(\"Kathie Lee Gifford's\", sentence=14960, chars=[10,29], words=[2,5]))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = sv.get_selected() if sv else list(fp.union(fn))[0]\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily see the labels that the LFs gave to this candidate using simple ORM-enabled syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Label (LF_no_spouse_in_sentence = -1)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also now explore some of the additional functionalities of the `lf_stats` method for our dev set LF labels, `L_dev`: we can plug in the gold labels that we have, and the accuracies that our generative model has learned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>TP</th>\n",
       "      <th>FP</th>\n",
       "      <th>FN</th>\n",
       "      <th>TN</th>\n",
       "      <th>Empirical Acc.</th>\n",
       "      <th>Learned Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>0</td>\n",
       "      <td>0.001066</td>\n",
       "      <td>0.001066</td>\n",
       "      <td>0.000355</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.540976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>1</td>\n",
       "      <td>0.006397</td>\n",
       "      <td>0.006397</td>\n",
       "      <td>0.003198</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.388889</td>\n",
       "      <td>0.523514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife</th>\n",
       "      <td>2</td>\n",
       "      <td>0.092751</td>\n",
       "      <td>0.062189</td>\n",
       "      <td>0.019545</td>\n",
       "      <td>94</td>\n",
       "      <td>151</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.383673</td>\n",
       "      <td>0.556170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>3</td>\n",
       "      <td>0.069652</td>\n",
       "      <td>0.055082</td>\n",
       "      <td>0.013859</td>\n",
       "      <td>82</td>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.433862</td>\n",
       "      <td>0.557460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_same_last_name</th>\n",
       "      <td>4</td>\n",
       "      <td>0.024165</td>\n",
       "      <td>0.018834</td>\n",
       "      <td>0.013859</td>\n",
       "      <td>19</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.279412</td>\n",
       "      <td>0.538473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>5</td>\n",
       "      <td>0.624023</td>\n",
       "      <td>0.098081</td>\n",
       "      <td>0.015636</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>1630</td>\n",
       "      <td>0.967359</td>\n",
       "      <td>0.793239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_and_married</th>\n",
       "      <td>6</td>\n",
       "      <td>0.002843</td>\n",
       "      <td>0.001777</td>\n",
       "      <td>0.001777</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.551017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>7</td>\n",
       "      <td>0.117982</td>\n",
       "      <td>0.103767</td>\n",
       "      <td>0.029495</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>306</td>\n",
       "      <td>0.953271</td>\n",
       "      <td>0.583724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.074982</td>\n",
       "      <td>0.068941</td>\n",
       "      <td>0.012438</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>203</td>\n",
       "      <td>0.995098</td>\n",
       "      <td>0.575739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_other_relationship</th>\n",
       "      <td>9</td>\n",
       "      <td>0.008884</td>\n",
       "      <td>0.008529</td>\n",
       "      <td>0.002843</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>0.809524</td>\n",
       "      <td>0.541944</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   j  Coverage  Overlaps  Conflicts  TP   FP  \\\n",
       "LF_distant_supervision             0  0.001066  0.001066   0.000355   2    1   \n",
       "LF_distant_supervision_last_names  1  0.006397  0.006397   0.003198   7   11   \n",
       "LF_husband_wife                    2  0.092751  0.062189   0.019545  94  151   \n",
       "LF_husband_wife_left_window        3  0.069652  0.055082   0.013859  82  107   \n",
       "LF_same_last_name                  4  0.024165  0.018834   0.013859  19   49   \n",
       "LF_no_spouse_in_sentence           5  0.624023  0.098081   0.015636   0    0   \n",
       "LF_and_married                     6  0.002843  0.001777   0.001777   1    7   \n",
       "LF_familial_relationship           7  0.117982  0.103767   0.029495   0    0   \n",
       "LF_family_left_window              8  0.074982  0.068941   0.012438   0    0   \n",
       "LF_other_relationship              9  0.008884  0.008529   0.002843   0    0   \n",
       "\n",
       "                                   FN    TN  Empirical Acc.  Learned Acc.  \n",
       "LF_distant_supervision              0     0        0.666667      0.540976  \n",
       "LF_distant_supervision_last_names   0     0        0.388889      0.523514  \n",
       "LF_husband_wife                     0     0        0.383673      0.556170  \n",
       "LF_husband_wife_left_window         0     0        0.433862      0.557460  \n",
       "LF_same_last_name                   0     0        0.279412      0.538473  \n",
       "LF_no_spouse_in_sentence           55  1630        0.967359      0.793239  \n",
       "LF_and_married                      0     0        0.125000      0.551017  \n",
       "LF_familial_relationship           15   306        0.953271      0.583724  \n",
       "LF_family_left_window               1   203        0.995098      0.575739  \n",
       "LF_other_relationship               4    17        0.809524      0.541944  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_dev.lf_stats(session, L_gold_dev, gen_model.learned_lf_stats()['Accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for labeling functions with low coverage, our learned accuracies are closer to our prior of 70% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving our training labels\n",
    "\n",
    "Finally, we'll save the `training_marginals`, which are our **probabilistic training labels**, so that we can use them in the next tutorial to train our end extraction model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 22276 marginals\n",
      "CPU times: user 12.3 s, sys: 116 ms, total: 12.5 s\n",
      "Wall time: 13 s\n"
     ]
    }
   ],
   "source": [
    "from snorkel.annotations import save_marginals\n",
    "%time save_marginals(session, L_train, train_marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part III, we'll use these probabilistic training labels to train a deep neural network."
   ]
  }
 ],
 "metadata": {
  "anaconda-butt": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  },
  "widgets": {
   "state": {
    "5c30a7935aa84f14b282753d32a7c2a8": {
     "views": [
      {
       "cell_index": 49
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}