examples/text-processing/text_preprocessing_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demos some functionality in ConvoKit to preprocess text, and store the results. In particular, it shows examples of:\n",
    "\n",
    "* A `TextProcessor` base class that maps per-utterance attributes to per-utterance outputs;\n",
    "* A `TextParser` class that does dependency parsing;\n",
    "* Selective and decoupled data storage and loading;\n",
    "* Per-utterance calls to a transformer;\n",
    "* Pipelining transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries: loading an existing corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we load a clean version of a corpus. For speed we will use a 200-utterance subset of the tennis corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import convokit\n",
    "from convokit import download, Speaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPTION 1: DOWNLOAD CORPUS \n",
    "# UNCOMMENT THESE LINES TO DOWNLOAD CORPUS\n",
    "# DATA_DIR = '<YOUR DIRECTORY>'\n",
    "# ROOT_DIR = download('tennis-corpus')\n",
    "\n",
    "# OPTION 2: READ PREVIOUSLY-DOWNLOADED CORPUS FROM DISK\n",
    "# UNCOMMENT THIS LINE AND REPLACE WITH THE DIRECTORY WHERE THE TENNIS-CORPUS IS LOCATED\n",
    "# ROOT_DIR = '<YOUR DIRECTORY>'\n",
    "\n",
    "corpus = convokit.Corpus(ROOT_DIR, utterance_end_index=199)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Speakers: 9\n",
      "Number of Utterances: 200\n",
      "Number of Conversations: 100\n"
     ]
    }
   ],
   "source": [
    "corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SET YOUR OWN OUTPUT DIRECTORY HERE. \n",
    "# OUT_DIR = '<YOUR DIRECTORY>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example of an utterance from this corpus (questions asked to tennis players after matches, and the answers they give):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_utt_id = '1681_14.a'\n",
    "utt = corpus.get_utterance(test_utt_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Yeah, but many friends went with me, Japanese guy. So I wasn't -- I wasn't like homesick. But now sometimes I get homesick.\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now, `utt.meta` contains the following fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'is_answer': True, 'is_question': False, 'pair_idx': '1681_14'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The TextProcessor class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of our transformers are per-utterance mappings of one attribute of an utterance to another. To facilitate these calls, we use a `TextProcessor` class that inherits from `Transformer`. \n",
    "\n",
    "`TextProcessor` is initialized with the following arguments:\n",
    "\n",
    "* `proc_fn`: the mapping function. Supports one of two function signatures: `proc_fn(input)` and `proc_fn(input, auxiliary_info)`. \n",
    "* `input_field`: the attribute of the utterance that `proc_fn` will take as input. If set to `None`, will default to reading `utt.text`, as seems to be presently done.\n",
    "* `output_field`: the name of the attribute that the output of `proc_fn` will be written to. \n",
    "* `aux_input`: any auxiliary input that `proc_fn` needs (e.g., a pre-loaded model); passed in as a dict.\n",
    "* `input_filter`: a boolean function of signature `input_filter(utterance, aux_input)`, where `aux_input` is again passed as a dict. If this returns `False` then the particular utterance will be skipped; by default it will always return `True`.\n",
    "\n",
    "Both `input_field` and `output_field` support multiple items -- that is, `proc_fn` could take in multiple attributes of an utterance and output multiple attributes. I'll show how this works in advanced usage, below.\n",
    "\n",
    "\"Attribute\" is a deliberately generic term. `TextProcessor` could produce \"features\" as we may conventionally think of them (e.g., wordcount, politeness strategies). It can also be used to pre-process text, i.e., generate alternate representations of the text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit.text_processing import TextProcessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### simple example: cleaning the text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a simple example, suppose we want to remove hyphens \"`--`\" from the text as a preprocessing step. To use `TextProcessor` to do this for us, we'd define the following as a `proc_fn`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_text(text):\n",
    "    text = text.replace(' -- ', ' ')\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we initialize `prep`, a `TextProcessor` object that will run `preprocess_text` on each utterance.\n",
    "\n",
    "When we call `prep.transform()`, the following will occur:\n",
    "\n",
    "* Because we didn't specify an input field, `prep` will pass `utterance.text` into `preprocess_text`\n",
    "* It will write the output -- the text minus the hyphens -- to a field called `clean_text` that will be stored in the utterance meta and that can be accessed as `utt.meta['clean_text']` or `utt.get_info('clean_text')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "prep = TextProcessor(proc_fn=preprocess_text, output_field='clean_text')\n",
    "corpus = prep.transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And as desired, we now have a new field attached to `utt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Yeah, but many friends went with me, Japanese guy. So I wasn't I wasn't like homesick. But now sometimes I get homesick.\""
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('clean_text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing text with the TextParser class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common utterance-level thing we want to do is parse the text. In practice, in increasing order of (computational) difficulty, this typically entails:\n",
    "\n",
    "* proper tokenizing of words and sentences;\n",
    "* POS-tagging;\n",
    "* dependency-parsing. \n",
    "\n",
    "As such, we provide a `TextParser` class that inherits from `TextProcessor` to do all of this, taking in the following arguments:\n",
    "\n",
    "* `output_field`: defaults to `'parsed'`\n",
    "* `input_field`\n",
    "* `mode`: whether we want to go through all of the above steps (which may be expensive) or stop mid-way through. Supports the following options: `'tokenize'`, `'tag'`, `'parse'` (the default).\n",
    "\n",
    "Under the surface, `TextParser` actually uses two separate models: a `spacy` object that does word tokenization, tagging and parsing _per sentence_, and `nltk`'s sentence tokenizer. The rationale is:\n",
    "\n",
    "* `spacy` doesn't support sentence tokenization without dependency-parsing, and we often want sentence tokenization without having to go through the effort of parsing.\n",
    "* We want to be consistent (as much as possible, given changes to spacy and nltk) in the tokenizations we produce, between runs where we don't want parsing and runs where we do.\n",
    "\n",
    "If we've pre-loaded these models, we can pass them into the constructor too, as:\n",
    "\n",
    "* `spacy_nlp`\n",
    "* `sent_tokenizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit.text_processing import TextParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = TextParser(input_field='clean_text', verbosity=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "050/200 utterances processed\n",
      "100/200 utterances processed\n",
      "150/200 utterances processed\n",
      "200/200 utterances processed\n"
     ]
    }
   ],
   "source": [
    "corpus = parser.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### parse output\n",
    "\n",
    "A parse produced by `TextParser` is serialized in text form. It is a list consisting of sentences, where each sentence is a dict with\n",
    "\n",
    "* `toks`: a list of tokens (i.e., words) in the sentence;\n",
    "* `rt`: the index of the root of the dependency tree (i.e., `sentence['toks'][sentence['rt']` gives the root)\n",
    "\n",
    "Each token, in turn, contains the following:\n",
    "\n",
    "* `tok`: the text of the token;\n",
    "* `tag`: the tag;\n",
    "* `up`: the index of the parent of the token in the dependency tree (no entry for the root);\n",
    "* `down`: the indices of the children of the token;\n",
    "* `dep`: the dependency of the edge between the token and its parent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_parse = utt.retrieve_meta('parsed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'rt': 5,\n",
       " 'toks': [{'tok': 'Yeah', 'tag': 'UH', 'dep': 'intj', 'up': 5, 'dn': []},\n",
       "  {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "  {'tok': 'but', 'tag': 'CC', 'dep': 'cc', 'up': 5, 'dn': []},\n",
       "  {'tok': 'many', 'tag': 'JJ', 'dep': 'amod', 'up': 4, 'dn': []},\n",
       "  {'tok': 'friends', 'tag': 'NNS', 'dep': 'nsubj', 'up': 5, 'dn': [3, 10]},\n",
       "  {'tok': 'went', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 1, 2, 4, 6, 8, 11]},\n",
       "  {'tok': 'with', 'tag': 'IN', 'dep': 'prep', 'up': 5, 'dn': [7]},\n",
       "  {'tok': 'me', 'tag': 'PRP', 'dep': 'pobj', 'up': 6, 'dn': []},\n",
       "  {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "  {'tok': 'Japanese', 'tag': 'JJ', 'dep': 'amod', 'up': 10, 'dn': []},\n",
       "  {'tok': 'guy', 'tag': 'NN', 'dep': 'appos', 'up': 4, 'dn': [9]},\n",
       "  {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 5, 'dn': []}]}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_parse[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we didn't want to go through the trouble of dependency-parsing (which could be expensive) we could initialize `TextParser` with `mode='tag'`, which only POS-tags tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "texttagger = TextParser(output_field='tagged', input_field='clean_text', mode='tag')\n",
    "corpus = texttagger.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'toks': [{'tok': 'Yeah', 'tag': 'UH'},\n",
       "  {'tok': ',', 'tag': ','},\n",
       "  {'tok': 'but', 'tag': 'CC'},\n",
       "  {'tok': 'many', 'tag': 'JJ'},\n",
       "  {'tok': 'friends', 'tag': 'NNS'},\n",
       "  {'tok': 'went', 'tag': 'VBD'},\n",
       "  {'tok': 'with', 'tag': 'IN'},\n",
       "  {'tok': 'me', 'tag': 'PRP'},\n",
       "  {'tok': ',', 'tag': ','},\n",
       "  {'tok': 'Japanese', 'tag': 'JJ'},\n",
       "  {'tok': 'guy', 'tag': 'NN'},\n",
       "  {'tok': '.', 'tag': '.'}]}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('tagged')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing and loading corpora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now computed a bunch of utterance-level attributes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['is_answer', 'is_question', 'pair_idx', 'clean_text', 'parsed', 'tagged']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(utt.meta.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, calling `corpus.dump` will write all of these attributes to disk, within the file that stores utterances; later calling `corpus.load` will load all of these attributes back into a new corpus. For big objects like parses, this incurs a high computational burden (especially if in a later use case you might not even need to look at parses)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid this, `corpus.dump`  takes an optional argument `fields_to_skip`, which is a dict of object type (`'utterance'`, `'conversation'`, `'speaker'`, `'corpus'`) to a list of fields that we do not want to write to disk. \n",
    "\n",
    "The following call will write the corpus to disk, without any of the preprocessing output we generated above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus.dump(os.path.basename(OUT_DIR), base_path=os.path.dirname(OUT_DIR), \n",
    "            fields_to_skip={'utterance': ['parsed','tagged','clean_text']})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For attributes we want to keep around, but that we don't want to read and write to disk in a big batch with all the other corpus data, `corpus.dump_info` will dump fields of a Corpus object into separate files. This takes the following arguments as input:\n",
    "\n",
    "* `obj_type`: which type of Corpus object you're dealing with.\n",
    "* `fields`: a list of the fields to write. \n",
    "* `dir_name`: which directory to write to; by default will write to the directory you read the corpus from.\n",
    "\n",
    "This function will write each field in `fields` to a separate file called `info.<field>.jsonl` where each line of the file is a json-serialized dict: `{\"id\": <ID of object>, \"value\": <object.get_info(field)>}`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus.dump_info('utterance',['parsed','tagged'], dir_name = OUT_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, we now have the following files in the output directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conversations.json  index.json         info.tagged.jsonl  users.json\n",
      "corpus.json         info.parsed.jsonl  speakers.json      utterances.jsonl\n"
     ]
    }
   ],
   "source": [
    "ls $OUT_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/kitchen/convokit_corpora_lf/tennis-corpus/'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now initialize a new corpus by reading from this directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_corpus = convokit.Corpus(OUT_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_utt = new_corpus.get_utterance(test_utt_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that things that we've omitted in the `corpus.dump` call will not be read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KeysView({'is_answer': True, 'is_question': False, 'pair_idx': '1681_14'})"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_utt.meta.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a counterpart to `corpus.dump_info` we can also load auxiliary information on-demand. Here, this call will look for `info.<field>.jsonl` in the directory of `new_corpus` (or an optionally-specified `dir_name`) and attach the value specified in each line of the file to the utterance with the associated id:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_corpus.load_info('utterance',['parsed'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rt': 5,\n",
       "  'toks': [{'tok': 'Yeah', 'tag': 'UH', 'dep': 'intj', 'up': 5, 'dn': []},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "   {'tok': 'but', 'tag': 'CC', 'dep': 'cc', 'up': 5, 'dn': []},\n",
       "   {'tok': 'many', 'tag': 'JJ', 'dep': 'amod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'friends', 'tag': 'NNS', 'dep': 'nsubj', 'up': 5, 'dn': [3, 10]},\n",
       "   {'tok': 'went', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 1, 2, 4, 6, 8, 11]},\n",
       "   {'tok': 'with', 'tag': 'IN', 'dep': 'prep', 'up': 5, 'dn': [7]},\n",
       "   {'tok': 'me', 'tag': 'PRP', 'dep': 'pobj', 'up': 6, 'dn': []},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "   {'tok': 'Japanese', 'tag': 'JJ', 'dep': 'amod', 'up': 10, 'dn': []},\n",
       "   {'tok': 'guy', 'tag': 'NN', 'dep': 'appos', 'up': 4, 'dn': [9]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 5, 'dn': []}]},\n",
       " {'rt': 2,\n",
       "  'toks': [{'tok': 'So', 'tag': 'RB', 'dep': 'advmod', 'up': 2, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 2, 'dn': []},\n",
       "   {'tok': 'was', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 1, 3, 5, 9]},\n",
       "   {'tok': \"n't\", 'tag': 'RB', 'dep': 'neg', 'up': 2, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 5, 'dn': []},\n",
       "   {'tok': 'was', 'tag': 'VBD', 'dep': 'ccomp', 'up': 2, 'dn': [4, 6, 8]},\n",
       "   {'tok': \"n't\", 'tag': 'RB', 'dep': 'neg', 'up': 5, 'dn': []},\n",
       "   {'tok': 'like', 'tag': 'UH', 'dep': 'intj', 'up': 8, 'dn': []},\n",
       "   {'tok': 'homesick', 'tag': 'JJ', 'dep': 'acomp', 'up': 5, 'dn': [7]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 2, 'dn': []}]},\n",
       " {'rt': 4,\n",
       "  'toks': [{'tok': 'But', 'tag': 'CC', 'dep': 'cc', 'up': 4, 'dn': []},\n",
       "   {'tok': 'now', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'sometimes', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 4, 'dn': []},\n",
       "   {'tok': 'get', 'tag': 'VBP', 'dep': 'ROOT', 'dn': [0, 1, 2, 3, 5, 6]},\n",
       "   {'tok': 'homesick', 'tag': 'JJ', 'dep': 'acomp', 'up': 4, 'dn': []},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 4, 'dn': []}]}]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_utt.retrieve_meta('parsed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Per-utterance calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TextProcessor` objects also support calls per-utterance via `TextProcessor.transform_utterance()`. These calls take in raw strings as well as utterances, and will return an utterance:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_str = \"I played -- a tennis match.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Utterance({'obj_type': 'utterance', 'meta': {'clean_text': 'I played a tennis match.'}, 'vectors': [], 'speaker': Speaker({'obj_type': 'speaker', 'meta': {}, 'vectors': [], 'owner': None, 'id': 'speaker'}), 'conversation_id': None, 'reply_to': None, 'timestamp': None, 'text': 'I played -- a tennis match.', 'owner': None, 'id': None})"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prep.transform_utterance(test_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "adhoc_utt = prep.transform_utterance(corpus.random_utterance())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I played a tennis match.'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adhoc_utt.retrieve_meta('clean_text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can string together multiple transformers, and hence `TextProcessors`, into a pipeline, using a `ConvokitPipeline` object. This is analogous to (and in fact inherits from) scikit-learn's `Pipeline` class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit.convokitPipeline import ConvokitPipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, suppose we want to both clean the text and parse it. We can chain the required steps to get there by initializing `ConvokitPipeline` with a list of steps, represented as a tuple of `(<step name>, initialized transformer-like object)`:\n",
    "\n",
    "* `'prep'`, our de-hyphenator\n",
    "* `'parse'`, our parser\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "parse_pipe = ConvokitPipeline([('prep', TextProcessor(preprocess_text, 'clean_text_pipe')),\n",
    "                ('parse', TextParser('parsed_pipe', input_field='clean_text_pipe',\n",
    "                                    verbosity=50))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "050/200 utterances processed\n",
      "100/200 utterances processed\n",
      "150/200 utterances processed\n",
      "200/200 utterances processed\n"
     ]
    }
   ],
   "source": [
    "corpus = parse_pipe.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rt': 5,\n",
       "  'toks': [{'tok': 'Yeah', 'tag': 'UH', 'dep': 'intj', 'up': 5, 'dn': []},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "   {'tok': 'but', 'tag': 'CC', 'dep': 'cc', 'up': 5, 'dn': []},\n",
       "   {'tok': 'many', 'tag': 'JJ', 'dep': 'amod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'friends', 'tag': 'NNS', 'dep': 'nsubj', 'up': 5, 'dn': [3, 10]},\n",
       "   {'tok': 'went', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 1, 2, 4, 6, 8, 11]},\n",
       "   {'tok': 'with', 'tag': 'IN', 'dep': 'prep', 'up': 5, 'dn': [7]},\n",
       "   {'tok': 'me', 'tag': 'PRP', 'dep': 'pobj', 'up': 6, 'dn': []},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 5, 'dn': []},\n",
       "   {'tok': 'Japanese', 'tag': 'JJ', 'dep': 'amod', 'up': 10, 'dn': []},\n",
       "   {'tok': 'guy', 'tag': 'NN', 'dep': 'appos', 'up': 4, 'dn': [9]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 5, 'dn': []}]},\n",
       " {'rt': 2,\n",
       "  'toks': [{'tok': 'So', 'tag': 'RB', 'dep': 'advmod', 'up': 2, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 2, 'dn': []},\n",
       "   {'tok': 'was', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 1, 3, 5, 9]},\n",
       "   {'tok': \"n't\", 'tag': 'RB', 'dep': 'neg', 'up': 2, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 5, 'dn': []},\n",
       "   {'tok': 'was', 'tag': 'VBD', 'dep': 'ccomp', 'up': 2, 'dn': [4, 6, 8]},\n",
       "   {'tok': \"n't\", 'tag': 'RB', 'dep': 'neg', 'up': 5, 'dn': []},\n",
       "   {'tok': 'like', 'tag': 'UH', 'dep': 'intj', 'up': 8, 'dn': []},\n",
       "   {'tok': 'homesick', 'tag': 'JJ', 'dep': 'acomp', 'up': 5, 'dn': [7]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 2, 'dn': []}]},\n",
       " {'rt': 4,\n",
       "  'toks': [{'tok': 'But', 'tag': 'CC', 'dep': 'cc', 'up': 4, 'dn': []},\n",
       "   {'tok': 'now', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'sometimes', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 4, 'dn': []},\n",
       "   {'tok': 'get', 'tag': 'VBP', 'dep': 'ROOT', 'dn': [0, 1, 2, 3, 5, 6]},\n",
       "   {'tok': 'homesick', 'tag': 'JJ', 'dep': 'acomp', 'up': 4, 'dn': []},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 4, 'dn': []}]}]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('parsed_pipe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As promised, the pipeline also works to transform utterances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_utt = parse_pipe.transform_utterance(test_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rt': 1,\n",
       "  'toks': [{'tok': 'I', 'tag': 'PRP', 'dep': 'nsubj', 'up': 1, 'dn': []},\n",
       "   {'tok': 'played', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [0, 4, 5]},\n",
       "   {'tok': 'a', 'tag': 'DT', 'dep': 'det', 'up': 4, 'dn': []},\n",
       "   {'tok': 'tennis', 'tag': 'NN', 'dep': 'compound', 'up': 4, 'dn': []},\n",
       "   {'tok': 'match', 'tag': 'NN', 'dep': 'dobj', 'up': 1, 'dn': [2, 3]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 1, 'dn': []}]}]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_utt.retrieve_meta('parsed_pipe')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some advanced usage: playing around with parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The point of the following is to demonstrate more elaborate calls to `TextProcessor`. As an example, we will count words in an utterance.\n",
    "\n",
    "First, we'll initialize a `TextProcessor` that does wordcounts (i.e., `len(x.split())`) on just the raw text (`utt.text`), writing output to field `wc_raw`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc_raw = TextProcessor(proc_fn=lambda x: len(x.split()), output_field='wc_raw')\n",
    "corpus = wc_raw.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('wc_raw')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we instead wanted to wordcount our preprocessed text, with the hyphens removed, we can specify `input_field='clean_text'` -- as such, the `TextProcessor` will read from `utt.get_info('clean_text')` instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc = TextProcessor(proc_fn=lambda x: len(x.split()), output_field='wc', input_field='clean_text')\n",
    "corpus = wc.transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that we are no longer counting the extra hyphen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('wc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we can count characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "chars = TextProcessor(proc_fn=lambda x: len(x), output_field='ch', input_field='clean_text')\n",
    "corpus = chars.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('ch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that for some reason we now wanted to calculate:\n",
    "\n",
    "* characters per word\n",
    "* words per character (the reciprocal)\n",
    "\n",
    "This requires:\n",
    "\n",
    "* a `TextProcessor` that takes in multiple input fields, `'ch'` and `'wc'`;\n",
    "* and that writes to multiple output fields, `'char_per_word'` and `'word_per_char'`.\n",
    "\n",
    "Here's how the resultant object, `char_per_word`, handles this:\n",
    "\n",
    "* in `transform()`, we pass `proc_fn` a dict mapping input field name to value, e.g., `{'wc': 22, 'ch': 120}`\n",
    "* `proc_fn` will be written to return a tuple, where each element of that tuple corresponds to each element of the list we've passed to `output_field`, e.g., \n",
    "\n",
    "```out0, out1 = proc_fn(input)\n",
    "utt.set_info('char_per_word', out0) \n",
    "utt.set_info('word_per_char', out1)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "char_per_word = TextProcessor(proc_fn=lambda x: (x['ch']/x['wc'], x['wc']/x['ch']), \n",
    "                              output_field=['char_per_word', 'word_per_char'], input_field=['ch','wc'])\n",
    "corpus = char_per_word.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.454545454545454"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('char_per_word')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18333333333333332"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utt.retrieve_meta('word_per_char')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some advanced usage: input filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just for the sake of demonstration, suppose we wished to save some computation time and only parse the questions in a corpus. We can do this by specifying `input_filter` (which, recall discussion above, takes as argument an `Utterance` object). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_question(utt, aux={}):\n",
    "    return utt.meta['is_question']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "qparser = TextParser(output_field='qparsed', input_field='clean_text', input_filter=is_question, verbosity=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "050/200 utterances processed\n",
      "100/200 utterances processed\n",
      "150/200 utterances processed\n",
      "200/200 utterances processed\n"
     ]
    }
   ],
   "source": [
    "corpus = qparser.transform(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our test utterance is not a question, `qparser.transform()` will skip over it, and hence the utterance won't have the 'qparsed' attribute (and `get_info` returns `None`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "utt.retrieve_meta('qparsed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we take an utterance that's a question, we see that it is indeed parsed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'How hard was it for you when, 13 years, left your parents, left Japan to go to the States. Was it a big step for you?'"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_utt_id = '1681_14.q'\n",
    "q_utt = corpus.get_utterance(q_utt_id)\n",
    "q_utt.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rt': 11,\n",
       "  'toks': [{'tok': 'How', 'tag': 'WRB', 'dep': 'advmod', 'up': 1, 'dn': []},\n",
       "   {'tok': 'hard', 'tag': 'RB', 'dep': 'acomp', 'up': 2, 'dn': [0]},\n",
       "   {'tok': 'was', 'tag': 'VBD', 'dep': 'advcl', 'up': 11, 'dn': [1, 3, 4, 9]},\n",
       "   {'tok': 'it', 'tag': 'PRP', 'dep': 'nsubj', 'up': 2, 'dn': []},\n",
       "   {'tok': 'for', 'tag': 'IN', 'dep': 'prep', 'up': 2, 'dn': [5]},\n",
       "   {'tok': 'you', 'tag': 'PRP', 'dep': 'pobj', 'up': 4, 'dn': []},\n",
       "   {'tok': 'when', 'tag': 'WRB', 'dep': 'advmod', 'up': 9, 'dn': [7]},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 6, 'dn': []},\n",
       "   {'tok': '13', 'tag': 'CD', 'dep': 'nummod', 'up': 9, 'dn': []},\n",
       "   {'tok': 'years', 'tag': 'NNS', 'dep': 'npadvmod', 'up': 2, 'dn': [6, 8]},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 11, 'dn': []},\n",
       "   {'tok': 'left', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [2, 10, 13, 14, 15, 22]},\n",
       "   {'tok': 'your', 'tag': 'PRP$', 'dep': 'poss', 'up': 13, 'dn': []},\n",
       "   {'tok': 'parents', 'tag': 'NNS', 'dep': 'dobj', 'up': 11, 'dn': [12]},\n",
       "   {'tok': ',', 'tag': ',', 'dep': 'punct', 'up': 11, 'dn': []},\n",
       "   {'tok': 'left', 'tag': 'VBD', 'dep': 'conj', 'up': 11, 'dn': [16, 18]},\n",
       "   {'tok': 'Japan', 'tag': 'NNP', 'dep': 'dobj', 'up': 15, 'dn': []},\n",
       "   {'tok': 'to', 'tag': 'TO', 'dep': 'aux', 'up': 18, 'dn': []},\n",
       "   {'tok': 'go', 'tag': 'VB', 'dep': 'xcomp', 'up': 15, 'dn': [17, 19]},\n",
       "   {'tok': 'to', 'tag': 'IN', 'dep': 'prep', 'up': 18, 'dn': [21]},\n",
       "   {'tok': 'the', 'tag': 'DT', 'dep': 'det', 'up': 21, 'dn': []},\n",
       "   {'tok': 'States', 'tag': 'NNP', 'dep': 'pobj', 'up': 19, 'dn': [20]},\n",
       "   {'tok': '.', 'tag': '.', 'dep': 'punct', 'up': 11, 'dn': []}]},\n",
       " {'rt': 0,\n",
       "  'toks': [{'tok': 'Was', 'tag': 'VBD', 'dep': 'ROOT', 'dn': [1, 4, 7]},\n",
       "   {'tok': 'it', 'tag': 'PRP', 'dep': 'nsubj', 'up': 0, 'dn': []},\n",
       "   {'tok': 'a', 'tag': 'DT', 'dep': 'det', 'up': 4, 'dn': []},\n",
       "   {'tok': 'big', 'tag': 'JJ', 'dep': 'amod', 'up': 4, 'dn': []},\n",
       "   {'tok': 'step', 'tag': 'NN', 'dep': 'attr', 'up': 0, 'dn': [2, 3, 5]},\n",
       "   {'tok': 'for', 'tag': 'IN', 'dep': 'prep', 'up': 4, 'dn': [6]},\n",
       "   {'tok': 'you', 'tag': 'PRP', 'dep': 'pobj', 'up': 5, 'dn': []},\n",
       "   {'tok': '?', 'tag': '.', 'dep': 'punct', 'up': 0, 'dn': []}]}]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_utt.retrieve_meta('qparsed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}