constructing_dataset_for_generative_ASQ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Constructing dataset for inferring advice-seeking intentions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit import Corpus, Utterance, Speaker, download\n",
    "from convokit.text_processing import TextProcessor, TextCleaner, TextParser\n",
    "from cleantext import clean\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corpus we will use is from the subreddit r/Advice, which is a place where ''a place where anyone can seek advice on any subject\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset already exists at /belafonte_sauna/liye_translations/convokit_asq/subreddit-Advice\n"
     ]
    }
   ],
   "source": [
    "advice_corpus = Corpus(download('subreddit-Advice', \\\n",
    "                         data_dir='/belafonte_sauna/liye_translations/convokit_asq/'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Speakers: 297673\n",
      "Number of Utterances: 2373849\n",
      "Number of Conversations: 405056\n"
     ]
    }
   ],
   "source": [
    "advice_corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Clean and parse posts\n",
    "\n",
    "We start off by doing a little bit of preprocessing:\n",
    "\n",
    "* We will use a TextCleaner to do some basic cleaning such as fixing unicode errors, striping line breaks and convering text to lower case. We slightly deviate from the ConvoKit's default by allowing numbers and currency symbols, since they might provide relevant contexts for advice-seeking. \n",
    "\n",
    "* We will then tokenize the utterances using a TextParser, with the mode *tokenize*, since we will not be needing dependence parses (which can take a while to compute). \n",
    "\n",
    "We slightly deviate from the ConvoKit's default TextCleaner behavior by allowing numbers and currency symbols, since they might provide relevant contexts for advice-seeking. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input filter we will use throughout \n",
    "post_filter = lambda utt:utt.reply_to is None\n",
    "\n",
    "# path for where we'd like to save the output to\n",
    "OUTPUT_PATH = \"/belafonte_sauna/liye_translations/convokit_asq/filtered_pairs_oct6.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000/2373849 utterances processed\n",
      "100000/2373849 utterances processed\n",
      "150000/2373849 utterances processed\n",
      "200000/2373849 utterances processed\n",
      "250000/2373849 utterances processed\n",
      "300000/2373849 utterances processed\n",
      "350000/2373849 utterances processed\n",
      "400000/2373849 utterances processed\n",
      "450000/2373849 utterances processed\n",
      "500000/2373849 utterances processed\n",
      "550000/2373849 utterances processed\n",
      "600000/2373849 utterances processed\n",
      "650000/2373849 utterances processed\n",
      "700000/2373849 utterances processed\n",
      "750000/2373849 utterances processed\n",
      "800000/2373849 utterances processed\n",
      "850000/2373849 utterances processed\n",
      "900000/2373849 utterances processed\n",
      "950000/2373849 utterances processed\n",
      "1000000/2373849 utterances processed\n",
      "1050000/2373849 utterances processed\n",
      "1100000/2373849 utterances processed\n",
      "1150000/2373849 utterances processed\n",
      "1200000/2373849 utterances processed\n",
      "1250000/2373849 utterances processed\n",
      "1300000/2373849 utterances processed\n",
      "1350000/2373849 utterances processed\n",
      "1400000/2373849 utterances processed\n",
      "1450000/2373849 utterances processed\n",
      "1500000/2373849 utterances processed\n",
      "1550000/2373849 utterances processed\n",
      "1600000/2373849 utterances processed\n",
      "1650000/2373849 utterances processed\n",
      "1700000/2373849 utterances processed\n",
      "1750000/2373849 utterances processed\n",
      "1800000/2373849 utterances processed\n",
      "1850000/2373849 utterances processed\n",
      "1900000/2373849 utterances processed\n",
      "1950000/2373849 utterances processed\n",
      "2000000/2373849 utterances processed\n",
      "2050000/2373849 utterances processed\n",
      "2100000/2373849 utterances processed\n",
      "2150000/2373849 utterances processed\n",
      "2200000/2373849 utterances processed\n",
      "2250000/2373849 utterances processed\n",
      "2300000/2373849 utterances processed\n",
      "2350000/2373849 utterances processed\n",
      "2373849/2373849 utterances processed\n"
     ]
    }
   ],
   "source": [
    "clean_fn = lambda s: clean(s, no_numbers=False, no_digits=False, \\\n",
    "                           no_currency_symbols=False, no_punct=False)\n",
    "\n",
    "cleaner = TextCleaner(text_cleaner=clean_fn, input_filter=post_filter, verbosity=50000)\n",
    "\n",
    "advice_corpus = cleaner.transform(advice_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000/2373849 utterances processed\n",
      "100000/2373849 utterances processed\n",
      "150000/2373849 utterances processed\n",
      "200000/2373849 utterances processed\n",
      "250000/2373849 utterances processed\n",
      "300000/2373849 utterances processed\n",
      "350000/2373849 utterances processed\n",
      "400000/2373849 utterances processed\n",
      "450000/2373849 utterances processed\n",
      "500000/2373849 utterances processed\n",
      "550000/2373849 utterances processed\n",
      "600000/2373849 utterances processed\n",
      "650000/2373849 utterances processed\n",
      "700000/2373849 utterances processed\n",
      "750000/2373849 utterances processed\n",
      "800000/2373849 utterances processed\n",
      "850000/2373849 utterances processed\n",
      "900000/2373849 utterances processed\n",
      "950000/2373849 utterances processed\n",
      "1000000/2373849 utterances processed\n",
      "1050000/2373849 utterances processed\n",
      "1100000/2373849 utterances processed\n",
      "1150000/2373849 utterances processed\n",
      "1200000/2373849 utterances processed\n",
      "1250000/2373849 utterances processed\n",
      "1300000/2373849 utterances processed\n",
      "1350000/2373849 utterances processed\n",
      "1400000/2373849 utterances processed\n",
      "1450000/2373849 utterances processed\n",
      "1500000/2373849 utterances processed\n",
      "1550000/2373849 utterances processed\n",
      "1600000/2373849 utterances processed\n",
      "1650000/2373849 utterances processed\n",
      "1700000/2373849 utterances processed\n",
      "1750000/2373849 utterances processed\n",
      "1800000/2373849 utterances processed\n",
      "1850000/2373849 utterances processed\n",
      "1900000/2373849 utterances processed\n",
      "1950000/2373849 utterances processed\n",
      "2000000/2373849 utterances processed\n",
      "2050000/2373849 utterances processed\n",
      "2100000/2373849 utterances processed\n",
      "2150000/2373849 utterances processed\n",
      "2200000/2373849 utterances processed\n",
      "2250000/2373849 utterances processed\n",
      "2300000/2373849 utterances processed\n",
      "2350000/2373849 utterances processed\n",
      "2373849/2373849 utterances processed\n"
     ]
    }
   ],
   "source": [
    "parser = TextParser(mode='tokenize', input_filter=post_filter, verbosity=50000)\n",
    "advice_corpus = parser.transform(advice_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Seperating questions from narratives \n",
    "\n",
    "The first step to construct the (narrative, questions) pairs is by seperating questions from the original advice-seeking post. We consider the question-stripped text as the __narrative__ that provides enough contexts and we consider the identified questions are possible candidates for indicating the OP's advice-seeking intentions. \n",
    "\n",
    "We will use a custom `QuestionCloze` transformer to produce, for each utterance (i.e., an advice-seeking post), the question-masked narrative (saved under `masked_text`) and questions found in the utterance (saved under `questions`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# seperating questions and narratives\n",
    "from questionCloze import QuestionCloze"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000/2373849 utterances processed\n",
      "100000/2373849 utterances processed\n",
      "150000/2373849 utterances processed\n",
      "200000/2373849 utterances processed\n",
      "250000/2373849 utterances processed\n",
      "300000/2373849 utterances processed\n",
      "350000/2373849 utterances processed\n",
      "400000/2373849 utterances processed\n",
      "450000/2373849 utterances processed\n",
      "500000/2373849 utterances processed\n",
      "550000/2373849 utterances processed\n",
      "600000/2373849 utterances processed\n",
      "650000/2373849 utterances processed\n",
      "700000/2373849 utterances processed\n",
      "750000/2373849 utterances processed\n",
      "800000/2373849 utterances processed\n",
      "850000/2373849 utterances processed\n",
      "900000/2373849 utterances processed\n",
      "950000/2373849 utterances processed\n",
      "1000000/2373849 utterances processed\n",
      "1050000/2373849 utterances processed\n",
      "1100000/2373849 utterances processed\n",
      "1150000/2373849 utterances processed\n",
      "1200000/2373849 utterances processed\n",
      "1250000/2373849 utterances processed\n",
      "1300000/2373849 utterances processed\n",
      "1350000/2373849 utterances processed\n",
      "1400000/2373849 utterances processed\n",
      "1450000/2373849 utterances processed\n",
      "1500000/2373849 utterances processed\n",
      "1550000/2373849 utterances processed\n",
      "1600000/2373849 utterances processed\n",
      "1650000/2373849 utterances processed\n",
      "1700000/2373849 utterances processed\n",
      "1750000/2373849 utterances processed\n",
      "1800000/2373849 utterances processed\n",
      "1850000/2373849 utterances processed\n",
      "1900000/2373849 utterances processed\n",
      "1950000/2373849 utterances processed\n",
      "2000000/2373849 utterances processed\n",
      "2050000/2373849 utterances processed\n",
      "2100000/2373849 utterances processed\n",
      "2150000/2373849 utterances processed\n",
      "2200000/2373849 utterances processed\n",
      "2250000/2373849 utterances processed\n",
      "2300000/2373849 utterances processed\n",
      "2350000/2373849 utterances processed\n",
      "2373849/2373849 utterances processed\n"
     ]
    }
   ],
   "source": [
    "qcloze = QuestionCloze(input_filter=post_filter, verbosity=50000)\n",
    "advice_corpus = qcloze.transform(advice_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, utterances that are posts should be annotated with *masked_text* and *questions*, see below for an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi , i am a 16 yr old guy , nearly 17and i think there s something mentally wrong with me . i want to discuss it here , but i m not sure if i could get banned discussing the problem . i just came here cause i ca n't tell anyone irl , i would be looked down upon , and get zero sympathy , which i do n't expect to get any here either , which i totally understand .\n",
      "\t []\n"
     ]
    }
   ],
   "source": [
    "utt = advice_corpus.random_utterance()\n",
    "while utt.reply_to:\n",
    "    utt = advice_corpus.random_utterance()\n",
    "\n",
    "print(utt.meta['masked_text'])\n",
    "print('\\t', utt.meta['questions'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Selecting appropriate question\n",
    "\n",
    "Next, among all the possible question candidates, we use a `QuestionSelector` to find one \"most appropriate\" question for each narrative. \n",
    "\n",
    "Our selection criteria is to consider the specificity of the question---as approximated by max idf of the words---and the position it appears (earlier is better). This is not necessarily the best approach, but seems to produce good enough results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TfidfVectorizer(min_df=20)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = TfidfVectorizer(min_df=20)\n",
    "\n",
    "# We use the original text collection to compute the idf\n",
    "texts = [utt.text for utt in advice_corpus.iter_utterances() if utt.reply_to is None]\n",
    "vectorizer.fit(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2idf = defaultdict(int)\n",
    "for k, v in zip(vectorizer.get_feature_names(), vectorizer.idf_):\n",
    "    word2idf[k] = v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now select questions, keeping both the selected question (saved under `sel_question`), and its position in the post (saved under `sel_question_pos`), in terms of sentence index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from questionSelector import QuestionSelector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "qselector = QuestionSelector(word2idf=word2idf, input_filter=post_filter, verbosity=50000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000/2373849 utterances processed\n",
      "100000/2373849 utterances processed\n",
      "150000/2373849 utterances processed\n",
      "200000/2373849 utterances processed\n",
      "250000/2373849 utterances processed\n",
      "300000/2373849 utterances processed\n",
      "350000/2373849 utterances processed\n",
      "400000/2373849 utterances processed\n",
      "450000/2373849 utterances processed\n",
      "500000/2373849 utterances processed\n",
      "550000/2373849 utterances processed\n",
      "600000/2373849 utterances processed\n",
      "650000/2373849 utterances processed\n",
      "700000/2373849 utterances processed\n",
      "750000/2373849 utterances processed\n",
      "800000/2373849 utterances processed\n",
      "850000/2373849 utterances processed\n",
      "900000/2373849 utterances processed\n",
      "950000/2373849 utterances processed\n",
      "1000000/2373849 utterances processed\n",
      "1050000/2373849 utterances processed\n",
      "1100000/2373849 utterances processed\n",
      "1150000/2373849 utterances processed\n",
      "1200000/2373849 utterances processed\n",
      "1250000/2373849 utterances processed\n",
      "1300000/2373849 utterances processed\n",
      "1350000/2373849 utterances processed\n",
      "1400000/2373849 utterances processed\n",
      "1450000/2373849 utterances processed\n",
      "1500000/2373849 utterances processed\n",
      "1550000/2373849 utterances processed\n",
      "1600000/2373849 utterances processed\n",
      "1650000/2373849 utterances processed\n",
      "1700000/2373849 utterances processed\n",
      "1750000/2373849 utterances processed\n",
      "1800000/2373849 utterances processed\n",
      "1850000/2373849 utterances processed\n",
      "1900000/2373849 utterances processed\n",
      "1950000/2373849 utterances processed\n",
      "2000000/2373849 utterances processed\n",
      "2050000/2373849 utterances processed\n",
      "2100000/2373849 utterances processed\n",
      "2150000/2373849 utterances processed\n",
      "2200000/2373849 utterances processed\n",
      "2250000/2373849 utterances processed\n",
      "2300000/2373849 utterances processed\n",
      "2350000/2373849 utterances processed\n",
      "2373849/2373849 utterances processed\n"
     ]
    }
   ],
   "source": [
    "advice_corpus = qselector.transform(advice_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also compute some helpful stats, to help further filter valid (narrative, question) pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scnt = TextProcessor(proc_fn=lambda parses: len(parses), \\\n",
    "                     output_field='num_sents', input_field='parsed', \\\n",
    "                     input_filter=post_filter)\n",
    "\n",
    "wcnt = TextProcessor(proc_fn=lambda text: len(text.split()), \\\n",
    "                     output_field='num_words', input_field='masked_text', \\\n",
    "                     input_filter=lambda utt:utt.reply_to is None)\n",
    "\n",
    "advice_corpus = wcnt.transform(advice_corpus)\n",
    "advice_corpus = scnt.transform(advice_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Exporting valid (narrative, question) pairs\n",
    "\n",
    "Now we will get all information we need into a dataframe for some minor filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "narrative_df = advice_corpus.get_utterances_dataframe(selector=post_filter)\n",
    "narrative_df = narrative_df[['text', 'meta.masked_text', 'meta.questions', 'meta.sel_question', 'meta.sel_question_pos', \\\n",
    "                             'meta.num_words', 'meta.num_sents']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to apply various filters and see their effects on the data size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1: 405056/189583 (46.80%)\n",
      "step 2: 405056/66537 (16.43%)\n",
      "step 3: 405056/33745 (8.33%)\n",
      "step 4: 405056/31872 (7.87%)\n",
      "step 5: 405056/31562 (7.79%)\n"
     ]
    }
   ],
   "source": [
    "# filter narratives for which no question is retrieved \n",
    "base_cnt = len(narrative_df)\n",
    "filtered_cnts = []\n",
    "\n",
    "# 1. keep only instances in which explicit questions are identified, i.e., 'questions' isn't empty\n",
    "filtered_df = narrative_df[narrative_df['meta.questions'].map(bool)]\n",
    "filtered_cnts.append(len(filtered_df))\n",
    "\n",
    "# 2. keep only instances in which question is selected successfully \n",
    "filtered_df = filtered_df[(filtered_df['meta.sel_question'].notnull())]\n",
    "filtered_cnts.append(len(filtered_df))\n",
    "\n",
    "# 3. keep only texts where the masked narrative is within a given length range \n",
    "filtered_df = filtered_df[(filtered_df['meta.num_words']>=100) & (filtered_df['meta.num_words']<=400)]\n",
    "filtered_cnts.append(len(filtered_df))\n",
    "\n",
    "# 4. filtering out instances where tl;dr is present\n",
    "filtered_df = filtered_df[(~filtered_df['meta.masked_text'].str.contains('tl;dr')) & \\\n",
    "                          (~filtered_df['meta.masked_text'].str.contains('tldr'))]\n",
    "filtered_cnts.append(len(filtered_df))\n",
    "\n",
    "# 5. filtering out most common questions (at sentence level)\n",
    "common_qns = {qn for qn, _ in Counter(filtered_df['meta.sel_question']).most_common(20)}\n",
    "filtered_df = filtered_df[~filtered_df['meta.sel_question'].isin(common_qns)]\n",
    "filtered_cnts.append(len(filtered_df))\n",
    "\n",
    "for i, cnt in enumerate(filtered_cnts):\n",
    "    print(\"step {}: {}/{} ({:.2f}%)\".format(i+1, base_cnt, cnt, 100*cnt/base_cnt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, since our filters are based on some heuristics, some very generic questions can still escape, we will inspect from some frequent questions to ignore some very generic ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are some example (narrative, question) pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===Narrative===\n",
      "so , a month ago , i got hired as a cook at golden corral . it was in * really * poor shape and the regular paycheck there really helped me out . i was able to get some food and keep my bills relatively under control . i hate the job at golden corral . it is miserable , unsatisfying work that leaves me coming home at night smelling terrible with my covered with burns ( i get to work the fryers ! : d ) . the only benefit from working at the place is that i 've met some cool people . my situation is under control , and i have opportunity to get a better job elsewhere . more satisfying , better pay and wo n't leave me hating everything . i want to quit , but i also feel bad doing so so soon after i started working . i also feel like i 'm gon na hurt the people i 've met there that have helped me out to figure out everything that i need to do . i do n't know exactly what i 'm looking for with this post .\n",
      "------------\n",
      "Question: ['how can i leave this job without feeling guilty about it ?']\n",
      "Selected question: how can i leave this job without feeling guilty about it ?\n",
      "\n",
      "===Narrative===\n",
      "my wife and i have been separated for a few weeks now . i 've gone to counseling and have been speaking with friends regarding the split . she has done the same . since then she says that she has gone through the hurt process and felt neglected . i suffer from depression and have been dealing with that . this was the root cause of our distance . since then we have come together for conversation . she felt that she did n't love me . that same night she kissed me . then , she stated she did n't want to complicate things . now , she has agreed to possibly go to counseling . she still says she is confused about her feelings and that she was feeling the distance for months on end . however , she texts me when we do n't talk for about a day . she said that she is 100 % honest with me , and that she does n't want to bring these feelings to work with her .\n",
      "------------\n",
      "Question: ['am i missing something ?', 'does it sound like there is someone else ?']\n",
      "Selected question: does it sound like there is someone else ?\n",
      "\n",
      "===Narrative===\n",
      "preface : i am aware that this forum is not a substitute for medical advice . just do your best . thanks . my girlfriend has been feeling really tired , regardless of getting a lot of sleep , and getting frequent headaches to boot . she 's been avoiding school ( she 's a senior in college ) because she said she ca n't think and feels weak all day . now , i have theories about what she could be suffering from , but there 's a bigger issue here : she does n't trust doctors . she grew up in a sort of new age - y household so she 's generally been raised to believe that doctors are scammers who will prescribe her a bunch of pills she does n't need and overcharge her for everything . she 's still on her mom 's insurance but she insists that \" it will pay for hardly anything . \" her solution thus far has been to eat unprocessed foods ( good ) and take a lot of expensive vitamins ( questionable ) . even if this method was useful she has been feeling this way for about a month , fluctuating , but with no noticeable improvement .\n",
      "------------\n",
      "Question: ['how can i convince her to get a checkup ?']\n",
      "Selected question: how can i convince her to get a checkup ?\n",
      "\n",
      "===Narrative===\n",
      "so i 'm buying a new macbook ( please no rhetoric about how it 's a rip off , etc . i 've heard it all before ) . but , i 'm torn between the air and the retina display . i 'm also confused about what specs to add to my computer . i 'm a college student right now and when i 'm not , i 'm just a regular user . so , i primarily use my computer for school work ( standard word , ppt , excel ) and entertainment . if you have any advice , i 'd really appreciate it .\n",
      "------------\n",
      "Question: ['should i add more storage , should i not ?']\n",
      "Selected question: should i add more storage , should i not ?\n",
      "\n",
      "===Narrative===\n",
      "last semester i cut loose and found that i was basically failing at the end of the semester . i knew i did n't have it in me to turn it around with finals and i talked to the school counselor that i had started seeing that semester . he recommended i take a leave of absence , so i 'm at home working now . my dad lost his job this year and my fafsa would only be 12,500 . tuition is 30,000 . my parents say they will pay if i want to go , but that 's a big commitment for a guy who has never had it all together with school and has occasional doubts about his profession . and who pretty much failed out once . i do n't know what my parents financial situation really is . my mom has her job and it 's okay , but my dad losing his is a loss of about 2/3 of our income . i 'm not \" passionate \" about my field . i have n't had a very positive experience with my school so far . all that aside , i am relatively sure i could graduate . i also frequently struggle with my emotions and i feel that they impact my ability to succeed . i fear that left to my own devices i would be completely irresponsible and useless , taking the path of least resistance to my eventual self - destruction . edit : i have 4 years left . my fafsa should be about 20,000 for the last 2 years . i need both a pragmatic answer based on financial cost and a personal answer . anyone who could offer experiences with debt and expensive tuition would be appreciated .\n",
      "------------\n",
      "Question: ['is the risk of driving my parents into poverty and taking on ridiculous debt worth it ?', 'am i making a mistake going back ?', 'should i try to find a cheaper easier route ?']\n",
      "Selected question: should i try to find a cheaper easier route ?\n",
      "\n",
      "===Narrative===\n",
      "i 'm a junior right now . my gpa is 2.9 . i 'm a smart kid , i just had a shit freshman year and slacked off too much my sophomore year . i 'm taking two ap classes , my math class is honors as well as my human anatomy class . i wanted to take an ap science class , but combining with a new school fucked up everyone 's schedules . next year , i 'll be in ap gov / econ , ap statistics , ap english lit , ap biology and ap environmental issues . by the end of my senior year , judging by the hard work that i 've put in this year , my gpa should be a 3.3 or 3.4 . the problem is , when i apply for colleges , they 'll see my cumulative gpa for fm , sm , and jr year . my act is a 29 , but that was without studying or a calculator and i 'm taking it again ( i 've studied and bought a calculator . ) my plan is to take a gap year and apply for colleges with my gpa that 'll include senior year . during my gap year , i 'll find something awesome and significant to do ( some sort of program ) that 'll make me look better for colleges . my questions are ... \n",
      " rather than deferring my freshman year at an \" okay \" school , i want to apply to good schools after my gap year .\n",
      "------------\n",
      "Question: ['would they like me less for the gap year ?', 'would it lower my odds of getting in ?', 'are there a lot of programs for gap year students ?', 'internships ?', 'when applying to colleges during a gap year , how do i get all of my old information from my high school ?', 'will it still be there or do i need to get it at the end of my senior year ?', 'like what about the things that my high school has to send directly to the colleges like my transcript ?', \"i know i already asked this , but it 's the most important part : will i be less likely to get into good schools because of my gap year ?\"]\n",
      "Selected question: are there a lot of programs for gap year students ?\n",
      "\n",
      "===Narrative===\n",
      "so i wanted to pitch an idea to this company i really want to work for . i think it has potential to make a lot of money for them . anyways , i emailed them and the pr manager said he would call me today at 2 to discuss my idea . its about to be 6 and he has yet to call me back.i do not have the guys phone number . the company does nt have a phone number for anyone but general customer service . they already signed an nda waiver for me . sorry i am business noob . thanks for your help !\n",
      "------------\n",
      "Question: ['should i say eff them and go to another company to pitch or wait for them to call ?']\n",
      "Selected question: should i say eff them and go to another company to pitch or wait for them to call ?\n",
      "\n",
      "===Narrative===\n",
      "so i 'm trying to get over my fear of spiders and have been allowing this one spider to live in my room . i just got home and it was spazzing out everywhere and i noticed it has an egg sack . i do n't want to kill it , it 's kind of like my spiderbro now , but am too afraid to catch and release it and have no one to do it for me right now . i already feel bad because i think i killed the father a few days ago before trying to get over my fear . i do n't want my room turned into a nursery !\n",
      "------------\n",
      "Question: ['if i squish it will babies come out ?', 'how long is the spider gestation period ?', 'will there be babies everywhere tomorrow when i wake up ?', 'tl;dr : will baby spiders explode everywhere in my room ?']\n",
      "Selected question: will there be babies everywhere tomorrow when i wake up ?\n",
      "\n",
      "===Narrative===\n",
      "i 'm a pre - optometry major and i 've had a really bad semester . i 've gotten only 2 as in 3 credit hour classes , 2 cs in 4 credit hour classes , and possibly a b ( hopefully i can pull off an a ) in another 4 credit hour class . my gpa is already much lower than i want it to be . i 'm really concerned about acceptance into optometry school in the future and maintaining my scholarship right now . i 'm currently considering another major because i 'm not feeling smart enough for this major , but i am extremely attached to it and i do not have any other ideas for majors and careers .\n",
      "------------\n",
      "Question: ['did any of you have a less - than - impressive semester ?', 'how did it affect your future plans and how did you try to mend the situation ?']\n",
      "Selected question: how did it affect your future plans and how did you try to mend the situation ?\n",
      "\n",
      "===Narrative===\n",
      "my so and i just got an apartment on campus at my university . we checked in thursday but we did n't plan to move in for another week or two . when we went to check out the apartment we noticed two small roaches . we killed them , disposed of their bodies , and then left . later on that night i took my friend over to the new apartment to show her . we counted over 15 little roaches . we brought over some raid she had , and sprayed in the corners and such . friday i bought two packs of those little black roach - killing traps , and some egg - killing traps . i put them everywhere . they pretty much cover the floor . i also bought another type of spray , and sprayed along the perimeter of every wall . i emailed the housing office because they 're supposed to take care of this , but there office was already closed . i 'm fairly sure this apartment has n't been occupied in a while .\n",
      "------------\n",
      "Question: ['do you think that the roaches will be dead in time for our move - in ( ~2 weeks ) ?', 'what else can i do to kill them and prevent them ?', \"so what do i do if this does n't kill them ?\"]\n",
      "Selected question: what else can i do to kill them and prevent them ?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for item in filtered_df[0:10].itertuples():\n",
    "    print(\"===Narrative===\")\n",
    "    print(item[2])\n",
    "    print('------------')\n",
    "    print(\"Question:\", [qn for qn, _ in item[3]])\n",
    "    print(\"Selected question:\", item[4])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One may be curious where our selected questions come from. Perhaps not surprising, most of them are found nearing the end. The final drop is most likely due to our deliberate (but somewhat arbitrary) decision to give priority over earlier questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 328.,  247.,  405.,  702.,  994., 2633., 3888., 6494., 9176.,\n",
       "        7005.]),\n",
       " array([0.0, 0.0975, 0.195, 0.2925, 0.39, 0.48750000000000004, 0.585,\n",
       "        0.6825, 0.78, 0.8775000000000001, 0.975], dtype=object),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPFUlEQVR4nO3df6zdd13H8efLluEAJ527W8bt8A5TgW6RwOqsoAQZyQozdiYsqQpryJLGORGNRjr+kD9MkxINwakbaQaui0jTjMVV59ClimjYD+/YoHR1rrLZ1dX1ggoTk2HL2z/O94+T9t7e7+3uPWf3fp6P5OR8v5/v5/s9n3du87qffs4535uqQpLUju8b9wAkSaNl8EtSYwx+SWqMwS9JjTH4Jakxq8c9gPlccMEFNTU1Ne5hSNKy8sgjj3yjqiZmO/aSD/6pqSmmp6fHPQxJWlaS/Ntcx1zqkaTGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxrzkv7krSVPb7x3baz+985qxvfZSccYvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5Ia0yv4k/xGkoNJvpbks0m+P8n5Se5P8mT3vGao/81JDid5IsnVQ+1XJDnQHbslSZaiKEnS3OYN/iSTwK8BG6rqcmAVsAXYDuyvqnXA/m6fJOu745cBm4Bbk6zqLncbsA1Y1z02LWo1kqR59V3qWQ2cm2Q18ArgWWAzsLs7vhu4ttveDOypqheq6ingMHBlkouB86rqgaoq4M6hcyRJIzJv8FfVvwO/DxwBjgHfqqq/AS6qqmNdn2PAhd0pk8AzQ5c42rVNdtuntp8mybYk00mmZ2ZmFlaRJOmM+iz1rGEwi78UeA3wyiTvO9Mps7TVGdpPb6zaVVUbqmrDxMTEfEOUJC1An6WedwFPVdVMVf0fcDfwVuC5bvmG7vl41/8ocMnQ+WsZLA0d7bZPbZckjVCf4D8CbEzyiu5TOFcBh4B9wNauz1bgnm57H7AlycuTXMrgTdyHu+Wg55Ns7K5z/dA5kqQRWT1fh6p6KMldwJeBE8CjwC7gVcDeJDcw+OVwXdf/YJK9wONd/5uq6mR3uRuBO4Bzgfu6hyRphOYNfoCq+ijw0VOaX2Aw+5+t/w5gxyzt08DlCxyjJGkR+c1dSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5Jakyvv8AlSa2a2n7vWF736Z3XLNm1nfFLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5Jaoy3bJDU27huX6DF5Yxfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TG9Ar+JK9OcleSf05yKMlPJjk/yf1Jnuye1wz1vznJ4SRPJLl6qP2KJAe6Y7ckyVIUJUmaW98Z/x8An6+qNwBvAg4B24H9VbUO2N/tk2Q9sAW4DNgE3JpkVXed24BtwLrusWmR6pAk9TRv8Cc5D3g78CmAqvpuVf03sBnY3XXbDVzbbW8G9lTVC1X1FHAYuDLJxcB5VfVAVRVw59A5kqQR6TPjfx0wA/xJkkeT3J7klcBFVXUMoHu+sOs/CTwzdP7Rrm2y2z61/TRJtiWZTjI9MzOzoIIkSWfWJ/hXA28BbquqNwPfoVvWmcNs6/Z1hvbTG6t2VdWGqtowMTHRY4iSpL76BP9R4GhVPdTt38XgF8Fz3fIN3fPxof6XDJ2/Fni2a187S7skaYTmDf6q+g/gmSSv75quAh4H9gFbu7atwD3d9j5gS5KXJ7mUwZu4D3fLQc8n2dh9muf6oXMkSSPS97bMHwQ+k+Qc4OvABxj80tib5AbgCHAdQFUdTLKXwS+HE8BNVXWyu86NwB3AucB93UOSNEK9gr+qHgM2zHLoqjn67wB2zNI+DVy+kAFKkhaX39yVpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmN6B3+SVUkeTfKX3f75Se5P8mT3vGao781JDid5IsnVQ+1XJDnQHbslSRa3HEnSfBYy4/8QcGhofzuwv6rWAfu7fZKsB7YAlwGbgFuTrOrOuQ3YBqzrHpte1OglSQvWK/iTrAWuAW4fat4M7O62dwPXDrXvqaoXquop4DBwZZKLgfOq6oGqKuDOoXMkSSPSd8b/CeC3ge8NtV1UVccAuucLu/ZJ4Jmhfke7tslu+9R2SdIIzRv8SX4WOF5Vj/S85mzr9nWG9tlec1uS6STTMzMzPV9WktRHnxn/24CfS/I0sAd4Z5I/BZ7rlm/ono93/Y8ClwydvxZ4tmtfO0v7aapqV1VtqKoNExMTCyhHkjSf1fN1qKqbgZsBkrwD+K2qel+S3wO2Aju753u6U/YBf5bk48BrGLyJ+3BVnUzyfJKNwEPA9cAfLnI9UhOmtt877iFoGZs3+M9gJ7A3yQ3AEeA6gKo6mGQv8DhwAripqk5259wI3AGcC9zXPSRJI7Sg4K+qLwBf6La/CVw1R78dwI5Z2qeByxc6SEnS4vGbu5LUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUmNXjHoC0XE1tv3fcQ5DOijN+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMfMGf5JLkvxdkkNJDib5UNd+fpL7kzzZPa8ZOufmJIeTPJHk6qH2K5Ic6I7dkiRLU5YkaS59ZvwngN+sqjcCG4GbkqwHtgP7q2odsL/bpzu2BbgM2ATcmmRVd63bgG3Auu6xaRFrkST1MG/wV9Wxqvpyt/08cAiYBDYDu7tuu4Fru+3NwJ6qeqGqngIOA1cmuRg4r6oeqKoC7hw6R5I0Igta408yBbwZeAi4qKqOweCXA3Bh120SeGbotKNd22S3fWr7bK+zLcl0kumZmZmFDFGSNI/ewZ/kVcDngF+vqm+fqessbXWG9tMbq3ZV1Yaq2jAxMdF3iJKkHnoFf5KXMQj9z1TV3V3zc93yDd3z8a79KHDJ0OlrgWe79rWztEuSRqjPp3oCfAo4VFUfHzq0D9jabW8F7hlq35Lk5UkuZfAm7sPdctDzSTZ217x+6BxJ0oj0+dOLbwPeDxxI8ljX9hFgJ7A3yQ3AEeA6gKo6mGQv8DiDTwTdVFUnu/NuBO4AzgXu6x6SpBGaN/ir6h+ZfX0e4Ko5ztkB7JilfRq4fCEDlCQtLr+5K0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqTJ+/wCW9pE1tv3fcQ5CWFWf8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTF+nFOLwo9USsuHM35JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGD/OucL4sUpJ81nRwT+uEHx65zVjeV1J6sOlHklqzIqe8Y+Lyy2SXsqc8UtSYwx+SWrMyIM/yaYkTyQ5nGT7qF9fklo30uBPsgr4Y+DdwHrgF5KsH+UYJKl1o57xXwkcrqqvV9V3gT3A5hGPQZKaNupP9UwCzwztHwV+4tROSbYB27rd/0nyxFm+3gXAN87y3OWs1bqh3dpbrRtWaO352Lxd5qv7h+c6MOrgzyxtdVpD1S5g14t+sWS6qja82OssN63WDe3W3mrd0G7tL6buUS/1HAUuGdpfCzw74jFIUtNGHfz/BKxLcmmSc4AtwL4Rj0GSmjbSpZ6qOpHkV4G/BlYBn66qg0v4ki96uWiZarVuaLf2VuuGdms/67pTddoSuyRpBfObu5LUGINfkhqzIoJ/vttAZOCW7vhXk7xlHONcbD3q/qWu3q8m+VKSN41jnIut720/kvx4kpNJ3jvK8S2lPrUneUeSx5IcTPL3ox7jUujxb/0Hk/xFkq90dX9gHONcbEk+neR4kq/Ncfzssq2qlvWDwZvE/wq8DjgH+Aqw/pQ+7wHuY/A9go3AQ+Me94jqfiuwptt+dyt1D/X7W+CvgPeOe9wj/Jm/GngceG23f+G4xz2iuj8CfKzbngD+Ezhn3GNfhNrfDrwF+Nocx88q21bCjL/PbSA2A3fWwIPAq5NcPOqBLrJ5666qL1XVf3W7DzL43sRy1/e2Hx8EPgccH+Xgllif2n8RuLuqjgBU1Uqov0/dBfxAkgCvYhD8J0Y7zMVXVV9kUMtczirbVkLwz3YbiMmz6LPcLLSmGxjMDJa7eetOMgn8PPDJEY5rFPr8zH8UWJPkC0keSXL9yEa3dPrU/UfAGxl8IfQA8KGq+t5ohjdWZ5VtK+EvcPW5DUSvW0UsM71rSvIzDIL/p5Z0RKPRp+5PAB+uqpODCeCK0af21cAVwFXAucADSR6sqn9Z6sEtoT51Xw08BrwT+BHg/iT/UFXfXurBjdlZZdtKCP4+t4FYibeK6FVTkh8DbgfeXVXfHNHYllKfujcAe7rQvwB4T5ITVfXnoxnikun7b/0bVfUd4DtJvgi8CVjOwd+n7g8AO2uw8H04yVPAG4CHRzPEsTmrbFsJSz19bgOxD7i+ewd8I/Ctqjo26oEusnnrTvJa4G7g/ct8xjds3rqr6tKqmqqqKeAu4FdWQOhDv3/r9wA/nWR1klcwuPvtoRGPc7H1qfsIg//lkOQi4PXA10c6yvE4q2xb9jP+muM2EEl+uTv+SQaf7HgPcBj4Xwazg2WtZ92/A/wQcGs3+z1Ry/wuhj3rXpH61F5Vh5J8Hvgq8D3g9qqa9aOAy0XPn/nvAnckOcBg+ePDVbXsb9Wc5LPAO4ALkhwFPgq8DF5ctnnLBklqzEpY6pEkLYDBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhrz/6SkBHtX2ornAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(filtered_df['meta.sel_question_pos'] / filtered_df['meta.num_sents'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the output to \n",
    "filtered_df.to_json(OUTPUT_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}