4.5-Bayes_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy, pandas\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "import matplotlib.pyplot as plot\n",
    "import seaborn\n",
    "%matplotlib inline\n",
    "#%config InlineBackend.figure_format = 'svg'\n",
    "#plot.rcParams['figure.figsize'] = [4, 4]\n",
    "\n",
    "# from https://stackoverflow.com/a/21009774/6871666\n",
    "numpy.set_printoptions(formatter={'float_kind': lambda x: \"%.3f\" % x})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Predictions\n",
    "\n",
    "All of our examples so far have been about numbers: all of our features have been numeric and the kind of thing that we find the mean and standard deviation of.\n",
    "\n",
    "What if we want to look at features that aren't numbers? Today, what if that is natural language?\n",
    "\n",
    "**Natural languages** are things like English, French, and Chinese: the languages that humans write and speak. It's perfectly reasonable to want to take some text and decide that it's in the class \"angry\" or \"about the weather\", but we need some new tools for that.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An example: what's the topic\n",
    "\n",
    "One really common problem we might want to tackle with natural language: given some text, can we determine what its \"about\"? What's the topic?\n",
    "\n",
    "As always with supervised machine learning, we need some training data: a collection of text with corresponding topics that we know. We can use that to train a classifier and make predictions in the future.\n",
    "\n",
    "Scikit-Learn comes with some sample data where we can do this: a collection of 11,000 postings from [Usenet newsgroups](https://en.wikipedia.org/wiki/Usenet_newsgroup) in the 1990s (guessing from the topics being discussed).\n",
    "\n",
    "\\[This example (and a little code) is borrowed from [Python Data Science Handbook by Jake VanderPlas](https://github.com/jakevdp/PythonDataScienceHandbook), which is a **great** book about using Python for manipulating data and doing machine learning.\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "data = fetch_20newsgroups()\n",
    "data.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will select a few of the newsgroups that we will try to predict between"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = ['alt.atheism', 'comp.sys.mac.hardware', 'comp.sys.ibm.pc.hardware',\n",
    "           'rec.sport.baseball', 'rec.sport.hockey', 'soc.religion.christian']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = fetch_20newsgroups(categories=classes, remove=('headers', 'footers'))\n",
    "X = data.data\n",
    "y = data.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have a look at one of the \"features\": a message that somebody wrote one time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print((X[5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the correct class for that input: the newsgroup where it was posted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.target_names[y[5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same train/test split, and then we're ready to have a pipeline..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transforming Text\n",
    "\n",
    "In the previous transformer examples, we did some fairly simple transformations: one set of numeric features into another. If we want to work with text, we're going to need to do something more advanced.\n",
    "\n",
    "One way or another, we need to get the text turned into a sequence of number before we can try to use a Bayes classifier (or any other machine learning classifier) on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way we might get numbers out of our text: start counting.\n",
    "\n",
    "If we counted how many times each word appeared, we would get some numbers. For each piece of text, count how many times \"hello\" appears, and how many times \"left\", and how many times \"disambiguate\", .... Out of that, we could produce a big matrix of numbers and put that into a ML model.\n",
    "\n",
    "The core idea of \"just count the words\" isn't bad, but we need to be a little more sophisticated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problems with counting words: we will see \"the\" in every example of English text, so that's probably not so useful. If we have longer and shorted passages, we would expect to see words appear more or less often, but we probably can't predict anything from that.\n",
    "\n",
    "The transformation [**tf-idf**](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) will help us make seense of text in a way that will actually be useful to making predictions. The idea: count words, but give them a score that says something about how \"important\" they are in that passage of text.\n",
    "\n",
    "There are two steps in calculating td-idf:\n",
    "1. Count the words in each piece of text (and throw away words that appear too often). The idea: if a word occurs more times, then it must be more important to that text. **Term Frequency**.\n",
    "2. Weight words as less important if they appear more often overall: it's not very interesting to see \"the\" or \"hello\" in text because they appear in lots of examples. It's interesting to see \"disambiguate\" because it's rare. **Inverse document frequency**.\n",
    "\n",
    "This is one of those things that sounds worse than it is: \"term frequency-inverse document frequency\" is a mouthful, but don't let that scare you. Word occurs more often in one example: it's more important. Word occurs in more of the examples: it's less important. A bunch of arithmetic happens to make the more/less important work out to numbers that are actually useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn will take care of the calculations for us, and do it in a way we can use in a pipeline model. The `TfidfVectorizer` will do what we need. \\[The two steps mentioned above are actually available as `CountVectorizer` and `TfidfTransformer` if you want them separately, but `TfidfVectorizer` combines them automatically.\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at what tf-idf does on some simple sample text..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_example = [\n",
    "    'I am sam.',\n",
    "    'Sam I am.',\n",
    "    'Do you like green eggs and ham?',\n",
    "    'I do not like them, Sam-I-am!',\n",
    "    'I do not like green eggs and ham!',\n",
    "    'Would you like them here or there?',\n",
    "    'Would you eat them in a box?',\n",
    "    'Would you eat them with a fox?',\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, `TfidfVectorizer` will go in a pipeline, but we can try it on its own to examine what happens. We will ask it to automatically ignore English words that are very common (like \"I\" and \"am\" and \"not\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf = TfidfVectorizer(max_df=0.5, stop_words='english')\n",
    "transformed = tfidf.fit_transform(X_example).toarray()\n",
    "transformed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those are the feature values that tf-idf produces for this input. Each row represents one of the input sentences. Each column is a word from the \"vocabulary\" it selected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have a more visual look at what it just calculated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_sorted = [w for (p,w) in sorted([(p, w) for w, p in list(tfidf.vocabulary_.items())])]\n",
    "seaborn.heatmap(transformed, annot=True, fmt='.2f', cbar=False,\n",
    "            xticklabels=vocab_sorted, yticklabels=X_example, cmap='viridis');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There were not words selected that occur too often to matter at all. Try changing to `max_df=0.25` above and see what gets ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf.stop_words_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting With Text\n",
    "\n",
    "Let's remember the goal: given a text post, predict the newsgroup (topic) it came from.\n",
    "\n",
    "We will create a pipeline with a `TfidfVectorizer` that will do the tf-idf calculation for us and transform the text into numbers.\n",
    "\n",
    "Then we will use a Naive Baysean classifier like before, but we will use one that assumes a different probability distribution. The `GaussianNB` that we have been using assumes data follows a normal (Gaussian) distribution.\n",
    "\n",
    "For natural language problems, it's more common to assume the data follows a **multinomial distribution**, which is implemented by `MultinomialNB`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = make_pipeline(\n",
    "    TfidfVectorizer(),\n",
    "    MultinomialNB()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But as always, we train with the training data and test with the testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not a bad score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checking Our Model\n",
    "\n",
    "The accuracy score looks pretty good, but not perfect. Let's have a look at the model and see what kind of predictions it's making.\n",
    "\n",
    "The **confusion matrix** lets us look at exactly where predictions were right and wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion = confusion_matrix(y_test, model.predict(X_test), labels=model.classes_)\n",
    "confusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Large numbers along the diagonal are good: those were test values that were predicted correctly. Off the diagonal were predicted incorrectly.\n",
    "\n",
    "Or more visually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seaborn.heatmap(confusion, square=True, annot=True, fmt='d', cbar=False,\n",
    "            xticklabels=classes, yticklabels=classes, cmap='viridis')\n",
    "plot.xlabel('predicted label')\n",
    "plot.ylabel('true label');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you have seen before, the classification report can give you a values for precision (fraction found that were real) and recall (fraction of values in that category that were found)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print((classification_report(y_test, model.predict(X_test), target_names=classes)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also just test a few sample inputs and see if we get sensible results predicted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_category(s):\n",
    "    pred = model.predict([s])\n",
    "    return data.target_names[pred[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('that is one heck of an ADB port')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('that was a great game')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('my computer has a virus')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}