ang-ex6-spam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spam Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.io as io\n",
    "import sklearn.svm as svm # For SVM.\n",
    "import re # For matching regular expressions.\n",
    "import nltk.tokenize as tkn # For tokenizing.\n",
    "import nltk.stem.porter as stem # For stemming.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "emailSample1.txt:\n",
      "> Anyone knows how much it costs to host a web portal ?\r\n",
      ">\r\n",
      "Well, it depends on how many visitors you're expecting.\r\n",
      "This can be anywhere from less than 10 bucks a month to a couple of $100. \r\n",
      "You should checkout http://www.rackspace.com/ or perhaps Amazon EC2 \r\n",
      "if youre running something big..\r\n",
      "\r\n",
      "To unsubscribe yourself from this mailing list, send an email to:\r\n",
      "groupname-unsubscribe@egroups.com\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "# Glimpse of the data.\n",
    "print(\"emailSample1.txt:\")\n",
    "!cat data/emailSample1.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Email preprocessing\n",
    "- Lower-casing: The entire email is converted into lower case.\n",
    "- Stripping HTML: All HTML tags are removed from the emails. \n",
    "- Normalizing URLs: All URLs are replaced with the text “httpaddr”.\n",
    "- Normalizing Email Addresses: All email addresses are replaced with the text “emailaddr”.\n",
    "- Normalizing Numbers: All numbers are replaced with the text \"number\".\n",
    "- Normalizing Dollars: All dollar signs are replaced with the text “dollar”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessor(email):\n",
    "    # Lower casing\n",
    "    email = email.lower()\n",
    "    \n",
    "    # Stripping HTML tags (<..> .... </..>)\n",
    "    email = re.sub('<[^<>]+>', ' ', email)\n",
    "    \n",
    "    # Normalizing URLs\n",
    "    email = re.sub('(http|https)://[^\\s]*', 'httpaddr', email)\n",
    "    \n",
    "    # Normalizing Email Adrresses\n",
    "    email = re.sub('[^\\s]+@[^\\s]+', 'emailaddr', email)\n",
    "    \n",
    "    # Normalizing Numbers\n",
    "    email = re.sub('[0-9]+', 'number', email)\n",
    "    \n",
    "    # Normalizig Dollars\n",
    "    email = re.sub('[$]+', 'dollar', email)\n",
    "    \n",
    "    return email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing and stemming\n",
    "\n",
    "The following function takes in a raw email, preprocesses it using the above preprocessor, then tokenizes it, stems each word and returns an ordered list of tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def email_to_tokens(email):\n",
    "    # Preprocessing\n",
    "    email = preprocessor(email)\n",
    "    \n",
    "    # Tokenizing\n",
    "    tokenizer = tkn.RegexpTokenizer('\\w+')\n",
    "    tokens = tokenizer.tokenize(email)\n",
    "    \n",
    "    # Stemming\n",
    "    stemmer = stem.PorterStemmer()\n",
    "    stemmed_tokens = [stemmer.stem(token) for token in tokens]\n",
    "    \n",
    "    return sorted(set(stemmed_tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary and indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabulary = {}\n",
    "with open('data/vocab.txt') as f:\n",
    "    for line in f:\n",
    "        val, key = line.split()\n",
    "        vocabulary[key] = int(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we convert the tokens in the processed email to the indices corresponding to those in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokens_to_indices(email, vocabulary):\n",
    "    tokens = email_to_tokens(email)\n",
    "    indices = [vocabulary[token] for token in tokens if token in vocabulary]\n",
    "    return indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[71, 86, 89, 162, 181, 238, 370, 375, 431, 479, 530, 531, 592, 688, 790, 794, 799, 810, 883, 916, 945, 961, 992, 1002, 1062, 1077, 1120, 1162, 1171, 1182, 1237, 1364, 1440, 1477, 1510, 1547, 1663, 1676, 1699, 1758, 1822, 1831, 1893, 1895, 1896]\n"
     ]
    }
   ],
   "source": [
    "email = open('data/emailSample1.txt').read()\n",
    "print(tokens_to_indices(email, vocabulary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature vector creation\n",
    "The feature vectors show if a word in the email is present in the vocabulary. The function below will create a feature vector from a list of indices, if the $i$-th word from the vocabulary appears in the email then the corresponding entry in the vector will be $1$, else $0$.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feature_vector_from_email(email, vocabulary):\n",
    "    total_words = len(vocabulary)\n",
    "    email_indices = tokens_to_indices(email, vocabulary)\n",
    "    feature_vector = np.zeros(total_words)\n",
    "    feature_vector[email_indices] = 1\n",
    "    return feature_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of feature vector: 1899\n",
      "Number of non-zero entries: 45\n"
     ]
    }
   ],
   "source": [
    "email = open('data/emailSample1.txt').read()\n",
    "fv = feature_vector_from_email(email, vocabulary)\n",
    "print(f'Length of feature vector: {fv.size}')\n",
    "print(f'Number of non-zero entries: {int(np.sum(fv))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM as spam classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data\n",
    "training_data = io.loadmat(os.path.join('data', 'spamTrain.mat'))\n",
    "X, y = training_data['X'], training_data['y'].ravel()\n",
    "\n",
    "test_data = io.loadmat(os.path.join('data', 'spamTest.mat'))\n",
    "Xtest, ytest = test_data['Xtest'], test_data['ytest'].ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of training emails 4000\n",
      "Total number of spam emails 1277\n",
      "Total number of nonspam emails 2723\n"
     ]
    }
   ],
   "source": [
    "pos, neg = X[y==1], X[y==0]\n",
    "print(f'Total number of training emails {X.shape[0]}')\n",
    "print(f'Total number of spam emails {pos.shape[0]}')\n",
    "print(f'Total number of nonspam emails {neg.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=0.1, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto_deprecated',\n",
       "  kernel='linear', max_iter=-1, probability=False, random_state=None,\n",
       "  shrinking=True, tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Linear SVM, with C = 0.1\n",
    "linear_svm = svm.SVC(C=0.1, kernel='linear')\n",
    "linear_svm.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training accuracy is 99.825.\n",
      "The test accuracy is 98.9.\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "training_prediction = linear_svm.predict(X)\n",
    "test_prediction = linear_svm.predict(Xtest)\n",
    "\n",
    "training_accuracy = (np.sum(training_prediction == y) / y.size) * 100\n",
    "test_accuracy = (np.sum(test_prediction == ytest) / ytest.size) * 100\n",
    "\n",
    "print(f'The training accuracy is {training_accuracy}.')\n",
    "print(f'The test accuracy is {test_accuracy}.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top predictors for spam\n",
    "Now we check which words, according to the classifier, are the most predictive of spam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The fifteen top predictors for spam are:\n",
      "otherwis clearli remot gt visa base doesn wife previous player mortgag natur ll futur hot\n",
      "The fifteen bottom predictors for spam are:\n",
      "http toll xp ratio august unsubscrib useless numberth round linux datapow wrong urgent that spam\n"
     ]
    }
   ],
   "source": [
    "flipped_vocabulary = {k:v for v,k in vocabulary.items()}\n",
    "sorted_indices = np.argsort(linear_svm.coef_, axis=None)[::-1]\n",
    "\n",
    "print('The fifteen top predictors for spam are:')\n",
    "print(*[flipped_vocabulary[index] for index in sorted_indices[:15]])\n",
    "\n",
    "print('The fifteen bottom predictors for spam are:')\n",
    "print(*[flipped_vocabulary[index] for index in sorted_indices[-15:]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional predictions\n",
    "def prediction(email_file):\n",
    "    email = open(email_file).read()\n",
    "    fv = feature_vector_from_email(email, vocabulary)\n",
    "    prediction = linear_svm.predict(fv.reshape(1, -1))\n",
    "    if (prediction[0] == 1):\n",
    "        print('Spam')\n",
    "    else:\n",
    "        print('Not Spam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not Spam\n"
     ]
    }
   ],
   "source": [
    "prediction('data/emailSample2.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not Spam\n"
     ]
    }
   ],
   "source": [
    "prediction('data/spamSample1.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spam\n"
     ]
    }
   ],
   "source": [
    "prediction('data/spamSample2.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus the classifier correctly classifies the second non-spam and spam email samples but fails to classify the first spam email sample."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}