OkNLP-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OkNLP\n",
    "\n",
    "This notebook demonstrates the algorithm we used in our project. It shows an example of how we clustered using Nonnegative Matrix Factorization. We manually inspect the output of NMF to determine the best number of clusters for each group. Then, we create word clouds for specific groups and demographic splits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.sparse import hstack\n",
    "from sklearn.cross_validation import cross_val_predict\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "from utils.categorize_demographics import recategorize\n",
    "from utils.clean_up import clean_up, col_to_data_matrix\n",
    "from utils.distinctive_tokens import log_odds_ratio\n",
    "from utils.happyfuntokenizing import Tokenizer\n",
    "from utils.nonnegative_matrix_factorization import nmf_labels\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "essay_dict = {'essay0' : 'My self summary',\n",
    "              'essay1' : 'What I\\'m doing with my life',\n",
    "              'essay2' : 'I\\'m really good at',\n",
    "              'essay3' : 'The first thing people notice about me',\n",
    "              'essay4' : 'Favorite books, movies, tv, food',\n",
    "              'essay5' : 'The six things I could never do without',\n",
    "              'essay6' : 'I spend a lot of time thinking about',\n",
    "              'essay7' : 'On a typical Friday night I am',\n",
    "              'essay8' : 'The most private thing I am willing to admit',\n",
    "              'essay9' : 'You should message me if'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n",
    "\n",
    "First we read in the data frame and re-categorize some of the demographic information. \n",
    "\n",
    "We'll have two separate dataframes, one for essay0 and one for essay4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/profiles.20120630.csv')\n",
    "\n",
    "essay_list = ['essay4']\n",
    "df_4 = clean_up(df, essay_list)\n",
    "df_4 = recategorize(df_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_4_y = df_4[df_4.drugs == 'yes'] #take only users with yes/no drug status\n",
    "df_4_n = df_4[df_4.drugs == 'no']\n",
    "\n",
    "df_4_y = df_4_y.sample(6500, random_state=42) #subsample data for both y and no\n",
    "df_4_n = df_4_n.sample(6500, random_state=42)\n",
    "\n",
    "drugs = df_4_y.append(df_4_n) #combine dfs \n",
    "\n",
    "drugs['y'] = drugs['drugs'].apply(lambda x: 1 if x == 'yes' else 0) #add column for 1/0 if drug use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering\n",
    "\n",
    "Convert the users' essays into a tfidf matrix and use NMF to cluster the data points into 25 groups.\n",
    "\n",
    "\n",
    "Vocabulary includes unigrams, bigrams, and trigrams without redundancies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "K = 25\n",
    "count_matrix, tfidf_matrix, vocab = col_to_data_matrix(drugs, 'essay4', min_df=0.001)\n",
    "drugs['group'] = nmf_labels(tfidf_matrix, K) #group assignment per user (group with maximum weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Featurize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = drugs.y.values #1/0 vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = tfidf_matrix.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Log Odds Ratio features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "count_0 = count_matrix[np.array(drugs.drugs=='yes'), :].sum(axis=0)\n",
    "count_1 = count_matrix[np.array(drugs.drugs=='no'), :].sum(axis=0)\n",
    "counts = np.array(np.vstack((count_0, count_1)))\n",
    "log_odds = log_odds_ratio(counts, vocab, use_variance=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "n = 2000\n",
    "top = log_odds.sort('log_odds_ratio', ascending=False)['features'].tolist()[:n]\n",
    "bottom = log_odds.sort('log_odds_ratio', ascending=False)['features'].tolist()[-n:]\n",
    "log_odds_features = top + bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "log_odds_mask = np.array([t in log_odds_features for t in vocab])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = X[:,log_odds_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NMF features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# nmf = pd.get_dummies(drugs.group, prefix='nmf').values\n",
    "# X = hstack([X, nmf], format='csr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Validated Estimates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic Regression, naive Bayes, SVM, Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clf0 = LogisticRegression()\n",
    "clf1 = MultinomialNB()\n",
    "clf2 = LinearSVC()\n",
    "clf3 = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7293 [Logistic Regression]\n",
      "Accuracy: 0.7326 [naive Bayes]\n",
      "Accuracy: 0.7299 [SVM]\n",
      "Accuracy: 0.6456 [Random Forest]\n"
     ]
    }
   ],
   "source": [
    "for clf, name in zip([clf0, clf1, clf2, clf3],\n",
    "                     ['Logistic Regression', 'naive Bayes', 'SVM', 'Random Forest']):\n",
    "    yhat = cross_val_predict(clf, X, y, cv=10)\n",
    "    print(\"Accuracy: %0.4f [%s]\" % (accuracy_score(y, yhat), name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without feature selection:\n",
      "\n",
      "Accuracy: 0.6715 [Logistic Regression]\n",
      "Accuracy: 0.6738 [naive Bayes]\n",
      "Accuracy: 0.6387 [SVM]\n",
      "Accuracy: 0.6305 [Random Forest]\n"
     ]
    }
   ],
   "source": [
    "print(\"\"\"Without feature selection:\n",
    "\n",
    "Accuracy: 0.6715 [Logistic Regression]\n",
    "Accuracy: 0.6738 [naive Bayes]\n",
    "Accuracy: 0.6387 [SVM]\n",
    "Accuracy: 0.6305 [Random Forest]\"\"\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.3.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}