Module3_Lab3_FMML20210660.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SreenadhSingamaneni/IIITH-FMML20210660/blob/main/Module3_Lab3_FMML20210660.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAKrrEmgLZ_J"
      },
      "source": [
        "# **FOUNDATIONS OF MODERN MACHINE LEARNING, IIIT Hyderabad**\n",
        "### MODULE: CLASSIFICATION-1\n",
        "### LAB-3 : Using KNN for Text Classification\n",
        "#### Module Coordinator: Sahil Manoj Bhatt\n",
        "\n",
        "NOTE: YOU ONLY NEED TO MAKE CHANGES/WRITE CODE IN CELLS THAT SPECIFICALLY MENTION TASK-1, TASK-2, etc.\n",
        "\n",
        "WRITE ANY OBSERVATION(S), IF REQUIRED BY THE TASK, IN A SEPARATE CELL AT THE BOTTOM OF THE NOTEBOOK.  \n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wFOZHWV3_ABt",
        "outputId": "cc5fef33-52d4-4df6-fa8f-d0bdf3b22af5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# NLTK (or Natural Language Tool Kit) is a commonly used library for processing text.\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMJhTF89MmT_"
      },
      "source": [
        "### Data Cleaning and Preprocessing step"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRXVjJHk3urf"
      },
      "source": [
        "Raw text must be processed and converted into a form so that it is suitable to use with various machine-learning algorithms.  \n",
        "In case of text, there are lots of things that need to be taken into account.  \n",
        "\n",
        "\n",
        "1.   Removing numbers from the text\n",
        "2.   Handling capitalization and punctuation.\n",
        "3.   Stemming and Lemmatizing text.  \n",
        "\n",
        "And most importantly, one can't just use words or images directly in algorithms; they need to be converted into vectors- a form that algorithms can understand.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gpgq3SQK5IOr"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "import numpy\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.stem import SnowballStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk import pos_tag\n",
        "from nltk.corpus import wordnet\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "def cleanText(text, lemmatize, stemmer):\n",
        "    \"\"\"Method for cleaning text from train and test data. Removes numbers, punctuation, and capitalization. Stems or lemmatizes text.\"\"\"\n",
        "\n",
        "    if isinstance(text, float):\n",
        "        text = str(text)\n",
        "    if isinstance(text, numpy.int64):\n",
        "        text = str(text)\n",
        "    try:\n",
        "        text = text.decode()\n",
        "    except AttributeError:\n",
        "        pass\n",
        "\n",
        "    soup = BeautifulSoup(text, \"lxml\")\n",
        "    text = soup.get_text()\n",
        "    text = re.sub(r\"[^A-Za-z]\", \" \", text)\n",
        "    text = text.lower()\n",
        "\n",
        "\n",
        "    if lemmatize:\n",
        "        wordnet_lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "        def get_tag(tag):\n",
        "            if tag.startswith('J'):\n",
        "                return wordnet.ADJ\n",
        "            elif tag.startswith('V'):\n",
        "                return wordnet.VERB\n",
        "            elif tag.startswith('N'):\n",
        "                return wordnet.NOUN\n",
        "            elif tag.startswith('R'):\n",
        "                return wordnet.ADV\n",
        "            else:\n",
        "                return ''\n",
        "\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)  # Generate list of tokens\n",
        "        tagged = pos_tag(tokens)\n",
        "        for t in tagged:\n",
        "            try:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0], get_tag(t[1][:2])))\n",
        "            except:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0]))\n",
        "        return text_result\n",
        "\n",
        "    if stemmer:\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)\n",
        "        snowball_stemmer = SnowballStemmer('english')\n",
        "        for t in tokens:\n",
        "            text_result.append(snowball_stemmer.stem(t))\n",
        "        return text_result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZuBxfzbuzrs1",
        "outputId": "1333bd42-ca12-49d8-e449-70d55e5e9d3c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Troubling\n",
            "troubl\n",
            "trouble\n"
          ]
        }
      ],
      "source": [
        "sample_text = \"Troubling\"\n",
        "sample_text_result = cleanText(sample_text, lemmatize=False, stemmer=True)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text)\n",
        "print(sample_text_result)\n",
        "sample_text_result = cleanText(sample_text, lemmatize=True, stemmer=False)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcqQi34UoPvq"
      },
      "source": [
        "### BAG OF WORDS \n",
        "\n",
        "A bag-of-words model, or BoW for short, is a way of extracting features from text for use in modeling, such as with machine learning algorithms.\n",
        "\n",
        "The approach is very simple and flexible, and can be used in many ways for extracting features from documents.\n",
        "\n",
        "A bag-of-words is a representation of text that describes the occurrence of words within a document. \n",
        "It is called a “bag” of words, because any information about the order or structure of words in the document is discarded. The model is only concerned with whether known words occur in the document, not where in the document."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "3YYSpQzIM05l"
      },
      "outputs": [],
      "source": [
        "# Functions to convert document(s) to a list of words, with the option of removing stopwords. Returns document-term matrix.\n",
        "\n",
        "def createBagOfWords(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    bag_of_words_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    bag_of_words_test = vectorizer.transform(clean_test).toarray()\n",
        "    return bag_of_words_train, bag_of_words_test\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-v0iCUBqoX82"
      },
      "source": [
        "### TF-IDF \n",
        "TF-IDF technique is used to find meaning of sentences consisting of words and cancels out the incapabilities of Bag of Words technique which is good for text classification or for helping a machine read words in numbers.\n",
        "\n",
        "The number of times a term occurs in a document is called its Term frequency (TF).\n",
        "\n",
        " Document frequency is the number of documents in which the word is present.  Inverse DF (IDF) is the inverse of the document frequency which measures the informativeness of term *t*. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L6_-DDMQoXEX"
      },
      "outputs": [],
      "source": [
        "def createTFIDF(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = TfidfVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer =  TfidfVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    tfidf_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    tfidf_test = vectorizer.transform(clean_test).toarray()\n",
        "    return tfidf_train, tfidf_test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7g0jS45epcC5"
      },
      "source": [
        "### UNDERSTANDING THE DATA : A REVIEWS DATASET\n",
        "\n",
        "Sentiment analysis is the interpretation and classification of emotions (such as positive, negative and neutral) within text data using text analysis techniques.  \n",
        "Given below is a dataset consisting of reviews along with sentiment class (positive or negative)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jU6875-NxrHw"
      },
      "outputs": [],
      "source": [
        "# Upload the Reviews CSV file that has been shared with you.\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hmI-b6KkAApW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        },
        "outputId": "59314925-692b-4413-92d8-c48918d31a10"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-3f6e4894-f342-40ec-b8f2-025313505bfe\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Not sure who was more lost - the flat characte...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Attempting artiness with black &amp; white and cle...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Very little music or anything to speak of.</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>The best scene in the movie was when Gerardo i...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The rest of the movie lacks art, charm, meanin...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Wasted two hours.</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Saw the movie today and thought it was a good ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>A bit predictable.</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Loved the casting of Jimmy Buffet as the scien...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>And those baby owls were adorable.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>The movie showed a lot of Florida at it's best...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>The Songs Were The Best And The Muppets Were S...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>It Was So Cool.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>This is a very \"right on case\" movie that deli...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>It had some average acting from the main perso...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>This review is long overdue, since I consider ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>I'll put this gem up against any movie in term...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>It's practically perfect in all of them  a tr...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>\" The structure of this film is easily the mos...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>I can think of no other film where something v...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f6e4894-f342-40ec-b8f2-025313505bfe')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3f6e4894-f342-40ec-b8f2-025313505bfe button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3f6e4894-f342-40ec-b8f2-025313505bfe');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                             sentence  sentiment\n",
              "0   Not sure who was more lost - the flat characte...          0\n",
              "1   Attempting artiness with black & white and cle...          0\n",
              "2          Very little music or anything to speak of.          0\n",
              "3   The best scene in the movie was when Gerardo i...          1\n",
              "4   The rest of the movie lacks art, charm, meanin...          0\n",
              "5                                   Wasted two hours.          0\n",
              "6   Saw the movie today and thought it was a good ...          1\n",
              "7                                  A bit predictable.          0\n",
              "8   Loved the casting of Jimmy Buffet as the scien...          1\n",
              "9                  And those baby owls were adorable.          1\n",
              "10  The movie showed a lot of Florida at it's best...          1\n",
              "11  The Songs Were The Best And The Muppets Were S...          1\n",
              "12                                    It Was So Cool.          1\n",
              "13  This is a very \"right on case\" movie that deli...          1\n",
              "14  It had some average acting from the main perso...          0\n",
              "15  This review is long overdue, since I consider ...          1\n",
              "16  I'll put this gem up against any movie in term...          1\n",
              "17  It's practically perfect in all of them  a tr...          1\n",
              "18  \" The structure of this film is easily the mos...          1\n",
              "19  I can think of no other film where something v...          1"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('reviews.csv',sep='\\t')\n",
        "df.head(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QXo11Bxvytu"
      },
      "source": [
        "### KNN MODEL\n",
        "\n",
        "Given below are two KNN models; in the first case we are using Bag-of-Words and in the second case we are using TF-IDF.\n",
        "Note the different metrics and parameters used in each."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R5yZXboZ92Z4"
      },
      "outputs": [],
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 1: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv',sep='\\t')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=6, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='manhattan', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv',sep='\\t')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"],\n",
        "                                                        test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=6, weights='distance', algorithm='brute', leaf_size=30, p=2,\n",
        "                                         metric='manhattan', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uPuI3wrL-8ZJ"
      },
      "source": [
        "Note: Cross-validation will be discussed in detail in the upcoming lab session."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xzno1rRNHoFT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32492bce-7735-469e-e99b-1bac51f20f1d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with BOW accuracy = 65.5%\n",
            "Cross Validation Accuracy: 0.64\n",
            "[0.66292135 0.63533835 0.63157895]\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "## KNN accuracy after using BoW\n",
        "predicted, y_test = bow_knn()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FI5NMP8L-4eW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f394f87-1249-4b41-be7a-fa00d89b6156"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with TFIDF accuracy = 66.0%\n",
            "Cross Validation Accuracy: 0.62\n",
            "[0.70037453 0.59022556 0.56390977]\n"
          ]
        }
      ],
      "source": [
        "## KNN accuracy after using TFIDF\n",
        "predicted, y_test = tfidf_knn()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vm2vYDVfa5AP"
      },
      "source": [
        "### SPAM TEXT DATASET\n",
        "Now let's use what we've learnt to classify texts as spam or not spam."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c1GZgvXCsKX1"
      },
      "outputs": [],
      "source": [
        "# Upload the spam text data CSV file that has been shared with you.\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qRiS7dT7piTE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        },
        "outputId": "08cd497b-df65-4410-8c73-6e3990ba253b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-68601e6d-6ed1-4424-8030-1cb30b94b354\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Category</th>\n",
              "      <th>Message</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>spam</td>\n",
              "      <td>FreeMsg Hey there darling it's been 3 week's n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>ham</td>\n",
              "      <td>Even my brother is not like to speak with me. ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>ham</td>\n",
              "      <td>As per your request 'Melle Melle (Oru Minnamin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>spam</td>\n",
              "      <td>WINNER!! As a valued network customer you have...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>spam</td>\n",
              "      <td>Had your mobile 11 months or more? U R entitle...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>ham</td>\n",
              "      <td>I'm gonna be home soon and i don't want to tal...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>spam</td>\n",
              "      <td>SIX chances to win CASH! From 100 to 20,000 po...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>spam</td>\n",
              "      <td>URGENT! You have won a 1 week FREE membership ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>ham</td>\n",
              "      <td>I've been searching for the right words to tha...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>ham</td>\n",
              "      <td>I HAVE A DATE ON SUNDAY WITH WILL!!</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>spam</td>\n",
              "      <td>XXXMobileMovieClub: To use your credit, click ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>ham</td>\n",
              "      <td>Oh k...i'm watching here:)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>ham</td>\n",
              "      <td>Eh u remember how 2 spell his name... Yes i di...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>ham</td>\n",
              "      <td>Fine if thatÂ’s the way u feel. ThatÂ’s the wa...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>spam</td>\n",
              "      <td>England v Macedonia - dont miss the goals/team...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-68601e6d-6ed1-4424-8030-1cb30b94b354')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-68601e6d-6ed1-4424-8030-1cb30b94b354 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-68601e6d-6ed1-4424-8030-1cb30b94b354');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   Category                                            Message\n",
              "0       ham  Go until jurong point, crazy.. Available only ...\n",
              "1       ham                      Ok lar... Joking wif u oni...\n",
              "2      spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3       ham  U dun say so early hor... U c already then say...\n",
              "4       ham  Nah I don't think he goes to usf, he lives aro...\n",
              "5      spam  FreeMsg Hey there darling it's been 3 week's n...\n",
              "6       ham  Even my brother is not like to speak with me. ...\n",
              "7       ham  As per your request 'Melle Melle (Oru Minnamin...\n",
              "8      spam  WINNER!! As a valued network customer you have...\n",
              "9      spam  Had your mobile 11 months or more? U R entitle...\n",
              "10      ham  I'm gonna be home soon and i don't want to tal...\n",
              "11     spam  SIX chances to win CASH! From 100 to 20,000 po...\n",
              "12     spam  URGENT! You have won a 1 week FREE membership ...\n",
              "13      ham  I've been searching for the right words to tha...\n",
              "14      ham                I HAVE A DATE ON SUNDAY WITH WILL!!\n",
              "15     spam  XXXMobileMovieClub: To use your credit, click ...\n",
              "16      ham                         Oh k...i'm watching here:)\n",
              "17      ham  Eh u remember how 2 spell his name... Yes i di...\n",
              "18      ham  Fine if thatÂ’s the way u feel. ThatÂ’s the wa...\n",
              "19     spam  England v Macedonia - dont miss the goals/team..."
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('spam_text_data.csv', error_bad_lines=False)\n",
        "df.head(20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OsvBm4luNCME"
      },
      "outputs": [],
      "source": [
        "df['Category'] = df['Category'].map({'ham': 0, 'spam': 1})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KiEHuMypWyb6"
      },
      "outputs": [],
      "source": [
        "df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WRJU9rFy1XQR"
      },
      "outputs": [],
      "source": [
        "len(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wnv0v4T6sqJQ"
      },
      "outputs": [],
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 2: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam_text_data.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=6, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam_text_data.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=6, weights='distance', algorithm='brute', leaf_size=30, p=2, metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W8PwydHYs1h_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "19cc336c-a2c3-4754-dcea-161146b584ff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with BOW accuracy = 97.48878923766816%\n",
            "Cross Validation Accuracy: 0.96\n",
            "[0.95289367 0.95962315 0.95824916]\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = bow_knn()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "zf8i2P1nxpl8",
        "outputId": "46d58ef1-55b6-4807-bdff-741ca0d544f9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "KNN with TFIDF accuracy = 98.56502242152466%\n",
            "Cross Validation Accuracy: 0.97\n",
            "[0.96837147 0.96769852 0.96363636]\n"
          ]
        }
      ],
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = tfidf_knn()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9xvbzS4yLTr"
      },
      "source": [
        "### Questions to Think About and Answer\n",
        "1. Why does the TF-IDF approach generally result in a better accuracy than Bag-of-Words ?\n",
        "2. Can you think of techniques that are better than both BoW and TF-IDF ?\n",
        "3. Read about Stemming and Lemmatization from the resources given below. Think about the pros/cons of each."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Answers**\n",
        "\n",
        "**1st answer**\n",
        "Bag of Words just creates a set of vectors containing the count of word occurrences in the document  where as the TF-IDF model contains information on the more important words and the less important ones as well.\n",
        "\n",
        "**2nd answer**\n",
        "Binary,\n",
        "Latent Semantic Analysis,\n",
        "Word2Vec\n",
        "\n",
        "**3rd answer**\n",
        "stemming \n",
        "pros: \n",
        "reduce inflectional forms and sometimes derivationally related forms of a word to a common base form\n",
        "cons:\n",
        "there is lot of ambiguity which may cause several false positives.\n",
        "stemmers have a much higher error rate when compared with lemmatization \n",
        "\n",
        "lemmatization:\n",
        "pros:\n",
        " it takes into consideration the context of the word to determine which is the intended meaning the user is looking for. This process allows to decrease noise and speed up the user’s task.\n",
        "cons:\n",
        "it is slower when compared with stemmers\n"
      ],
      "metadata": {
        "id": "a0GDujJAX_LC"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T6xPL6smyWG-"
      },
      "source": [
        "### Useful Resources for further reading\n",
        "1. Stemming and Lemmatization: https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html\n",
        "2. TF-IDF and BoW : https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/\n",
        "3. TF-IDF: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Module3_Lab3_FMML20210660.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}