data_cleaning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "data-cleaning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9_jO2mrKWM_U"
      },
      "source": [
        "### Notebook part 1\n",
        "\n",
        "# Data Cleaning\n",
        "_By **Avi Patel**_\n",
        "\n",
        "## Overview\n",
        "This project seeks to create a model that creates one line abstractive summary of a given news article. This model will help to get a quick idea on what the article is all about without having to read all whole news.\n",
        "\n",
        "\n",
        "## Business Problem\n",
        "News/Media/Blogging Companies always wants a catchy  headlines for the articles they post so that they can get maximum number of clicks. A machine learning algorithm can be  used to create a headline based on the context of the article.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ESeYmNwDrfch"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re\n",
        "!pip install contractions\n",
        "import contractions\n",
        "from time import time\n",
        "import spacy\n",
        "spacy.prefer_gpu()\n",
        "nlp = spacy.load('en', disable=['ner', 'parser']) # disabling Named Entity Recognition for speed"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WABglY4TrmFy"
      },
      "source": [
        "df1 = pd.read_csv('/content/drive/MyDrive/news_summary.csv', encoding='iso-8859-1')\n",
        "df2 = pd.read_csv('/content/drive/MyDrive/news_summary_more.csv', encoding='iso-8859-1')"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        },
        "id": "pILiDEgl7GgP",
        "outputId": "de9e169a-ae21-49d5-a339-c61cca5d207e"
      },
      "source": [
        "df1.head(2)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>author</th>\n",
              "      <th>date</th>\n",
              "      <th>headlines</th>\n",
              "      <th>read_more</th>\n",
              "      <th>text</th>\n",
              "      <th>ctext</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Chhavi Tyagi</td>\n",
              "      <td>03 Aug 2017,Thursday</td>\n",
              "      <td>Daman &amp; Diu revokes mandatory Rakshabandhan in...</td>\n",
              "      <td>http://www.hindustantimes.com/india-news/raksh...</td>\n",
              "      <td>The Administration of Union Territory Daman an...</td>\n",
              "      <td>The Daman and Diu administration on Wednesday ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Daisy Mowke</td>\n",
              "      <td>03 Aug 2017,Thursday</td>\n",
              "      <td>Malaika slams user who trolled her for 'divorc...</td>\n",
              "      <td>http://www.hindustantimes.com/bollywood/malaik...</td>\n",
              "      <td>Malaika Arora slammed an Instagram user who tr...</td>\n",
              "      <td>From her special numbers to TV?appearances, Bo...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         author  ...                                              ctext\n",
              "0  Chhavi Tyagi  ...  The Daman and Diu administration on Wednesday ...\n",
              "1   Daisy Mowke  ...  From her special numbers to TV?appearances, Bo...\n",
              "\n",
              "[2 rows x 6 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "p3axndCb7RXh",
        "outputId": "af9a28d9-8036-46ef-c6cc-7b2436afd779"
      },
      "source": [
        "df2.head(2)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>headlines</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>upGrad learner switches to career in ML &amp; Al w...</td>\n",
              "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Delhi techie wins free food from Swiggy for on...</td>\n",
              "      <td>Kunal Shah's credit card bill payment platform...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                           headlines                                               text\n",
              "0  upGrad learner switches to career in ML & Al w...  Saurav Kant, an alumnus of upGrad and IIIT-B's...\n",
              "1  Delhi techie wins free food from Swiggy for on...  Kunal Shah's credit card bill payment platform..."
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tmmL1yUmw2b1"
      },
      "source": [
        "pre1 = df1.copy()\n",
        "pre2 = df2.copy()\n",
        "pre1['text'] =  pre1['text'].str.cat(pre1['ctext'], sep = \" \")"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lKwaNLq0yqV9"
      },
      "source": [
        "df = pd.DataFrame()\n",
        "df['text'] = pd.concat([pre2['text'], pre1['text']], ignore_index=True)\n",
        "df['summary'] = pd.concat([pre2['headlines'],pre1['headlines']],ignore_index = True)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "5sQVApvu9M62",
        "outputId": "a61ab237-972c-49e3-a912-8c92e606ab29"
      },
      "source": [
        "df.shape"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(102915, 2)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "1nbssKSA9QBe",
        "outputId": "9dd209a9-c6f8-4fc7-ca8f-b85a3a466337"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>summary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's...</td>\n",
              "      <td>upGrad learner switches to career in ML &amp; Al w...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Kunal Shah's credit card bill payment platform...</td>\n",
              "      <td>Delhi techie wins free food from Swiggy for on...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>New Zealand defeated India by 8 wickets in the...</td>\n",
              "      <td>New Zealand end Rohit Sharma-led India's 12-ma...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>With Aegon Life iTerm Insurance plan, customer...</td>\n",
              "      <td>Aegon life iTerm insurance plan helps customer...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Speaking about the sexual harassment allegatio...</td>\n",
              "      <td>Have known Hirani for yrs, what if MeToo claim...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text                                            summary\n",
              "0  Saurav Kant, an alumnus of upGrad and IIIT-B's...  upGrad learner switches to career in ML & Al w...\n",
              "1  Kunal Shah's credit card bill payment platform...  Delhi techie wins free food from Swiggy for on...\n",
              "2  New Zealand defeated India by 8 wickets in the...  New Zealand end Rohit Sharma-led India's 12-ma...\n",
              "3  With Aegon Life iTerm Insurance plan, customer...  Aegon life iTerm insurance plan helps customer...\n",
              "4  Speaking about the sexual harassment allegatio...  Have known Hirani for yrs, what if MeToo claim..."
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfgjU_ruBd87"
      },
      "source": [
        "**Cleaning Text and Summary**\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pmdkT_4Vy88z"
      },
      "source": [
        "def text_clean(column):\n",
        "    for row in column:\n",
        "        \n",
        "        #REGEX IN ORDER\n",
        "        \n",
        "        row=re.sub(\"(\\\\t)\", ' ', str(row)).lower() #remove escape charecters\n",
        "        row=re.sub(\"(\\\\r)\", ' ', str(row)).lower() \n",
        "        row=re.sub(\"(\\\\n)\", ' ', str(row)).lower()\n",
        "        \n",
        "        row=re.sub(\"(__+)\", ' ', str(row)).lower()   #remove _ if it occors more than one time consecutively\n",
        "        row=re.sub(\"(--+)\", ' ', str(row)).lower()   #remove - if it occors more than one time consecutively\n",
        "        row=re.sub(\"(~~+)\", ' ', str(row)).lower()   #remove ~ if it occors more than one time consecutively\n",
        "        row=re.sub(\"(\\+\\++)\", ' ', str(row)).lower()   #remove + if it occors more than one time consecutively\n",
        "        row=re.sub(\"(\\.\\.+)\", ' ', str(row)).lower()   #remove . if it occors more than one time consecutively\n",
        "        \n",
        "        row=re.sub(r\"[<>()|&©ø\\[\\]\\'\\\",;?~*!]\", ' ', str(row)).lower() #remove <>()|&©ø\"',;?~*!\n",
        "        \n",
        "        row=re.sub(\"(mailto:)\", ' ', str(row)).lower() #remove mailto:\n",
        "        row=re.sub(r\"(\\\\x9\\d)\", ' ', str(row)).lower() #remove \\x9* in text\n",
        "        row=re.sub(\"([iI][nN][cC]\\d+)\", 'INC_NUM', str(row)).lower() #replace INC nums to INC_NUM\n",
        "        row=re.sub(\"([cC][mM]\\d+)|([cC][hH][gG]\\d+)\", 'CM_NUM', str(row)).lower() #replace CM# and CHG# to CM_NUM\n",
        "        \n",
        "        \n",
        "        row=re.sub(\"(\\.\\s+)\", ' ', str(row)).lower() #remove full stop at end of words(not between)\n",
        "        row=re.sub(\"(\\-\\s+)\", ' ', str(row)).lower() #remove - at end of words(not between)\n",
        "        row=re.sub(\"(\\:\\s+)\", ' ', str(row)).lower() #remove : at end of words(not between)\n",
        "        \n",
        "        row=re.sub(\"(\\s+.\\s+)\", ' ', str(row)).lower() #remove any single charecters hanging between 2 spaces\n",
        "        \n",
        "        #Replace any url as such https://abc.xyz.net/browse/sdf-5327 ====> abc.xyz.net\n",
        "        try:\n",
        "            url = re.search(r'((https*:\\/*)([^\\/\\s]+))(.[^\\s]+)', str(row))\n",
        "            repl_url = url.group(3)\n",
        "            row = re.sub(r'((https*:\\/*)([^\\/\\s]+))(.[^\\s]+)',repl_url, str(row))\n",
        "        except:\n",
        "            pass #there might be emails with no url in them\n",
        "        \n",
        "        row = re.sub(\"(\\s+)\",' ',str(row)).lower() #remove multiple spaces\n",
        "        \n",
        "        #Should always be last\n",
        "        row=re.sub(\"(\\s+.\\s+)\", ' ', str(row)).lower() #remove any single charecters hanging between 2 spaces\n",
        "\n",
        "        yield row"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gsHgOgW9y_7c"
      },
      "source": [
        "cleaned_text = text_clean(df['text'])\n",
        "cleaned_summary = text_clean(df['summary'])"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H506RNC-zC_E",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "9c282d65-7b14-4986-81d7-160652f3188d"
      },
      "source": [
        "t = time()\n",
        "\n",
        "text = [str(doc) for doc in nlp.pipe(cleaned_text, batch_size=5000, n_threads=-1)]\n",
        "\n",
        "print('Time to clean up everything: {} mins'.format(round((time() - t) / 60, 2)))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time to clean up everything: 4.66 mins\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vOJejmHl0aNU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "7b0254ad-888d-4124-9954-a582293085a5"
      },
      "source": [
        "t = time()\n",
        "\n",
        "summary = [str(doc) for doc in nlp.pipe(cleaned_summary, batch_size=5000, n_threads=-1)]\n",
        "\n",
        "print('Time to clean up everything: {} mins'.format(round((time() - t) / 60, 2)))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time to clean up everything: 0.89 mins\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-OvnAigSzcQ2"
      },
      "source": [
        "df['cleaned_text'] = pd.Series(text)\n",
        "df['cleaned_summary'] = pd.Series(summary)\n"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nndigrj_CimB"
      },
      "source": [
        "**Contractions to Expansion**\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UmadQOUADaJn"
      },
      "source": [
        "df.cleaned_summary = df.cleaned_summary.apply(lambda x: contractions.fix(x))"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CdaYNWy2Djh6"
      },
      "source": [
        "df.cleaned_text = df.cleaned_text.apply(lambda x: contractions.fix(x))"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "z3ugL_Ao_qZ4",
        "outputId": "45ae4ec5-122c-4cdd-c2da-7c81951175c8"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>summary</th>\n",
              "      <th>cleaned_text</th>\n",
              "      <th>cleaned_summary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's...</td>\n",
              "      <td>upGrad learner switches to career in ML &amp; Al w...</td>\n",
              "      <td>saurav kant an alumnus of upgrad and iiit-b pg...</td>\n",
              "      <td>upgrad learner switches to career in ml al wit...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Kunal Shah's credit card bill payment platform...</td>\n",
              "      <td>Delhi techie wins free food from Swiggy for on...</td>\n",
              "      <td>kunal shah credit card bill payment platform c...</td>\n",
              "      <td>delhi techie wins free food from swiggy for on...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>New Zealand defeated India by 8 wickets in the...</td>\n",
              "      <td>New Zealand end Rohit Sharma-led India's 12-ma...</td>\n",
              "      <td>new zealand defeated india by wickets in the f...</td>\n",
              "      <td>new zealand end rohit sharma-led india 12-matc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>With Aegon Life iTerm Insurance plan, customer...</td>\n",
              "      <td>Aegon life iTerm insurance plan helps customer...</td>\n",
              "      <td>with aegon life iterm insurance plan customers...</td>\n",
              "      <td>aegon life iterm insurance plan helps customer...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Speaking about the sexual harassment allegatio...</td>\n",
              "      <td>Have known Hirani for yrs, what if MeToo claim...</td>\n",
              "      <td>speaking about the sexual harassment allegatio...</td>\n",
              "      <td>have known hirani for yrs what if metoo claims...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  ...                                    cleaned_summary\n",
              "0  Saurav Kant, an alumnus of upGrad and IIIT-B's...  ...  upgrad learner switches to career in ml al wit...\n",
              "1  Kunal Shah's credit card bill payment platform...  ...  delhi techie wins free food from swiggy for on...\n",
              "2  New Zealand defeated India by 8 wickets in the...  ...  new zealand end rohit sharma-led india 12-matc...\n",
              "3  With Aegon Life iTerm Insurance plan, customer...  ...  aegon life iterm insurance plan helps customer...\n",
              "4  Speaking about the sexual harassment allegatio...  ...  have known hirani for yrs what if metoo claims...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkSjM4lcDqie"
      },
      "source": [
        "**Save Cleaned DataFrame**\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lTHXhSzCAI6k"
      },
      "source": [
        "#df.to_csv('cleaned_news_summary.csv', index=False)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "udlJ_LI-WR1M"
      },
      "source": [
        "**Next Step**\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "The Next Step Involves modeling of the data using Seq2Se1 LSTM which can be found in (LSTM.ipynb)"
      ]
    }
  ]
}