wwps_sentenceBERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb338ad0",
   "metadata": {
    "id": "eb338ad0"
   },
   "source": [
    "<a name =\"top\"></a>\n",
    "# Welcome to WWPS (What would Presidents say?)!\n",
    "\n",
    "We have access to all the US Presidential Speeches from April 30th, 1789 to Sept. 25th, 2019. Presidential speeches are an opportunity for Presidents to set the tone of the country, put forth their agenda, influence policy and sway public opinion. This dataset can give us some good insight into what a President would say about various topics. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ce831a",
   "metadata": {},
   "source": [
    "# Challenge: Build a pipeline that, when given a sentence, would spit out the top 3 similar sentences previoulsy uttered by different US presidents.\n",
    "\n",
    "\n",
    "So in other words we're going to create WWPS i.e. \"What would Presidents say?\" \n",
    "\n",
    "Source data: https://www.kaggle.com/littleotter/united-states-presidential-speeches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d60077",
   "metadata": {
    "id": "34d60077"
   },
   "source": [
    "# Results\n",
    "\n",
    "\n",
    "\n",
    "### Sentence Embedding Model using SentenceBERT\n",
    "1. The results were good! The top 3 sentences accurately conveyed the meaning of the input sentence.\n",
    "    - query = \"I believe I our children are our future.\"\n",
    "2. The top 3 results gave a **cosine similarity of 0.889 to 0.956**\n",
    "3. The model took over 24 hours to run. \n",
    "4. The long run time of the model prevented further exploration of the model for example with different queries..\n",
    "\n",
    "### Word Embedding Model using GloVe\n",
    "1. The results did **not** resemble the query in spite of the soft cosine similarity being 0.982 to 1.0.\n",
    "    - query = 'I believe that the young people in America are our future.'\n",
    "2. The model was much less complex.\n",
    "3. The model did not convey the meaning of the input sentences. \n",
    "\n",
    "\n",
    "### Limitations\n",
    "\n",
    "1. The dataset requires more lot of cleaning.\n",
    "    - Some was attempted. But for the most part this would require significantly more time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2699722",
   "metadata": {
    "id": "a2699722"
   },
   "source": [
    "# TABLE OF CONTENTS\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39bd972",
   "metadata": {
    "id": "d39bd972"
   },
   "source": [
    "PART A. [EXPLORING THE DATASET](#step0)  \n",
    "[Step 0: Solution Approach](#stepm)   \n",
    "[Step i: Import Libraries and Load the Data](#stepi)  \n",
    "[Step ii: Work with a subset](#stepii)  \n",
    "[Step iii: Characteristics of the dataset](#stepiii)   \n",
    "[Step iv: Data Cleaning](#stepiv)\n",
    "\n",
    "PART B. [SENTENCE EMBEDDING](#stepb)  \n",
    "1. [Define Cosine Similarity Function](#step1)\n",
    "2. [SentenceBERT](#step2)\n",
    "3. [Results](#results_bert)\n",
    "\n",
    "PART C. [WORD EMBEDDING](#stepc)  \n",
    "1. [GLoVe](#step3) \n",
    "2. [Results](#results_glove)\n",
    "\n",
    "PART D. LIMITATIONS & REFERENCES  \n",
    "1. [Limitations](#limitations)\n",
    "2. [References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aabdf898",
   "metadata": {
    "id": "aabdf898"
   },
   "source": [
    "<a name =step0></a>\n",
    "# PART A - EXPLORING THE DATASET\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebeb68b",
   "metadata": {},
   "source": [
    "<a name =stepm></a>\n",
    "    \n",
    "# Step 0: Solution Approach\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d816333a",
   "metadata": {},
   "source": [
    "#### Understanding the problem\n",
    "The first thing we need to do is understand the type of problem we are solving. The main features of this problem are:\n",
    "\n",
    "- Unsupervised Learning\n",
    "    - The data is not labelled and we don't have a test set to validate our results.\n",
    "- Text Similarity\n",
    "    - We need to compare whether one set of text with another and determine their similarity.\n",
    "- Information Retrieval\n",
    "    - We need to fetch relevant sources of information from a corpus just like a quer or search on a search engine. Now that we know that we understand which steps to take to develop a suitable algorithm.\n",
    "\n",
    "#### Text Similarity Approach\n",
    "\n",
    "There are 2 main ways to analyse Text Similarity; Lexical Similarity and Semantical Similarity. We will select Semantical Similarity as in addition to syntax, the algorithm will consider context.\n",
    "\n",
    "The steps involved in Text Similarity:\n",
    "\n",
    "- Information Retrieval\n",
    "    - We need to define search term(query) and find a way to get input from the user.\n",
    "- Text Normalization\n",
    "    - Tokenize the sentences.\n",
    "    - Develop a method that can be used for both the search_ and the corpus.\n",
    "- Feature Engineering\n",
    "    - Our options here include Bag of Words, TF-IDF and Word Vectorization\n",
    "    - We will use a Word Embebbding Model and a Sentence Embedding Model\n",
    "- Similarity Measure\n",
    "    - We need to find an optimal measure.\n",
    "    - Options include Hellinger-Bhattcharya Distance and Okapi BM25 ranking\n",
    "    - We will use Soft Cosine Measure"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bd0eeb7",
   "metadata": {
    "id": "3bd0eeb7"
   },
   "source": [
    "<a name =stepi></a>\n",
    "    \n",
    "# Step i: Import Libraries and Load the Data\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a7bef5d4",
   "metadata": {
    "id": "a7bef5d4"
   },
   "outputs": [],
   "source": [
    "#import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import re\n",
    "import numpy as np\n",
    "\n",
    "from io import StringIO\n",
    "\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "from sklearn import svm\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "VwP_wAD3rBDu",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VwP_wAD3rBDu",
    "outputId": "a5f8c296-08b3-44b0-e90d-da0b2dc8a76c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "mOH0KXiU3Ulq",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mOH0KXiU3Ulq",
    "outputId": "f66bb7af-0cd3-43f8-8cbf-db209ab0873f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "53fdf209",
   "metadata": {
    "id": "53fdf209"
   },
   "outputs": [],
   "source": [
    "def read_corpus():\n",
    "    corpus_df = pd.read_csv('corpus.csv')\n",
    "    return corpus_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "377e51d9",
   "metadata": {
    "id": "377e51d9"
   },
   "outputs": [],
   "source": [
    "corpus_df = read_corpus()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0faee5b6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "0faee5b6",
    "outputId": "512a2945-e7a9-40d6-e89e-e371c0cb082e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>John Quincy Adams</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Andrew Jackson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: About to undertake the arduou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Martin Van Buren</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: The practice of all my predec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>William Harrison</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Called from a retirement which I had supposed ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>John Tyler</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>To the People of the United States Before my a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Unnamed: 0                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "5  John Quincy Adams  Democratic-Republican   \n",
       "6     Andrew Jackson             Democratic   \n",
       "7   Martin Van Buren             Democratic   \n",
       "8   William Harrison                   Whig   \n",
       "9         John Tyler           Unaffiliated   \n",
       "\n",
       "                                         transcripts  \n",
       "0  Fellow Citizens of the Senate and the House of...  \n",
       "1  When it was first perceived, in early times, t...  \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...  \n",
       "3  Unwilling to depart from examples of the most ...  \n",
       "4  I should be destitute of feeling if I was not ...  \n",
       "5  AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...  \n",
       "6  Fellow Citizens: About to undertake the arduou...  \n",
       "7  Fellow Citizens: The practice of all my predec...  \n",
       "8  Called from a retirement which I had supposed ...  \n",
       "9  To the People of the United States Before my a...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df.iloc[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "132741f7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "132741f7",
    "outputId": "a79866cc-c817-4403-d0ad-afd2cbbb5076",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           President                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "\n",
       "                                         transcripts  \n",
       "0  Fellow Citizens of the Senate and the House of...  \n",
       "1  When it was first perceived, in early times, t...  \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...  \n",
       "3  Unwilling to depart from examples of the most ...  \n",
       "4  I should be destitute of feeling if I was not ...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df.rename( columns={\"Unnamed: 0\" :\"President\"}, inplace=True )\n",
    "corpus_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e402d70d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e402d70d",
    "outputId": "86252e2c-cf08-4f79-aec4-5d58d9ee414d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(44, 3)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c112e8",
   "metadata": {
    "id": "65c112e8"
   },
   "source": [
    "<a name =stepii></a>\n",
    "    \n",
    "# Step ii:  Work with a subset\n",
    "To speed up processing\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d39abacc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "d39abacc",
    "outputId": "81195c0a-8311-4511-9cfd-05227d3167b5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>transcripts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>To Chairman Dean and my great friend Dick Durb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Chief Justice Roberts, President Carter, Presi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       President                                        transcripts\n",
       "42  Barack Obama  To Chairman Dean and my great friend Dick Durb...\n",
       "43  Donald Trump  Chief Justice Roberts, President Carter, Presi..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_df = corpus_df.loc[42:43, (\"President\",\"transcripts\")]\n",
    "subset_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34934d36",
   "metadata": {
    "id": "34934d36"
   },
   "source": [
    "<a name =stepiii></a>\n",
    "    \n",
    "# Step iii:  Characteristics of the dataset\n",
    "Let's examine the length (no. of characters, words and sentences) in each corpora.\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e91e29",
   "metadata": {
    "id": "01e91e29"
   },
   "source": [
    "## Length"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4AzN4zyb5UeR",
   "metadata": {
    "id": "4AzN4zyb5UeR"
   },
   "source": [
    "### Length -  Number of Speeches\n",
    "\n",
    "Could not easily split into individal speeches.\n",
    "The empty_tokens found later may be a way to count and number the number of speeches and I dntify them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40bdfec8",
   "metadata": {
    "id": "40bdfec8"
   },
   "source": [
    "### Length by character i.e. letters, spaces and punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f41c081",
   "metadata": {
    "id": "7f41c081",
    "outputId": "81b949c5-8bca-4a50-9449-64dea07d21b4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1155148"
      ]
     },
     "execution_count": 761,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(subset_df.transcripts.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "769535d8",
   "metadata": {
    "id": "769535d8",
    "outputId": "81937d1a-8ab9-426e-b138-877b961a7cda"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 762,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Includes letters, spaces and punctuation\n",
    "len('Hi how are you? Do you?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "624fe9f5",
   "metadata": {
    "id": "624fe9f5",
    "outputId": "5b7c4b37-a069-4235-c6e5-1baee055058d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1155148, 475677]"
      ]
     },
     "execution_count": 763,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(subset_df.transcripts.iloc[i]) for i in range(subset_df.shape[0])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e176f64e",
   "metadata": {
    "id": "e176f64e",
    "outputId": "7ffbc67c-3a01-4a19-9b81-335fc3ea4243",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[190015,\n",
       " 88556,\n",
       " 211059,\n",
       " 216989,\n",
       " 297306,\n",
       " 222338,\n",
       " 933812,\n",
       " 389851,\n",
       " 49669,\n",
       " 411510,\n",
       " 620263,\n",
       " 68280,\n",
       " 234451,\n",
       " 385312,\n",
       " 485593,\n",
       " 597666,\n",
       " 569655,\n",
       " 621306,\n",
       " 410483,\n",
       " 17722,\n",
       " 305870,\n",
       " 919540,\n",
       " 836204,\n",
       " 566308,\n",
       " 756190,\n",
       " 709967,\n",
       " 490304,\n",
       " 174762,\n",
       " 452057,\n",
       " 546456,\n",
       " 1166892,\n",
       " 207143,\n",
       " 105898,\n",
       " 698750,\n",
       " 1421446,\n",
       " 382082,\n",
       " 240004,\n",
       " 588246,\n",
       " 1188847,\n",
       " 512239,\n",
       " 768815,\n",
       " 630362,\n",
       " 1155148,\n",
       " 475677]"
      ]
     },
     "execution_count": 764,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len_char = [len(corpus_df.transcripts.iloc[i]) for i in range(corpus_df.shape[0])]\n",
    "len_char"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa86df4d",
   "metadata": {
    "id": "aa86df4d"
   },
   "source": [
    "#### Plots for Length Total Characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad411c6",
   "metadata": {
    "id": "9ad411c6",
    "outputId": "4e59811c-507f-4b62-8c2d-08f060f893a7",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>characters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>190015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>88556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>211059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>216989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>297306</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           President                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "\n",
       "                                         transcripts  characters  \n",
       "0  Fellow Citizens of the Senate and the House of...      190015  \n",
       "1  When it was first perceived, in early times, t...       88556  \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...      211059  \n",
       "3  Unwilling to depart from examples of the most ...      216989  \n",
       "4  I should be destitute of feeling if I was not ...      297306  "
      ]
     },
     "execution_count": 765,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df[\"characters\"] = len_char\n",
    "corpus_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc60e785",
   "metadata": {
    "id": "bc60e785",
    "outputId": "726c80a5-001c-40dc-ac6d-23b85a101460"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 766,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEFCAYAAADKeq1sAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOwUlEQVR4nO3dcYxl5VnH8e9TtujCIBCXTszSutgALbLSsKNWMGQGTAS2sTFBRbEGgm78o0jM1hRNFBPTiH9gWmNrs8GGmNZOLJBGRamNZQraLu0uAgusGIRNu1RBagEHSei0j3/cuzA7zux9Z+eee5/bfj/Jhrlz3nvmN4f7/vbsufecE5mJJKmuN4w7gCTp2CxqSSrOopak4ixqSSrOopak4jZ1sdItW7bktm3bmsa+/PLLnHzyyV3EGLpJygqTldes3ZmkvJOUFYabd//+/c9n5hmrLszMof/ZsWNHtrr33nubx47bJGXNnKy8Zu3OJOWdpKyZw80L7Ms1OtVDH5JUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUXCenkE+ibTfdPXDM7u1LXNswbr0O3bJz6OuU9J3DPWpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKq6pqCPiNyPisYh4NCI+GRHf23UwSVLPwKKOiK3AbwAzmXk+cAJwddfBJEk9rYc+NgGbI2ITcBLwte4iSZKWi8wcPCjiRuADwCvAP2TmNauM2QXsApient4xPz/fFGBxcZGpqan1ZO7EgWdeHDhmejM8+8rwf/b2racOf6UM3rYtv3NXVv7OVV4HLSYpK0xW3knKCsPNOzc3tz8zZ1ZbNrCoI+J04E7gF4AXgE8Bd2Tmx9d6zszMTO7bt68p3MLCArOzs01ju7TtprsHjtm9fYlbD2wa+s8+dMvOoa8TBm/blt+5Kyt/5yqvgxaTlBUmK+8kZYXh5o2INYu65dDHTwFPZ+Z/ZeY3gbuAi4aSTJI0UEtRfwV4Z0ScFBEBXAYc7DaWJOmIgUWdmQ8AdwAPAgf6z9nTcS5JUl/TAdfMvBm4ueMskqRVeGaiJBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBU3/FtqSw1W3gF99/Ylrh3BXdG7uuO71CX3qCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpOItakoqzqCWpuKaijojTIuKOiPjXiDgYET/RdTBJUk/rzW0/BNyTmVdFxInASR1mkiQtM7CoI+L7gEuAawEy81Xg1W5jSZKOiMw89oCIdwB7gMeBC4D9wI2Z+fKKcbuAXQDT09M75ufnmwIsLi4yNTW17uDDduCZFweOmd4Mz74ygjBDMkl5R5V1+9ZTN7yOKq/ZVpOUd5KywnDzzs3N7c/MmdWWtRT1DLAXuDgzH4iIDwEvZebvrvWcmZmZ3LdvX1O4hYUFZmdnm8Z2adtNdw8cs3v7ErceaD1aNH6TlHdUWQ/dsnPD66jymm01SXknKSsMN29ErFnULW8mHgYOZ+YD/cd3ABcOJZkkaaCBRZ2Z/wl8NSLO7X/rMnqHQSRJI9D6b80bgE/0P/HxFHBdd5EkScs1FXVmPgSseuxEktQtz0yUpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOIsakkqzqKWpOLK3aK65W7gkvTdxD1qSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSrOopak4ixqSSquuagj4oSI+JeI+NsuA0mSjraePeobgYNdBZEkra6pqCPiTGAncFu3cSRJK0VmDh4UcQfwh8ApwPsy812rjNkF7AKYnp7eMT8/3xRgcXGRqamp1x4feObFpueNw/RmePaVcadoN0l5R5V1+9ZTN7yOla/Z6iYp7yRlheHmnZub25+ZM6st2zToyRHxLuC5zNwfEbNrjcvMPcAegJmZmZydXXPoURYWFlg+9tqb7m563jjs3r7ErQcGbrIyJinvqLIeumZ2w+tY+ZqtbpLyTlJWGF3elkMfFwM/ExGHgHng0oj4eKepJEmvGVjUmfnbmXlmZm4DrgY+l5m/3HkySRLg56glqbx1HRTMzAVgoZMkkqRVuUctScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVZ1JJUnEUtScVNxi2qpSHZNoS73O/evsS161zPoVt2bvjnTprj2dbHs21X8522vd2jlqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiBhZ1RLw5Iu6NiIMR8VhE3DiKYJKknpab2y4BuzPzwYg4BdgfEZ/NzMc7ziZJomGPOjP/IzMf7H/9P8BBYGvXwSRJPZGZ7YMjtgH3Aedn5ksrlu0CdgFMT0/vmJ+fb1rn4uIiU1NTrz0+8MyLzXlGbXozPPvKuFO0m6S8Zu3OWaeecNQcG5XjmcvD2rbbt5668ZU0WNlfGzE3N7c/M2dWW9Zc1BExBXwe+EBm3nWssTMzM7lv376m9S4sLDA7O/va42033d30vHHYvX2JWw+0HC2qYZLymrU7t19+8lFzbFSOZy4Pa9seumXnhtfRYmV/bURErFnUTZ/6iIg3AncCnxhU0pKk4Wr51EcAfw4czMw/7j6SJGm5lj3qi4H3AJdGxEP9P1d2nEuS1DfwYFBm/hMQI8giSVqFZyZKUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnEWtSQVZ1FLUnGTcytlSWp0PHdAPx67ty9x7bKf1dXdz92jlqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTiLGpJKs6ilqTimoo6Ii6PiCci4smIuKnrUJKk1w0s6og4AfgwcAVwHvCLEXFe18EkST0te9Q/BjyZmU9l5qvAPPDubmNJko6IzDz2gIirgMsz81f7j98D/HhmvnfFuF3Arv7Dc4EnGjNsAZ5fT+gxmqSsMFl5zdqdSco7SVlhuHl/MDPPWG3BpoYnxyrf+3/tnpl7gD3rDEZE7MvMmfU+bxwmKStMVl6zdmeS8k5SVhhd3pZDH4eBNy97fCbwtW7iSJJWainqLwNnR8RZEXEicDXw193GkiQdMfDQR2YuRcR7gc8AJwAfy8zHhphh3YdLxmiSssJk5TVrdyYp7yRlhRHlHfhmoiRpvDwzUZKKs6glqbiRFfWg09Cj50/6yx+JiAtHlW2VLIOyXtPP+EhEfCEiLhhHzn6WptP7I+JHI+Jb/c/Fj01L3oiYjYiHIuKxiPj8qDMuyzHodXBqRPxNRDzcz3rdOHL2s3wsIp6LiEfXWF5pfg3KWmZ+9fMcM++ycd3Nsczs/A+9NyH/Hfgh4ETgYeC8FWOuBP6e3ue23wk8MIpsx5n1IuD0/tdXVM66bNzngL8DrhpH1nVs29OAx4G39B+/qXDW3wH+qP/1GcB/AyeOKe8lwIXAo2ssLzG/GrOWmF+teZe9XjqbY6Pao245Df3dwF9kz17gtIj4gRHlW25g1sz8QmZ+o/9wL73Plo9D6+n9NwB3As+NMtwqWvL+EnBXZn4FIDPHlbklawKnREQAU/SKemm0MftBMu/r//y1VJlfA7MWml9H8gzattDxHBtVUW8Fvrrs8eH+99Y7ZhTWm+N6ensq4zAwa0RsBX4W+OgIc62lZdueA5weEQsRsT8ifmVk6Y7WkvVPgbfTOwHsAHBjZn57NPHWrcr8Wq9xzq8mo5hjLaeQD0PLaehNp6qPQHOOiJij90L6yU4Tra0l6weB92fmt3o7fmPVkncTsAO4DNgMfDEi9mbmv3UdboWWrD8NPARcCrwV+GxE3J+ZL3Wc7XhUmV/NCsyvVh+k4zk2qqJuOQ29yqnqTTki4keA24ArMvPrI8q2UkvWGWC+/wLaAlwZEUuZ+emRJDxa6+vg+cx8GXg5Iu4DLgBGXdQtWa8DbsneQconI+Jp4G3Al0YTcV2qzK8mReZXq+7n2IgOxm8CngLO4vU3Zn54xZidHP1mx5fG9MZBS9a3AE8CF40j43qyrhh/O+N9M7Fl274d+Mf+2JOAR4Hzi2b9M+D3+19PA88AW8a4fbex9ht0JeZXY9YS86s174pxncyxkexR5xqnoUfEr/eXf5Teu6VX9v8H/S+9vZWRa8z6e8D3Ax/p/y26lGO44ldj1jJa8mbmwYi4B3gE+DZwW2Ye82NR48oK/AFwe0QcoFeA78/MsVyiMyI+CcwCWyLiMHAz8MZlWUvML2jKWmJ+HdGQt/sM/b8FJElFeWaiJBVnUUtScRa1JBVnUUtScRa1JG1Q64Wblo3/+Yh4vH8xr78cON5PfUjSxkTEJcAiveupnD9g7NnAXwGXZuY3IuJNOeCaNu5RS9IG5SoXboqIt0bEPf1r1twfEW/rL/o14MPZv/DUoJIGi1qSurIHuCEzdwDvAz7S//45wDkR8c8RsTciLh+0olFd60OSvmtExBS962p/atmFmr6n/99NwNn0znY8E7g/Is7PzBfWWp9FLUnD9wbghcx8xyrLDgN7M/ObwNMR8QS94v7ysVYmSRqi7F3q9umI+Dl47VZoR24p9mlgrv/9LfQOhTx1rPVZ1JK0Qf0LN30RODciDkfE9cA1wPUR8TDwGK/fIegzwNcj4nHgXuC3csClXP14niQV5x61JBVnUUtScRa1JBVnUUtScRa1JBVnUUtScRa1JBX3f+uc2+V+Xl6lAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "corpus_df['characters'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c2fdf25",
   "metadata": {
    "id": "4c2fdf25",
    "outputId": "a5a49672-1597-4684-b373-9e5e48105218"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='characters'>"
      ]
     },
     "execution_count": 767,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEDCAYAAAA4FgP0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaU0lEQVR4nO3de5Cd9X3f8fdnhZx1dTGJtAJHYrNSRzYBD8j2cpkkJEDtWMix1XhsAr4kZaCqZgxyTNMYN9O4rSczTTyhtmIwI7sMZuJYlWtKVFcR8SR20ARwtHIFRhAcsRawwWgXGdCFbJA43/5xzhFHyzlnn13tc57b5zWj0T6Xs/vdR0fn+3x/t0cRgZmZVVdf1gGYmVm2nAjMzCrOicDMrOKcCMzMKs6JwMys4pwIzMwqrpCJQNKdksYlPZrw/KslPSZpn6Q/Szs+M7MiURHnEUj6ZeAocHdEvG2ac1cD24ArI+IFScsiYrwXcZqZFUEhK4KIuB/4Ses+Sf9S0k5JeyTtknRu49C/BW6LiBcar3USMDNrUchE0MEW4KaIeCfwO8Dtjf1vAd4i6W8lPSRpbWYRmpnl0BlZBzAXJC0EfgH4hqTm7p9q/H0GsBq4HFgB7JL0toh4scdhmpnlUikSAfXK5sWIWNPm2BjwUEQcB34k6QnqiWF3D+MzM8utUjQNRcRh6h/yHwJQ3YWNw/cCVzT2L6XeVDSaRZxmZnlUyEQg6evAg8BbJY1Juh74CHC9pIeBfcD6xun3AYckPQZ8B/gPEXEoi7jNzPKokMNHzcxs7hSyIjAzs7lTuM7ipUuXxtDQUNZhmJkVyp49e56PiIF2xwqXCIaGhhgZGck6DDOzQpH0VKdjbhoyM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOruMKNGjKz6qnVggOHjnHw8CRnLe5naMkC+vo0/QstEScCM8u1Wi3Yue85bt62l8njNfrn93Hr1WtYe/7ZTgZzxE1DZpZrBw4dO5kEACaP17h5214OHDqWcWTlkVoiSPpcYUkXSXpV0gfTisXMiuvg4cmTSaBp8niN8SOTGUVUPmlWBHcBXZ8GJmke8IfUVwg1M3udsxb30z//1I+q/vl9LFvUn1FE5ZNaImj3XOE2bgK+Cfg5wmbW1tCSBdx69ZqTyaDZRzC0ZEHGkZVHZp3FkpYDvw5cCVw0zbkbgA0Ag4OD6QdnZrnR1yfWnn825266jPEjkyxb5FFDcy3LUUOfBz4VEa+2PGe4rYjYQv3h9AwPD/sBCmYV09cnVg0sZNXAwqxDKaUsE8EwsLWRBJYC6ySdiIh7M4zJzKxyMksEEbGy+bWku4BvOQmYmfVeaomg8Vzhy4GlksaAzwDzASLijrR+rpmZzUxqiSAirp3Buf8mrTjMzKw7zyw2M6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6s4JwIzs4pzIjAzqzgnAjOzinMiMDOrOCcCM7OKcyIwM6u41BKBpDsljUt6tMPxj0h6pPHnAUkXphWLmZl1lmZFcBewtsvxHwG/EhEXAJ8FtqQYi5mZdXBGWt84Iu6XNNTl+AMtmw8BK9KKxcxsJmq14MChYxw8PMlZi/sZWrKAvj5lHVZqUksEM3Q98BedDkraAGwAGBwc7FVMZlZBtVqwc99z3LxtL5PHa/TP7+PWq9ew9vyzS5sMMu8slnQF9UTwqU7nRMSWiBiOiOGBgYHeBWdmlXPg0LGTSQBg8niNm7ft5cChYxlHlp5ME4GkC4CvAOsj4lCWsZhZcrVaMDpxlAeffJ7RiaPUapF1SHPm4OHJk0mgafJ4jfEjkxlFlL7MmoYkDQL3AB+LiB9mFYeZzUzZm07OWtxP//y+U5JB//w+li3qzzCqdKU5fPTrwIPAWyWNSbpe0kZJGxun/D6wBLhd0l5JI2nFYmZzp+xNJ0NLFnDr1Wvon1//eGwmuqElC1L9uVlWWWmOGrp2muM3ADek9fPNLB3dmk5WDSzMKKq509cn1p5/NuduuozxI5MsW5T+qKGsq6zMO4vNrFiaTSetytZ00tcnVg0s5NJVS1k1sDD1D+OsqywnAjObkayaTsos6w7qvMwjMLOCyKLppOyy7qB2RWBmM9brppOyy7rKckVgZpaxrKssJwIzsxxoVllZjLxy05CZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcWl+fD6OyWNS3q0w3FJ2ixpv6RHJL0jrVjMzKyzNCuCu4C1XY5fBaxu/NkAfCnFWMzMrIPUEkFE3A/8pMsp64G7o+4h4ExJb04rHjMzay/LPoLlwDMt22ONfWZm1kNZJoJ2z2CLtidKGySNSBqZmJhIOSwzs2rJMhGMAee0bK8Anm13YkRsiYjhiBgeGBjoSXBmZlWRZSLYDvxmY/TQpcBLEfHjDOMxM6uk1B5eL+nrwOXAUkljwGeA+QARcQewA1gH7AdeBq5LKxYzM+sstUQQEddOczyAj6f1883MLBnPLDYzq7jUKgKbnVotOHDoGAcPT3LW4n6Gliygr6/dACszs7nhRJAjtVqwc99z3LxtL5PHa/TP7+PWq9ew9vyznQzMLDVuGsqRA4eOnUwCAJPHa9y8bS8HDh3LODKbqlYLRieO8uCTzzM6cZRare0UGLNCcEWQIwcPT55MAk2Tx2uMH5lk1cDCjKKyqVy5Wdm4IsiRsxb30z//1H+S/vl9LFvUn1FE1o4rNysbJ4IcGVqygFuvXnMyGTTvNIeWLMg4MmvVrXIzKyI3DeVIX59Ye/7ZnLvpMsaPTLJskUcN5VGzcmtNBq7crMhcEeRMX59YNbCQS1ctZdXAQieBHHLlZmXjisBshly5WdkkSgSSfhHYGxHHJH0UeAfwhYh4KtXozHKqWbl5NJeVQdKmoS8BL0u6EPhd4Cng7tSiMjOznkmaCE40FolbT70S+AKwKL2wzMysV5L2ERyR9Gngo8AvS5pHY0lpMzMrtqQVwW8A/wxcHxHPUX+28OdSi8rMzHpm2oqgcff/pxHxrua+iHga9xGYteUVZK1opk0EEfGqpJclvSkiXupFUGZF5XWIrIiS9hFMAj+Q9G3g5IIqEbEplajMCqrTOkTnbrrMQ00tt5Imgv/b+GNmXXgFWSuiRIkgIr4q6Y3AYEQ8kXJMZoXldYisiBKNGpL0PmAvsLOxvUbS9gSvWyvpCUn7Jd3S5vibJP0fSQ9L2ifpuhnGb5YrXofIiihp09B/Bi4GvgsQEXslrez2gsZoo9uAdwNjwG5J2yPisZbTPg48FhHvkzQAPCHpaxHxysx+DbN88DpEVkRJE8GJiHhJOuXNPN2z+S4G9kfEKICkrdRnJrcmggAWqf6NFwI/AU4kjMksl7wOkRVN0gllj0r6MDBP0mpJfwI8MM1rlgPPtGyPNfa1+iLw88CzwA+AT0REbco5SNogaUTSyMTERMKQzcwsiaSJ4CbgfOqzi/8MeAn4xDSvaVcLT60i3kO97+FngTXAFyUtft2LIrZExHBEDA8MDCQM2cys92q1YHTiKA8++TyjE0ep1aZrPMle0qah90bE7wG/19wh6UPAN7q8Zgw4p2V7BfU7/1bXAf+tsaDdfkk/As4F/i5hXGZmuVHUCYVJK4JPJ9zXajewWtJKSW8ArgGmjjR6GvhXAJLOAt4KjCaMyaxUingnaafqNKHwwKFj07wyW10rAklXAeuA5ZI2txxazDSduhFxQtKNwH3APODOiNgnaWPj+B3AZ4G7JP2AelPSpyLi+Vn/NmYFVdQ7STtVUScUTtc09CwwArwf2NOy/wjwyem+eUTsAHZM2XdHy9fPAr+aNFizsvLSFOVQ1AmFXRNBRDwMPCzpfwPHIuJVODlH4Kd6EJ9ZJRT1TtJO1ZxQOLWyy/uEwqSdxX8JvAs42th+Y2PfL6QRlFnVFPVO0k5V1AmFSTuL+yOimQRofP0v0gnJrHq8NEV5NCcUXrpqKasGFuY+CUDyiuCYpHdExPcBJL0T+Kf0wjKrlunuJP2wG0tT0kTw28A3JDXnAbyZ+uMrzWyOdFqawiOKLG1Jl6HeLelc6uP8Bfx9RBxPNTIzA6ozoshVT3aSVgRQTwLnAf3A2yUREX5usVnKqjCiyFVPtpI+j+AzwJ80/lwB/BH1uQVmlrLmiKJWZRtRVNQZuWWRdNTQB6kvBfFcRFwHXIjnEZj1RBVGFHWreix9SZuG/ikiapJONFYHHQdWpRiXmTUUdWz6THgeRbaSVgQjks4Evkx9qYnv4xVCzXqmiGPTZ6IKVU+eqb4CdJcT6k8PWxERzzS2h4DFEfFI+uG93vDwcIyMjGTxo80sRc1RQ2WterImaU9EDLc7Nm3TUESEpHuBdza2D8xpdGZm+BGfWUraNPSQpItSjcTMzDKRtLP4CuDfSXoKOEZ9UllExAWpRWZmZj2RNBFclWoUZmYVkNfZ00mXmHgKQNIy6jOLrSTy+sacqbL8HlZeeZ49nSgRSHo/8MfAz1KfQ/BzwOPA+emFZmnL8xtzJsrye1i55XnNqKSdxZ8FLgV+GBErqc8y/tvUorKeKMu0/rL8HlZueZ49nTQRHI+IQ0CfpL6I+A6wZroXSVor6QlJ+yXd0uGcyyXtlbRP0t8kD91OVx7fmLVaMDpxlAeffJ7RiaPUat3nuUA+fw+zqfK8ZlTSzuIXJS0E7ge+JmkcONHtBY3nGt8GvBsYA3ZL2h4Rj7WccyZwO7A2Ip5u9EFYj+RtWv9sm3jy9nuYtZPn5xknrQjWU38i2SeBncCTwPumec3FwP6IGI2IV4Ctje/T6sPAPRHxNEBEjCcN3E5f3qb1z7aJJ2+/hxXDbKrP09FcM2rHpsvYuuESdmy6LDf9WElHDbX+T/xqwu+9HHimZXsMuGTKOW8B5kv6LrAI+EK7ZxxI2gBsABgcHEz44206eVvMbLbr7uft97D8y2qAQV5nTyd9HsEHJP2DpJckHZZ0RNLh6V7WZt/UlHsG9aUr3gu8B/hPkt7yuhdFbImI4YgYHhgYSBKyJZSnxcyma0PtdgeXp9/D8s8DDE6VtGnoj4D3R8SbImJxRCyKiMXTvGYMOKdlewXwbJtzdkbEsYh4nnofxIUJY7KS6dbE07yDW7d5F9d++Xus27yLnfueS72cn2u9bo6w9jzA4FRJO4sPRsTjM/zeu4HVklYC/whcQ71PoNWfA1+UdAbwBupNR/99hj/HSqJbE8/oxNHcjsFOyvMd8sMDDE7VtSJoNAl9gPrzCP6npGub+xr7O4qIE8CNwH3UJ59ti4h9kjZK2tg453Hqnc+PUH++wVci4tE5+L2soDo18ZThDq5ozRFlrl48wOBU01UEzZFBAbwM/GrLsQDu6fbiiNgB7Jiy744p258DPpckWKuuMtzBFekh9GWvXjzA4FRdK4KIuK7xjOJ5wCdbtm/uSXRmDWW4g8vzhKKpila9zIYHGLwmaR/BBRHxYnMjIl6Q9PZ0QjJ7vTLcweV5QtFURape7PQlTQR9kn46Il4AkPQzM3it2ZzI6xjspIqUzMrQFGfJJf0w/2PgAUn/i3rfwNXAH6QWlVlJFSWZFal66bUyLnmedGbx3ZJGgCupTxT7QOuaQWZWLkWqXnqprJ3oiijWkLDh4eEYGRnJOgwzq6DRiaOs27zrdU1mOwown0XSnogYbncs6cxiM8upMo/3z5syzGdpxx2+1nNlbGPNSlmbKvKqrJ3orgisp8qyZlBeVGG8f55kNZ8l7arPFYH1VJ6f21pEHu/fW1l0ovei6nNFYD1V1jbWrBRptnJZ9HpGci+qPicC6yl/cM2tMiy9Yd314ubJTUPWU56oNLdOt6nCHff514sOas8jsJ5rfvh4olK2POKoGObq36nbPAInArOKKvLkqKqZi5unbonATUNmFeURR8WR9hpV7iw2qyh33FuTE0EFeAkCa8cjjqzJTUMl5w5Bj4zpxCuMWlOqFYGktZKekLRf0i1dzrtI0quSPphmPFVU9SUIvKRFd35co0GKiUDSPOA24CrgPOBaSed1OO8PgfvSiqXKqj6Tt+qJ0CyJNCuCi4H9ETEaEa8AW4H1bc67CfgmMJ5iLKUwm7b+LDoE89QnUfVEaJZEmn0Ey4FnWrbHgEtaT5C0HPh16k8+u6jTN5K0AdgAMDg4OOeBFsFs2/p7PZM3b30SZV022F7jPqDTl2ZF0O5fYuqt4eeBT0XEq92+UURsiYjhiBgeGBiYq/gKZbZNHM0OwR2bLmPrhkvYsemyVD+U89YU45Ex5eY+oLmRZkUwBpzTsr0CeHbKOcPAVkkAS4F1kk5ExL0pxlVIpzP5p5cPTM/bJCWPjCk3L2s+N9JMBLuB1ZJWAv8IXAN8uPWEiFjZ/FrSXcC3nATaK0oTRx7j7GUitN7K241HUaXWNBQRJ4AbqY8GehzYFhH7JG2UtDGtn1tWRWniKEqcVg6eHT03vOhcgRRl1c6ixGnFl7fBCXnm1UfNrLR845GMVx81s9JyH9Dp86JzZmYV50RgZlZxbhoy68AzVq0qnAjM2vBoFKsSNw2lJE8Lr9nM5W2pDLM0uSJIge8mi88zVq1KXBGkwHeTxecZq1YlTgQp8Br4xeelMqxK3DSUgjwuvGYz41VLrUpcEaTAd5Pl4Of5WlW4IkiB7ybNrEicCFLi9U/MrCicCGxWPOvWisDv02ScCGzGPE/CisDv0+TcWZyBos869jwJKwK/T5NzRXAaZlN2luEuxbNurQj8Pk3OiWAanT7sZ/uB3uku5dxNlxXmzel5EtZNXtrl/T5NLtWmIUlrJT0hab+kW9oc/4ikRxp/HpB0YZrxzFTzw37d5l1c++XvsW7zLnbue+7kG302ZWcZZh17noR10u3/TK/5fZpcahWBpHnAbcC7gTFgt6TtEfFYy2k/An4lIl6QdBWwBbgkrZhmqtvd+2zLzjLcpXiehHWSp4rX79Pk0qwILgb2R8RoRLwCbAXWt54QEQ9ExAuNzYeAFWkEMtvO2W4f9rNdlKwsdymedWvt5K3i9fs0mTT7CJYDz7Rsj9H9bv964C/aHZC0AdgAMDg4OKMgTqdzttvde/MDfer3ne4D3XcpVmZlqHirSBHptN1J+hDwnoi4obH9MeDiiLipzblXALcDvxQRh7p93+Hh4RgZGUkcx+jEUdZt3vW6N+aOBKXqdEmk2VfgD3TLs1523pZhVFxZSdoTEcPtjqVZEYwB57RsrwCenXqSpAuArwBXTZcEZuN0hpBNd/fuZSQs73r9weyKt5jS7CPYDayWtFLSG4BrgO2tJ0gaBO4BPhYRP0wjiNN9wIjbGK3IsphU5f8zxZNaIoiIE8CNwH3A48C2iNgnaaOkjY3Tfh9YAtwuaa+k5G0+CZWlc9ZsNvLWeWv5lOqEsojYAeyYsu+Olq9vAG5IMwaXqlZl7ry1JCqx1pBLVasqV8SWhJeYMCsxV8SWhBOBWcl5dJtNpxJNQ2Zm1pkrAjPrmbysTGqnciIws57wrOP8ctOQmfWEnxiWX5VPBEV/bKRZUXhyW35VumnIpapZ73hyW35VuiJwqWrWO57cll+Vrgj8cOvuPMLD5pInt+VXpROBS9XO3GxmafDktnyqdNNQmUrVue70drOZWXVUuiIoS6maxt27m83MqqPSFQGUY2XSNO7eT/eBPmZWHJVPBGWQxvjsMjWbmVl3lW4aKos0Or3L0mxmZtNzRVACad29l6HZbDqeWW7miqAUfPc+Ox4ia1aXakUgaa2kJyTtl3RLm+OStLlx/BFJ70gznjKrwt37XPMQWbO61BKBpHnAbcBVwHnAtZLOm3LaVcDqxp8NwJfSisdsKi+CZlaXZkVwMbA/IkYj4hVgK7B+yjnrgbuj7iHgTElvTjEms5M8RNasLs1EsBx4pmV7rLFvpucgaYOkEUkjExMTcx6oVZOHyJrVpdlZ3K6ReuqQjCTnEBFbgC0Aw8PDHtZhc8Kd7GZ1aSaCMeCclu0VwLOzOMcsNV4EzSzdpqHdwGpJKyW9AbgG2D7lnO3AbzZGD10KvBQRP04xJjMzmyK1iiAiTki6EbgPmAfcGRH7JG1sHL8D2AGsA/YDLwPXpRWPmZm1l+qEsojYQf3DvnXfHS1fB/DxNGMwM7PuvMSEmVnFORGYmVWc6q0zxSFpAnhqli9fCjw/h+GUia9NZ742nfnadJa3a/NzETHQ7kDhEsHpkDQSEcNZx5FHvjad+dp05mvTWZGujZuGzMwqzonAzKziqpYItmQdQI752nTma9OZr01nhbk2leojMDOz16taRWBmZlM4EZiZVVxlEsF0j82sEkl3ShqX9GjLvp+R9G1J/9D4+6ezjDELks6R9B1Jj0vaJ+kTjf2+NlK/pL+T9HDj2vyXxv7KX5smSfMk/T9J32psF+baVCIRJHxsZpXcBaydsu8W4K8iYjXwV43tqjkB/PuI+HngUuDjjfeJrw38M3BlRFwIrAHWNlYM9rV5zSeAx1u2C3NtKpEISPbYzMqIiPuBn0zZvR74auPrrwL/upcx5UFE/Dgivt/4+gj1/9TL8bWh8TjZo43N+Y0/ga8NAJJWAO8FvtKyuzDXpiqJINEjMSvurOazIBp/L8s4nkxJGgLeDnwPXxvgZNPHXmAc+HZE+Nq85vPA7wK1ln2FuTZVSQSJHolpBiBpIfBN4Lcj4nDW8eRFRLwaEWuoP0nwYklvyzikXJD0a8B4ROzJOpbZqkoi8CMxp3dQ0psBGn+PZxxPJiTNp54EvhYR9zR2+9q0iIgXge9S72fytYFfBN4v6QD1ZucrJf0pBbo2VUkESR6bWXXbgd9qfP1bwJ9nGEsmJAn4H8DjEXFryyFfG2lA0pmNr98IvAv4e3xtiIhPR8SKiBii/tny1xHxUQp0bSozs1jSOurteM3HZv5BthFlR9LXgcupL5N7EPgMcC+wDRgEngY+FBFTO5RLTdIvAbuAH/BaW+9/pN5PUPVrcwH1Ds951G8gt0XEf5W0hIpfm1aSLgd+JyJ+rUjXpjKJwMzM2qtK05CZmXXgRGBmVnFOBGZmFedEYGZWcU4EZmYV50RgZlZxTgRmZhX3/wGK8VGbc6+BbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(data=corpus_df, x=corpus_df.index, y=\"characters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15e92ecc",
   "metadata": {
    "id": "15e92ecc"
   },
   "source": [
    "<a name=\"words\"></a>\n",
    "### Length by words\n",
    "Removing punctuation and irrelevant terms where possible as determined in the [Data Cleaning](#stepiv) section below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "463da9d1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "463da9d1",
    "outputId": "b44dd9d5-7ff0-4963-d9a2-9a84abd5904d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "232121"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(word_tokenize(subset_df.transcripts.iloc[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "57c08521",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "57c08521",
    "outputId": "61a5d3aa-e175-4004-b191-8de8c390c3c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Hi', 'how', 'are', 'you', '?', 'Do', 'you', '?'], 8)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Counts words includes punctation\n",
    "word_tokenize('Hi how are you? Do you?'), len(word_tokenize('Hi how are you? Do you?'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d7e2e519",
   "metadata": {
    "id": "d7e2e519"
   },
   "outputs": [],
   "source": [
    "def clean(doc):\n",
    "    # Tokenize, clean up input document string\n",
    "    doc = re.sub(r'\\[[^()]*\\]',\"\", doc)#removes [Applause][\"Laughter\"]\n",
    "    doc = re.sub(r'\\([^()]*\\)',\"\", doc)#removes (Applause)(\"Laughter\")\n",
    "    doc = re.sub(r'[^\\w\\s]', \"\", doc)# removes punctuation\n",
    "    doc = re.sub(r'\\b[A-Z][A-Z0-9]+\\b',\"\",doc) #remove ALL CAPS\n",
    "    doc=\" \".join(doc.split()) #remove double whitespace\n",
    "    return doc.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b712a223",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "b712a223",
    "outputId": "ae24b0e0-5fa5-4ee2-d0f3-45d0b54b8e91"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'hi how are cow you dont you do you'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean(\"Hi how are Cow ; 'you don't' \"\" THOMAS JEFF you (laughter) [Applause]? Do you?'=\") #len(clean('Hi how are Cow THOMAS JEFF you [Applause]? Do you?'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "28580b16",
   "metadata": {
    "id": "28580b16"
   },
   "outputs": [],
   "source": [
    "# remove empty tokens\n",
    "speeches=(subset_df.transcripts.iloc[0])\n",
    "words = word_tokenize(speeches)\n",
    "cleaned_speeches = [clean(speech) for speech in words]\n",
    "cleanup_tokens = [ tok for tok in cleaned_speeches if tok ]\n",
    "empty_tokens = [ tok for tok in cleaned_speeches if not tok ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f57a295a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f57a295a",
    "outputId": "d096b5b0-99f3-4e84-e36c-af3ed611d933"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(232121, 207813)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cleaned_speeches), len(cleanup_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dRnp04YmbfR5",
   "metadata": {
    "id": "dRnp04YmbfR5"
   },
   "source": [
    "#### Empty Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "FjgmJBhvTWnd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FjgmJBhvTWnd",
    "outputId": "ef2b1e8a-b438-4b02-ea0c-ed4c8087d30a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " ...]"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uY-ZDfYnb0g1",
   "metadata": {
    "id": "uY-ZDfYnb0g1"
   },
   "source": [
    "### Length by words cont'd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8824a3d4",
   "metadata": {
    "id": "8824a3d4"
   },
   "outputs": [],
   "source": [
    "def len_speech(df): \n",
    "    length_speeches=[] \n",
    "    for i in range(df.shape[0]):\n",
    "        speeches=(df.transcripts.iloc[i])\n",
    "        words = word_tokenize(speeches)\n",
    "        cleaned_speeches = [clean(speech) for speech in words]\n",
    "        cleanup_tokens = [tok for tok in cleaned_speeches if tok] \n",
    "        length_speeches.append(len(cleanup_tokens))\n",
    "    return length_speeches\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "484910b7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "484910b7",
    "outputId": "3aba22a7-0b85-4bda-f662-18701290cdb8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[207813, 85452]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length_speeches_pres = len_speech(subset_df)\n",
    "length_speeches_pres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6f75f424",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6f75f424",
    "outputId": "1d3a78bf-8e4b-4afc-bf34-f6e003458529",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[31521,\n",
       " 14653,\n",
       " 35885,\n",
       " 36076,\n",
       " 50197,\n",
       " 37038,\n",
       " 158016,\n",
       " 64877,\n",
       " 8461,\n",
       " 69743,\n",
       " 104737,\n",
       " 11377,\n",
       " 39000,\n",
       " 63465,\n",
       " 81102,\n",
       " 103386,\n",
       " 95429,\n",
       " 103752,\n",
       " 67815,\n",
       " 2978,\n",
       " 50036,\n",
       " 150587,\n",
       " 140194,\n",
       " 92798,\n",
       " 132240,\n",
       " 116406,\n",
       " 85472,\n",
       " 28917,\n",
       " 74147,\n",
       " 89836,\n",
       " 197224,\n",
       " 37078,\n",
       " 18117,\n",
       " 122162,\n",
       " 251584,\n",
       " 68204,\n",
       " 40739,\n",
       " 102470,\n",
       " 209691,\n",
       " 91875,\n",
       " 137663,\n",
       " 108811,\n",
       " 207813,\n",
       " 85453]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length_speeches_pres = len_speech(corpus_df)\n",
    "length_speeches_pres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2f683ed6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2f683ed6",
    "outputId": "f36478ed-d1c2-4437-86ad-c54436e3dcf7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(length_speeches_pres)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de8c6f2",
   "metadata": {
    "id": "9de8c6f2"
   },
   "source": [
    "#### Plots for Length Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "03f9b359",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "03f9b359",
    "outputId": "6965f007-5b64-4a3b-a25c-f9bcf6ddd443",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>total_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>31521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>14653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>35885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>36076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>50197</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           President                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "\n",
       "                                         transcripts  total_words  \n",
       "0  Fellow Citizens of the Senate and the House of...        31521  \n",
       "1  When it was first perceived, in early times, t...        14653  \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...        35885  \n",
       "3  Unwilling to depart from examples of the most ...        36076  \n",
       "4  I should be destitute of feeling if I was not ...        50197  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df[\"total_words\"] = length_speeches_pres\n",
    "corpus_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "b2ef31c6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "id": "b2ef31c6",
    "outputId": "8fa441d8-ca24-42c9-fb5d-4ce9130e44a6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7ff69193f2d0>"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEHCAYAAACncpHfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASDklEQVR4nO3de5AlZX3G8e8DKxqVCIQJhcuSRWORUMQLGY2CmnhHYmI0JEC84CUuidF4iVpQVkUrf2k0lklMlFUJuVBIolCaGC+oqCQQcEHkKgFvsIAwakUtjaUrv/zRveYw7swexj3dM/N+P1Wnts/bPef9vdNn55nu0/N2qgpJUnv2GrsASdI4DABJapQBIEmNMgAkqVEGgCQ1asPYBUzjwAMPrM2bN49dhiStKZdddtnXqmpuqfVrIgA2b97Mtm3bxi5DktaUJF9Zbr2ngCSpUQaAJDXKAJCkRhkAktQoA0CSGmUASFKjDABJapQBIEmNMgAkqVEGgFZs46ZDSbJmHhs3HTr2t0xaVdbEVBBanW7dfjMnnH7R2GVM7ZxTjh67BGlV8QhAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqNmFgBJzkhyR5KrJ9relOTzSa5Mcl6S/WbVvyRpebM8AjgTOHZR2/nAkVX1YOC/gdNm2L8kaRkzC4Cq+jTwjUVtH62qHf3T/wIOmVX/kqTljfkZwAuAD43YvyQ1bZQASPJaYAdw1jLbbEmyLcm2hYWF4YqTpEYMHgBJngc8DXhWVdVS21XV1qqar6r5ubm5weqTpFYMekOYJMcCrwF+taq+O2TfkqS7muVloGcDFwOHJ9me5IXA24B9gfOTXJHkHbPqX5K0vJkdAVTVSbtofves+pMk3T3+JbAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktSodR8AGzcdSpI189i46dCxv2WSGjHodNBjuHX7zZxw+kVjlzG1c045euwSJDVi3R8BSJJ2zQCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGzSwAkpyR5I4kV0+0HZDk/CQ39P/uP6v+JUnLm+URwJnAsYvaTgU+XlUPAj7eP5ckjWBmAVBVnwa+saj56cDf98t/D/zWrPqXJC1v6M8ADqqq2/rlrwIHDdy/JKk32ofAVVVALbU+yZYk25JsW1hYGLAySWrD0AFwe5KDAfp/71hqw6raWlXzVTU/Nzc3WIGS1IqhA+ADwMn98snA+wfuX5LUm+VloGcDFwOHJ9me5IXAG4AnJbkBeGL/XJI0gpndErKqTlpi1RNm1ackaXr+JbAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktSomc0FpBXaawNJxq5CUgMMgNXmzh2ccPpFY1cxlXNOOXrsEiT9BDwFJEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRowRAklckuSbJ1UnOTnKvMeqQpJYNHgBJNgJ/DMxX1ZHA3sCJQ9chSa0b6xTQBuCnkmwA7g3cOlIdktSswQOgqm4B3gzcBNwGfLOqPrp4uyRbkmxLsm1hYWHoMiVp3RvjFND+wNOBw4D7A/dJ8uzF21XV1qqar6r5ubm5ocuUpHVvjFNATwS+VFULVfUD4FzAieUlaWBjBMBNwCOT3Dvdra+eAFw3Qh2S1LQxPgO4BHgvcDlwVV/D1qHrkKTWjXJLyKp6HfC6MfqWJHX8S2BJatRUAZDkmGnaJElrx7RHAH89ZZskaY1Y9jOAJI+iu0RzLskrJ1b9NN0UDpKkNWp3HwLvA9y3327fifZvAcfPqihJ0uwtGwBV9SngU0nOrKqvDFSTJGkA014Ges8kW4HNk19TVY+fRVGSpNmbNgD+BXgH8C7gh7MrR5I0lGkDYEdVvX2mlUiSBjXtZaD/muTFSQ5OcsDOx0wrkyTN1LRHACf3/756oq2AB+zZciRJQ5kqAKrqsFkXIkka1lQBkOS5u2qvqn/Ys+VIkoYy7Smgh08s34tuDv/LAQNAktaoaU8BvXTyeZL9gPfMpCJJ0iBWOh30d+ju6StJWqOm/QzgX+mu+oFuErhfBP55VkVJkmZv2s8A3jyxvAP4SlVtn0E9kqSBTHUKqJ8U7vN0M4LuD3x/lkVJkmZv2juC/S5wKfA7wO8ClyRxOmhJWsOmPQX0WuDhVXUHQJI54GPAe2dVmCRptqa9CmivnT/8e1+/G18rSVqFpj0C+HCSjwBn989PAP59NiVJkoawu3sC/zxwUFW9OskzgUf3qy4Gzlppp/0fkr0LOJLu8tIXVNXFK309SdLdt7sjgLcCpwFU1bnAuQBJfqlf9xsr7PcvgQ9X1fFJ9gHuvcLXkSSt0O4C4KCqumpxY1VdlWTzSjpMcj/gscDz+tf6Pl5WKkmD290Hufsts+6nVtjnYcAC8HdJPpvkXUnus3ijJFuSbEuybWFhYYVdSZKWsrsA2JbkRYsbk/w+cNkK+9wAHAW8vaoeRjev0KmLN6qqrVU1X1Xzc3NzK+xKkrSU3Z0CejlwXpJn8f8/8OeBfYBnrLDP7cD2qrqkf/5edhEAkqTZWjYAqup24Ogkj6O7Ygfgg1X1iZV2WFVfTXJzksOr6nq6ewtcu9LXkyStzLT3A7gAuGAP9vtS4Kz+CqAvAs/fg68tSZrCtH8ItkdV1RV0p5IkSSNxOgdJapQBIEmNMgAkqVEGgCQ1ygCQpEYZAJLUKANAkhplAEhSowwASWqUASBJjRplKghpFHttIMnYVUzl/ods4pabbxq7DK1zBoDacecOTjj9orGrmMo5pxw9dglqgKeAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRo0WAEn2TvLZJP82Vg2S1LIxjwBeBlw3Yv+S1LRRAiDJIcCvA+8ao39J0nhHAG8FXgPcudQGSbYk2ZZk28LCwnCVSVIjBg+AJE8D7qiqy5bbrqq2VtV8Vc3Pzc0NVJ0ktWOMI4BjgN9M8mXgPcDjk/zTCHVIUtMGD4CqOq2qDqmqzcCJwCeq6tlD1yFJrfPvACSpUaPeE7iqPgl8cswaJKlVHgFIUqMMAElqlAEgSY0yACSpUQaAJDXKAJCkRhkAktQoA0CSGmUASFKjDABJapQBIOkntnHToSRZE4+Nmw4d+9u1aow6F5Ck9eHW7TdzwukXjV3GVM455eixS1g1PAKQpEYZAJLUKANAkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaNXgAJNmU5IIk1ya5JsnLhq5BkjTOZHA7gD+pqsuT7AtcluT8qrp2hFokqVmDHwFU1W1VdXm//G3gOmDj0HVIUutGnQ46yWbgYcAlu1i3BdgCcOihzt+txuy1gSRjV6F1brQASHJf4H3Ay6vqW4vXV9VWYCvA/Px8DVyeNK47d6yZ+fXBOfbXqlGuAkpyD7of/mdV1blj1CBJrRvjKqAA7wauq6q3DN2/JKkzxhHAMcBzgMcnuaJ/HDdCHZLUtME/A6iq/wD8dEuSRuZfAktSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElq1Kj3A5Ckwa2xey3c/5BN3HLzTTN5bQNAUlu818KPeApIkhplAEhSowwASWqUASBJjTIAJKlRBoAkNcoAkKRGGQCS1CgDQJIaZQBIUqMMAElq1CgBkOTYJNcnuTHJqWPUIEmtGzwAkuwN/A3wVOAI4KQkRwxdhyS1bowjgEcAN1bVF6vq+8B7gKePUIckNS1VNWyHyfHAsVX1+/3z5wC/UlUvWbTdFmBL//Rw4Pq70c2BwNf2QLlriWNug2Nux54Y989V1dxSK1ft/QCqaiuwdSVfm2RbVc3v4ZJWNcfcBsfcjiHGPcYpoFuATRPPD+nbJEkDGiMAPgM8KMlhSfYBTgQ+MEIdktS0wU8BVdWOJC8BPgLsDZxRVdfs4W5WdOpojXPMbXDM7Zj5uAf/EFiStDr4l8CS1CgDQJIata4CYD1MMZHky0muSnJFkm192wFJzk9yQ//v/n17kvxVP94rkxw18Ton99vfkOTkifZf7l//xv5rM8IYz0hyR5KrJ9pmPsal+hhxzK9Pcku/r69IctzEutP6+q9P8pSJ9l2+x/uLKi7p28/pL7AgyT375zf26zcPM2JIsinJBUmuTXJNkpf17et2Xy8z5tW5r6tqXTzoPlD+AvAAYB/gc8ARY9e1gnF8GThwUdufA6f2y6cCb+yXjwM+BAR4JHBJ334A8MX+3/375f37dZf226b/2qeOMMbHAkcBVw85xqX6GHHMrwdetYttj+jfv/cEDuvf13sv9x4H/hk4sV9+B/CH/fKLgXf0yycC5ww45oOBo/rlfYH/7se2bvf1MmNelft60P/4M/7GPwr4yMTz04DTxq5rBeP4Mj8eANcDB0+8wa7vl08HTlq8HXAScPpE++l928HA5yfa77LdwOPczF1/GM58jEv1MeKYl/qhcJf3Lt0Vc49a6j3e//D7GrChb//Rdju/tl/e0G+Xkfb5+4EntbCvdzHmVbmv19MpoI3AzRPPt/dta00BH01yWbrpMAAOqqrb+uWvAgf1y0uNebn27btoXw2GGONSfYzpJf3pjjMmTlPc3TH/DPA/VbVjUftdXqtf/81++0H1pyMeBlxCI/t60ZhhFe7r9RQA68Wjq+ooutlS/yjJYydXVhfv6/ra3SHGuEq+j28HHgg8FLgN+Itxy5mNJPcF3ge8vKq+Nbluve7rXYx5Ve7r9RQA62KKiaq6pf/3DuA8utlTb09yMED/7x395kuNebn2Q3bRvhoMMcal+hhFVd1eVT+sqjuBd9Lta7j7Y/46sF+SDYva7/Ja/fr79dsPIsk96H4QnlVV5/bN63pf72rMq3Vfr6cAWPNTTCS5T5J9dy4DTwauphvHzisfTqY7r0jf/tz+6olHAt/sD3s/Ajw5yf79oeaT6c4T3gZ8K8kj+6slnjvxWmMbYoxL9TGKnT+ges+g29fQ1Xlif1XHYcCD6D7s3OV7vP8N9wLg+P7rF3//do75eOAT/fYz13//3w1cV1VvmVi1bvf1UmNetft6jA9GZviBy3F0n7p/AXjt2PWsoP4H0H3a/zngmp1joDuP93HgBuBjwAF9e+hurvMF4CpgfuK1XgDc2D+eP9E+37/5vgC8jRE+EATOpjsM/gHdOcwXDjHGpfoYccz/2I/pyv4/78ET27+2r/96Jq7UWuo93r93Lu2/F/8C3LNvv1f//MZ+/QMGHPOj6U69XAlc0T+OW8/7epkxr8p97VQQktSo9XQKSJJ0NxgAktQoA0CSGmUASFKjDABJapQBIEmNMgC0riTZL8mLd7PN5iS/N8Vrbc7E9M1DSXJmkuN3v6X0kzEAtN7sRzct7nI2A7sNgCFM/Em/NDgDQOvNG4AH9jfdeFP/uDrdTUNOmNjmMf02r+h/078wyeX94+hpOkrywSQP7pc/m+RP++U/S/KifkqDH+s/ya/1/X0AuLbf7m3pbv7xMeBnJ/p4Q7qbi1yZ5M178Psk4W8fWm9OBY6sqocm+W3gD4CHAAcCn0ny6X6bV1XV0wCS3Bt4UlV9L8mD6KZtmJ+irwvpguQrwA7gmL79MX2/z6Sb/XFx/9DdHObIqvpSkmcCh9PdHOQg4FrgjCQ/QzdvzC9UVSXZb+XfFunHeQSg9ezRwNnVzcJ4O/Ap4OG72O4ewDuTXEU3l8oRU77+hXR3+joG+CBw3z5MDquq63fT/6VV9aV++bET290KfKJv/ybwPeDdfUh8d+qRS1PwCECCVwC30/2mvhfdD91pfIbuSOGLwPl0v+W/CLhsiq/9zu42qKodSR4BPIFudseXAI+fsjZptzwC0Hrzbbp7sUL3G/oJSfZOMkf3m/ali7aBbt7026qbq/05dPdj3a2q+j7dHZh+B7i47+9VwM7TPEv1v9inJ7Y7GHgc/OimIverqn+nC6mHTFOXNC2PALSuVNXXk/xnf/nmh+im3/0c3RS9r6mqryb5OvDDJJ8DzgT+FnhfkucCH2aK384nXAg8oar+N8mFdDfouLBfdx7dPVsX9/8Li17jPLrf7K8FbqILE+hC6v1J7kU3VfIr70Zd0m45HbQkNcpTQJLUKE8BSbuR5CnAGxc1f6mqnjFGPdKe4ikgSWqUp4AkqVEGgCQ1ygCQpEYZAJLUqP8Dz4NSRhmGYbQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.histplot(data=corpus_df, x='total_words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "3cd005fe",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "id": "3cd005fe",
    "outputId": "1ebe6ab7-3596-40d7-f44d-1915e304b929"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7ff691c3a990>"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAD4CAYAAADRuPC7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcaUlEQVR4nO3df5TV9X3n8edrEDsekCowoiuwIw27FnO6JE6UnoRda6pBki0msUS2jSTrhp6NLuaYpppsujY/9qzpaUglzbpFw4o9iYY1ZmVThFJiK3s2okPCatD2SAiucPglGEFyxqjz3j/uZ+Ayzty5zNzv/X7v974e59wz937u98eH753hfT/vz4+vIgIzM7NG68i7AmZmVk4OMGZmlgkHGDMzy4QDjJmZZcIBxszMMnFG3hUokqlTp0Z3d3fe1TAzaynbtm17KSK6Bpc7wFTp7u6mt7c372qYmbUUSS8MVe4UmZmZZcIBxszMMuEAY2ZmmXCAMTOzTDjAmJlZJjyKzMzaWn9/sPvwcQ4c7WPapE66p0ygo0N5V6sUMm3BSJoh6TFJz0raIemWVP4nkvZK2p4eC6v2+ayknZL+UdL7qsoXpLKdkm6vKr9I0tZU/h1JZ6byX0mvd6b3u7P8t5pZ6+nvDzbs2M/ClVtYcs9WFq7cwoYd++nv9yrzjZB1iuwN4NMRMQeYB9wkaU5672sRMTc91gOk964HLgEWAP9V0jhJ44BvANcAc4AlVcf5SjrW24CXgRtT+Y3Ay6n8a2k7M7MTdh8+zq1rt9P3ej8Afa/3c+va7ew+fDznmpVDpgEmIvZFxI/S82PAc8CFNXZZBDwYEa9FxM+AncBl6bEzInZFxC+BB4FFkgRcCTyU9l8DXFt1rDXp+UPAe9P2ZmYAHDjadyK4DOh7vZ+Dx/pyqlG5NK2TP6Wo3gFsTUU3S3pa0mpJ56ayC4EXq3bbk8qGK58C/Dwi3hhUfsqx0vuvpO3NzACYNqmTzvGn/jfYOb6D887uzKlG5dKUACNpIvBd4FMRcRS4G/g1YC6wD/hqM+oxTN2WSeqV1Hvo0KG8qmFmOeieMoEVi+eeCDKd4ztYsXgu3VMm5Fyzcsh8FJmk8VSCy7ci4mGAiDhQ9f49wPfTy73AjKrdp6cyhik/DJwj6YzUSqnefuBYeySdAfxq2v4UEbEKWAXQ09Pjnj2zNtLRIRZccj4XL5/PwWN9nHe2R5E1UtajyAR8E3guIlZUlV9QtdkHgZ+k5+uA69MIsIuA2cCTwFPA7DRi7EwqAwHWRUQAjwHXpf2XAo9UHWtpen4d8IO0vZnZCR0dYlbXRObNmsqsrokOLg2UdQvm3cBHgWckbU9ln6MyCmwuEMBu4A8AImKHpLXAs1RGoN0UEW8CSLoZ2AiMA1ZHxI50vNuAByV9GfgxlYBG+vlXknYCR6gEJTMzaxL5S/1JPT094eX6zcxOj6RtEdEzuNxLxZiZWSYcYMzMLBMOMGZmlgkHGDMzy4QDjJmZZcIBxszMMuEAY2ZmmXCAMTOzTDjAmJlZJhxgzMwsEw4wZmaWCQcYMzPLhAOMmZllwgHGzMwy4QBjZmaZcIAxM7NMOMCYmVkmHGDMzCwTDjBmZpYJBxgzM8uEA4yZmWXCAcbMzDLhAGNmZplwgDEzs0yckXcFzMzaSX9/sPvwcQ4c7WPapE66p0ygo0N5VysTDjBmZk3S3x9s2LGfW9dup+/1fjrHd7Bi8VwWXHJ+ZkEmz4DmFJmZWZPsPnz8RHAB6Hu9n1vXbmf34eOZnG8goC1cuYUl92xl4cotbNixn/7+yOR8gznAmJk1yYGjfSeCy4C+1/s5eKwvk/M1O6AN5gBjZtYk0yZ10jn+1P92O8d3cN7ZnZmcr9kBbTAHGDOzJumeMoEVi+eeCDIDfTDdUyZkcr5mB7TB3MlvZtYkHR1iwSXnc/Hy+Rw81sd5Z2fb6T4Q0AYPKsgqoA2miOw6eyTNAO4HpgEBrIqIuyRNBr4DdAO7gcUR8bIkAXcBC4FfAB+LiB+lYy0FPp8O/eWIWJPKLwXuA84C1gO3REQMd45a9e3p6Yne3t6G/NvNzIpgYBRZlgFN0raI6BlcnnWK7A3g0xExB5gH3CRpDnA7sDkiZgOb02uAa4DZ6bEMuDtVfjJwB3A5cBlwh6Rz0z53A5+o2m9BKh/uHGZmbaOjQ8zqmsi8WVOZ1TWxqXNuMg0wEbFvoAUSEceA54ALgUXAmrTZGuDa9HwRcH9UPAGcI+kC4H3Apog4klohm4AF6b1JEfFEVJpi9w861lDnMLOC6u8Pdh16lR/+9CV2HXq1acNpLRtN64OR1A28A9gKTIuIfemt/VRSaFAJPi9W7bYnldUq3zNEOTXOMbhey6i0lpg5c+Zp/qvMrFHymIRo2WrKKDJJE4HvAp+KiKPV76WWR6ZfU2qdIyJWRURPRPR0dXVlWQ0zqyHvORvWeJkHGEnjqQSXb0XEw6n4QEpvkX4eTOV7gRlVu09PZbXKpw9RXuscZlZAec/ZsMbLNMCkUWHfBJ6LiBVVb60DlqbnS4FHqspvUMU84JWU5toIXC3p3NS5fzWwMb13VNK8dK4bBh1rqHOYWQHlPWfDGi/rFsy7gY8CV0ranh4LgTuBqyQ9D/x2eg2VYca7gJ3APcAnASLiCPAl4Kn0+GIqI21zb9rnp8CjqXy4c5hZATV7EqJlL9N5MK3G82DM8tWMORvWeMPNg/FMfjMrjIE5G7O6JuZdFWsAr0VmZmaZcIAxM7NMOMCYmVkmHGDMzCwTDjBmZpYJBxgzM8uEA4yZmWXCAcbMzDLhAGNmZplwgDEzs0w4wJiZWSYcYMzMLBMOMGZmlgkHGDMzy4QDjJmZZcIBxszMMuEAY2ZmmXCAMTOzTDjAmJlZJhxgzMwsEw4wZmaWCQcYMzPLhAOMmZllwgHGzMwycUbeFbDm6e8Pdh8+zoGjfUyb1En3lAl0dCjvaplZSTnAtIn+/mDDjv3cunY7fa/30zm+gxWL57LgkvMdZMwsE3WlyCT9qaRJksZL2izpkKTfz7py1ji7Dx8/EVwA+l7v59a129l9+HjONTOzsqq3D+bqiDgKfADYDbwN+ExWlbLGO3C070RwGdD3ej8Hj/XlVCMzK7t6A8xAKu39wP+IiFcyqo9lZNqkTjrHn/pxd47v4LyzO3OqkZmVXb0B5vuS/gG4FNgsqQvwV98W0j1lAisWzz0RZAb6YLqnTMi5ZmZWVoqI+jaUJgOvRMSbkiYAZ0fE/hH2WU0lrXYwIt6eyv4E+ARwKG32uYhYn977LHAj8CawPCI2pvIFwF3AOODeiLgzlV8EPAhMAbYBH42IX0r6FeB+KgHxMPCRiNg90r+xp6cnent767oerWhgFNnBY32cd7ZHkRWVR/tZq5G0LSJ6BpfXHEUm6UNDlFW/fHiE894H/AWV/+yrfS0i/mzQcecA1wOXAP8E+FtJ/yy9/Q3gKmAP8JSkdRHxLPCVdKwHJf03KsHp7vTz5Yh4m6Tr03YfGaGupdfRIWZ1TWRW18S8q2LD8Gg/K5ORUmT/Oj1uBL4J/F563Av825EOHhGPA0fqrMsi4MGIeC0ifgbsBC5Lj50RsSsifkmlxbJIlUh3JfBQ2n8NcG3Vsdak5w8B79WgyGhWRB7tZ2VSM8BExMcj4uPAeGBORHw4Ij5MpZUxfgznvVnS05JWSzo3lV0IvFi1zZ5UNlz5FODnEfHGoPJTjpXefyVt/xaSlknqldR76NChoTYxaxqP9rMyqbeTf0ZE7Kt6fQCYOcpz3g38GjAX2Ad8dZTHaYiIWBURPRHR09XVlWdVzDzaz0ql3gCzWdJGSR+T9DHgr4G/Hc0JI+JARLwZEf3APVRSYAB7gRlVm05PZcOVHwbOkXTGoPJTjpXe/9W0vVmhebSflUldS8VExM2SPgj8y1S0KiK+N5oTSrqgqjX0QeAn6fk64NuSVlDp5J8NPAkImJ1GjO2lMhDg30RESHoMuI5Kv8xS4JGqYy0Ffpje/0HUO1zOLEcdHWLBJedz8fL5Hu1nLW/EACNpHLAjIi4GTiuoSHoAuAKYKmkPcAdwhaS5QFBZFeAPACJih6S1wLPAG8BNEfFmOs7NwEYqw5RXR8SOdIrbgAclfRn4MZWBCKSffyVpJ5VBBtefTr3N8uTRflYWdc2DkfQI8B8i4v9lX6X8lH0ejLU+z5GxIhrVPJgq5wI7JD0JnBgvGRG/06D6mdkIPEfGWk29AeaPM62FmY1ouDkyFy+f73RaG2jF1mu9nfx/L2ka8K5U9GREHMyuWmY2WK05Mg4w5daqrdd67wezmMqIrt8FFgNbJV2XZcXM7FSeI9O+WnWFh3rnwfxH4F0RsTQibqAyd8VpM7Mm8hyZ9tWqKzzU2wfTMSgldpj6g5OZNcBY5si0Yv7eThpovVYHmVZovdYbYDZI2gg8kF5/BFifTZXMbDijmSPTqvl7O2mg9Tr4Myx66/V07gfzIeA96eWW0c7kLzLPg7Ey2nXoVRau3PKWb7/rPfqspRT5fk5jmgcj6Ubg8YgY6f4vZlYwHn1WDq24wkO9KbKZwF9K6qZy58jHqbRitmdULzNrkFbN31vrq6ujPiLuiIgrqdwHZgvwGSqBxswKzqPPLC/1psg+D7wbmEhlUck/pBJozKzgvEKz5aXeFNmHqKxw/NfA3wM/jIjXMquVmTVUK+bvrfXVmyJ7J/DbVGbzXwU8I+l/Z1kxM7NG6e8Pdh16lR/+9CV2HXqV/n7fHqoZ6k2RvR2YD/wroIfK/e6dIjOzwvM8oPzUOxv/TuBsYCXw6xHxWxHxn7KrlplZY7TqOl5lUO9qyh+o9b6k70bEhxtTJTOzxvE8oPw0aj2xWQ06jplZQ3kV6vw0KsC4x8ysRZW9A9zzgPJT7zBlMyuhdugA9zyg/DSqBeNPyqwFtUsH+MA8oHmzpjKra6KDS5M0KsDc1qDjmFkTteqNrKw11EyRSXqGoftXBERE/AaVJ3+TQd3MLGNeCNOyNFIfTM3hyWbW2lr1RlbWGmoGmIh4oVkVMbPmcwd4ORT1ltj1LhUzD/g68OvAmcA44HhETMqwbmbWBF4Is7UVeSRgvZ38fwEsAZ4HzgL+HfCNrCplZmb1KfJIwLpHkUXETmBcRLwZEf8dWJBdtawMyj6Bz6wIijwSsN6Jlr+QdCawXdKfAvto3BBnK6GiNdtHm6Muam7bbECRRwLWGyQ+mra9GTgOzKByEzKzIRWp2T4Q7Bau3MKSe7aycOUWNuzYP2KLarT7mTWz9V7kpXDqbcFcGxF3AX3AFwAk3QLclVXFrLUVaQXb4YLdxcvn16zLaPez9tbs1nuRRwLW24JZOkTZx0baSdJqSQcl/aSqbLKkTZKeTz/PTeWStFLSTklPS3pn1T5L0/bPS1paVX6ppGfSPislqdY5rHmKtILtaHPURc5tW3Hl0Xov6lI4NQOMpCWS/hdwkaR1VY+/A47Ucfz7eOtggNuBzRExG9icXgNcA8xOj2XA3akOk4E7gMuBy4A7qgLG3cAnqvZbMMI5rEmK1GwfbbArUpC01uEvJieNlCL7P1Q69KcCX60qPwY8PdLBI+JxSd2DihcBV6Tna4C/o7KW2SLg/ogI4AlJ50i6IG27KSKOAEjaBCxIQW5SRDyRyu8HrgUerXEOa5IiNdtHO1vds9xtNIrc6d5s9czkfwH4TUnTgHelt56LiDdGec5pEbEvPd8PTEvPLwRerNpuTyqrVb5niPJa53gLScuotJiYOXPm6f5brIaiTOAbKdgNN1KsSEHSWoe/mJxU70z+3wX+jEpLQMDXJX0mIh4ay8kjIiRlOiRnpHNExCpgFUBPT4+HB5XUcMFupA7ZogRJax3+YnJSvZ38nwfeFRFLI+IGKn0hfzzKcx5IqS/Sz4OpfC+V4c8DpqeyWuXThyivdQ6zUxRpOPVYeWJrcRS1073Z6g0wHRFR/Z/04dPYd7B1nByVthR4pKr8hjSabB7wSkpzbQSulnRu6ty/GtiY3jsqaV4aPXbDoGMNdQ6zU5SlQ9ZzdsqhbF8S6p0H86ikjcAD6fVHgPUj7STpASqd7VMl7aEyGuxOYK2kG6n07yxOm68HFgI7gV8AHweIiCOSvgQ8lbb74kCHP/BJKiPVzqLSuf9oKh/uHGanKEuHrOfstL6irX7RCPUGmAD+EnhPer0KmDfiThFLhnnrvUNsG8BNwxxnNbB6iPJe4O1DlB8e6hxmg5WlQ7ZIE1tH4uV3hlbGLwn1BpirIuI24OGBAklfwEN/rcWVpUO2VVpiZfyW3iit9CWhXiNNtPz36bbJ/zzNrh94/Iw65sGYtYIydMgWaWJrLWUaVNFoZZzYO1IL5ttU+jX+C6fOhj9W1Q9iZjnLqiXW6HRWGb+lN0oe6dqs05UjTbR8BXiFys3GzKzAGj1nJ4t0Vquk8vLQ7HRtM9KVvqeLmQ0pi3RWq6Ty8tLMdG0z0pX1dvKbWZvJIp1VlkEVZdCMdKUDjJWGh782VlbpLC+/UwzNSFc6RWal4Jnsjed0Vrk14/NVZX6jQWWxy97e3ryrYaOw69CrLFy55S3fxta38CS1IhhoFTqdVU6N+nwlbYuInsHlTpFZKXj4azacziq3rD9fp8isFMo4Sc2s1TnAWCm4v6BYyrYqsI2OU2RWCh7+Whxeb8wGuAVjpVGGNcXKwOuN2QAHGDNrqLLcxM3GzikyGxNPbrTBvN6YDXCAsVFr91y7g+vQynITN/BnPFaeaFnFEy1PTztPbmz34DqSMkzQ9Gdcv+EmWroPxkatnXPt7siurQwDLvwZj50DjAGjm7fQzpMb2zm4tgt/xmPnAGOjXigyj8mNRZnA187BtV34Mx4798FUadc+mLH0pTQz116knHiR6mLZ8GdcPy92acMay0KRzVwMcbic+MU5DCrwygHl58947BxgrGXmLRRtxWSvNFx+/ozHxn0w1jILRTonbtZa3IKxlkkFlGkCn1k7cCd/lXbt5G8lZZjAZ1Y27uS3UnBO3Kx1uA/GzMwy4QBjZmaZyC3ASNot6RlJ2yX1prLJkjZJej79PDeVS9JKSTslPS3pnVXHWZq2f17S0qryS9Pxd6Z9S5GoL8pMdhsbf47WDvLug/mtiHip6vXtwOaIuFPS7en1bcA1wOz0uBy4G7hc0mTgDqAHCGCbpHUR8XLa5hPAVmA9sAB4tDn/rGx4ZnE5+HO0dlG0FNkiYE16vga4tqr8/qh4AjhH0gXA+4BNEXEkBZVNwIL03qSIeCIqw+TurzpWy/LqruXgz9HaRZ4BJoC/kbRN0rJUNi0i9qXn+4Fp6fmFwItV++5JZbXK9wxR3tK8ums5+HO0dpFniuw9EbFX0nnAJkn/UP1mRISkzBPTKbgtA5g5c2bWpxuTVlnSxWrz52jtIrcWTETsTT8PAt8DLgMOpPQW6efBtPleYEbV7tNTWa3y6UOUD1WPVRHRExE9XV1dY/1nZapVlnSx2vw5WrvIZSa/pAlAR0QcS883AV8E3gscrurknxwRfyTp/cDNwEIqnfwrI+Ky1Mm/DRgYVfYj4NKIOCLpSWA5Jzv5vx4R62vVqxVm8nsmezn4c7QyKdpM/mnA99LI4TOAb0fEBklPAWsl3Qi8ACxO26+nElx2Ar8APg6QAsmXgKfSdl+MiCPp+SeB+4CzqIwea+kRZAM8k70c/DlaO/BaZFVaoQVjZsUw0Ao9cLSPaZPauxVatBaM2bD8h2tF57lM9SnaPBhrcwN/uAtXbmHJPVtZuHILG3bs90x3KxTPZaqPA0yJlGH5Ef/h2nCK9PvtuUz1cYosJ7XSQKNJEZWlyV602yJbMRTt99tzmerjFkwOaqWBRpsiKss3f98W2YZStN9vz2WqjwNMDmr9sYz2D6ksTXb/4dpQivb7PXCb8fXL5/PgsstZv3x+y2ULmsEpsjEaTTqr1h9LBKNKEZWlyT7wh3vx8vmehGgnFPH323OZRuYWzBiMNp1VKw002hRRmb75D/zhzps1lVldEx1crFS/3+3EEy2rnO5Ey12HXmXhyi1v+Va1fvn8mt9qanVYAqPuzPTyI9ZMzZ6v5N/v4vJEywyMdsTTSGmg0aaI3GS3ZsljVJd/v1uPU2RjMJYRT7XSQE4RWdEVbVSXFZMDzBg4L2ztqmijuqyYnCIbA494snZVxFFdVjxuwYyR01nWjtx6t3q4BWNmp82td6uHA4yZjYpHddlInCIzM7NMuAVjZqXgG9UVjwOMmbW8oi3nbxVOkZlZy/PEz2JygMlQke7AZ1ZmnvhZTE6RZcRNdrPm8cTPYnILJiNustfm1p01kid+FpNbMBnxveWH59adNZonfhaTWzAZ8b3lh+fWnWXByzYVjwNMRsrUZG90OssdsmbtwSmyjJSlyZ5FOssdsmbtwS2YDJWhyZ5FOqtMrTszG55bMFZTFoMVytK6M7PaHGCspqzSWe2wEq/XxrJ25xSZ1eR01ugM9F0tXLmFJfdsZeHKLWzYsd/zfaytKKK8v/CSFgB3AeOAeyPizlrb9/T0RG9vb1Pq1koGvok7nVW/XYdeZeHKLW9p+a1fPr/UrTZrT5K2RUTP4PLSpsgkjQO+AVwF7AGekrQuIp7Nt2atpx3SWY3mibZm5U6RXQbsjIhdEfFL4EFgUc51sjbhibZm5Q4wFwIvVr3ek8pOIWmZpF5JvYcOHWpa5azc3HdlVuIUWb0iYhWwCip9MDlXx0rCQ7HNyh1g9gIzql5PT2VmTeG+K2t3ZU6RPQXMlnSRpDOB64F1OdfJzKxtlLYFExFvSLoZ2EhlmPLqiNiRc7XMzNpGaQMMQESsB9bnXQ8zs3ZU5hSZmZnlyAHGzMwyUeqlYk6XpEPAC6PcfSrwUgOrUya+NsPztRmar8vwinht/mlEdA0udIBpEEm9Q63FY742tfjaDM3XZXitdG2cIjMzs0w4wJiZWSYcYBpnVd4VKDBfm+H52gzN12V4LXNt3AdjZmaZcAvGzMwy4QBjZmaZcIBpAEkLJP2jpJ2Sbs+7PnmStFrSQUk/qSqbLGmTpOfTz3PzrGMeJM2Q9JikZyXtkHRLKve1kTolPSnp/6Zr84VUfpGkrenv6jtp0dq2I2mcpB9L+n563TLXxQFmjKpuzXwNMAdYImlOvrXK1X3AgkFltwObI2I2sDm9bjdvAJ+OiDnAPOCm9HviawOvAVdGxL8A5gILJM0DvgJ8LSLeBrwM3JhjHfN0C/Bc1euWuS4OMGPnWzNXiYjHgSODihcBa9LzNcC1Ta1UAUTEvoj4UXp+jMp/GBfia0NUvJpejk+PAK4EHkrlbXltJE0H3g/cm16LFrouDjBjV9etmdvctIjYl57vB6blWZm8SeoG3gFsxdcGOJEG2g4cBDYBPwV+HhFvpE3a9e/qz4E/AvrT6ym00HVxgLGmisq4+LYdGy9pIvBd4FMRcbT6vXa+NhHxZkTMpXLn2cuAi3OuUu4kfQA4GBHb8q7LaJX6fjBN4lszj+yApAsiYp+kC6h8S207ksZTCS7fioiHU7GvTZWI+Lmkx4DfBM6RdEb6tt6Of1fvBn5H0kKgE5gE3EULXRe3YMbOt2Ye2TpgaXq+FHgkx7rkIuXOvwk8FxErqt7ytZG6JJ2Tnp8FXEWlj+ox4Lq0Wdtdm4j4bERMj4huKv+v/CAifo8Wui6eyd8A6RvGn3Py1sz/Oecq5UbSA8AVVJYUPwDcAfxPYC0wk8rtEBZHxOCBAKUm6T3AFuAZTubTP0elH6bdr81vUOmsHkflS+/aiPiipFlUBs1MBn4M/H5EvJZfTfMj6QrgDyPiA610XRxgzMwsE06RmZlZJhxgzMwsEw4wZmaWCQcYMzPLhAOMmZllwgHGzMwy4QBjZmaZ+P84E7OlCL873wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(data=corpus_df, x=corpus_df.index, y=\"total_words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "WfoJwxLMZfig",
   "metadata": {
    "id": "WfoJwxLMZfig"
   },
   "source": [
    "#### Most used Words by each President\n",
    "Can be used as a basis for stop words for model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "lpVRTykFh7dJ",
   "metadata": {
    "id": "lpVRTykFh7dJ"
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "h4R0Yq6xaYWK",
   "metadata": {
    "id": "h4R0Yq6xaYWK"
   },
   "outputs": [],
   "source": [
    "def most_occur(a_list,j): \n",
    "  c = Counter(a_list)\n",
    "  most_occur = c.most_common(j)\n",
    "  return most_occur\n",
    "\n",
    "def cleanup_words(df):\n",
    "  cleanup_token_all=[]  \n",
    "  for i in range(df.shape[0]):\n",
    "      speeches=(df.transcripts.iloc[i])\n",
    "      words = word_tokenize(speeches)\n",
    "      cleaned_speeches = [clean(speech) for speech in words]\n",
    "      cleanup_tokens = [tok for tok in cleaned_speeches if tok]\n",
    "      cleanup_token_all.append(cleanup_tokens) \n",
    "  #df[\"Cleaned Tokens\"]=cleanup_token_all\n",
    "  return cleanup_token_all\n",
    "\n",
    "def most_common_w(df,j):\n",
    "  cleaned_words=[]\n",
    "  for i in range(df.shape[0]): \n",
    "    most_common_words=dict(most_occur(cleanup_words(df)[i],j))\n",
    "    cleaned_words.append(most_common_words)\n",
    "  df[\"Cleaned Words\"]=cleaned_words\n",
    "  return df\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "elCKiu1IsoyM",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "elCKiu1IsoyM",
    "outputId": "f413f7e0-5b56-4477-a764-8329d24bc43a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-dbb8eaf8-9700-43b9-9f46-bce7074c4e32\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>Cleaned Words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>To Chairman Dean and my great friend Dick Durb...</td>\n",
       "      <td>{'the': 9491, 'and': 7277, 'to': 6763, 'of': 5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Chief Justice Roberts, President Carter, Presi...</td>\n",
       "      <td>{'the': 3456, 'and': 3309, 'to': 2543, 'we': 2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dbb8eaf8-9700-43b9-9f46-bce7074c4e32')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-dbb8eaf8-9700-43b9-9f46-bce7074c4e32 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-dbb8eaf8-9700-43b9-9f46-bce7074c4e32');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "       President  ...                                      Cleaned Words\n",
       "42  Barack Obama  ...  {'the': 9491, 'and': 7277, 'to': 6763, 'of': 5...\n",
       "43  Donald Trump  ...  {'the': 3456, 'and': 3309, 'to': 2543, 'we': 2...\n",
       "\n",
       "[2 rows x 3 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_common_w(subset_df,100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "lmxgDcdrN4WH",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "lmxgDcdrN4WH",
    "outputId": "76c96e36-519a-4dd5-904d-b0a9625e96d5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>total_words</th>\n",
       "      <th>Cleaned Words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>31521</td>\n",
       "      <td>{'the': 2698, 'of': 1883, 'to': 1328, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>14653</td>\n",
       "      <td>{'the': 1244, 'of': 914, 'and': 640, 'to': 553...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>35885</td>\n",
       "      <td>{'the': 2663, 'of': 1940, 'to': 1389, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>36076</td>\n",
       "      <td>{'the': 3613, 'of': 2308, 'and': 1275, 'to': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>50197</td>\n",
       "      <td>{'the': 4688, 'of': 2709, 'to': 1890, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>John Quincy Adams</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...</td>\n",
       "      <td>37038</td>\n",
       "      <td>{'the': 4010, 'of': 3109, 'to': 1244, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Andrew Jackson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: About to undertake the arduou...</td>\n",
       "      <td>158016</td>\n",
       "      <td>{'the': 15572, 'of': 9439, 'to': 6253, 'and': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Martin Van Buren</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: The practice of all my predec...</td>\n",
       "      <td>64877</td>\n",
       "      <td>{'the': 6129, 'of': 3962, 'to': 2444, 'and': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>William Harrison</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Called from a retirement which I had supposed ...</td>\n",
       "      <td>8461</td>\n",
       "      <td>{'the': 829, 'of': 604, 'to': 318, 'and': 231,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>John Tyler</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>To the People of the United States Before my a...</td>\n",
       "      <td>69743</td>\n",
       "      <td>{'the': 6870, 'of': 4452, 'to': 2789, 'and': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>James K. Polk</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow citizens: Without solicitation on my pa...</td>\n",
       "      <td>104737</td>\n",
       "      <td>{'the': 9968, 'of': 6482, 'and': 3778, 'to': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Zachary Taylor</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Elected by the American people to the highest ...</td>\n",
       "      <td>11377</td>\n",
       "      <td>{'the': 1093, 'of': 757, 'to': 483, 'and': 388...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Millard Fillmore</td>\n",
       "      <td>Whig</td>\n",
       "      <td>To the Senate and House of Representatives: I ...</td>\n",
       "      <td>39000</td>\n",
       "      <td>{'the': 3830, 'of': 2534, 'to': 1406, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Franklin Pierce</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My Countrymen: It a relief to feel that no hea...</td>\n",
       "      <td>63465</td>\n",
       "      <td>{'the': 6284, 'of': 4600, 'to': 2430, 'and': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>James Buchanan</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: I appear before you this day ...</td>\n",
       "      <td>81102</td>\n",
       "      <td>{'the': 7481, 'of': 4373, 'to': 3077, 'and': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Abraham Lincoln</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Honors To Henry ClayOn the fourth day of July,...</td>\n",
       "      <td>103386</td>\n",
       "      <td>{'the': 7991, 'of': 4671, 'and': 3282, 'to': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Andrew Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I must be permitted to say that I have been al...</td>\n",
       "      <td>95429</td>\n",
       "      <td>{'the': 9266, 'of': 5769, 'to': 3287, 'and': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ulysses S. Grant</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Citizens of the United States: Your suffrages ...</td>\n",
       "      <td>103752</td>\n",
       "      <td>{'the': 10135, 'of': 7030, 'to': 3779, 'and': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Rutherford B. Hayes</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We have assembled to repeat t...</td>\n",
       "      <td>67815</td>\n",
       "      <td>{'the': 7187, 'of': 4703, 'and': 2362, 'to': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>James A. Garfield</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We stand to-day upon an emine...</td>\n",
       "      <td>2978</td>\n",
       "      <td>{'the': 317, 'of': 182, 'and': 119, 'to': 80, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chester A. Arthur</td>\n",
       "      <td>Republican</td>\n",
       "      <td>For the fourth time in the history of the Repu...</td>\n",
       "      <td>50036</td>\n",
       "      <td>{'the': 4885, 'of': 3297, 'to': 1518, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Grover Cleveland</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: In the presence of this vast ...</td>\n",
       "      <td>150587</td>\n",
       "      <td>{'the': 13661, 'of': 9427, 'and': 5332, 'to': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Benjamin Harrison</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: There is no constitutional or...</td>\n",
       "      <td>140194</td>\n",
       "      <td>{'the': 13697, 'of': 8696, 'and': 4733, 'to': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>William McKinley</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: In obedience to the will of t...</td>\n",
       "      <td>92798</td>\n",
       "      <td>{'the': 9383, 'of': 5940, 'and': 3405, 'to': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Franklin D. Roosevelt</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>President Hoover, Mr. Chief Justice, my friend...</td>\n",
       "      <td>132240</td>\n",
       "      <td>{'the': 9602, 'of': 6444, 'and': 4752, 'to': 4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>William Taft</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Citizens: Anyone who has taken the o...</td>\n",
       "      <td>116406</td>\n",
       "      <td>{'the': 11517, 'of': 7154, 'to': 4046, 'and': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Woodrow Wilson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>There has been a change of government. It bega...</td>\n",
       "      <td>85472</td>\n",
       "      <td>{'the': 6396, 'of': 4667, 'and': 3503, 'to': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Warren G. Harding</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My countrymen, the surpassing war of all times...</td>\n",
       "      <td>28917</td>\n",
       "      <td>{'the': 2208, 'of': 1457, 'and': 1074, 'to': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Calvin Coolidge</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Since the close of the last Congress the Natio...</td>\n",
       "      <td>74147</td>\n",
       "      <td>{'the': 5594, 'of': 3913, 'to': 2346, 'and': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Herbert Hoover</td>\n",
       "      <td>Republican</td>\n",
       "      <td>This campaign now draws near a close. The plat...</td>\n",
       "      <td>89836</td>\n",
       "      <td>{'the': 7127, 'of': 5149, 'and': 2906, 'to': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Theodore Roosevelt</td>\n",
       "      <td>Republican</td>\n",
       "      <td>By the President of the United States of Ameri...</td>\n",
       "      <td>197224</td>\n",
       "      <td>{'the': 17315, 'of': 10495, 'to': 6570, 'and':...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Harry S. Truman</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Mr. Speaker, Mr. President, Members of the Con...</td>\n",
       "      <td>37078</td>\n",
       "      <td>{'the': 2705, 'of': 1429, 'and': 1269, 'to': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Dwight D. Eisenhower</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My friends, before I begin the expression of t...</td>\n",
       "      <td>18117</td>\n",
       "      <td>{'the': 1325, 'of': 937, 'and': 653, 'to': 565...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>John F. Kennedy</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Governor Stevenson, Senator Johnson, Mr. Butle...</td>\n",
       "      <td>122162</td>\n",
       "      <td>{'the': 7353, 'of': 4895, 'and': 4547, 'to': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Lyndon B. Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>On this hallowed ground, heroic deeds were per...</td>\n",
       "      <td>251584</td>\n",
       "      <td>{'the': 15291, 'of': 8801, 'to': 8711, 'and': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Richard M. Nixon</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Americans: I come before you tonight...</td>\n",
       "      <td>68204</td>\n",
       "      <td>{'the': 4361, 'of': 2381, 'to': 2370, 'and': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Gerald Ford</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Mr. Chief Justice, my dear friends, my fellow ...</td>\n",
       "      <td>40739</td>\n",
       "      <td>{'the': 2417, 'of': 1538, 'and': 1427, 'to': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I am Edwin Newman, moderator of this first deb...</td>\n",
       "      <td>102470</td>\n",
       "      <td>{'the': 6239, 'of': 3397, 'to': 3263, 'and': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Ronald Reagan</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Thank you. Thank you very much. Thank you and ...</td>\n",
       "      <td>209691</td>\n",
       "      <td>{'the': 11873, 'and': 7250, 'of': 7040, 'to': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>I have many friends to thank tonight. I thank ...</td>\n",
       "      <td>91875</td>\n",
       "      <td>{'the': 5191, 'and': 3387, 'to': 3049, 'of': 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Bill Clinton</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My fellow citizens, today we celebrate the mys...</td>\n",
       "      <td>137663</td>\n",
       "      <td>{'the': 6917, 'to': 5534, 'and': 4843, 'of': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>President Clinton, distinguished guests and my...</td>\n",
       "      <td>108811</td>\n",
       "      <td>{'the': 5838, 'and': 4835, 'to': 3589, 'of': 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>To Chairman Dean and my great friend Dick Durb...</td>\n",
       "      <td>207813</td>\n",
       "      <td>{'the': 9491, 'and': 7277, 'to': 6763, 'of': 5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Chief Justice Roberts, President Carter, Presi...</td>\n",
       "      <td>85453</td>\n",
       "      <td>{'the': 3456, 'and': 3309, 'to': 2543, 'we': 2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                President                  Party  \\\n",
       "0       George Washington           Unaffiliated   \n",
       "1              John Adams             Federalist   \n",
       "2        Thomas Jefferson  Democratic-Republican   \n",
       "3           James Madison  Democratic-Republican   \n",
       "4            James Monroe  Democratic-Republican   \n",
       "5       John Quincy Adams  Democratic-Republican   \n",
       "6          Andrew Jackson             Democratic   \n",
       "7        Martin Van Buren             Democratic   \n",
       "8        William Harrison                   Whig   \n",
       "9              John Tyler           Unaffiliated   \n",
       "10          James K. Polk             Democratic   \n",
       "11         Zachary Taylor                   Whig   \n",
       "12       Millard Fillmore                   Whig   \n",
       "13        Franklin Pierce             Democratic   \n",
       "14         James Buchanan             Democratic   \n",
       "15        Abraham Lincoln             Republican   \n",
       "16         Andrew Johnson             Democratic   \n",
       "17       Ulysses S. Grant             Republican   \n",
       "18    Rutherford B. Hayes             Republican   \n",
       "19      James A. Garfield             Republican   \n",
       "20      Chester A. Arthur             Republican   \n",
       "21       Grover Cleveland             Democratic   \n",
       "22      Benjamin Harrison             Republican   \n",
       "23       William McKinley             Republican   \n",
       "24  Franklin D. Roosevelt             Democratic   \n",
       "25           William Taft             Republican   \n",
       "26         Woodrow Wilson             Democratic   \n",
       "27      Warren G. Harding             Republican   \n",
       "28        Calvin Coolidge             Republican   \n",
       "29         Herbert Hoover             Republican   \n",
       "30     Theodore Roosevelt             Republican   \n",
       "31        Harry S. Truman             Democratic   \n",
       "32   Dwight D. Eisenhower             Republican   \n",
       "33        John F. Kennedy             Democratic   \n",
       "34      Lyndon B. Johnson             Democratic   \n",
       "35       Richard M. Nixon             Republican   \n",
       "36            Gerald Ford             Republican   \n",
       "37           Jimmy Carter             Democratic   \n",
       "38          Ronald Reagan             Republican   \n",
       "39      George H. W. Bush             Republican   \n",
       "40           Bill Clinton             Democratic   \n",
       "41         George W. Bush             Republican   \n",
       "42           Barack Obama             Democratic   \n",
       "43           Donald Trump             Republican   \n",
       "\n",
       "                                          transcripts  total_words  \\\n",
       "0   Fellow Citizens of the Senate and the House of...        31521   \n",
       "1   When it was first perceived, in early times, t...        14653   \n",
       "2   FRIENDS AND FELLOW-CITIZENS, Called upon to un...        35885   \n",
       "3   Unwilling to depart from examples of the most ...        36076   \n",
       "4   I should be destitute of feeling if I was not ...        50197   \n",
       "5   AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...        37038   \n",
       "6   Fellow Citizens: About to undertake the arduou...       158016   \n",
       "7   Fellow Citizens: The practice of all my predec...        64877   \n",
       "8   Called from a retirement which I had supposed ...         8461   \n",
       "9   To the People of the United States Before my a...        69743   \n",
       "10  Fellow citizens: Without solicitation on my pa...       104737   \n",
       "11  Elected by the American people to the highest ...        11377   \n",
       "12  To the Senate and House of Representatives: I ...        39000   \n",
       "13  My Countrymen: It a relief to feel that no hea...        63465   \n",
       "14  Fellow Citizens: I appear before you this day ...        81102   \n",
       "15  Honors To Henry ClayOn the fourth day of July,...       103386   \n",
       "16  I must be permitted to say that I have been al...        95429   \n",
       "17  Citizens of the United States: Your suffrages ...       103752   \n",
       "18  Fellow Citizens: We have assembled to repeat t...        67815   \n",
       "19  Fellow Citizens: We stand to-day upon an emine...         2978   \n",
       "20  For the fourth time in the history of the Repu...        50036   \n",
       "21  Fellow Citizens: In the presence of this vast ...       150587   \n",
       "22  Fellow Citizens: There is no constitutional or...       140194   \n",
       "23  Fellow Citizens: In obedience to the will of t...        92798   \n",
       "24  President Hoover, Mr. Chief Justice, my friend...       132240   \n",
       "25  My Fellow Citizens: Anyone who has taken the o...       116406   \n",
       "26  There has been a change of government. It bega...        85472   \n",
       "27  My countrymen, the surpassing war of all times...        28917   \n",
       "28  Since the close of the last Congress the Natio...        74147   \n",
       "29  This campaign now draws near a close. The plat...        89836   \n",
       "30  By the President of the United States of Ameri...       197224   \n",
       "31  Mr. Speaker, Mr. President, Members of the Con...        37078   \n",
       "32  My friends, before I begin the expression of t...        18117   \n",
       "33  Governor Stevenson, Senator Johnson, Mr. Butle...       122162   \n",
       "34  On this hallowed ground, heroic deeds were per...       251584   \n",
       "35  My Fellow Americans: I come before you tonight...        68204   \n",
       "36  Mr. Chief Justice, my dear friends, my fellow ...        40739   \n",
       "37  I am Edwin Newman, moderator of this first deb...       102470   \n",
       "38  Thank you. Thank you very much. Thank you and ...       209691   \n",
       "39  I have many friends to thank tonight. I thank ...        91875   \n",
       "40  My fellow citizens, today we celebrate the mys...       137663   \n",
       "41  President Clinton, distinguished guests and my...       108811   \n",
       "42  To Chairman Dean and my great friend Dick Durb...       207813   \n",
       "43  Chief Justice Roberts, President Carter, Presi...        85453   \n",
       "\n",
       "                                        Cleaned Words  \n",
       "0   {'the': 2698, 'of': 1883, 'to': 1328, 'and': 1...  \n",
       "1   {'the': 1244, 'of': 914, 'and': 640, 'to': 553...  \n",
       "2   {'the': 2663, 'of': 1940, 'to': 1389, 'and': 1...  \n",
       "3   {'the': 3613, 'of': 2308, 'and': 1275, 'to': 1...  \n",
       "4   {'the': 4688, 'of': 2709, 'to': 1890, 'and': 1...  \n",
       "5   {'the': 4010, 'of': 3109, 'to': 1244, 'and': 1...  \n",
       "6   {'the': 15572, 'of': 9439, 'to': 6253, 'and': ...  \n",
       "7   {'the': 6129, 'of': 3962, 'to': 2444, 'and': 2...  \n",
       "8   {'the': 829, 'of': 604, 'to': 318, 'and': 231,...  \n",
       "9   {'the': 6870, 'of': 4452, 'to': 2789, 'and': 2...  \n",
       "10  {'the': 9968, 'of': 6482, 'and': 3778, 'to': 3...  \n",
       "11  {'the': 1093, 'of': 757, 'to': 483, 'and': 388...  \n",
       "12  {'the': 3830, 'of': 2534, 'to': 1406, 'and': 1...  \n",
       "13  {'the': 6284, 'of': 4600, 'to': 2430, 'and': 2...  \n",
       "14  {'the': 7481, 'of': 4373, 'to': 3077, 'and': 2...  \n",
       "15  {'the': 7991, 'of': 4671, 'and': 3282, 'to': 3...  \n",
       "16  {'the': 9266, 'of': 5769, 'to': 3287, 'and': 3...  \n",
       "17  {'the': 10135, 'of': 7030, 'to': 3779, 'and': ...  \n",
       "18  {'the': 7187, 'of': 4703, 'and': 2362, 'to': 2...  \n",
       "19  {'the': 317, 'of': 182, 'and': 119, 'to': 80, ...  \n",
       "20  {'the': 4885, 'of': 3297, 'to': 1518, 'and': 1...  \n",
       "21  {'the': 13661, 'of': 9427, 'and': 5332, 'to': ...  \n",
       "22  {'the': 13697, 'of': 8696, 'and': 4733, 'to': ...  \n",
       "23  {'the': 9383, 'of': 5940, 'and': 3405, 'to': 2...  \n",
       "24  {'the': 9602, 'of': 6444, 'and': 4752, 'to': 4...  \n",
       "25  {'the': 11517, 'of': 7154, 'to': 4046, 'and': ...  \n",
       "26  {'the': 6396, 'of': 4667, 'and': 3503, 'to': 2...  \n",
       "27  {'the': 2208, 'of': 1457, 'and': 1074, 'to': 1...  \n",
       "28  {'the': 5594, 'of': 3913, 'to': 2346, 'and': 2...  \n",
       "29  {'the': 7127, 'of': 5149, 'and': 2906, 'to': 2...  \n",
       "30  {'the': 17315, 'of': 10495, 'to': 6570, 'and':...  \n",
       "31  {'the': 2705, 'of': 1429, 'and': 1269, 'to': 1...  \n",
       "32  {'the': 1325, 'of': 937, 'and': 653, 'to': 565...  \n",
       "33  {'the': 7353, 'of': 4895, 'and': 4547, 'to': 3...  \n",
       "34  {'the': 15291, 'of': 8801, 'to': 8711, 'and': ...  \n",
       "35  {'the': 4361, 'of': 2381, 'to': 2370, 'and': 1...  \n",
       "36  {'the': 2417, 'of': 1538, 'and': 1427, 'to': 1...  \n",
       "37  {'the': 6239, 'of': 3397, 'to': 3263, 'and': 3...  \n",
       "38  {'the': 11873, 'and': 7250, 'of': 7040, 'to': ...  \n",
       "39  {'the': 5191, 'and': 3387, 'to': 3049, 'of': 2...  \n",
       "40  {'the': 6917, 'to': 5534, 'and': 4843, 'of': 3...  \n",
       "41  {'the': 5838, 'and': 4835, 'to': 3589, 'of': 3...  \n",
       "42  {'the': 9491, 'and': 7277, 'to': 6763, 'of': 5...  \n",
       "43  {'the': 3456, 'and': 3309, 'to': 2543, 'we': 2...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_common_w(corpus_df,100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "P1BB3e2D9dCR",
   "metadata": {
    "id": "P1BB3e2D9dCR"
   },
   "source": [
    "#### Most used Words by all President\n",
    "Shall be used as stop words for model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "I_tY046BBw_W",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "I_tY046BBw_W",
    "outputId": "ddc883bb-2968-4a31-d58b-eb8de1530202"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 295710),\n",
       " ('of', 186742),\n",
       " ('and', 130829),\n",
       " ('to', 129621),\n",
       " ('in', 81502),\n",
       " ('a', 62747),\n",
       " ('that', 58416),\n",
       " ('is', 39010),\n",
       " ('be', 37900),\n",
       " ('for', 37775),\n",
       " ('it', 36836),\n",
       " ('we', 35989),\n",
       " ('our', 34063),\n",
       " ('i', 31620),\n",
       " ('have', 28478),\n",
       " ('by', 27973),\n",
       " ('this', 27816),\n",
       " ('as', 26032),\n",
       " ('with', 24308),\n",
       " ('not', 23023),\n",
       " ('which', 21524),\n",
       " ('are', 21064),\n",
       " ('will', 20755),\n",
       " ('on', 20122),\n",
       " ('has', 16800),\n",
       " ('their', 16472),\n",
       " ('all', 16296),\n",
       " ('or', 15955),\n",
       " ('been', 15752),\n",
       " ('from', 15414),\n",
       " ('but', 14664),\n",
       " ('they', 14471),\n",
       " ('at', 13673),\n",
       " ('was', 13588),\n",
       " ('states', 13147),\n",
       " ('you', 12214),\n",
       " ('government', 12212),\n",
       " ('an', 12146),\n",
       " ('can', 11513),\n",
       " ('people', 11298),\n",
       " ('united', 10125),\n",
       " ('its', 9665),\n",
       " ('would', 9490),\n",
       " ('who', 9155),\n",
       " ('so', 9138),\n",
       " ('more', 9108),\n",
       " ('these', 8947),\n",
       " ('no', 8850),\n",
       " ('s', 8754),\n",
       " ('there', 8491),\n",
       " ('them', 8304),\n",
       " ('if', 7840),\n",
       " ('any', 7544),\n",
       " ('other', 7448),\n",
       " ('congress', 7386),\n",
       " ('should', 7362),\n",
       " ('now', 7328),\n",
       " ('do', 7189),\n",
       " ('those', 7124),\n",
       " ('one', 7064),\n",
       " ('country', 6975),\n",
       " ('than', 6931),\n",
       " ('such', 6753),\n",
       " ('upon', 6605),\n",
       " ('my', 6571),\n",
       " ('were', 6332),\n",
       " ('time', 6165),\n",
       " ('may', 5977),\n",
       " ('great', 5736),\n",
       " ('us', 5606),\n",
       " ('world', 5502),\n",
       " ('american', 5472),\n",
       " ('must', 5417),\n",
       " ('his', 5196),\n",
       " ('new', 5040),\n",
       " ('what', 5009),\n",
       " ('when', 4958),\n",
       " ('made', 4765),\n",
       " ('had', 4665),\n",
       " ('he', 4578),\n",
       " ('under', 4230),\n",
       " ('every', 4198),\n",
       " ('public', 4079),\n",
       " ('war', 4076),\n",
       " ('only', 3889),\n",
       " ('shall', 3811),\n",
       " ('year', 3641),\n",
       " ('america', 3636),\n",
       " ('years', 3618),\n",
       " ('president', 3537),\n",
       " ('your', 3486),\n",
       " ('some', 3434),\n",
       " ('peace', 3371),\n",
       " ('nt', 3330),\n",
       " ('state', 3124),\n",
       " ('nation', 2807),\n",
       " ('about', 2782),\n",
       " ('law', 2766),\n",
       " ('into', 2716),\n",
       " ('last', 2513)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import functools, operator\n",
    "most_common_sum = dict(functools.reduce(operator.add,\n",
    "         map(collections.Counter, corpus_df[\"Cleaned Words\"])))\n",
    "\n",
    "stop_words_real =most_common_sum\n",
    "\n",
    "stop_words_sorted = sorted(stop_words_real.items(), key=lambda kv: kv[1], reverse=True)\n",
    "stop_words_sorted[:100]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "MCdGNjZNbkVU",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MCdGNjZNbkVU",
    "outputId": "95e8d050-c169-4275-a8a4-a9dc0091e3b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41,\n",
       " 67,\n",
       " ['the',\n",
       "  'of',\n",
       "  'and',\n",
       "  'to',\n",
       "  'in',\n",
       "  'a',\n",
       "  'that',\n",
       "  'is',\n",
       "  'be',\n",
       "  'for',\n",
       "  'it',\n",
       "  'we',\n",
       "  'our',\n",
       "  'i',\n",
       "  'have',\n",
       "  'by',\n",
       "  'this',\n",
       "  'as',\n",
       "  'with',\n",
       "  'not',\n",
       "  'which',\n",
       "  'are',\n",
       "  'will',\n",
       "  'on',\n",
       "  'has',\n",
       "  'their',\n",
       "  'all',\n",
       "  'or',\n",
       "  'been',\n",
       "  'from',\n",
       "  'but',\n",
       "  'they',\n",
       "  'at',\n",
       "  'was',\n",
       "  'states',\n",
       "  'you',\n",
       "  'government',\n",
       "  'an',\n",
       "  'can',\n",
       "  'people',\n",
       "  'united',\n",
       "  'its',\n",
       "  'would',\n",
       "  'who',\n",
       "  'so',\n",
       "  'more',\n",
       "  'these',\n",
       "  'no',\n",
       "  's',\n",
       "  'there',\n",
       "  'them',\n",
       "  'if',\n",
       "  'any',\n",
       "  'other',\n",
       "  'congress',\n",
       "  'should',\n",
       "  'now',\n",
       "  'do',\n",
       "  'those',\n",
       "  'one',\n",
       "  'country',\n",
       "  'than',\n",
       "  'such',\n",
       "  'upon',\n",
       "  'my',\n",
       "  'were',\n",
       "  'time'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def stops_used(m):\n",
    "  stop_words_used=[]\n",
    "  for item in stop_words_sorted:\n",
    "    if item[1]>m:\n",
    "      stop_words_used.append(item[0])\n",
    "  return(stop_words_used)\n",
    "\n",
    "len(stops_used(10000)), len(stops_used(6000)), stops_used(6000) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b08bbc8",
   "metadata": {
    "id": "1b08bbc8"
   },
   "source": [
    "## No. of Sentences\n",
    "1. Normalize text (No. of sentences)\n",
    "2. No. of unique sentences\n",
    "3. Most used/repeated sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b244e59",
   "metadata": {
    "id": "4b244e59"
   },
   "source": [
    "<a name=\"normalize\"></a>\n",
    "#### 1. Normalize text\n",
    "Start with a subset again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "976f2095",
   "metadata": {
    "id": "976f2095",
    "outputId": "6a50e42e-f13e-45c3-d149-8f254930f6c7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Barack Obama',\n",
       " ['To Chairman Dean and my great friend Dick Durbin; and to all my fellow citizens of this great nation; With profound gratitude and great humility, I accept your nomination for the presidency of the United States.',\n",
       "  'Let me express my thanks to the historic slate of candidates who accompanied me on this journey, and especially the one who traveled the farthest, a champion for working Americans and an inspiration to my daughters and to yours, Hillary Rodham Clinton.',\n",
       "  'To President Clinton, who last night made the case for change as only he can make it; to Ted Kennedy, who embodies the spirit of service; and to the next Vice President of the United States, Joe Biden, I thank you.',\n",
       "  'I am grateful to finish this journey with one of the finest statesmen of our time, a man at ease with everyone from world leaders to the conductors on the Amtrak train he still takes home every night.',\n",
       "  'To the love of my life, our next First Lady, Michelle Obama, and to Sasha and Malia, I love you so much, and be: ( 1 so proud of all of you.',\n",
       "  \"Four years ago, I stood before you and told you my story, of the brief union between a young man from Kenya and a young woman from Kansas who weren't well off or well known, but shared a belief that in America, their son could achieve whatever he put his mind to.\",\n",
       "  'It is that promise that has always set this country apart, that through hard work and sacrifice, each of us can pursue our individual dreams but still come together as one American family, to ensure that the next generation can pursue their dreams as well.',\n",
       "  \"That's why I stand here tonight.\",\n",
       "  'Because for two hundred and thirty two years, at each moment when that promise was in jeopardy, ordinary men and women, students and soldiers, farmers and teachers, nurses and janitors, found the courage to keep it alive.',\n",
       "  'We meet at one of those defining moments, a moment when our nation is at war, our economy is in turmoil, and the American promise has been threatened once more.',\n",
       "  'Tonight, more Americans are out of work and more are working harder for less.',\n",
       "  'More of you have lost your homes and even more are watching your home values plummet.',\n",
       "  \"More of you have cars you can't afford to drive, credit card bills you can't afford to pay, and tuition that's beyond your reach.\",\n",
       "  \"These challenges are not all of government's making.\",\n",
       "  'But the failure to respond is a direct result of a broken politics in Washington and the failed policies of George W. Bush.',\n",
       "  'America, we are better than these last eight years.',\n",
       "  'We are a better country than this.',\n",
       "  'This country is more decent than one where a woman in Ohio, on the brink of retirement, finds herself one illness away from disaster after a lifetime of hard work.',\n",
       "  \"This country is more generous than one where a man in Indiana has to pack up the equipment he's worked on for twenty years and watch it shipped off to China, and then chokes up as he explains how he felt like a failure when he went home to tell his family the news.\",\n",
       "  'We are more compassionate than a government that lets veterans sleep on our streets and families slide into poverty; that sits on its hands while a major American city drowns before our eyes.',\n",
       "  'Tonight, I say to the American people, to Democrats and Republicans and Independents across this great land, enough!',\n",
       "  'This moment, this election, is our chance to keep, in the 21st century, the American promise alive.',\n",
       "  'Because next week, in Minnesota, the same party that brought you two terms of George Bush and Dick Cheney will ask this country for a third.',\n",
       "  'And we are here because we love this country too much to let the next four years look like the last eight.',\n",
       "  'On November 4th, we must stand up and say: “Eight is enough.” Now let there be no doubt.',\n",
       "  'The Republican nominee, John McCain, has worn the uniform of our country with bravery and distinction, and for that we owe him our gratitude and respect.',\n",
       "  \"And next week, we'll also hear about those occasions when he's broken with his party as evidence that he can deliver the change that we need.\",\n",
       "  \"But the record's clear: John McCain has voted with George Bush ninety percent of the time.\",\n",
       "  'Senator McCain likes to talk about judgment, but really, what does it say about your judgment when you think George Bush has been right more than ninety percent of the time?',\n",
       "  \"I don't know about you, but be: ( 1 not ready to take a ten percent chance on change.\",\n",
       "  'The truth is, on issue after issue that would make a difference in your lives, on health care and education and the economy, Senator McCain has been anything but independent.',\n",
       "  'He said that our economy has made “great progress” under this President.',\n",
       "  'He said that the fundamentals of the economy are strong.',\n",
       "  \"And when one of his chief advisors, the man who wrote his economic plan, was talking about the anxiety Americans are feeling, he said that we were just suffering from a “mental recession,” and that we've become, and I quote, “a nation of whiners.” A nation of whiners?\",\n",
       "  'Tell that to the proud auto workers at a Michigan plant who, after they found out it was closing, kept showing up every day and working as hard as ever, because they knew there were people who counted on the brakes that they made.',\n",
       "  'Tell that to the military families who shoulder their burdens silently as they watch their loved ones leave for their third or fourth or fifth tour of duty.',\n",
       "  'These are not whiners.',\n",
       "  'They work hard and give back and keep going without complaint.',\n",
       "  'These are the Americans that I know.',\n",
       "  \"Now, I don't believe that Senator McCain doesn't care what's going on in the lives of Americans.\",\n",
       "  \"I just think he doesn't know.\",\n",
       "  'Why else would he define middle class as someone making under five million dollars a year?',\n",
       "  'How else could he propose hundreds of billions in tax breaks for big corporations and oil companies but not one penny of tax relief to more than one hundred million Americans?',\n",
       "  \"How else could he offer a health care plan that would actually tax people's benefits, or an education plan that would do nothing to help families pay for college, or a plan that would privatize Social Security and gamble your retirement?\",\n",
       "  \"It's not because John McCain doesn't care.\",\n",
       "  \"It's because John McCain doesn't get it.\",\n",
       "  \"For over two decades, he's subscribed to that old, discredited Republican philosophy, give more and more to those with the most and hope that prosperity trickles down to everyone else.\",\n",
       "  \"In Washington, they call this the Ownership Society, but what it really means is, you're on your own.\",\n",
       "  'Out of work?',\n",
       "  'Tough luck.',\n",
       "  'No health care?',\n",
       "  'The market will fix it.',\n",
       "  'Born into poverty?',\n",
       "  \"Pull yourself up by your own bootstraps, even if you don't have boots.\",\n",
       "  \"You're on your own.\",\n",
       "  \"Well it's time for them to own their failure.\",\n",
       "  \"It's time for us to change America.\",\n",
       "  'You see, we Democrats have a very different measure of what constitutes progress in this country.',\n",
       "  'We measure progress by how many people can find a job that pays the mortgage; whether you can put a little extra money away at the end of each month so you can someday watch your child receive her college diploma.',\n",
       "  'We measure progress in the 23 million new jobs that were created when Bill Clinton was President, when the average American family saw its income go up $ 7,500 instead of down $ 2,000 like it has under George Bush.',\n",
       "  'We measure the strength of our economy not by the number of billionaires we have or the profits of the Fortune 500, but by whether someone with a good idea can take a risk and start a new business, or whether the waitress who lives on tips can take a day off to look after a sick kid without losing her job, an economy that honors the dignity of work.',\n",
       "  'The fundamentals we use to measure economic strength are whether we are living up to that fundamental promise that has made this country great, a promise that is the only reason I am standing here tonight.',\n",
       "  \"Because in the faces of those young veterans who come back from Iraq and Afghanistan, I see my grandfather, who signed up after Pearl Harbor, marched in Patton's Army, and was rewarded by a grateful nation with the chance to go to college on the GI Bill.\",\n",
       "  'In the face of that young student who sleeps just three hours before working the night shift, I think about my mom, who raised my sister and me on her own while she worked and earned her degree; who once turned to food stamps but was still able to send us to the best schools in the country with the help of student loans and scholarships.',\n",
       "  'When I listen to another worker tell me that his factory has shut down, I remember all those men and women on the South Side of Chicago who I stood by and fought for two decades ago after the local steel plant closed.',\n",
       "  'And when I hear a woman talk about the difficulties of starting her own business, I think about my grandmother, who worked her way up from the secretarial pool to middle management, despite years of being passed over for promotions because she was a woman.',\n",
       "  \"She's the one who taught me about hard work.\",\n",
       "  \"She's the one who put off buying a new car or a new dress for herself so that I could have a better life.\",\n",
       "  'She poured everything she had into me.',\n",
       "  \"And although she can no longer travel, I know that she's watching tonight, and that tonight is her night as well.\",\n",
       "  \"I don't know what kind of lives John McCain thinks that celebrities lead, but this has been mine.\",\n",
       "  'These are my heroes.',\n",
       "  'Theirs are the stories that shaped me.',\n",
       "  'And it is on their behalf that I intend to win this election and keep our promise alive as President of the United States.',\n",
       "  'What is that promise?',\n",
       "  \"It's a promise that says each of us has the freedom to make of our own lives what we will, but that we also have the obligation to treat each other with dignity and respect.\",\n",
       "  \"It's a promise that says the market should reward drive and innovation and generate growth, but that businesses should live up to their responsibilities to create American jobs, look out for American workers, and play by the rules of the road.\",\n",
       "  'Ours is a promise that says government can not solve all our problems, but what it should do is that which we can not do for ourselves, protect us from harm and provide every child a decent education; keep our water clean and our toys safe; invest in new schools and new roads and new science and technology.',\n",
       "  'Our government should work for us, not against us.',\n",
       "  'It should help us, not hurt us.',\n",
       "  \"It should ensure opportunity not just for those with the most money and influence, but for every American who's willing to work.\",\n",
       "  \"That's the promise of America, the idea that we are responsible for ourselves, but that we also rise or fall as one nation; the fundamental belief that I am my brother's keeper; I am my sister's keeper.\",\n",
       "  \"That's the promise we need to keep.\",\n",
       "  \"That's the change we need right now.\",\n",
       "  'So let me spell out exactly what that change would mean if I am President.',\n",
       "  \"Change means a tax code that doesn't reward the lobbyists who wrote it, but the American workers and small businesses who deserve it.\",\n",
       "  'Unlike John McCain, I will stop giving tax breaks to corporations that ship jobs overseas, and I will start giving them to companies that create good jobs right here in America.',\n",
       "  'I will eliminate capital gains taxes for the small businesses and the start-ups that will create the high-wage, high-tech jobs of tomorrow.',\n",
       "  'I will cut taxes, cut taxes, for 95 percent of all working families.',\n",
       "  'Because in an economy like this, the last thing we should do is raise taxes on the middle class.',\n",
       "  'And for the sake of our economy, our security, and the future of our planet, I will set a clear goal as President: in ten years, we will finally end our dependence on oil from the Middle East.',\n",
       "  \"Washington's been talking about our oil addiction for the last thirty years, and John McCain has been there for twenty-six of them.\",\n",
       "  \"In that time, he's said no to higher fuel-efficiency standards for cars, no to investments in renewable energy, no to renewable fuels.\",\n",
       "  'And today, we import triple the amount of oil as the day that Senator McCain took office.',\n",
       "  'Now is the time to end this addiction, and to understand that drilling is a stop-gap measure, not a long term solution.',\n",
       "  'Not even close.',\n",
       "  'As President, I will tap our natural gas reserves, invest in clean coal technology, and find ways to safely harness nuclear power.',\n",
       "  \"I'll help our auto companies re tool, so that the fuel-efficient cars of the future are built right here in America.\",\n",
       "  \"I'll make it easier for the American people to afford these new cars.\",\n",
       "  \"And I'll invest 150 billion dollars over the next decade in affordable, renewable sources of energy, wind power and solar power and the next generation of biofuels; an investment that will lead to new industries and five million new jobs that pay well and can't ever be outsourced.\",\n",
       "  'America, now is not the time for small plans.',\n",
       "  'Now is the time to finally meet our moral obligation to provide every child a world class education, because it will take nothing less to compete in the global economy.',\n",
       "  'Michelle and I are only here tonight because we were given a chance at an education.',\n",
       "  \"And I will not settle for an America where some kids don't have that chance.\",\n",
       "  \"I'll invest in early childhood education.\",\n",
       "  \"I'll recruit an army of new teachers, and pay them higher salaries and give them more support.\",\n",
       "  \"And in exchange, I'll ask for higher standards and more accountability.\",\n",
       "  'And we will keep our promise to every young American - if you commit to serving your community or your country, we will make sure you can afford a college education.',\n",
       "  'Now is the time to finally keep the promise of affordable, accessible health care for every single American.',\n",
       "  'If you have health care, my plan will lower your premiums.',\n",
       "  \"If you don't, you'll be able to get the same kind of coverage that members of Congress give themselves.\",\n",
       "  'And as someone who watched my mother argue with insurance companies while she lay in bed dying of cancer, I will make certain those companies stop discriminating against those who are sick and need care the most.',\n",
       "  'Now is the time to help families with paid sick days and better family leave, because nobody in America should have to choose between keeping their jobs and caring for a sick child or ailing parent.',\n",
       "  'Now is the time to change our bankruptcy laws, so that your pensions are protected ahead of CEO bonuses; and the time to protect Social Security for future generations.',\n",
       "  \"And now is the time to keep the promise of equal pay for an equal day's work, because I want my daughters to have exactly the same opportunities as your sons.\",\n",
       "  \"Now, many of these plans will cost money, which is why I've laid out how I'll pay for every dime, by closing corporate loopholes and tax havens that don't help America grow.\",\n",
       "  'But I will also go through the federal budget, line by line, eliminating programs that no longer work and making the ones we do need work better and cost less, because we can not meet twenty-first century challenges with a twentieth century bureaucracy.',\n",
       "  \"And Democrats, we must also admit that fulfilling America's promise will require more than just money.\",\n",
       "  'It will require a renewed sense of responsibility from each of us to recover what John F. Kennedy called our “intellectual and moral strength.” Yes, government must lead on energy independence, but each of us must do our part to make our homes and businesses more efficient.',\n",
       "  'Yes, we must provide more ladders to success for young men who fall into lives of crime and despair.',\n",
       "  \"But we must also admit that programs alone can't replace parents; that government can't turn off the television and make a child do her homework; that fathers must take more responsibility for providing the love and guidance their children need.\",\n",
       "  \"Individual responsibility and mutual responsibility, that's the essence of America's promise.\",\n",
       "  \"And just as we keep our keep our promise to the next generation here at home, so must we keep America's promise abroad.\",\n",
       "  \"If John McCain wants to have a debate about who has the temperament, and judgment, to serve as the next Commander-in-Chief, that's a debate be: ( 1 ready to have.\",\n",
       "  'For while Senator McCain was turning his sights to Iraq just days after 9/11, I stood up and opposed this war, knowing that it would distract us from the real threats we face.',\n",
       "  'When John McCain said we could just “muddle through” in Afghanistan, I argued for more resources and more troops to finish the fight against the terrorists who actually attacked us on 9/11, and made clear that we must take out Osama bin Laden and his lieutenants if we have them in our sights.',\n",
       "  \"John McCain likes to say that he'll follow bin Laden to the Gates of Hell, but he won't even go to the cave where he lives.\",\n",
       "  \"And today, as my call for a time frame to remove our troops from Iraq has been echoed by the Iraqi government and even the Bush Administration, even after we learned that Iraq has a $ 79 billion surplus while we're wallowing in deficits, John McCain stands alone in his stubborn refusal to end a misguided war.\",\n",
       "  \"That's not the judgment we need.\",\n",
       "  \"That won't keep America safe.\",\n",
       "  'We need a President who can face the threats of the future, not keep grasping at the ideas of the past.',\n",
       "  \"You don't defeat a terrorist network that operates in eighty countries by occupying Iraq.\",\n",
       "  \"You don't protect Israel and deter Iran just by talking tough in Washington.\",\n",
       "  \"You can't truly stand up for Georgia when you've strained our oldest alliances.\",\n",
       "  'If John McCain wants to follow George Bush with more tough talk and bad strategy, that is his choice, but it is not the change we need.',\n",
       "  'We are the party of Roosevelt.',\n",
       "  'We are the party of Kennedy.',\n",
       "  \"So don't tell me that Democrats won't defend this country.\",\n",
       "  \"Don't tell me that Democrats won't keep us safe.\",\n",
       "  'The Bush-McCain foreign policy has squandered the legacy that generations of Americans, Democrats and Republicans, have built, and we are here to restore that legacy.',\n",
       "  \"As Commander-in-Chief, I will never hesitate to defend this nation, but I will only send our troops into harm's way with a clear mission and a sacred commitment to give them the equipment they need in battle and the care and benefits they deserve when they come home.\",\n",
       "  'I will end this war in Iraq responsibly, and finish the fight against al Qaeda and the Taliban in Afghanistan.',\n",
       "  'I will rebuild our military to meet future conflicts.',\n",
       "  'But I will also renew the tough, direct diplomacy that can prevent Iran from obtaining nuclear weapons and curb Russian aggression.',\n",
       "  'I will build new partnerships to defeat the threats of the 21st century: terrorism and nuclear proliferation; poverty and genocide; climate change and disease.',\n",
       "  'And I will restore our moral standing, so that America is once again that last, best hope for all who are called to the cause of freedom, who long for lives of peace, and who yearn for a better future.',\n",
       "  'These are the policies I will pursue.',\n",
       "  'And in the weeks ahead, I look forward to debating them with John McCain.',\n",
       "  'But what I will not do is suggest that the Senator takes his positions for political purposes.',\n",
       "  \"Because one of the things that we have to change in our politics is the idea that people can not disagree without challenging each other's character and patriotism.\",\n",
       "  'The times are too serious, the stakes are too high for this same partisan playbook.',\n",
       "  'So let us agree that patriotism has no party.',\n",
       "  'I love this country, and so do you, and so does John McCain.',\n",
       "  'The men and women who serve in our battlefields may be Democrats and Republicans and Independents, but they have fought together and bled together and some died together under the same proud flag.',\n",
       "  'They have not served a Red America or a Blue America, they have served the United States of America.',\n",
       "  \"So I've got news for you, John McCain.\",\n",
       "  'We all put our country first.',\n",
       "  'America, our work will not be easy.',\n",
       "  'The challenges we face require tough choices, and Democrats as well as Republicans will need to cast off the worn out ideas and politics of the past.',\n",
       "  \"For part of what has been lost these past eight years can't just be measured by lost wages or bigger trade deficits.\",\n",
       "  'What has also been lost is our sense of common purpose, our sense of higher purpose.',\n",
       "  \"And that's what we have to restore.\",\n",
       "  'We may not agree on abortion, but surely we can agree on reducing the number of unwanted pregnancies in this country.',\n",
       "  \"The reality of gun ownership may be different for hunters in rural Ohio than for those plagued by gang-violence in Cleveland, but don't tell me we can't uphold the Second Amendment while keeping AK-47s out of the hands of criminals.\",\n",
       "  'I know there are differences on same-sex marriage, but surely we can agree that our gay and lesbian brothers and sisters deserve to visit the person they love in the hospital and to live lives free of discrimination.',\n",
       "  \"Passions fly on immigration, but I don't know anyone who benefits when a mother is separated from her infant child or an employer undercuts American wages by hiring illegal workers.\",\n",
       "  \"This too is part of America's promise, the promise of a democracy where we can find the strength and grace to bridge divides and unite in common effort.\",\n",
       "  'I know there are those who dismiss such beliefs as happy talk.',\n",
       "  'They claim that our insistence on something larger, something firmer and more honest in our public life is just a Trojan Horse for higher taxes and the abandonment of traditional values.',\n",
       "  \"And that's to be expected.\",\n",
       "  \"Because if you don't have any fresh ideas, then you use stale tactics to scare the voters.\",\n",
       "  \"If you don't have a record to run on, then you paint your opponent as someone people should run from.\",\n",
       "  'You make a big election about small things.',\n",
       "  \"And you know what, it's worked before.\",\n",
       "  'Because it feeds into the cynicism we all have about government.',\n",
       "  \"When Washington doesn't work, all its promises seem empty.\",\n",
       "  \"If your hopes have been dashed again and again, then it's best to stop hoping, and settle for what you already know.\",\n",
       "  'I get it.',\n",
       "  'I realize that I am not the likeliest candidate for this office.',\n",
       "  \"I don't fit the typical pedigree, and I haven't spent my career in the halls of Washington.\",\n",
       "  'But I stand before you tonight because all across America something is stirring.',\n",
       "  \"What the nay-sayers don't understand is that this election has never been about me.\",\n",
       "  \"It's been about you.\",\n",
       "  'For eighteen long months, you have stood up, one by one, and said enough to the politics of the past.',\n",
       "  'You understand that in this election, the greatest risk we can take is to try the same old politics with the same old players and expect a different result.',\n",
       "  \"You have shown what history teaches us, that at defining moments like this one, the change we need doesn't come from Washington.\",\n",
       "  'Change comes to Washington.',\n",
       "  'Change happens because the American people demand it, because they rise up and insist on new ideas and new leadership, a new politics for a new time.',\n",
       "  'America, this is one of those moments.',\n",
       "  'I believe that as hard as it will be, the change we need is coming.',\n",
       "  \"Because I've seen it.\",\n",
       "  \"Because I've lived it.\",\n",
       "  \"I've seen it in Illinois, when we provided health care to more children and moved more families from welfare to work.\",\n",
       "  \"I've seen it in Washington, when we worked across party lines to open up government and hold lobbyists more accountable, to give better care for our veterans and keep nuclear weapons out of terrorist hands.\",\n",
       "  \"And I've seen it in this campaign.\",\n",
       "  'In the young people who voted for the first time, and in those who got involved again after a very long time.',\n",
       "  \"In the Republicans who never thought they'd pick up a Democratic ballot, but did.\",\n",
       "  \"I've seen it in the workers who would rather cut their hours back a day than see their friends lose their jobs, in the soldiers who re enlist after losing a limb, in the good neighbors who take a stranger in when a hurricane strikes and the floodwaters rise.\",\n",
       "  \"This country of ours has more wealth than any nation, but that's not what makes us rich.\",\n",
       "  \"We have the most powerful military on Earth, but that's not what makes us strong.\",\n",
       "  \"Our universities and our culture are the envy of the world, but that's not what keeps the world coming to our shores.\",\n",
       "  'Instead, it is that American spirit, that American promise, that pushes us forward even when the path is uncertain; that binds us together in spite of our differences; that makes us fix our eye not on what is seen, but what is unseen, that better place around the bend.',\n",
       "  'That promise is our greatest inheritance.',\n",
       "  \"It's a promise I make to my daughters when I tuck them in at night, and a promise that you make to yours, a promise that has led immigrants to cross oceans and pioneers to travel west; a promise that led workers to picket lines, and women to reach for the ballot.\",\n",
       "  \"And it is that promise that forty five years ago today, brought Americans from every corner of this land to stand together on a Mall in Washington, before Lincoln's Memorial, and hear a young preacher from Georgia speak of his dream.\",\n",
       "  \"The men and women who gathered there could've heard many things.\",\n",
       "  \"They could've heard words of anger and discord.\",\n",
       "  \"They could've been told to succumb to the fear and frustration of so many dreams deferred.\",\n",
       "  'But what the people heard instead, people of every creed and color, from every walk of life, is that in America, our destiny is inextricably linked.',\n",
       "  'That together, our dreams can be one.',\n",
       "  '“We can not walk alone,” the preacher cried.',\n",
       "  '“And as we walk, we must make the pledge that we shall always march ahead.',\n",
       "  'We can not turn back.” America, we can not turn back.',\n",
       "  'Not with so much work to be done.',\n",
       "  'Not with so many children to educate, and so many veterans to care for.',\n",
       "  'Not with an economy to fix and cities to rebuild and farms to save.',\n",
       "  'Not with so many families to protect and so many lives to mend.',\n",
       "  'America, we can not turn back.',\n",
       "  'We can not walk alone.',\n",
       "  'At this moment, in this election, we must pledge once more to march into the future.',\n",
       "  'Let us keep that promise - that American promise - and in the words of Scripture hold firmly, without wavering, to the hope that we confess.',\n",
       "  'Thank you, God Bless you, and God Bless the United States of America If there is anyone out there who still doubts that America is a place where all things are possible; who still wonders if the dream of our founders is alive in our time; who still questions the power of our democracy, tonight is your answer.',\n",
       "  \"It's the answer told by lines that stretched around schools and churches in numbers this nation has never seen; by people who waited three hours and four hours, many for the very first time in their lives, because they believed that this time must be different; that their voice could be that difference.\",\n",
       "  \"It's the answer spoken by young and old, rich and poor, Democrat and Republican, black, white, Latino, Asian, Native American, gay, straight, disabled and not disabled, Americans who sent a message to the world that we have never been a collection of Red States and Blue States: we are, and always will be, the United States of America.\",\n",
       "  \"It's the answer that led those who have been told for so long by so many to be cynical, and fearful, and doubtful of what we can achieve to put their hands on the arc of history and bend it once more toward the hope of a better day.\",\n",
       "  \"It's been a long time coming, but tonight, because of what we did on this day, in this election, at this defining moment, change has come to America.\",\n",
       "  'I just received a very gracious call from Senator McCain.',\n",
       "  \"He fought long and hard in this campaign, and he's fought even longer and harder for the country he loves.\",\n",
       "  'He has endured sacrifices for America that most of us can not begin to imagine, and we are better off for the service rendered by this brave and selfless leader.',\n",
       "  \"I congratulate him and Governor Palin for all they have achieved, and I look forward to working with them to renew this nation's promise in the months ahead.\",\n",
       "  'I want to thank my partner in this journey, a man who campaigned from his heart and spoke for the men and women he grew up with on the streets of Scranton and rode with on that train home to Delaware, the Vice President-elect of the United States, Joe Biden.',\n",
       "  \"I would not be standing here tonight without the unyielding support of my best friend for the last sixteen years, the rock of our family and the love of my life, our nation's next First Lady, Michelle Obama.\",\n",
       "  \"Sasha and Malia, I love you both so much, and you have earned the new puppy that's coming with us to the White House.\",\n",
       "  \"And while she's no longer with us, I know my grandmother is watching, along with the family that made me who I am.\",\n",
       "  'I miss them tonight, and know that my debt to them is beyond measure.',\n",
       "  \"To my campaign manager David Plouffe, my chief strategist David Axelrod, and the best campaign team ever assembled in the history of politics, you made this happen, and I am forever grateful for what you've sacrificed to get it done.\",\n",
       "  'But above all, I will never forget who this victory truly belongs to, it belongs to you.',\n",
       "  'I was never the likeliest candidate for this office.',\n",
       "  \"We didn't start with much money or many endorsements.\",\n",
       "  'Our campaign was not hatched in the halls of Washington, it began in the backyards of Des Moines and the living rooms of Concord and the front porches of Charleston.',\n",
       "  'It was built by working men and women who dug into what little savings they had to give five dollars and ten dollars and twenty dollars to this cause.',\n",
       "  \"It grew strength from the young people who rejected the myth of their generation's apathy; who left their homes and their families for jobs that offered little pay and less sleep; from the not-so-young people who braved the bitter cold and scorching heat to knock on the doors of perfect strangers; from the millions of Americans who volunteered, and organized, and proved that more than two centuries later, a government of the people, by the people and for the people has not perished from this Earth.\",\n",
       "  'This is your victory.',\n",
       "  \"I know you didn't do this just to win an election and I know you didn't do it for me.\",\n",
       "  'You did it because you understand the enormity of the task that lies ahead.',\n",
       "  'For even as we celebrate tonight, we know the challenges that tomorrow will bring are the greatest of our lifetime, two wars, a planet in peril, the worst financial crisis in a century.',\n",
       "  'Even as we stand here tonight, we know there are brave Americans waking up in the deserts of Iraq and the mountains of Afghanistan to risk their lives for us.',\n",
       "  \"There are mothers and fathers who will lie awake after their children fall asleep and wonder how they'll make the mortgage, or pay their doctor's bills, or save enough for college.\",\n",
       "  'There is new energy to harness and new jobs to be created; new schools to build and threats to meet and alliances to repair.',\n",
       "  'The road ahead will be long.',\n",
       "  'Our climb will be steep.',\n",
       "  'We may not get there in one year or even one term, but America, I have never been more hopeful than I am tonight that we will get there.',\n",
       "  'I promise you, we as a people will get there.',\n",
       "  'There will be setbacks and false starts.',\n",
       "  \"There are many who won't agree with every decision or policy I make as President, and we know that government can't solve every problem.\",\n",
       "  'But I will always be honest with you about the challenges we face.',\n",
       "  'I will listen to you, especially when we disagree.',\n",
       "  \"And above all, I will ask you join in the work of remaking this nation the only way it's been done in America for two hundred and twenty-one years, block by block, brick by brick, calloused hand by calloused hand.\",\n",
       "  'What began twenty-one months ago in the depths of winter must not end on this autumn night.',\n",
       "  'This victory alone is not the change we seek, it is only the chance for us to make that change.',\n",
       "  'And that can not happen if we go back to the way things were.',\n",
       "  'It can not happen without you.',\n",
       "  'So let us summon a new spirit of patriotism; of service and responsibility where each of us resolves to pitch in and work harder and look after not only ourselves, but each other.',\n",
       "  \"Let us remember that if this financial crisis taught us anything, it's that we can not have a thriving Wall Street while Main Street suffers, in this country, we rise or fall as one nation; as one people.\",\n",
       "  'Let us resist the temptation to fall back on the same partisanship and pettiness and immaturity that has poisoned our politics for so long.',\n",
       "  'Let us remember that it was a man from this state who first carried the banner of the Republican Party to the White House, a party founded on the values of self reliance, individual liberty, and national unity.',\n",
       "  'Those are values we all share, and while the Democratic Party has won a great victory tonight, we do so with a measure of humility and determination to heal the divides that have held back our progress.',\n",
       "  'As Lincoln said to a nation far more divided than ours, “We are not enemies, but friends.. though passion may have strained it must not break our bonds of affection.” And to those Americans whose support I have yet to earn, I may not have won your vote, but I hear your voices, I need your help, and I will be your President too.',\n",
       "  'And to all those watching tonight from beyond our shores, from parliaments and palaces to those who are huddled around radios in the forgotten corners of our world, our stories are singular, but our destiny is shared, and a new dawn of American leadership is at hand.',\n",
       "  'To those who would tear this world down, we will defeat you.',\n",
       "  'To those who seek peace and security, we support you.',\n",
       "  \"And to all those who have wondered if America's beacon still burns as bright, tonight we proved once more that the true strength of our nation comes not from our the might of our arms or the scale of our wealth, but from the enduring power of our ideals: democracy, liberty, opportunity, and unyielding hope.\",\n",
       "  'For that is the true genius of America, that America can change.',\n",
       "  'Our union can be perfected.',\n",
       "  'And what we have already achieved gives us hope for what we can and must achieve tomorrow.',\n",
       "  'This election had many firsts and many stories that will be told for generations.',\n",
       "  \"But one that's on my mind tonight is about a woman who cast her ballot in Atlanta.\",\n",
       "  \"She's a lot like the millions of others who stood in line to make their voice heard in this election except for one thing, Ann Nixon Cooper is 106 years old.\",\n",
       "  \"She was born just a generation past slavery; a time when there were no cars on the road or planes in the sky; when someone like her couldn't vote for two reasons, because she was a woman and because of the color of her skin.\",\n",
       "  \"And tonight, I think about all that she's seen throughout her century in America, the heartache and the hope; the struggle and the progress; the times we were told that we can't, and the people who pressed on with that American creed: Yes we can.\",\n",
       "  \"At a time when women's voices were silenced and their hopes dismissed, she lived to see them stand up and speak out and reach for the ballot.\",\n",
       "  'Yes we can.',\n",
       "  'When there was despair in the dust bowl and depression across the land, she saw a nation conquer fear itself with a New Deal, new jobs and a new sense of common purpose.',\n",
       "  'Yes we can.',\n",
       "  'When the bombs fell on our harbor and tyranny threatened the world, she was there to witness a generation rise to greatness and a democracy was saved.',\n",
       "  'Yes we can.',\n",
       "  'She was there for the buses in Montgomery, the hoses in Birmingham, a bridge in Selma, and a preacher from Atlanta who told a people that “We Shall Overcome.” Yes we can.',\n",
       "  'A man touched down on the moon, a wall came down in Berlin, a world was connected by our own science and imagination.',\n",
       "  'And this year, in this election, she touched her finger to a screen, and cast her vote, because after 106 years in America, through the best of times and the darkest of hours, she knows how America can change.',\n",
       "  'Yes we can.',\n",
       "  'America, we have come so far.',\n",
       "  'We have seen so much.',\n",
       "  'But there is so much more to do.',\n",
       "  'So tonight, let us ask ourselves, if our children should live to see the next century; if my daughters should be so lucky to live as long as Ann Nixon Cooper, what change will they see?',\n",
       "  'What progress will we have made?',\n",
       "  'This is our chance to answer that call.',\n",
       "  'This is our moment.',\n",
       "  \"This is our time, to put our people back to work and open doors of opportunity for our kids; to restore prosperity and promote the cause of peace; to reclaim the American Dream and reaffirm that fundamental truth, that out of many, we are one; that while we breathe, we hope, and where we are met with cynicism, and doubt, and those who tell us that we can't, we will respond with that timeless creed that sums up the spirit of a people: Yes We Can.\",\n",
       "  'Thank you, God bless you, and may God Bless the United States of America I stand here today humbled by the task before us, grateful for the trust you have bestowed, mindful of the sacrifices borne by our ancestors.',\n",
       "  'I thank President Bush for his service to our nation, as well as the generosity and cooperation he has shown throughout this transition.',\n",
       "  'Forty-four Americans have now taken the presidential oath.',\n",
       "  'The words have been spoken during rising tides of prosperity and the still waters of peace.',\n",
       "  'Yet, every so often the oath is taken amidst gathering clouds and raging storms.',\n",
       "  'At these moments, America has carried on not simply because of the skill or vision of those in high office, but because We the People have remained faithful to the ideals of our forbearers, and true to our founding documents.',\n",
       "  'So it has been.',\n",
       "  'So it must be with this generation of Americans.',\n",
       "  'That we are in the midst of crisis is now well understood.',\n",
       "  'Our nation is at war, against a far-reaching network of violence and hatred.',\n",
       "  'Our economy is badly weakened, a consequence of greed and irresponsibility on the part of some, but also our collective failure to make hard choices and prepare the nation for a new age.',\n",
       "  'Homes have been lost; jobs shed; businesses shuttered.',\n",
       "  'Our health care is too costly; our schools fail too many; and each day brings further evidence that the ways we use energy strengthen our adversaries and threaten our planet.',\n",
       "  'These are the indicators of crisis, subject to data and statistics.',\n",
       "  \"Less measurable but no less profound is a sapping of confidence across our land, a nagging fear that America's decline is inevitable, and that the next generation must lower its sights.\",\n",
       "  'Today I say to you that the challenges we face are real.',\n",
       "  'They are serious and they are many.',\n",
       "  'They will not be met easily or in a short span of time.',\n",
       "  'But know this, America, they will be met.',\n",
       "  'On this day, we gather because we have chosen hope over fear, unity of purpose over conflict and discord.',\n",
       "  'On this day, we come to proclaim an end to the petty grievances and false promises, the recriminations and worn out dogmas, that for far too long have strangled our politics.',\n",
       "  'We remain a young nation, but in the words of Scripture, the time has come to set aside childish things.',\n",
       "  'The time has come to reaffirm our enduring spirit; to choose our better history; to carry forward that precious gift, that noble idea, passed on from generation to generation: the God given promise that all are equal, all are free, and all deserve a chance to pursue their full measure of happiness.',\n",
       "  'In reaffirming the greatness of our nation, we understand that greatness is never a given.',\n",
       "  'It must be earned.',\n",
       "  'Our journey has never been one of short-cuts or settling for less.',\n",
       "  'It has not been the path for the faint-hearted, for those who prefer leisure over work, or seek only the pleasures of riches and fame.',\n",
       "  'Rather, it has been the risk-takers, the doers, the makers of things, some celebrated but more often men and women obscure in their labor, who have carried us up the long, rugged path towards prosperity and freedom.',\n",
       "  'For us, they packed up their few worldly possessions and traveled across oceans in search of a new life.',\n",
       "  'For us, they toiled in sweatshops and settled the West; endured the lash of the whip and plowed the hard earth.',\n",
       "  'For us, they fought and died, in places like Concord and Gettysburg; Normandy and Khe Sahn.',\n",
       "  'Time and again these men and women struggled and sacrificed and worked till their hands were raw so that we might live a better life.',\n",
       "  'They saw America as bigger than the sum of our individual ambitions; greater than all the differences of birth or wealth or faction.',\n",
       "  'This is the journey we continue today.',\n",
       "  'We remain the most prosperous, powerful nation on Earth.',\n",
       "  'Our workers are no less productive than when this crisis began.',\n",
       "  'Our minds are no less inventive, our goods and services no less needed than they were last week or last month or last year.',\n",
       "  'Our capacity remains undiminished.',\n",
       "  'But our time of standing pat, of protecting narrow interests and putting off unpleasant decisions, that time has surely passed.',\n",
       "  'Starting today, we must pick ourselves up, dust ourselves off, and begin again the work of remaking America.',\n",
       "  'For everywhere we look, there is work to be done.',\n",
       "  'The state of the economy calls for action, bold and swift, and we will act, not only to create new jobs, but to lay a new foundation for growth.',\n",
       "  'We will build the roads and bridges, the electric grids and digital lines that feed our commerce and bind us together.',\n",
       "  \"We will restore science to its rightful place, and wield technology's wonders to raise health care's quality and lower its cost.\",\n",
       "  'We will harness the sun and the winds and the soil to fuel our cars and run our factories.',\n",
       "  'And we will transform our schools and colleges and universities to meet the demands of a new age.',\n",
       "  'All this we can do.',\n",
       "  'And all this we will do.',\n",
       "  'Now, there are some who question the scale of our ambitions, who suggest that our system can not tolerate too many big plans.',\n",
       "  'Their memories are short.',\n",
       "  'For they have forgotten what this country has already done; what free men and women can achieve when imagination is joined to common purpose, and necessity to courage.',\n",
       "  'What the cynics fail to understand is that the ground has shifted beneath them, that the stale political arguments that have consumed us for so long no longer apply.',\n",
       "  'The question we ask today is not whether our government is too big or too small, but whether it works, whether it helps families find jobs at a decent wage, care they can afford, a retirement that is dignified.',\n",
       "  'Where the answer is yes, we intend to move forward.',\n",
       "  'Where the answer is no, programs will end.',\n",
       "  \"And those of us who manage the public's dollars will be held to account, to spend wisely, reform bad habits, and do our business in the light of day, because only then can we restore the vital trust between a people and their government.\",\n",
       "  'Nor is the question before us whether the market is a force for good or ill. Its power to generate wealth and expand freedom is unmatched, but this crisis has reminded us that without a watchful eye, the market can spin out of control, and that a nation can not prosper long when it favors only the prosperous.',\n",
       "  'The success of our economy has always depended not just on the size of our Gross Domestic Product, but on the reach of our prosperity; on our ability to extend opportunity to every willing heart, not out of charity, but because it is the surest route to our common good.',\n",
       "  'As for our common defense, we reject as false the choice between our safety and our ideals.',\n",
       "  'Our Founding Fathers, faced with perils we can scarcely imagine, drafted a charter to assure the rule of law and the rights of man, a charter expanded by the blood of generations.',\n",
       "  \"Those ideals still light the world, and we will not give them up for expedience's sake.\",\n",
       "  'And so to all other peoples and governments who are watching today, from the grandest capitals to the small village where my father was born: know that America is a friend of each nation and every man, woman, and child who seeks a future of peace and dignity, and that we are ready to lead once more.',\n",
       "  'Recall that earlier generations faced down fascism and communism not just with missiles and tanks, but with sturdy alliances and enduring convictions.',\n",
       "  'They understood that our power alone can not protect us, nor does it entitle us to do as we please.',\n",
       "  'Instead, they knew that our power grows through its prudent use; our security emanates from the justness of our cause, the force of our example, the tempering qualities of humility and restraint.',\n",
       "  'We are the keepers of this legacy.',\n",
       "  'Guided by these principles once more, we can meet those new threats that demand even greater effort, even greater cooperation and understanding between nations.',\n",
       "  'We will begin to responsibly leave Iraq to its people, and forge a hard earned peace in Afghanistan.',\n",
       "  'With old friends and former foes, we will work tirelessly to lessen the nuclear threat, and roll back the specter of a warming planet.',\n",
       "  'We will not apologize for our way of life, nor will we waver in its defense, and for those who seek to advance their aims by inducing terror and slaughtering innocents, we say to you now that our spirit is stronger and can not be broken; you can not outlast us, and we will defeat you.',\n",
       "  'For we know that our patchwork heritage is a strength, not a weakness.',\n",
       "  'We are a nation of Christians and Muslims, Jews and Hindus and non believers.',\n",
       "  'We are shaped by every language and culture, drawn from every end of this Earth; and because we have tasted the bitter swill of civil war and segregation, and emerged from that dark chapter stronger and more united, we can not help but believe that the old hatreds shall someday pass; that the lines of tribe shall soon dissolve; that as the world grows smaller, our common humanity shall reveal itself; and that America must play its role in ushering in a new era of peace.',\n",
       "  'To the Muslim world, we seek a new way forward, based on mutual interest and mutual respect.',\n",
       "  \"To those leaders around the globe who seek to sow conflict, or blame their society's ills on the West, know that your people will judge you on what you can build, not what you destroy.\",\n",
       "  'To those who cling to power through corruption and deceit and the silencing of dissent, know that you are on the wrong side of history; but that we will extend a hand if you are willing to unclench your fist.',\n",
       "  'To the people of poor nations, we pledge to work alongside you to make your farms flourish and let clean waters flow; to nourish starved bodies and feed hungry minds.',\n",
       "  \"And to those nations like ours that enjoy relative plenty, we say we can no longer afford indifference to suffering outside our borders; nor can we consume the world's resources without regard to effect.\",\n",
       "  'For the world has changed, and we must change with it.',\n",
       "  'As we consider the road that unfolds before us, we remember with humble gratitude those brave Americans who, at this very hour, patrol far off deserts and distant mountains.',\n",
       "  'They have something to tell us today, just as the fallen heroes who lie in Arlington whisper through the ages.',\n",
       "  'We honor them not only because they are guardians of our liberty, but because they embody the spirit of service; a willingness to find meaning in something greater than themselves.',\n",
       "  'And yet, at this moment, a moment that will define a generation, it is precisely this spirit that must inhabit us all.',\n",
       "  'For as much as government can do and must do, it is ultimately the faith and determination of the American people upon which this nation relies.',\n",
       "  'It is the kindness to take in a stranger when the levees break, the selflessness of workers who would rather cut their hours than see a friend lose their job which sees us through our darkest hours.',\n",
       "  \"It is the firefighter's courage to storm a stairway filled with smoke, but also a parent's willingness to nurture a child, that finally decides our fate.\",\n",
       "  'Our challenges may be new.',\n",
       "  'The instruments with which we meet them may be new.',\n",
       "  'But those values upon which our success depends, hard work and honesty, courage and fair play, tolerance and curiosity, loyalty and patriotism, these things are old.',\n",
       "  'These things are true.',\n",
       "  'They have been the quiet force of progress throughout our history.',\n",
       "  'What is demanded then is a return to these truths.',\n",
       "  'What is required of us now is a new era of responsibility, a recognition, on the part of every American, that we have duties to ourselves, our nation, and the world, duties that we do not grudgingly accept but rather seize gladly, firm in the knowledge that there is nothing so satisfying to the spirit, so defining of our character, than giving our all to a difficult task.',\n",
       "  'This is the price and the promise of citizenship.',\n",
       "  'This is the source of our confidence, the knowledge that God calls on us to shape an uncertain destiny.',\n",
       "  'This is the meaning of our liberty and our creed, why men and women and children of every race and every faith can join in celebration across this magnificent mall, and why a man whose father less than sixty years ago might not have been served at a local restaurant can now stand before you to take a most sacred oath.',\n",
       "  'So let us mark this day with remembrance, of who we are and how far we have traveled.',\n",
       "  \"In the year of America's birth, in the coldest of months, a small band of patriots huddled by dying campfires on the shores of an icy river.\",\n",
       "  'The capital was abandoned.',\n",
       "  'The enemy was advancing.',\n",
       "  'The snow was stained with blood.',\n",
       "  'At a moment when the outcome of our revolution was most in doubt, the father of our nation ordered these words be read to the people: “Let it be told to the future world.',\n",
       "  'that in the depth of winter, when nothing but hope and virtue could survive.',\n",
       "  'that the city and the country, alarmed at one common danger, came forth to meet [ it ].” America.',\n",
       "  'In the face of our common dangers, in this winter of our hardship, let us remember these timeless words.',\n",
       "  'With hope and virtue, let us brave once more the icy currents, and endure what storms may come.',\n",
       "  \"Let it be said by our children's children that when we were tested we refused to let this journey end, that we did not turn back nor did we falter; and with eyes fixed on the horizon and God's grace upon us, we carried forth that great gift of freedom and delivered it safely to future generations.\",\n",
       "  'Thank you.',\n",
       "  'God bless you.',\n",
       "  \"And may God bless the United States of America It is fitting that with the very first bill I sign – the Lilly Ledbetter Fair Pay Restoration Act – we are upholding one of this nation's first principles: that we are all created equal and each deserve a chance to pursue our own version of happiness.\",\n",
       "  'It is also fitting that we are joined today by the woman after whom this bill is named – someone Michelle and I have had the privilege of getting to know for ourselves.',\n",
       "  \"Lilly Ledbetter didn't set out to be a trailblazer or a household name.\",\n",
       "  'She was just a good hard worker who did her job – and did it well – for nearly two decades before discovering that for years, she was paid less than her male colleagues for the very same work.',\n",
       "  'Over the course of her career, she lost more than $ 200,000 in salary, and even more in pension and Social Security benefits – losses she still feels today.',\n",
       "  'Now, Lilly could have accepted her lot and moved on.',\n",
       "  \"She could have decided that it wasn't worth the hassle and harassment that would inevitably come with speaking up for what she deserved.\",\n",
       "  'But instead, she decided that there was a principle at stake, something worth fighting for.',\n",
       "  'So she set out on a journey that would take more than ten years, take her all the way to the Supreme Court, and lead to this bill which will help others get the justice she was denied.',\n",
       "  \"Because while this bill bears her name, Lilly knows this story isn't just about her.\",\n",
       "  \"It's the story of women across this country still earning just 78 cents for every dollar men earn – women of color even less – which means that today, in the year 2009, countless women are still losing thousands of dollars in salary, income and retirement savings over the course of a lifetime.\",\n",
       "  \"But equal pay is by no means just a women's issue – it's a family issue.\",\n",
       "  \"It's about parents who find themselves with less money for tuition or child care; couples who wind up with less to retire on; households where, when one breadwinner is paid less than she deserves, that's the difference between affording the mortgage – or not; between keeping the heat on, or paying the doctor's bills – or not.\",\n",
       "  \"And in this economy, when so many folks are already working harder for less and struggling to get by, the last thing they can afford is losing part of each month's paycheck to simple discrimination.\",\n",
       "  'So in signing this bill today, I intend to send a clear message: That making our economy work means making sure it works for everyone.',\n",
       "  \"That there are no second class citizens in our workplaces, and that it's not just unfair and illegal – but bad for business – to pay someone less because of their gender, age, race, ethnicity, religion or disability.\",\n",
       "  \"And that justice isn't about some abstract legal theory, or footnote in a casebook – it's about how our laws affect the daily realities of people's lives: their ability to make a living and care for their families and achieve their goals.\",\n",
       "  \"Ultimately, though, equal pay isn't just an economic issue for millions of Americans and their families, it's a question of who we are – and whether we're truly living up to our fundamental ideals.\",\n",
       "  \"Whether we'll do our part, as generations before us, to ensure those words put to paper more than 200 years ago really mean something – to breathe new life into them with the more enlightened understandings of our time.\",\n",
       "  'That is what Lilly Ledbetter challenged us to do.',\n",
       "  'And today, I sign this bill not just in her honor, but in honor of those who came before her.',\n",
       "  'Women like my grandmother who worked in a bank all her life, and even after she hit that glass ceiling, kept getting up and giving her best every day, without complaint, because she wanted something better for me and my sister.',\n",
       "  'And I sign this bill for my daughters, and all those who will come after us, because I want them to grow up in a nation that values their contributions, where there are no limits to their dreams and they have opportunities their mothers and grandmothers never could have imagined.',\n",
       "  \"In the end, that's why Lilly stayed the course.\",\n",
       "  \"She knew it was too late for her – that this bill wouldn't undo the years of injustice she faced or restore the earnings she was denied.\",\n",
       "  'But this grandmother from Alabama kept on fighting, because she was thinking about the next generation.',\n",
       "  \"It's what we've always done in America – set our sights high for ourselves, but even higher for our children and grandchildren.\",\n",
       "  \"Now it's up to us to continue this work.\",\n",
       "  'This bill is an important step – a simple fix to ensure fundamental fairness to American workers – and I want to thank this remarkable and nonearning group of legislators who worked so hard to get it passed.',\n",
       "  'And this is only the beginning.',\n",
       "  \"I know that if we stay focused, as Lilly did – and keep standing for what's right, as Lilly did – we will close that pay gap and ensure that our daughters have the same rights, the same chances, and the same freedom to pursue their dreams as our sons.\",\n",
       "  'Thank you Thank you, everybody.',\n",
       "  'Please have a seat.',\n",
       "  'You guys can sit down, too.',\n",
       "  '( Laughter. )',\n",
       "  'Let me begin by saying thank you to a few people first of all, your outstanding Governor, Bill Ritter.',\n",
       "  'Please give Bill a big round of applause.',\n",
       "  \"Lieutenant Governor Barbara O'Brien.\",\n",
       "  'Secretary of State Bernie Buescher.',\n",
       "  'Your outstanding Mayor, John Hickenlooper.',\n",
       "  'Your new Senator, Michael Bennett.',\n",
       "  'Your old senator, now my Secretary of the Interior, Ken Salazar.',\n",
       "  'Mark Udall is not here, but give him a round of applause anyway.',\n",
       "  'One of the outstanding leaders who helped shepherd this process through in record time please give Max Baucus of Montana a big round of applause.',\n",
       "  'Thank you, Max.',\n",
       "  'To Secretary Federico Pena, one of my national reenact I would not be here if it were not for Federico.',\n",
       "  'Thank you.',\n",
       "  'To Representative Diana DeGette, who is a we are in her district.',\n",
       "  'So, thank you so much.',\n",
       "  'Representative Betsy Markey.',\n",
       "  'Representative Jared Polis.',\n",
       "  'Representative Ed Perlmutter.',\n",
       "  'To all the other elected officials and outstanding leaders who are here.',\n",
       "  'And to the whole Namaste family and Mr. Jones for outstanding work, congratulations.',\n",
       "  'Give them a big round of applause.',\n",
       "  \"And to the best Vice President that we've had in a long time Joe Biden.\",\n",
       "  'It is great to be back in Denver.',\n",
       "  'I was here last summer we had a good time to accept the nomination of my party and to make a promise to people of all parties that I would do all that I could to give every American the chance to make of their lives what they will; to see their children climb higher than they did.',\n",
       "  'And be: ( 1 back today to say that we have begun the difficult work of keeping that promise.',\n",
       "  'We have begun the essential work of keeping the American Dream alive in our time.',\n",
       "  \"And that's why we're here today.\",\n",
       "  \"Now, I don't want to pretend that today marks the end of our economic problems.\",\n",
       "  \"Nor does it constitute all of what we're going to have to do to turn our economy around.\",\n",
       "  \"But today does mark the beginning of the end the beginning of what we need to do to create jobs for Americans scrambling in the wake of layoffs; the beginning of what we need to do to provide relief for families worried they won't be able to pay next month's bills; the beginning of the first steps to set our economy on a firmer foundation, paving the way to long term growth and prosperity.\",\n",
       "  'The American Recovery and Reinvestment Act that I will sign today a plan that meets the principles I laid out in January is the most sweeping economic recovery package in our history.',\n",
       "  \"It's the product of broad consultation and the recipient of broad support from business leaders, unions, public interest groups, from the Chamber of Commerce and the National Association of Manufacturers, as well as the AFL-CIO.\",\n",
       "  'From Democrats and Republicans, mayors as well as governors.',\n",
       "  \"It's a rare thing in Washington for people with such diverse and different viewpoints to come together and support the same bill.\",\n",
       "  'And on behalf of our nation, I want to thank all of them for it, including your two outstanding Senators, Michael Bennett and Mark Udall, as well as all the members of your congressional delegation.',\n",
       "  'They did an outstanding job and they deserve a big round of applause.',\n",
       "  'I also want to thank Joe Biden for working behind the scenes from the very start to make this recovery act possible.',\n",
       "  'I want to thank Speaker Pelosi and Harry Reid for acting so quickly and for proving that Congress could step up to this challenge.',\n",
       "  \"I have special thanks to Max Baucus, who's the Chairman of the Finance Committee.\",\n",
       "  'Without Max, none of this would have happened.',\n",
       "  'He had to work overtime, and push his committee to work overtime.',\n",
       "  'And I want to thank all the committee chairs and members of Congress for coming up with a plan that is both bold and balanced enough to meet the demands of this moment.',\n",
       "  \"The American people were looking to them for leadership, and that's what they provided.\",\n",
       "  'Now, what makes this recovery plan so important is not just that it will create or save 3.5 million jobs over the next two years, including 60,000-plus here in Colorado.',\n",
       "  \"It's that we're putting Americans to work doing the work that America needs done – - in critical areas that have been neglected for too long; work that will bring real and lasting change for generations to come.\",\n",
       "  \"Because we know we can't build our economic future on the transportation and information networks of the past, we are remaking the American landscape with the largest new investment in our nation's infrastructure since Eisenhower built an Interstate Highway System in the 19and $ 4,575,397.97.\",\n",
       "  'Because of this investment, nearly 400,000 men and women will go to work rebuilding our crumbling roads and bridges, repairing our faulty dams and levees, bringing critical broadband connections to businesses and homes in nearly every community in America, upgrading mass transit, building high-speed rail lines that will improve travel and commerce throughout our nation.',\n",
       "  \"Because we know America can't out-compete the world tomorrow if our children are being out-educated today, we're making the largest investment in education in our nation's history.\",\n",
       "  \"It's an investment that will create jobs building 21st century classrooms and libraries and labs for millions of children across America.\",\n",
       "  'It will provide funds to train a new generation of math and science teachers, while giving aid to states and school districts to stop teachers from being laid off and education programs from being cut.',\n",
       "  'In a place like New York City, 14,000 teachers who were set to be let go may now be able to continue pursuing their critical mission.',\n",
       "  \"It's an investment that will create a new $ 2,500 annual tax credit to put the dream of a college degree within reach for middle class families and make college affordable for 7 million students helping more of our sons and daughters aim higher, reach further, fulfill their God given potential.\",\n",
       "  \"Because we know that spiraling health care costs are crushing families and businesses alike, we're taking the most meaningful steps in years towards modernizing our health care system.\",\n",
       "  \"It's an investment that will take the long overdue step of computerizing America's medical records to reduce the duplication and waste that costs billions of health care dollars, and medical errors that cost thousands of lives each year.\",\n",
       "  \"Further, thanks to the actions we've taken, 7 million Americans who lost their health care along the way will continue to get the coverage they need, and roughly 20 million more Americans can breathe a little easier knowing that their health care won't be cut due to a state budget shortfall.\",\n",
       "  \"And a historic commitment to wellness initiatives will keep millions of Americans from setting foot in the doctor's office in the first place because these are preventable diseases and we're going to invest in prevention.\",\n",
       "  'So taken together with the enactment earlier this month of a long delayed law to extend health care to millions more children of working families we have done more in 30 days to advance the cause of health care reform than this country has done in an entire decade.',\n",
       "  \"And that's something we should be proud of.\",\n",
       "  \"Because we know we can't power America's future on energy that's controlled by foreign dictators, we are taking big steps down the road to energy independence, laying the groundwork for new green energy economies that can create countless well paying jobs.\",\n",
       "  \"It's an investment that will double the amount of renewable energy produced over the next three years.\",\n",
       "  'Think about that double the amount of renewable energy in three years.',\n",
       "  'Provide tax credits and loan guarantees to companies like Namaste, a company that will be expanding, instead of laying people off, as a result of the plan that be: ( 1 about to sign.',\n",
       "  'And in the process, we will transform the way we use energy.',\n",
       "  \"Today, the electricity we use is carried along a grid of lines and wires that date back to Thomas Edison a grid that can't support the demands of this economy.\",\n",
       "  \"This means we're using 19th and 20th century technologies to battle 21st century problems like climate change and energy security.\",\n",
       "  \"It also means that places like North Dakota can that can produce a lot of wind energy can't deliver it to communities that want it, leading to a gap between how much clean energy we are using and how much we could be using.\",\n",
       "  \"The investment we're making today will create a newer, smarter electric grid that will allow for broader use of alternative energy.\",\n",
       "  \"We will build on the work that's being done in places like Boulder a community that's on its that's on pace to be the world's first Smart Grid city.\",\n",
       "  'This investment will place Smart Meters in homes to make our energy bills lower, make outages less likely, and make it easier to use clean energy.',\n",
       "  \"It's an investment that will save taxpayers over $ 1 billion by slashing energy costs in our federal buildings by 25 percent; save working families hundreds of dollars a year on their energy bills by weatherizing over 1 million homes.\",\n",
       "  \"And it's an investment that takes the important first step towards a national transmission superhighway that will connect our cities to the windy plains of the Dakotas and the sunny deserts of the Southwest.\",\n",
       "  \"Even beyond energy, from the National Institutes of Health to the National Science Foundation, this recovery act represents the biggest increase in basic research funding in the long history of America's noble endeavor to better understand our world.\",\n",
       "  \"And just as President Kennedy sparked an explosion of innovation when he set America's sights on the moon, I hope this investment will ignite our imagination once more, spurring new discoveries and breakthroughs in science, in medicine, in energy, to make our economy stronger and our nation more secure and our planet safer for our children.\",\n",
       "  'Now, while this package is composed mostly of critical investments, it also includes aid to state and local governments to prevent layoffs of firefighters or police recruits in recruits like the ones in Columbus, Ohio, who were told that instead of being sworn in as officers, they were about to be let go.',\n",
       "  'It includes help for those hardest hit by our economic crisis like the nearly 18 million Americans who will get larger unemployment checks in the mail.',\n",
       "  'About a third of this package comes in the forms of tax cuts by the way, the most progressive in our history not only spurring job creation, but putting money in the pockets of 95 percent of hardworking families in America.',\n",
       "  \"So unlike the tax cuts that we've seen in recent years, the vast majority of these tax benefits will go not to the wealthiest Americans, but to the middle class with those workers who make the least benefiting the most.\",\n",
       "  \"And it's a plan that rewards responsibility, lifting two million Americans from poverty by ensuring that anyone who works hard does not have to raise a child below the poverty line.\",\n",
       "  \"So as a whole, this plan will help poor and working Americans pull themselves into the middle class in a way we haven't seen in nearly 50 years.\",\n",
       "  'What be: ( 1 signing, then, is a balanced plan with a mix of tax cuts and investments.',\n",
       "  \"It's a plan that's been put together without earmarks or the usual pork barrel spending.\",\n",
       "  \"It's a plan that will be implemented with an unprecedented level of transparency and accountability.\",\n",
       "  'With a recovery package of this size comes a responsibility to assure every taxpayer that we are being careful with the money they work so hard to earn.',\n",
       "  \"And that's why be: ( 1 assigning a team of managers to ensure that the precious dollars we've invested are being spent wisely and well.\",\n",
       "  \"We will Governor Ritter, Mayor Hickenlooper, we're going to hold governors and local officials who receive the money to the same high standard.\",\n",
       "  'And we expect you, the American people, to hold us accountable for the results.',\n",
       "  \"And that's why we've created fiancée a web site so that every American can go online and see how this money is being spent and what kind of job is being created, where those jobs are being created.\",\n",
       "  'We want transparency and accountability throughout this process.',\n",
       "  'Now, as important as the step we take today is, this legislation represents only the first part of the broad strategy we need to address our economic crisis.',\n",
       "  \"In the coming days and weeks, I'll be launching other aspects of the plan.\",\n",
       "  'We will need to stabilize, repair, and reform our banking system, and get credit flowing again to families and businesses.',\n",
       "  'We will need to end the culture where we ignore problems until they become full-blown crises instead of recognizing that the only way to build a thriving economy is to set and enforce firm rules of the road.',\n",
       "  \"We must stem the spread of foreclosures and falling home values for all Americans, and do everything we can to help responsible homeowners stay in their homes something I'll talk more about tomorrow.\",\n",
       "  'And we will need to do everything in the short term to get our economy moving again, while at the same time recognizing that we have inherited a trillion-dollar deficit, and we need to begin restoring fiscal discipline and taming our exploding deficits over the long term.',\n",
       "  'None of this will be easy.',\n",
       "  'The road to recovery will not be straight.',\n",
       "  'We will make progress and there may be some slippage along the way.',\n",
       "  'It will demand courage and discipline.',\n",
       "  \"It will demand a new sense of responsibility that's been missing from Wall Street all the way to Washington.\",\n",
       "  'There will be hazards and reverses.',\n",
       "  'But I have every confidence that if we are willing to continue doing the critical work that must be done by each of us, by all of us then we will leave this struggling economy behind us, and come out on the other side, more prosperous as a people.',\n",
       "  'For our American story is not and has never been about things coming easy.',\n",
       "  \"It's about rising to the moment when the moment is hard, and converting crisis into opportunity, and seeing to it that we emerge from whatever trials we face stronger than we were before.\",\n",
       "  \"It's about rejecting the notion that our fate is somehow written for us, and instead laying claim to a destiny of our own making.\",\n",
       "  \"That's what earlier generations of Americans have done, that's what we owe our children, that's what we are doing today.\",\n",
       "  'Thank you, Colorado.',\n",
       "  \"Let's get to work.\",\n",
       "  \"Thank you Madam Speaker, Mr. Vice President, Members of Congress, the First Lady of the United States, she's around here somewhere: I have come here tonight not only to address the distinguished men and women in this great Chamber, but to speak frankly and directly to the men and women who sent us here.\",\n",
       "  'I know that for many Americans watching right now, the state of our economy is a concern that rises above all others, and rightly so.',\n",
       "  \"If you haven't been personally affected by this recession, you probably know someone who has: a friend, a neighbor, a member of your family.\",\n",
       "  \"You don't need to hear another list of statistics to know that our economy is in crisis, because you live it every day.\",\n",
       "  \"It's the worry you wake up with and the source of sleepless nights.\",\n",
       "  \"It's the job you thought you'd retire from but now have lost, the business you built your dreams upon that's now hanging by a thread, the college acceptance letter your child had to put back in the envelope.\",\n",
       "  'The impact of this recession is real, and it is everywhere.',\n",
       "  'But while our economy may be weakened and our confidence shaken, though we are living through difficult and uncertain times, tonight I want every American to know this: We will rebuild, we will recover, and the United States of America will emerge stronger than before.',\n",
       "  'The weight of this crisis will not determine the destiny of this Nation.',\n",
       "  \"The answers to our problems don't lie beyond our reach.\",\n",
       "  'They exist in our laboratories and our universities, in our fields and our factories, in the imaginations of our entrepreneurs and the pride of the hardest working people on Earth.',\n",
       "  'Those qualities that have made America the greatest force of progress and prosperity in human history, we still possess in ample measure.',\n",
       "  'What is required now is for this country to pull together, confront boldly the challenges we face, and take responsibility for our future once more.',\n",
       "  \"Now, if we're honest with ourselves, we'll admit that for too long, we have not always met these responsibilities as a Government or as a people.\",\n",
       "  \"I say this not to lay blame or to look backwards, but because it is only by understanding how we arrived at this moment that we'll be able to lift ourselves out of this predicament.\",\n",
       "  'The fact is, our economy did not fall into decline overnight, nor did all of our problems begin when the housing market collapsed or the stock market sank.',\n",
       "  'We have known for decades that our survival depends on finding new sources of energy, yet we import more oil today than ever before.',\n",
       "  'The cost of health care eats up more and more of our savings each year, yet we keep delaying reform.',\n",
       "  'Our children will compete for jobs in a global economy that too many of our schools do not prepare them for.',\n",
       "  'And though all these challenges went unsolved, we still managed to spend more money and pile up more debt, both as individuals and through our Government, than ever before.',\n",
       "  'In other words, we have lived through an era where too often short-term gains were prized over long term prosperity, where we failed to look beyond the next payment, the next quarter, or the next election.',\n",
       "  'A surplus became an excuse to transfer wealth to the wealthy instead of an opportunity to invest in our future.',\n",
       "  'Regulations were gutted for the sake of a quick profit at the expense of a healthy market.',\n",
       "  \"People bought homes they knew they couldn't afford from banks and lenders who pushed those bad loans anyway.\",\n",
       "  'And all the while, critical debates and difficult decisions were put off for some other time, on some other day.',\n",
       "  'Well, that day of reckoning has arrived, and the time to take charge of our future is here.',\n",
       "  'Now is the time to act boldly and wisely to not only revive this economy, but to build a new foundation for lasting prosperity.',\n",
       "  'Now is the time to jumpstart job creation, restart lending, and invest in areas like energy, health care, and education that will grow our economy, even as we make hard choices to bring our deficit down.',\n",
       "  \"That is what my economic agenda is designed to do, and that is what I'd like to talk to you about tonight.\",\n",
       "  \"It's an agenda that begins with jobs.\",\n",
       "  \"As soon as I took office, I asked this Congress to send me a recovery plan by President's Day that would put people back to work and put money in their pockets, not because I believe in bigger Government, I don't, not because be: ( 1 not mindful of the massive debt we've inherited, I am.\",\n",
       "  'I called for action because the failure to do so would have cost more jobs and caused more hardship.',\n",
       "  'In fact, a failure to act would have worsened our long term deficit by assuring weak economic growth for years.',\n",
       "  \"And that's why I pushed for quick action.\",\n",
       "  'And tonight I am grateful that this Congress delivered and pleased to say that the American Recovery and Reinvestment Act is now law.',\n",
       "  'Over the next 2 years, this plan will save or create 3.5 million jobs.',\n",
       "  'More than 90 percent of these jobs will be in the private sector: jobs rebuilding our roads and bridges, constructing wind turbines and solar panels, laying broadband and expanding mass transit.',\n",
       "  'Because of this plan, there are teachers who can now keep their jobs and educate our kids, health care professionals can continue caring for our sick.',\n",
       "  'There are 57 police officers who are still on the streets of Minneapolis tonight because this plan prevented the layoffs their department was about to make.',\n",
       "  'Because of this plan, 95 percent of working households in America will receive a tax cut; a tax cut that you will see in your paychecks beginning on April 1st.',\n",
       "  'Because of this plan, families who are struggling to pay tuition costs will receive a $ 2,500 tax credit for all 4 years of college, and Americans who have lost their jobs in this recession will be able to receive extended unemployment benefits and continued health care coverage to help them weather this storm.',\n",
       "  'Now, I know there are some in this Chamber and watching at home who are skeptical of whether this plan will work, and I understand that skepticism.',\n",
       "  \"Here in Washington, we've all seen how quickly good intentions can turn into broken promises and wasteful spending.\",\n",
       "  'And with a plan of this scale comes enormous responsibility to get it right.',\n",
       "  \"And that's why I've asked Vice President Biden to lead a tough, unprecedented oversight effort; because nobody messes with Joe.\",\n",
       "  'I, am I right?',\n",
       "  \"They don't mess with him.\",\n",
       "  'I have told each of my Cabinet, as well as mayors and Governors across the country, that they will be held accountable by me and the American people for every dollar they spend.',\n",
       "  \"I've appointed a proven and aggressive Inspector General to ferret out any and all cases of waste and fraud.\",\n",
       "  'And we have created a new web site called recovery.gov, so that every American can find out how and where their money is being spent.',\n",
       "  'So the recovery plan we passed is the first step in getting our economy back on track.',\n",
       "  'But it is just the first step.',\n",
       "  'Because even if we manage this plan flawlessly, there will be no real recovery unless we clean up the credit crisis that has severely weakened our financial system.',\n",
       "  \"I want to speak plainly and candidly about this issue tonight, because every American should know that it directly affects you and your family's well being.\",\n",
       "  \"You should also know that the money you've deposited in banks across the country is safe, your insurance is secure, you can rely on the continued operation of our financial system.\",\n",
       "  \"That's not the source of concern.\",\n",
       "  'The concern is that if we do not restart lending in this country, our recovery will be choked off before it even begins.',\n",
       "  'You see, the flow of credit is the lifeblood of our economy.',\n",
       "  'The ability to get a loan is how you finance the purchase of everything from a home to a car to a college education, how stores stock their shelves, farms buy equipment, and businesses make payroll.',\n",
       "  'But credit has stopped flowing the way it should.',\n",
       "  'Too many bad loans from the housing crisis have made their way onto the books of too many banks.',\n",
       "  'And with so much debt and so little confidence, these banks are now fearful of lending out any more money to households, to businesses, or even to each other.',\n",
       "  \"And when there is no lending, families can't afford to buy homes or cars, so businesses are forced to make layoffs.\",\n",
       "  'Our economy suffers even more, and credit dries up even further.',\n",
       "  'That is why this administration is moving swiftly and aggressively to break this destructive cycle, to restore confidence, and restart lending.',\n",
       "  'And we will do so in several ways.',\n",
       "  'First, we are creating a new lending fund that represents the largest effort ever to help provide auto loans, college loans, and small-business loans to the consumers and entrepreneurs who keep this economy running.',\n",
       "  'Second, we have launched a housing plan that will help responsible families facing the threat of foreclosure lower their monthly payments and refinance their mortgages.',\n",
       "  \"It's a plan that won't help speculators or that neighbor down the street who bought a house he could never hope to afford, but it will help millions of Americans who are struggling with declining home values; Americans who will now be able to take advantage of the lower interest rates that this plan has already helped to bring about.\",\n",
       "  'In fact, the average family who refinances today can save nearly $ 2,000 per year on their mortgage.',\n",
       "  'Third, we will act with the full force of the Federal Government to ensure that the major banks that Americans depend on have enough confidence and enough money to lend even in more difficult times.',\n",
       "  'And when we learn that a major bank has serious problems, we will hold accountable those responsible, force the necessary adjustments, provide the support to clean up their balance sheets, and assure the continuity of a strong, viable institution that can serve our people and our economy.',\n",
       "  'Now, I understand that on any given day, Wall Street may be more comforted by an approach that gives bank bailouts with no strings attached and that holds nobody accountable for their reckless decisions.',\n",
       "  \"But such an approach won't solve the problem, and our goal is to quicken the day when we restart lending to the American people and American business and end this crisis once and for all.\",\n",
       "  'And I intend to hold these banks fully accountable for the assistance they receive, and this time, they will have to clearly demonstrate how taxpayer dollars result in more lending for the American taxpayer.',\n",
       "  \"This time, CEOs won't be able to use taxpayer money to pad their paychecks or buy fancy drapes or disappear on a private jet.\",\n",
       "  'Those days are over.',\n",
       "  \"Still, this plan will require significant resources from the Federal Government, and, yes, probably more than we've already set aside.\",\n",
       "  'But while the cost of action will be great, I can assure you that the cost of inaction will be far greater, for it could result in an economy that sputters along for not months or years, but perhaps a decade.',\n",
       "  'That would be worse for our deficit, worse for business, worse for you, and worse for the next generation.',\n",
       "  'And I refuse to let that happen.',\n",
       "  'Now, I understand that when the last administration asked this Congress to provide assistance for struggling banks, Democrats and Republicans alike were infuriated by the mismanagement and the results that followed.',\n",
       "  'So were the American taxpayers; so was I.',\n",
       "  'So I know how unpopular it is to be seen as helping banks right now, especially when everyone is suffering in part from their bad decisions.',\n",
       "  'I promise you, I get it.',\n",
       "  'But I also know that in a time of crisis, we can not afford to govern out of anger or yield to the politics of the moment.',\n",
       "  'My job, our job is to solve the problem.',\n",
       "  'Our job is to govern with a sense of responsibility.',\n",
       "  \"I will not send, I will not spend a single penny for the purpose of rewarding a single Wall Street executive, but I will do whatever it takes to help the small business that can't pay its workers or the family that has saved and still can't get a mortgage.\",\n",
       "  \"That's what this is about.\",\n",
       "  \"It's not about helping banks; it's about helping people, [ applause ].\",\n",
       "  \"It's not about helping banks; it's about helping people.\",\n",
       "  'Because when credit is available again, that young family can finally buy a new home.',\n",
       "  'And then some company will hire workers to build it.',\n",
       "  'And then those workers will have money to spend.',\n",
       "  \"And if they can get a loan too, maybe they'll finally buy that car or open their own business.\",\n",
       "  'Investors will return to the market, and American families will see their retirement secured once more.',\n",
       "  'Slowly but surely, confidence will return and our economy will recover.',\n",
       "  'So I ask this Congress to join me in doing whatever proves necessary, because we can not consign our Nation to an open-ended recession.',\n",
       "  'And to ensure that a crisis of this magnitude never happens again, I ask Congress to move quickly on legislation that will finally reform our outdated regulatory system.',\n",
       "  'It is time to put in place tough, new, commonsense rules of the road so that our financial market rewards drive and innovation, and punishes shortcuts and abuse.',\n",
       "  \"The recovery plan and the financial stability plan are the immediate steps we're taking to revive our economy in the short term.\",\n",
       "  \"But the only way to fully restore America's economic strength is to make the long term investments that will lead to new jobs, new industries, and a renewed ability to compete with the rest of the world.\",\n",
       "  \"The only way this century will be another American century is if we confront at last the price of our dependence on oil and the high cost of health care, the schools that aren't preparing our children and the mountain of debt they stand to inherit.\",\n",
       "  'That is our responsibility.',\n",
       "  'In the next few days, I will submit a budget to Congress.',\n",
       "  \"So often, we've come to view these documents as simply numbers on a page or a laundry list of programs.\",\n",
       "  'I see this document differently.',\n",
       "  'I see it as a vision for America, as a blueprint for our future.',\n",
       "  'My budget does not attempt to solve every problem or address every issue.',\n",
       "  \"It reflects the stark reality of what we've inherited, a trillion-dollar deficit, a financial crisis, and a costly recession.\",\n",
       "  'Given these realities, everyone in this Chamber, Democrats and Republicans, will have to sacrifice some worthy priorities for which there are no dollars.',\n",
       "  'And that includes me.',\n",
       "  'But that does not mean we can afford to ignore our long term challenges.',\n",
       "  'I reject the view that says our problems will simply take care of themselves, that says Government has no role in laying the foundation for our common prosperity.',\n",
       "  'For history tells a different story.',\n",
       "  'History reminds us that at every moment of economic upheaval and transformation, this Nation has responded with bold action and big ideas.',\n",
       "  'In the midst of Civil War, we laid railroad tracks from one coast to another that spurred commerce and industry.',\n",
       "  'From the turmoil of the Industrial Revolution came a system of public high schools that prepared our citizens for a new age.',\n",
       "  'In the wake of war and depression, the GI bill sent a generation to college and created the largest middle class in history.',\n",
       "  'And a twilight struggle for freedom led to a nation of highways, an American on the Moon, and an explosion of technology that still shapes our world.',\n",
       "  \"In each case, Government didn't supplant private enterprise; it catalyzed private enterprise.\",\n",
       "  'It created the conditions for thousands of entrepreneurs and new businesses to adapt and to thrive.',\n",
       "  'We are a nation that has seen promise amid peril and claimed opportunity from ordeal.',\n",
       "  \"Now we must be that nation again, and that is why, even as it cuts back on programs we don't need, the budget I submit will invest in the three areas that are absolutely critical to our economic future: energy, health care, and education.\",\n",
       "  'It begins with energy.',\n",
       "  'We know the country that harnesses the power of clean, renewable energy will lead the 21st century.',\n",
       "  'And yet, it is China that has launched the largest effort in history to make their economy energy-efficient.',\n",
       "  \"We invented solar technology, but we've fallen behind countries like Germany and Japan in producing it.\",\n",
       "  'New plug in hybrids roll off our assembly lines, but they will run on batteries made in Korea.',\n",
       "  \"Well, I do not accept a future where the jobs and industries of tomorrow take root beyond our borders, and I know you don't either.\",\n",
       "  'It is time for America to lead again.',\n",
       "  \"Thanks to our recovery plan, we will double this Nation's supply of renewable energy in the next 3 years.\",\n",
       "  \"We've also made the largest investment in basic research funding in American history, an investment that will spur not only new discoveries in energy but breakthroughs in medicine and science and technology.\",\n",
       "  'We will soon lay down thousands of miles of power lines that can carry new energy to cities and towns across this country.',\n",
       "  'And we will put Americans to work making our homes and buildings more efficient so that we can save billions of dollars on our energy bills.',\n",
       "  'But to truly transform our economy, to protect our security, and save our planet from the ravages of climate change, we need to ultimately make clean, renewable energy the profitable kind of energy.',\n",
       "  'So I ask this Congress to send me legislation that places a market-based cap on carbon pollution and drives the production of more renewable energy in America.',\n",
       "  \"That's what we need.\",\n",
       "  'And to support that innovation, we will invest $ 15 billion a year to develop technologies like wind power and solar power, advanced biofuels, clean coal, and more efficient cars and trucks built right here in America.',\n",
       "  'Speaking of our auto industry, everyone recognizes that years of bad decisionmaking and a global recession have pushed our automakers to the brink.',\n",
       "  'We should not, and will not, protect them from their own bad practices.',\n",
       "  'But we are committed to the goal of a retooled, reimagined auto industry that can compete and win.',\n",
       "  'Millions of jobs depend on it; scores of communities depend on it.',\n",
       "  'And I believe the Nation that invented the automobile can not walk away from it.',\n",
       "  'Now, none of this will come without cost, nor will it be easy.',\n",
       "  'But this is America.',\n",
       "  \"We don't do what's easy.\",\n",
       "  \"We do what's necessary to move this country forward.\",\n",
       "  'And for that same reason, we must also address the crushing cost of health care.',\n",
       "  'This is a cost that now causes a bankruptcy in America every 30 seconds.',\n",
       "  'By the end of the year, it could cause 1.5 million Americans to lose their homes.',\n",
       "  'In the last 8 years, premiums have grown four times faster than wages.',\n",
       "  'And in each of these years, 1 million more Americans have lost their health insurance.',\n",
       "  'It is one of the major reasons why small businesses close their doors and corporations ship jobs overseas.',\n",
       "  \"And it's one of the largest and fastest growing parts of our budget.\",\n",
       "  'Given these facts, we can no longer afford to put health care reform on hold.',\n",
       "  \"We can't afford to do it.\",\n",
       "  \"It's time.\",\n",
       "  \"Already, we've done more to advance the cause of health care reform in the last 30 days than we've done in the last decade.\",\n",
       "  'When it was days old, this Congress passed a law to provide and protect health insurance for 11 million American children whose parents work full time.',\n",
       "  'Our recovery plan will invest in electronic health records, a new technology that will reduce errors, bring down costs, ensure privacy, and save lives.',\n",
       "  'It will launch a new effort to conquer a disease that has touched the life of nearly every American, including me, by seeking a cure for cancer in our time.',\n",
       "  \"And it makes the largest investment ever in preventive care, because that's one of the best ways to keep our people healthy and our costs under control.\",\n",
       "  'This budget builds on these reforms.',\n",
       "  'It includes a historic commitment to comprehensive health care reform, a down payment on the principle that we must have quality, affordable health care for every American.',\n",
       "  \"It's a commitment that's paid for in part by efficiencies in our system that are long overdue.\",\n",
       "  \"And it's a step we must take if we hope to bring down our deficit in the years to come.\",\n",
       "  \"Now, there will be many different opinions and ideas about how to achieve reform, and that's why be: ( 1 bringing together businesses and workers, doctors and health care providers, Democrats and Republicans to begin work on this issue next week.\",\n",
       "  'I suffer no illusions that this will be an easy process.',\n",
       "  'Once again, it will be hard.',\n",
       "  'But I also know that nearly a century after Teddy Roosevelt first called for reform, the cost of our health care has weighed down our economy and our conscience long enough.',\n",
       "  'So let there be no doubt: Health care reform can not wait, it must not wait, and it will not wait another year.',\n",
       "  'The third challenge we must address is the urgent need to expand the promise of education in America.',\n",
       "  'In a global economy where the most valuable skill you can sell is your knowledge, a good education is no longer just a pathway to opportunity, it is a prerequisite.',\n",
       "  'Right now, three quarters of the fastest growing occupations require more than a high school diploma.',\n",
       "  'And yet, just over half of our citizens have that level of education.',\n",
       "  'We have one of the highest high school dropout rates of any industrialized nation, and half of the students who begin college never finish.',\n",
       "  'This is a prescription for economic decline, because we know the countries that out-teach us today will out-compete us tomorrow.',\n",
       "  'That is why it will be the goal of this administration to ensure that every child has access to a complete and competitive education, from the day they are born to the day they begin a career.',\n",
       "  'That is a promise we have to make to the children of America.',\n",
       "  \"Already, we've made historic investment in education through the economic recovery plan.\",\n",
       "  \"We've dramatically expanded early childhood education and will continue to improve its quality, because we know that the most formative learning comes in those first years of life.\",\n",
       "  \"We've made college affordable for nearly 7 million more students, 7 million.\",\n",
       "  \"And we have provided the resources necessary to prevent painful cuts and teacher layoffs that would set back our children's progress.\",\n",
       "  \"But we know that our schools don't just need more resources, they need more reform.\",\n",
       "  'And that is why this budget creates new teachers, new incentives for teacher performance, pathways for advancement, and rewards for success.',\n",
       "  \"We'll invest in innovative programs that are already helping schools meet high standards and close achievement gaps, and we will expand our commitment to charter schools.\",\n",
       "  'It is our responsibility as lawmakers and as educators to make this system work.',\n",
       "  'But it is the responsibility of every citizen to participate in it.',\n",
       "  'So tonight I ask every American to commit to at least 1 year or more of higher education or career training.',\n",
       "  'This can be community college or a 4-year school, vocational training or an apprenticeship.',\n",
       "  'But whatever the training may be, every American will need to get more than a high school diploma.',\n",
       "  'And dropping out of high school is no longer an option.',\n",
       "  \"It's not just quitting on yourself, it's quitting on your country, and this country needs and values the talents of every American.\",\n",
       "  \"That's why we will support, we will provide the support necessary for all young Americans to complete college and meet a new goal.\",\n",
       "  'By 2020, America will once again have the highest proportion of college graduates in the world.',\n",
       "  'That is a goal we can meet.',\n",
       "  \"That's a goal we can meet.\",\n",
       "  'Now, I know that the price of tuition is higher than ever, which is why if you are willing to volunteer in your neighborhood or give back to your community or serve your country, we will make sure that you can afford a higher education.',\n",
       "  'And to encourage a renewed spirit of national service for this and future generations, I ask Congress to send me the bipartisan legislation that bears the name of Senator Orrin Hatch, as well as an American who has never stopped asking what he can do for his country, Senator Edward Kennedy.',\n",
       "  'These education policies will open the doors of opportunity for our children, but it is up to us to ensure they walk through them.',\n",
       "  'In the end, there is no program or policy that can substitute for a parent, for a mother or father who will attend those parent/teacher conferences or help with homework or turn off the TV, put away the video games, read to their child.',\n",
       "  \"I speak to you not just as a President, but as a father, when I say that responsibility for our children's education must begin at home.\",\n",
       "  \"That is not a Democratic issue or a Republican issue; that's an American issue.\",\n",
       "  'There is, of course, another responsibility we have to our children.',\n",
       "  \"And that's the responsibility to ensure that we do not pass on to them a debt they can not pay.\",\n",
       "  'That is critical.',\n",
       "  '[ Applause ] I agree, absolutely.',\n",
       "  'See, I know we can get some consensus in here.',\n",
       "  '[ Laughter ] With the deficit we inherited, the cost of the crisis we face, and the long term challenges we must meet, it has never been more important to ensure that as our economy recovers, we do what it takes to bring this deficit down.',\n",
       "  'That is critical.',\n",
       "  'Now, be: ( 1 proud that we passed a recovery plan free of earmarks, and I want to pass a budget next year that ensures that each dollar we spend reflects only our most important national priorities.',\n",
       "  'And yesterday I held a fiscal summit where I pledged to cut the deficit in half by the end of my first term in office.',\n",
       "  'My administration has also begun to go line by line through the Federal budget in order to eliminate wasteful and ineffective programs.',\n",
       "  'As you can imagine, this is a process that will take some time.',\n",
       "  'But we have already identified $ 2 trillion in savings over the next decade.',\n",
       "  \"In this budget, we will end education programs that don't work and end direct payments to large agribusiness that don't need them.\",\n",
       "  \"We'll eliminate the no-bid contracts that have wasted billions in Iraq and reform our defense budget so that we're not paying for cold war-era weapons systems we don't use.\",\n",
       "  \"We will root out the waste and fraud and abuse in our Medicare program that doesn't make our seniors any healthier.\",\n",
       "  'We will restore a sense of fairness and balance to our Tax Code by finally ending the tax breaks for corporations that ship our jobs overseas.',\n",
       "  'In order to save our children from a future of debt, we will also end the tax breaks for the wealthiest 2 percent of Americans.',\n",
       "  \"Now, let me be clear, let me be absolutely clear, because I know you'll end up hearing some of the same claims that rolling back these tax breaks means a massive tax increase on the American people: If your family earns less than $ 250,000 a year, a quarter million dollars a year, you will not see your taxes increased a single dime.\",\n",
       "  'I repeat: Not one single dime.',\n",
       "  \"In fact, not a dime, in fact, the recovery plan provides a tax cut, that's right, a tax cut, for 95 percent of working families.\",\n",
       "  'And by the way, these checks are on the way.',\n",
       "  'Now, to preserve our long term fiscal health, we must also address the growing costs in Medicare and Social Security.',\n",
       "  'Comprehensive health care reform is the best way to strengthen Medicare for years to come.',\n",
       "  'And we must also begin a conversation on how to do the same for Social Security, while creating tax-free universal savings accounts for all Americans.',\n",
       "  \"Finally, because we're also suffering from a deficit of trust, I am committed to restoring a sense of honesty and accountability to our budget.\",\n",
       "  'That is why this budget looks ahead 10 years and accounts for spending that was left out under the old rules.',\n",
       "  'And for the first time, that includes the full cost of fighting in Iraq and Afghanistan.',\n",
       "  'For 7 years, we have been a nation at war.',\n",
       "  'No longer will we hide its price.',\n",
       "  'Along with our outstanding national security team, be: ( 1 now carefully reviewing our policies in both wars, and I will soon announce a way forward in Iraq that leaves Iraq to its people and responsibly ends this war.',\n",
       "  'And with our friends and allies, we will forge a new and comprehensive strategy for Afghanistan and Pakistan to defeat Al Qaida and combat extremism, because I will not allow terrorists to plot against the American people from safe havens halfway around the world.',\n",
       "  'We will not allow it.',\n",
       "  'As we meet here tonight, our men and women in uniform stand watch abroad and more are readying to deploy.',\n",
       "  'To each and every one of them and to the families who bear the quiet burden of their absence, Americans are united in sending one message: We honor your service; we are inspired by your sacrifice; and you have our unyielding support.',\n",
       "  'To relieve the strain on our forces, my budget increases the number of our soldiers and marines.',\n",
       "  'And to keep our sacred trust with those who serve, we will raise their pay and give our veterans the expanded health care and benefits that they have earned.',\n",
       "  'To overcome extremism, we must also be vigilant in upholding the values our troops defend, because there is no force in the world more powerful than the example of America.',\n",
       "  'And that is why I have ordered the closing of the detention center at Guantanamo Bay and will seek swift and certain justice for captured terrorists.',\n",
       "  \"Because living our values doesn't make us weaker, it makes us safer and it makes us stronger.\",\n",
       "  'And that is why I can stand here tonight and say without exception or equivocation that the United States of America does not torture.',\n",
       "  'We can make that commitment here tonight.',\n",
       "  'In words and deeds, we are showing the world that a new era of engagement has begun.',\n",
       "  'For we know that America can not meet the threats of this century alone, but the world can not meet them without America.',\n",
       "  'We can not shun the negotiating table, nor ignore the foes or forces that could do us harm.',\n",
       "  'We are instead called to move forward with the sense of confidence and candor that serious times demand.',\n",
       "  'To seek progress towards a secure and lasting peace between Israel and her neighbors, we have appointed an envoy to sustain our effort.',\n",
       "  'To meet the challenges of the 21st century, from terrorism to nuclear proliferation, from pandemic disease to cyber threats to crushing poverty, we will strengthen old alliances, forge new ones, and use all elements of our national power.',\n",
       "  'And to respond to an economic crisis that is global in scope, we are working with the nations of the G-20 to restore confidence in our financial system, avoid the possibility of escalating protectionism, and spur demand for American goods in markets across the globe.',\n",
       "  \"For the world depends on us having a strong economy, just as our economy depends on the strength of the world's.\",\n",
       "  'As we stand at this crossroads of history, the eyes of all people in all nations are once again upon us, watching to see what we do with this moment, waiting for us to lead.',\n",
       "  'Those of us gathered here tonight have been called to govern in extraordinary times.',\n",
       "  'It is a tremendous burden, but also a great privilege, one that has been entrusted to few generations of Americans.',\n",
       "  'For in our hands lies the ability to shape our world for good or for ill.',\n",
       "  \"I know that it's easy to lose sight of this truth, to become cynical and doubtful, consumed with the petty and the trivial.\",\n",
       "  'But in my life, I have also learned that hope is found in unlikely places, that inspiration often comes not from those with the most power or celebrity, but from the dreams and aspirations of ordinary Americans who are anything but ordinary.',\n",
       "  'I think of Leonard Abess, a bank president from Miami who reportedly cashed out of his company, took a $ 60 million bonus, and gave it out to all 399 people who worked for him, plus another 72 who used to work for him.',\n",
       "  \"He didn't tell anyone, but when the local newspaper found out, he simply said, “I knew some of these people since I was 7 years old.\",\n",
       "  \"It didn't feel right getting the money myself.” I think about Greensburg, Kansas, a town that was completely destroyed by a tornado, but is being rebuilt by its residents as a global example of how clean energy can power an entire community, how it can bring jobs and businesses to a place where piles of bricks and rubble once lay.\",\n",
       "  '“The tragedy was terrible,” said one of the men who helped them rebuild.',\n",
       "  '“But the folks here know that it also provided an incredible opportunity.” I think about i.e Bethea, the young girl from that school I visited in Dillon, South Carolina, a place where the ceilings leak, the paint peels off the walls, and they have to stop teaching six times a day because the train barrels by their classroom.',\n",
       "  'She had been told that her school is hopeless, but the other day after class she went to the public library and typed up a letter to the people sitting in this Chamber.',\n",
       "  'She even asked her principal for the money to buy a stamp.',\n",
       "  'The letter asks us for help and says: “We are just students trying to become lawyers, doctors, Congressmen like yourself, and one day President, so we can make a change to not just the State of South Carolina but also the world.',\n",
       "  \"We are not quitters.” That's what she said: “We are not quitters.” These words and these stories tell us something about the spirit of the people who sent us here.\",\n",
       "  'They tell us that even in the most trying times, amid the most difficult circumstances, there is a generosity, a resilience, a decency, and a determination that perseveres, a willingness to take responsibility for our future and for posterity.',\n",
       "  'Their resolve must be our inspiration.',\n",
       "  'Their concerns must be our cause.',\n",
       "  'And we must show them and all our people that we are equal to the task before us.',\n",
       "  \"I know, look, I know that we haven't agreed on every issue thus far.\",\n",
       "  '[ Laughter ] There are surely times in the future where we will part ways.',\n",
       "  'But I also know that every American who is sitting here tonight loves this country and wants it to succeed.',\n",
       "  'I know that.',\n",
       "  'That must be the starting point for every debate we have in the coming months and where we return after those debates are done.',\n",
       "  'That is the foundation on which the American people expect us to build common ground.',\n",
       "  'And if we do, if we come together and lift this Nation from the depths of this crisis, if we put our people back to work and restart the engine of our prosperity, if we confront without fear the challenges of our time and summon that enduring spirit of an America that does not quit, then someday years from now our children can tell their children that this was the time when we performed, in the words that are carved into this very Chamber, “something worthy to be remembered.” Thank you.',\n",
       "  'God bless you, and may God bless the United States of America.',\n",
       "  'Thank you Thank you.',\n",
       "  'Well, be: ( 1 excited, too.',\n",
       "  '( Laughter. )',\n",
       "  'Of the many responsibilities granted to a President by our Constitution, few are more serious or more consequential than selecting a Supreme Court justice.',\n",
       "  'The members of our highest court are granted life tenure, often serving long after the Presidents who appointed them.',\n",
       "  'And they are charged with the vital task of applying principles put to paper more than 20 [ sic ] centuries ago to some of the most difficult questions of our time.',\n",
       "  \"So I don't take this decision lightly.\",\n",
       "  \"I've made it only after deep reflection and careful deliberation.\",\n",
       "  'While there are many qualities that I admire in judges across the spectrum of judicial philosophy, and that I seek in my own nominee, there are few that stand out that I just want to mention.',\n",
       "  'First and foremost is a rigorous intellect a mastery of the law, an ability to hone in on the key issues and provide clear answers to complex legal questions.',\n",
       "  \"Second is a recognition of the limits of the judicial role, an understanding that a judge's job is to interpret, not make, law; to approach decisions without any particular ideology or agenda, but rather a commitment to impartial justice; a respect for precedent and a determination to faithfully apply the law to the facts at hand.\",\n",
       "  \"These two qualities are essential, I believe, for anyone who would sit on our nation's highest court.\",\n",
       "  'And yet, these qualities alone are insufficient.',\n",
       "  'We need something more.',\n",
       "  'For as Supreme Court Justice Oliver Wendell Holmes once said, “The life of the law has not been logic; it has been experience.” Experience being tested by obstacles and barriers, by hardship and misfortune; experience insisting, persisting, and ultimately overcoming those barriers.',\n",
       "  'It is experience that can give a person a common touch and a sense of compassion; an understanding of how the world works and how ordinary people live.',\n",
       "  'And that is why it is a necessary ingredient in the kind of justice we need on the Supreme Court.',\n",
       "  'The process of reviewing and selecting a successor to Justice Souter has been rigorous and comprehensive, not least because of the standard that Justice Souter himself has set with his formidable intellect and fair-mindedness and decency.',\n",
       "  \"I've sought the advice of members of Congress on both sides of the aisle, including every member of the Senate Judiciary Committee.\",\n",
       "  'My team has reached out to constitutional scholars, advocacy organizations, and bar associations representing an array of interests and opinions.',\n",
       "  \"And I want to thank members of my staff and administration who've worked so hard and given so much of their time as part of this effort.\",\n",
       "  'After completing this exhaustive process, I have decided to nominate an inspiring woman who I believe will make a great justice: Judge Sonia Sotomayor of the great state of New York.',\n",
       "  'Over a distinguished career that spans three decades, Judge Sotomayor has worked at almost every level of our judicial system, providing her with a depth of experience and a breadth of perspective that will be invaluable as a Supreme Court justice.',\n",
       "  \"It's a measure of her qualities and her qualifications that Judge Sotomayor was nominated to the in 1881.\",\n",
       "  'District Court by a Republican President, George H.W.',\n",
       "  'Bush, and promoted to the Federal Court of Appeals by a Democrat, Bill Clinton.',\n",
       "  'Walking in the door she would bring more experience on the bench, and more varied experience on the bench, than anyone currently serving on the United States Supreme Court had when they were appointed.',\n",
       "  \"Judge Sotomayor is a distinguished graduate of two of America's leading universities.\",\n",
       "  \"She's been a nonessential prosecutor and a corporate litigator.\",\n",
       "  'She spent six years as a trial judge on the in 1881.',\n",
       "  'District Court, and would replace Justice Souter as the only justice with experience as a trial judge, a perspective that would enrich the judgments of the Court.',\n",
       "  'For the past 11 years she has been a judge on the Court of Appeals for the Second Circuit of New York, one of the most demanding circuits in the country.',\n",
       "  'There she has handed down decisions on a range of constitutional and legal questions that are notable for their careful reasoning, earning the respect of colleagues on the bench, the admiration of many lawyers who argue cases in her court, and the adoration of her clerks who look to her as a mentor.',\n",
       "  'During her tenure on the District Court, she presided over roughly 450 cases.',\n",
       "  'One case in particular involved a matter of enormous concern to many Americans, including me: the baseball strike of 1994 - 1995.',\n",
       "  '( Laughter. )',\n",
       "  'In a decision that reportedly took her just 15 minutes to announce, a swiftness much appreciated by baseball fans everywhere she issued an injunction that helped end the strike.',\n",
       "  'Some say that Judge Sotomayor saved baseball.',\n",
       "  'Judge Sotomayor came to the District Court from a law firm where she was a partner focused on complex commercial litigation, gaining insight into the workings of a global economy.',\n",
       "  \"Before that she was a prosecutor in the Manhattan DA's office, serving under the legendary Robert Morgenthau, an early mentor of Sonia's who still sings her praises today.\",\n",
       "  'There, Sonia learned what crime can do to a family and a community, and what it takes to fight it.',\n",
       "  \"It's a career that has given her not only a sweeping overview of the American judicial system, but a practical understanding of how the law works in the everyday lives of the American people.\",\n",
       "  \"But as impressive and meaningful as Judge Sotomayor's sterling credentials in the law is her own extraordinary journey.\",\n",
       "  \"Born in the South Bronx, she was raised in a housing project not far from Yankee Stadium, making her a lifelong Yankee's fan.\",\n",
       "  'I hope this will not disqualify her in the eyes of the New Englanders in the Senate.',\n",
       "  '( Laughter. )',\n",
       "  \"Sonia's parents came to New York from Puerto Rico during the second world war, her mother as part of the Women's Army Corps.\",\n",
       "  \"And, in fact, her mother is here today and I'd like us all to acknowledge Sonia's mom.\",\n",
       "  \"Sonia's mom has been a little choked up.\",\n",
       "  '( Laughter. )',\n",
       "  \"But she, Sonia's mother, began a family tradition of giving back to this country.\",\n",
       "  \"Sonia's father was a factory worker with a 3rd grade education who didn't speak English.\",\n",
       "  \"But like Sonia's mother, he had a willingness to work hard, a strong sense of family, and a belief in the American Dream.\",\n",
       "  'When Sonia was nine, her father passed away.',\n",
       "  'And her mother worked six days a week as a nurse to provide for Sonia and her brother who is also here today, is a doctor and a terrific success in his own right.',\n",
       "  \"But Sonia's mom bought the only set of encyclopedias in the neighborhood, sent her children to a Catholic school called Cardinal Spellman out of the belief that with a good education here in America all things are possible.\",\n",
       "  'With the support of family, friends, and teachers, Sonia earned scholarships to Princeton, where she graduated at the top of her class, and Yale Law School, where she was an editor of the Yale Law Journal, stepping onto the path that led her here today.',\n",
       "  \"Along the way she's faced down barriers, overcome the odds, lived out the American Dream that brought her parents here so long ago.\",\n",
       "  'And even as she has accomplished so much in her life, she has never forgotten where she began, never lost touch with the community that supported her.',\n",
       "  \"What Sonia will bring to the Court, then, is not only the knowledge and experience acquired over a course of a brilliant legal career, but the wisdom accumulated from an inspiring life's journey.\",\n",
       "  \"It's my understanding that Judge Sotomayor's interest in the law was sparked as a young girl by reading the Nancy Drew series and that when she was diagnosed with diabetes at the age of eight, she was informed that people with diabetes can't grow up to be police officers or private investigators like Nancy Drew.\",\n",
       "  \"And that's when she was told she'd have to scale back her dreams.\",\n",
       "  \"Well, Sonia, what you've shown in your life is that it doesn't matter where you come from, what you look like, or what challenges life throws your way no dream is beyond reach in the United States of America.\",\n",
       "  'And when Sonia Sotomayor ascends those marble steps to assume her seat on the highest court of the land, America will have taken another important step towards realizing the ideal that is etched above its entrance: Equal justice under the law.',\n",
       "  'I hope the Senate acts in a bipartisan fashion, as it has in confirming Judge Sotomayor twice before, and as swiftly as possible so that she can take her seat on the Court in September and participate in deliberations as the Court chooses which cases it will hear this coming year.',\n",
       "  \"And with that, I'd like all of you to give a warm greeting as I invite Judge Sotomayor to say a few words Thank you so much.\",\n",
       "  'Good afternoon.',\n",
       "  'I am honored to be in the timeless city of Cairo and to be hosted by two remarkable institutions.',\n",
       "  \"For over a thousand years, Al-Azhar has stood as a beacon of Islamic learning, and for over a century, Cairo University has been a source of Egypt's advancement.\",\n",
       "  'And together, you represent the harmony between tradition and progress.',\n",
       "  'be: ( 1 grateful for your hospitality and the hospitality of the people of Egypt.',\n",
       "  'And be: ( 1 also proud to carry with me the good will of the American people and a greeting of peace from Muslim communities in my country: As salaamu alaykum.',\n",
       "  'We meet at a time of great tension between the United States and Muslims around the world, tension rooted in historical forces that go beyond any current policy debate.',\n",
       "  'The relationship between Islam and the West includes centuries of coexistence and cooperation, but also conflict and religious wars.',\n",
       "  'More recently, tension has been fed by colonialism that denied rights and opportunities to many Muslims and a cold war in which Muslim majority countries were too often treated as proxies without regard to their own aspirations.',\n",
       "  'Moreover, the sweeping change brought by modernity and globalization led many Muslims to view the West as hostile to the traditions of Islam.',\n",
       "  'Violent extremists have exploited these tensions in a small, but potent minority of Muslims.',\n",
       "  'The attacks of September 11, 2001, and the continued efforts of these extremists to engage in violence against civilians has led some in my country to view Islam as inevitably hostile not only to America and Western countries, but also to human rights.',\n",
       "  'All this has bred more fear and more mistrust.',\n",
       "  'So long as our relationship is defined by our differences, we will empower those who sow hatred rather than peace, those who promote conflict rather than the cooperation that can help all of our people achieve justice and prosperity.',\n",
       "  'And this cycle of suspicion and discord must end.',\n",
       "  \"I've come here to Cairo to seek a new beginning between the United States and Muslims around the world, one based on mutual interest and mutual respect and one based upon the truth that America and Islam are not exclusive and need not be in competition.\",\n",
       "  'Instead, they overlap and share common principles, principles of justice and progress, tolerance and the dignity of all human beings.',\n",
       "  'I do so recognizing that change can not happen overnight.',\n",
       "  \"I know there's been a lot of publicity about this speech, but no single speech can eradicate years of mistrust.\",\n",
       "  'Nor can I answer in the time that I have this afternoon all the complex questions that brought us to this point.',\n",
       "  'But I am convinced that in order to move forward, we must say openly to each other the things we hold in our hearts and that too often are said only behind closed doors.',\n",
       "  'There must be a sustained effort to listen to each other, to learn from each other, to respect one another, and to seek common ground.',\n",
       "  'As the Holy Koran tells us: “Be conscious of God and speak always the truth.” That is what I will try to do today, to speak the truth as best I can, humbled by the task before us and firm in my belief that the interests we share as human beings are far more powerful than the forces that drive us apart.',\n",
       "  'Now part of this conviction is rooted in my own experience.',\n",
       "  'be: ( 1 a Christian, but my father came from a Kenyan family that includes generations of Muslims.',\n",
       "  'As a boy, I spent several years in Indonesia and heard the call of the azaan at the break of dawn and at the fall of dusk.',\n",
       "  'As a young man, I worked in Chicago communities where many found dignity and peace in their Muslim faith.',\n",
       "  \"As a student of history, I also know civilization's debt to Islam.\",\n",
       "  \"It was Islam, at places like Al-Azhar, that carried the light of learning through so many centuries, paving the way for Europe's renaissance and enlightenment.\",\n",
       "  'It was innovation in Muslim communities that developed the order of algebra, our magnetic compass and tools of navigation, our mastery of pens and printing, our understanding of how disease spreads and how it can be healed.',\n",
       "  'Islamic culture has given us majestic arches and soaring spires, timeless poetry and cherished music, elegant calligraphy and places of peaceful contemplation.',\n",
       "  'And throughout history, Islam has demonstrated through words and deeds the possibilities of religious tolerance and racial equality.',\n",
       "  \"I also know that Islam has always been a part of America's story.\",\n",
       "  'The first nation to recognize my country was Morocco.',\n",
       "  'In signing the Treaty of Tripoli in 1796, our second President, John Adams, wrote: “The United States has in itself no character of enmity against the laws, religion, or tranquillity of Muslims.” And since our founding, American Muslims have enriched the United States.',\n",
       "  \"They have fought in our wars; they have served in our government; they have stood for civil rights; they have started businesses; they have taught at our universities; they've excelled in our sports arenas; they've won Nobel Prizes, built our tallest building, and lit the Olympic Torch.\",\n",
       "  'And when the first Muslim American was recently elected to Congress, he took the oath to defend our Constitution using the same Holy Koran that one of our Founding Fathers, Thomas Jefferson, kept in his personal library.',\n",
       "  'So, I have known Islam on three continents before coming to the region where it was first revealed.',\n",
       "  \"That experience guides my conviction that partnership between America and Islam must be based on what Islam is, not what it isn't.\",\n",
       "  'And I consider it part of my responsibility as President of the United States to fight against negative stereotypes of Islam wherever they appear.',\n",
       "  'But that same principle must apply to Muslim perceptions of America.',\n",
       "  'Just as Muslims do not fit a crude stereotype, America is not the crude stereotype of a self interested empire.',\n",
       "  'The United States has been one of the greatest sources of progress that the world has ever known.',\n",
       "  'We were born out of revolution against an empire.',\n",
       "  'We were founded upon the ideal that all are created equal, and we have shed blood and struggled for centuries to give meaning to those words, within our borders and around the world.',\n",
       "  'We are shaped by every culture, drawn from every end of the Earth, and dedicated to a simple concept: E pluribus unum, “Out of many, one.” Now, much has been made of the fact that an African American with the name Barack Hussein Obama could be elected President.',\n",
       "  'But my personal story is not so unique.',\n",
       "  'The dream of opportunity for all people has not come true for everyone in America, but its promise exists for all who come to our shores, and that includes nearly 7 million American Muslims in our country today, who, by the way, enjoy incomes and educational levels that are higher than the American average.',\n",
       "  \"Moreover, freedom in America is indivisible from the freedom to practice one's religion.\",\n",
       "  'That is why there is a mosque in every State in our Union and over 1,200 mosques within our borders.',\n",
       "  \"That's why the United States Government has gone to court to protect the right of women and girls to wear the hijab and to punish those who would deny it.\",\n",
       "  'So let there be no doubt, Islam is a part of America.',\n",
       "  'And I believe that America holds within her the truth that regardless of race, religion, or station in life, all of us share common aspirations to live in peace and security, to get an education and to work with dignity, to love our families, our communities, and our God.',\n",
       "  'These things we share.',\n",
       "  'This is the hope of all humanity.',\n",
       "  'Of course, recognizing our common humanity is only the beginning of our task.',\n",
       "  'Words alone can not meet the needs of our people.',\n",
       "  'These needs will be met only if we act boldly in the years ahead and if we understand that the challenges we face are shared and our failure to meet them will hurt us all.',\n",
       "  'For we have learned from recent experience that when a financial system weakens in one country, prosperity is hurt everywhere.',\n",
       "  'When a new flu infects one human being, all are at risk.',\n",
       "  'When one nation pursues a nuclear weapon, the risk of nuclear attack rises for all nations.',\n",
       "  'When violent extremists operate in one stretch of mountains, people are endangered across an ocean.',\n",
       "  'When innocents in Bosnia and Darfur are slaughtered, that is a stain on our collective conscience.',\n",
       "  'That is what it means to share this world in the 21st century.',\n",
       "  'That is the responsibility we have to one another as human beings.',\n",
       "  'And this is a difficult responsibility to embrace, for human history has often been a record of nations and tribes and, yes, religions subjugating one another in pursuit of their own interests.',\n",
       "  'Yet in this new age, such attitudes are self defeating.',\n",
       "  'Given our interdependence, any world order that elevates one nation or group of people over another will inevitably fail.',\n",
       "  'So whatever we think of the past, we must not be prisoners to it.',\n",
       "  'Our problems must be dealt with through partnership; our progress must be shared.',\n",
       "  'Now, that does not mean we should ignore sources of tension.',\n",
       "  'Indeed, it suggests the opposite.',\n",
       "  'We must face these tensions squarely.',\n",
       "  'And so in that spirit, let me speak as clearly and as plainly as I can about some specific issues that I believe we must finally confront together.',\n",
       "  'The first issue that we have to confront is violent extremism in all of its forms.',\n",
       "  'In Ankara, I made clear that America is not, and never will be, at war with Islam.',\n",
       "  'We will, however, relentlessly confront violent extremists who pose a grave threat to our security, because we reject the same thing that people of all faiths reject: the killing of innocent men, women, and children.',\n",
       "  'And it is my first duty as President to protect the American people.',\n",
       "  \"The situation in Afghanistan demonstrates America's goals and our need to work together.\",\n",
       "  'Over 7 years ago, the United States pursued Al Qaida and the Taliban with broad international support.',\n",
       "  'We did not go by choice; we went because of necessity.',\n",
       "  \"be: ( 1 aware that there's still some who would question or even justify the events of 9/11.\",\n",
       "  'But let us be clear: Al Qaida killed nearly 3,000 people on that day.',\n",
       "  'The victims were innocent men, women, and children from America and many other nations who had done nothing to harm anybody.',\n",
       "  'And yet, Al Qaida chose to ruthlessly murder these people, claimed credit for the attack, and even now states their determination to kill on a massive scale.',\n",
       "  'They have affiliates in many countries and are trying to expand their reach.',\n",
       "  'These are not opinions to be debated; these are facts to be dealt with.',\n",
       "  'Make no mistake, we do not want to keep our troops in Afghanistan.',\n",
       "  'We see no, we seek no military bases there.',\n",
       "  'It is agonizing for America to lose our young men and women.',\n",
       "  'It is costly and politically difficult to continue this conflict.',\n",
       "  'We would gladly bring every single one of our troops home, if we could be confident that there were not violent extremists in Afghanistan and now Pakistan determined to kill as many Americans as they possibly can.',\n",
       "  'But that is not yet the case.',\n",
       "  \"And that's why we're partnering with a coalition of 46 countries.\",\n",
       "  \"And despite the costs involved, America's commitment will not weaken.\",\n",
       "  'Indeed, none of us should tolerate these extremists.',\n",
       "  'They have killed in many countries.',\n",
       "  'They have killed people of different faiths, but more than any other, they have killed Muslims.',\n",
       "  'Their actions are irreconcilable with the rights of human beings, the progress of nations, and with Islam.',\n",
       "  'The Holy Koran teaches that “whoever kills an innocent” is as, “it is as if he has killed all mankind.” And the Holy Koran also says, “whoever saves a person, it is as if he has saved all mankind.” The enduring faith of over a billion people is so much bigger than the narrow hatred of a few.',\n",
       "  'Islam is not part of the problem in combating violent extremism, it is an important part of promoting peace.',\n",
       "  'Now, we also know that military power alone is not going to solve the problems in Afghanistan and Pakistan.',\n",
       "  \"That's why we plan to invest $ 1.5 billion each year over the next 5 years to partner with Pakistanis to build schools and hospitals, roads and businesses, and hundreds of millions to help those who've been displaced.\",\n",
       "  \"That's why we are providing more than $ 2.8 billion to help Afghans develop their economy and deliver services that people depend on.\",\n",
       "  'Let me also address the issue of Iraq.',\n",
       "  ...])"
      ]
     },
     "execution_count": 790,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from re import sub\n",
    "#gensim simple_preprocess - This lowercases, tokenizes, removes punctuation, de-accents (optional). \n",
    "from gensim.utils import simple_preprocess\n",
    "\n",
    "speeches=(subset_df.transcripts.iloc[0])\n",
    "sentences=sent_tokenize(speeches)\n",
    "president =subset_df.President.iloc[0]\n",
    "president, sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "750d0073",
   "metadata": {
    "id": "750d0073",
    "outputId": "b65b8a64-63fe-43b3-b704-f710cfca400d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10672"
      ]
     },
     "execution_count": 791,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db1a4eba",
   "metadata": {
    "id": "db1a4eba",
    "outputId": "2f68d76d-dec6-4b16-e31f-e71afe32da24"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[786,\n",
       " 363,\n",
       " 938,\n",
       " 711,\n",
       " 1385,\n",
       " 878,\n",
       " 3700,\n",
       " 1530,\n",
       " 206,\n",
       " 1792,\n",
       " 2692,\n",
       " 275,\n",
       " 1008,\n",
       " 1464,\n",
       " 2459,\n",
       " 3951,\n",
       " 2532,\n",
       " 2870,\n",
       " 1865,\n",
       " 108,\n",
       " 1540,\n",
       " 4202,\n",
       " 4189,\n",
       " 2841,\n",
       " 5519,\n",
       " 3384,\n",
       " 2691,\n",
       " 1189,\n",
       " 3382,\n",
       " 3521,\n",
       " 6457,\n",
       " 1917,\n",
       " 809,\n",
       " 4768,\n",
       " 12715,\n",
       " 2802,\n",
       " 2023,\n",
       " 5345,\n",
       " 10300,\n",
       " 5536,\n",
       " 6695,\n",
       " 5939,\n",
       " 10672,\n",
       " 6276]"
      ]
     },
     "execution_count": 792,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len_sentences=[]\n",
    "for i in range(corpus_df.shape[0]):\n",
    "    speeches = (corpus_df.transcripts.iloc[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    len(sentences)\n",
    "    len_sentences.append(len(sentences))\n",
    "len_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b988bf69",
   "metadata": {
    "id": "b988bf69",
    "outputId": "eabf1f02-7219-4653-9c1c-e40607e0f207",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>characters</th>\n",
       "      <th>total_words</th>\n",
       "      <th>len_sentences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>190015</td>\n",
       "      <td>31502</td>\n",
       "      <td>786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>88556</td>\n",
       "      <td>14630</td>\n",
       "      <td>363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>211059</td>\n",
       "      <td>35823</td>\n",
       "      <td>938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>216989</td>\n",
       "      <td>35991</td>\n",
       "      <td>711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>297306</td>\n",
       "      <td>49931</td>\n",
       "      <td>1385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           President                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "\n",
       "                                         transcripts  characters  total_words  \\\n",
       "0  Fellow Citizens of the Senate and the House of...      190015        31502   \n",
       "1  When it was first perceived, in early times, t...       88556        14630   \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...      211059        35823   \n",
       "3  Unwilling to depart from examples of the most ...      216989        35991   \n",
       "4  I should be destitute of feeling if I was not ...      297306        49931   \n",
       "\n",
       "   len_sentences  \n",
       "0            786  \n",
       "1            363  \n",
       "2            938  \n",
       "3            711  \n",
       "4           1385  "
      ]
     },
     "execution_count": 793,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df[\"len_sentences\"] = len_sentences\n",
    "corpus_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10a2c346",
   "metadata": {
    "id": "10a2c346",
    "outputId": "e7bbfaa5-c29b-4b38-d102-5bfc278017b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='len_sentences', ylabel='Count'>"
      ]
     },
     "execution_count": 794,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEHCAYAAACp9y31AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAATUElEQVR4nO3df7RlZX3f8fdnZkR+qUC5UpgfDrQultbVBr1pZMyyLQRDjIWmpQJLFCNmWElj/NFooCQrK6urbWJciWmSKlNFbJgiipioSQSCJK4sCDIgv+SHGpWZAeNc64pKTCsj3/6x98jlzr1z7wxzzr7nPu/XWmedfZ6zz36++9yZz933Ofs8O1WFJKkdq4YuQJI0Xga/JDXG4Jekxhj8ktQYg1+SGrNm6AKW4thjj62NGzcOXYYkTZQ77rjjG1U1Nbd9IoJ/48aNbNu2begyJGmiJHl4vnaHeiSpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTErPvjXrt9AkmV3W7t+w9BvjaRGTcSUDU/Hozt3cO7ltwxdxl6uuXjT0CVIatSKP+KXJD2VwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmNGFvxJrkiyK8l98zz3i0kqybGj6l+SNL9RHvFfCZw5tzHJeuAMYPsI+5YkLWBkwV9VnwG+Oc9Tvw28A6hR9S1JWthYx/iTnAU8UlV3j7NfSdKTxjZJW5LDgcuAVyxx/c3AZoANG5zJUpIOlnEe8f8j4ETg7iRfBdYBdyb5h/OtXFVbqmq6qqanpqbGWKYkrWxjO+KvqnuB5+553If/dFV9Y1w1SJJGezrn1cCtwMlJdia5aFR9SZKWbmRH/FV1/iLPbxxV35KkhfnNXUlqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4Jakxo7zY+hVJdiW5b1bbbyZ5MMk9ST6W5KhR9S9Jmt8oj/ivBM6c03Yj8KKq+qfAF4BLR9i/JGkeIwv+qvoM8M05bTdU1e7+4V8B60bVvyRpfkOO8b8B+NOFnkyyOcm2JNtmZmbGWJYkrWyDBH+Sy4DdwNaF1qmqLVU1XVXTU1NT4ytOkla4NePuMMmFwKuA06uqxt2/JLVurMGf5Ezgl4B/UVXfHWffkqTOKE/nvBq4FTg5yc4kFwG/BzwLuDHJXUneO6r+JUnzG9kRf1WdP0/z+0fVnyRpafzmriQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxY5+rR71Va0gydBV7OWHdeh7ZsX3oMiSNkME/lCd2c+7ltwxdxV6uuXjT0CVIGjGHeiSpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTGjvNj6FUl2JblvVtsxSW5M8sX+/uhR9S9Jmt8oj/ivBM6c03YJcFNVPR+4qX8sSRqjkQV/VX0G+Oac5rOBD/bLHwT+zaj6lyTNb9xj/MdV1dcA+vvnLrRiks1JtiXZNjMzM7YCJWmlW7Yf7lbVlqqarqrpqampocuRpBVj3MH/9STHA/T3u8bcvyQ1b9zB/3Hgwn75QuCPxty/JDVvlKdzXg3cCpycZGeSi4BfB85I8kXgjP6xJGmMRnYFrqo6f4GnTh9Vn5KkxS3bD3clSaNh8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzJKCP8nLltImSVr+lnrE/7tLbJMkLXP7nJ0zyanAJmAqydtmPfVsYPUoC5MkjcZi0zIfAhzZr/esWe3fBs4ZVVGSpNHZZ/BX1V8Af5Hkyqp6eEw1SZJGaKkXYnlmki3AxtmvqarTRlGUJGl0lhr8HwHeC7wP+P7oypEkjdpSg393Vb3nYHWa5K3AG4EC7gV+uqr+78HaviRpYUs9nfMTSX4uyfFJjtlzO5AOk6wFfgGYrqoX0Z0ddN6BbEuStP+WesR/YX//9lltBZz0NPo9LMnjwOHAowe4HUnSflpS8FfViQerw6p6JMm7gO3A3wM3VNUNc9dLshnYDLBhw4aD1b0Ws2oNSYauYi8nrFvPIzu2D12GtCIsKfiTvG6+9qr6X/vbYZKjgbOBE4G/BT6S5IKqumrOtrcAWwCmp6drf/vRAXpiN+defsvQVezlmos3DV2CtGIsdajnh2ctHwqcDtwJ7HfwAz8GfKWqZgCSXEf37eCr9vkqSdJBsdShnjfNfpzkOcAfHGCf24GXJjmcbqjndGDbAW5LkrSfDnRa5u8Czz+QF1bVbcC1dH8x3NvXsOUA65Ak7aeljvF/gu4sHuhOv3wB8OED7bSqfhX41QN9vSTpwC11jP9ds5Z3Aw9X1c4R1CNJGrElDfX0k7U9SDdD59HA90ZZlCRpdJZ6Ba5XA58F/j3wauC2JE7LLEkTaKlDPZcBP1xVuwCSTAF/RvchrSRpgiz1rJ5Ve0K/93/247WSpGVkqUf8n0pyPXB1//hc4E9GU5IkaZQWu+buPwaOq6q3J/m3wI8CAW4Fto6hPknSQbbYcM27ge8AVNV1VfW2qnor3dH+u0dbmiRpFBYL/o1Vdc/cxqraRncZRknShFks+A/dx3OHHcxCJEnjsVjw357kZ+Y2JrkIuGM0JUmSRmmxs3reAnwsyWt4MuingUOAnxphXZKkEdln8FfV14FNSf4V8KK++Y+r6tMjr0ySNBJLnY//ZuDmEdciSRoDv30rSY0x+CWpMQa/JDXG4Jekxhj8ktSYQYI/yVFJrk3yYJIHkpw6RB2S1KKlTst8sP0O8KmqOifJIcDhA9UhSc0Ze/AneTbwcuD1AFX1PbyGrySNzRBDPScBM8AHknwuyfuSHDF3pSSbk2xLsm1mZmb8VUrSCjVE8K8BXgy8p6pOAf4OuGTuSlW1paqmq2p6ampq3DVK0oo1RPDvBHZW1W3942vpfhFIksZg7MFfVX8D7Ehyct90OnD/uOuQpFYNdVbPm4Ct/Rk9XwZ+eqA6JKk5gwR/Vd1FN6+/JGnM/OauJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTFDfXNX2j+r1pBk6Cr2csK69TyyY/vQZUj7xeDXZHhiN+defsvQVezlmos3DV2CtN8c6pGkxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1ZrDgT7I6yeeSfHKoGiSpRUMe8b8ZeGDA/iWpSYMEf5J1wE8C7xuif0lq2VBH/O8G3gE8MVD/ktSssQd/klcBu6rqjkXW25xkW5JtMzMzY6pOkla+IY74XwacleSrwIeA05JcNXelqtpSVdNVNT01NTXuGiVpxRp78FfVpVW1rqo2AucBn66qC8ZdhyS1yvP4Jakxg16Ipar+HPjzIWuQpNZ4xC9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1ZuzBn2R9kpuTPJDk80nePO4aJKllQ1xsfTfwH6vqziTPAu5IcmNV3T9ALZLUnLEf8VfV16rqzn75O8ADwNpx1yFJrRp0jD/JRuAU4LZ5ntucZFuSbTMzM2OvTVqSVWtIsuxua9dvGPqdmdfa9RsGf28m6f0alSGGegBIciTwUeAtVfXtuc9X1RZgC8D09HSNuTxpaZ7YzbmX3zJ0FXu55uJNQ5cwr0d37vD9WgYGOeJP8gy60N9aVdcNUYMktWqIs3oCvB94oKp+a9z9S1LrhjjifxnwWuC0JHf1t1cOUIckNWnsY/xV9ZdAxt2vJKnjN3clqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjRlskjZJWjb6WVaXoxPWreeRHdsP6jYNfklaprOswmhmDnWoR5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGjNI8Cc5M8lDSb6U5JIhapCkVo09+JOsBn4f+AnghcD5SV447jokqVVDHPH/c+BLVfXlqvoe8CHg7AHqkKQmparG22FyDnBmVb2xf/xa4Eeq6ufnrLcZ2Nw/PBl4aD+7Ohb4xtMsd0jWP6xJrx8mfx+s/+l7XlVNzW0cYnbO+eY+3eu3T1VtAbYccCfJtqqaPtDXD836hzXp9cPk74P1j84QQz07gfWzHq8DHh2gDklq0hDBfzvw/CQnJjkEOA/4+AB1SFKTxj7UU1W7k/w8cD2wGriiqj4/gq4OeJhombD+YU16/TD5+2D9IzL2D3clScPym7uS1BiDX5Ias+KCf7lOB5FkfZKbkzyQ5PNJ3ty3H5PkxiRf7O+PnvWaS/v9eCjJj89qf0mSe/vn/nuS+U6RHdV+rE7yuSSfnLT6kxyV5NokD/Y/h1Mnqf6+77f2/37uS3J1kkOX8z4kuSLJriT3zWo7aPUmeWaSa/r225JsHEP9v9n/G7onyceSHLVc619QVa2YG92HxX8NnAQcAtwNvHDouvrajgde3C8/C/gC3ZQV7wQu6dsvAX6jX35hX/8zgRP7/VrdP/dZ4FS670T8KfATY9yPtwH/G/hk/3hi6gc+CLyxXz4EOGrC6l8LfAU4rH/8YeD1y3kfgJcDLwbum9V20OoFfg54b798HnDNGOp/BbCmX/6N5Vz/gvs1jk7Gdevf2OtnPb4UuHTouhao9Y+AM+i+kXx833Y88NB8tdOdBXVqv86Ds9rPBy4fU83rgJuA03gy+CeifuDZdKGZOe0TUX/f11pgB3AM3Rl5n+xDaFnvA7BxTnAetHr3rNMvr6H7pmxGWf+c534K2Lqc65/vttKGevb8x9hjZ9+2rPR/zp0C3AYcV1VfA+jvn9uvttC+rO2X57aPw7uBdwBPzGqblPpPAmaAD/RDVe9LcgSTUz9V9QjwLmA78DXgW1V1AxO0D72DWe8PXlNVu4FvAf9gZJXv7Q10R/BPqaW3bOtfacG/pOkghpTkSOCjwFuq6tv7WnWettpH+0gleRWwq6ruWOpL5mkbrH66o6kXA++pqlOAv6MbZljIcquffiz8bLphhBOAI5JcsK+XzNM26D4s4kDqHfLncRmwG9i6SC3Lrv6VFvzLejqIJM+gC/2tVXVd3/z1JMf3zx8P7OrbF9qXnf3y3PZRexlwVpKv0s2oelqSq5ic+ncCO6vqtv7xtXS/CCalfoAfA75SVTNV9ThwHbCJydoHOLj1/uA1SdYAzwG+ObLKe0kuBF4FvKb6cRomqP6VFvzLdjqI/lP89wMPVNVvzXrq48CF/fKFdGP/e9rP6z/1PxF4PvDZ/k/j7yR5ab/N1816zchU1aVVta6qNtK9r5+uqgsmqP6/AXYkOblvOh24f1Lq720HXprk8L7v04EHJmwf9tR1sOqdva1z6P5djvSIOcmZwC8BZ1XVd2c9NRH1Ayvrw93+/Xol3Rkzfw1cNnQ9s+r6Ubo/4e4B7upvr6Qbz7sJ+GJ/f8ys11zW78dDzDrrApgG7uuf+z3G8GHQnH35lzz54e7E1A/8ELCt/xn8IXD0JNXf9/1rwIN9/39AdwbJst0H4Gq6zyMepzu6vehg1gscCnwE+BLdmTMnjaH+L9GNy+/5f/ze5Vr/QjenbJCkxqy0oR5J0iIMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8mkhJHhu6hvkk+U9D1yAtxvP4NZGSPFZVRw5dx1zLtS5pNo/4NfGSvD3J7f2FMX6tb9uY7mIr/zPdhUtuSHLYPrbxC0nu77fxob7tiP5CHLf3M3qe3be/Psl1ST6V7mIi7+zbfx04LMldSbb2bRck+WzfdnmS1X37Y0n+S5K7k/xVkuP69uP6i3vc3d82LbSd/nZluouy3JvkrSN8m7WSjOur5t68Hcwb8Fh//wpgC90sh6vo5qh/Od0c6ruBH+rX+zBwwT629yjwzH75qP7+v+55Dd1FW74AHEF38ZMv002odSjwMLB+dl398guATwDP6B//D+B1/XIB/7pffifwy/3yNXQzt0J3YaHnLLQd4CXAjbP6O2ron4u3ybiteZq/N6ShvaK/fa5/fCTd5Fjb6WayvKtvv4Pul8FC7gG2JvlDunl89mz7rCS/2D8+FNjQL99UVd8CSHI/8DyeOhc7dJOovQS4vZubi8N4cibK79H9ktpT2xn98ml0oU5VfR/4VpLXLrCdTwAnJfld4I+BG/axf9IPGPyadAH+W1Vd/pTG7mI3/29W0/fpAnMhP0n3l8JZwK8k+Sf9tv9dVT00Z9s/Ms+25/u/FOCDVXXpPM89XlV7PmBb6PWLbifJPwN+HPgPwKvpLgwi7ZNj/Jp01wNvSHeBG5KsTfLcRV7zFElW0Q3V3Ex3hbGj6P5yuB54Uz+VLklOWcLmHk933QXoZp48Z0896S4y/rxFXn8T8LP9+quTPHuh7SQ5FlhVVR8FfoXu+gLSojzi10SrqhuSvAC4tc/nx4AL6I6il2o1cFWS59AdXf92Vf1tkv9Md7nJe/rw/yrdxTf2ZUu//p1V9Zokvwzc0P9yeZzuyPzhfbz+zcCWJBf1+/CzVXXrAtv5e7pLSe45gJvvLwtpL57OKUmNcahHkhrjUI+akuT36a4fPNvvVNUHhqhHGoJDPZLUGId6JKkxBr8kNcbgl6TGGPyS1Jj/DwQGmntb18S6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.histplot(data=corpus_df, x='len_sentences')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0b41301",
   "metadata": {
    "id": "a0b41301",
    "outputId": "fd0fe6af-d8fc-461d-e7d6-ec41bb325375"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 795,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAO8UlEQVR4nO3df4zk9V3H8edbrpSDpQdIuzkP4kJCSAgbW26jUEwzC0UpENCERAhU0Db3h6liPaNHiGn8o5GqNNZorJcWJULZ2gMtgWhLaNfGRK/uUuweHAgtJ9yV3tHUHl28BC6+/WO+C+uyP2dmd+Ztn49ks9/5zHzn+/rM7bz4zndm+EZmIkmq58f6HUCS1BkLXJKKssAlqSgLXJKKssAlqahNG7mxM888M0dGRta0zquvvsopp5yyPoE2gPn7q3p+qD8H83dvenr6e5n5zoXjG1rgIyMjTE1NrWmdyclJWq3W+gTaAObvr+r5of4czN+9iPjPxcY9hCJJRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRW3oNzG7MbLrkb5t+8CdV/dt25K0FPfAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySilqxwCPi7og4EhH75o2dERGPRsSzze/T1zemJGmh1eyB/zVw5YKxXcBjmXke8FhzWZK0gVYs8Mz8GvD9BcPXAfc0y/cAv9DbWJKklXR6DHw4M18CaH6/q3eRJEmrEZm58o0iRoCHM/PC5vIPMvO0edf/V2Yuehw8InYAOwCGh4e3T0xMrCng7OwsQ0NDzBw6uqb1BsXwZjh8rLN1R7dt6W2YDsw9/lVVzw/152D+7o2Pj09n5tjC8U7PiXk4IrZm5ksRsRU4stQNM3M3sBtgbGwsW63WmjY0OTlJq9Xi1j6eE7MbO0ePc9dMZw/zgZtavQ3TgbnHv6rq+aH+HMy/fjo9hPIQcEuzfAvwxd7EkSSt1mo+Rng/8C/A+RFxMCI+BNwJXBERzwJXNJclSRtoxdf2mXnjEldd3uMskqQ18JuYklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklRUpyd00AYY6dNJLA7ceXVftitpbdwDl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiuirwiPhoRDwZEfsi4v6IOKlXwSRJy+u4wCNiG/AbwFhmXgicANzQq2CSpOV1ewhlE7A5IjYBJwPf6T6SJGk1IjM7XzniNuDjwDHgy5l50yK32QHsABgeHt4+MTGxpm3Mzs4yNDTEzKGjHefsp+HNcPhYv1Oszei2LW8szz3+VVXPD/XnYP7ujY+PT2fm2MLxjgs8Ik4HHgB+CfgB8AVgT2beu9Q6Y2NjOTU1tabtTE5O0mq1+nZ+yG7tHD3OXTO1Tj06/5yYc49/VdXzQ/05mL97EbFogXdzCOX9wPOZ+XJmvg48CLy3i/uTJK1BNwX+AnBxRJwcEQFcDuzvTSxJ0ko6LvDM3AvsAR4HZpr72t2jXJKkFXR1cDYzPwZ8rEdZJElr4DcxJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJakoC1ySirLAJamorgo8Ik6LiD0R8XRE7I+IS3oVTJK0vE1drv8p4B8z8/qIOBE4uQeZJEmr0HGBR8Q7gPcBtwJk5mvAa72JJUlaSWRmZytGvBvYDTwF/BQwDdyWma8uuN0OYAfA8PDw9omJiTVtZ3Z2lqGhIWYOHe0oZ78Nb4bDx/qdYm1Gt215Y3nu8a+qen6oPwfzd298fHw6M8cWjndT4GPAvwKXZubeiPgU8Epm/t5S64yNjeXU1NSatjM5OUmr1WJk1yMd5ey3naPHuWum2yNVG+vAnVe/sTz3+FdVPT/Un4P5uxcRixZ4N29iHgQOZube5vIe4KIu7k+StAYdF3hmfhd4MSLOb4Yup304RZK0Abp9bf/rwH3NJ1C+DfxK95EkSavRVYFn5hPAW47LSJLWn9/ElKSiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6SiLHBJKsoCl6Siap0qRv/v9frMSztHj3PrKu5z/lmIpCrcA5ekoixwSSrKApekoixwSSrKApekoixwSSrKApekoixwSSrKApekoixwSSrKApekoixwSSrKApekoixwSSqq6wKPiBMi4hsR8XAvAkmSVqcXe+C3Aft7cD+SpDXoqsAj4izgauAzvYkjSVqtyMzOV47YA/wBcCrw25l5zSK32QHsABgeHt4+MTGxpm3Mzs4yNDTEzKGjHefsp+HNcPhYv1N07kcl/+i2LesfpkNzz4GqzN+98fHx6cwcWzje8SnVIuIa4EhmTkdEa6nbZeZuYDfA2NhYtlpL3nRRk5OTtFqtVZ0WaxDtHD3OXTN1z1z3o5L/wE2t9Q/TobnnQFXmXz/dHEK5FLg2Ig4AE8BlEXFvT1JJklbUcYFn5u2ZeVZmjgA3AF/JzJt7lkyStCw/By5JRfXk4GZmTgKTvbgvSdLquAcuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUV1XOARcXZEfDUi9kfEkxFxWy+DSZKWt6mLdY8DOzPz8Yg4FZiOiEcz86keZZMkLaPjPfDMfCkzH2+WfwjsB7b1KpgkaXmRmd3fScQI8DXgwsx8ZcF1O4AdAMPDw9snJibWdN+zs7MMDQ0xc+ho1zn7YXgzHD7W7xSdM//6G922Zdnr554DVS2Wv1/P55Ue68UMwuM/Pj4+nZljC8e7LvCIGAL+Cfh4Zj643G3HxsZyampqTfc/OTlJq9ViZNcjXaTsn52jx7lrppsjVf1l/vV34M6rl71+7jlQ1WL5+/V8XumxXswgPP4RsWiBd/UplIh4G/AAcN9K5S1J6q1uPoUSwGeB/Zn5yd5FkiStRjd74JcCHwQui4gnmp+repRLkrSCjg8OZuY/A9HDLJKkNfCbmJJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJUlAUuSUVZ4JJU1GCfqkSSeqiTMwHtHD3OrT04g1AnZwNaiXvgklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklSUBS5JRVngklRUVwUeEVdGxDMR8VxE7OpVKEnSyjou8Ig4Afhz4APABcCNEXFBr4JJkpbXzR74TwPPZea3M/M1YAK4rjexJEkriczsbMWI64ErM/PDzeUPAj+TmR9ZcLsdwI7m4vnAM2vc1JnA9zoKORjM31/V80P9OZi/ez+Zme9cONjNOTFjkbG3/NcgM3cDuzveSMRUZo51un6/mb+/queH+nMw//rp5hDKQeDseZfPAr7TXRxJ0mp1U+D/BpwXEedExInADcBDvYklSVpJx4dQMvN4RHwE+BJwAnB3Zj7Zs2Rv6vjwy4Awf39Vzw/152D+ddLxm5iSpP7ym5iSVJQFLklFDWyBD+rX9CPi7Ij4akTsj4gnI+K2ZvyMiHg0Ip5tfp8+b53bm3k8ExE/P298e0TMNNf9aUQs9tHM9ZrHCRHxjYh4uFr+iDgtIvZExNPNv8MllfI32/5o8/ezLyLuj4iTBnkOEXF3RByJiH3zxnqWNyLeHhGfb8b3RsTIBuT/o+Zv6JsR8XcRcdqg5l9SZg7cD+03Rb8FnAucCPw7cEG/czXZtgIXNcunAv9B+38l8IfArmZ8F/CJZvmCJv/bgXOaeZ3QXPd14BLan6n/B+ADGziP3wI+BzzcXC6TH7gH+HCzfCJwWrH824Dngc3N5b8Fbh3kOQDvAy4C9s0b61le4NeATzfLNwCf34D8PwdsapY/Mcj5l5zXRmykgwf7EuBL8y7fDtze71xLZP0icAXtb5hubca2As8slp32p3YuaW7z9LzxG4G/3KDMZwGPAZfxZoGXyA+8g3b5xYLxEvmbbW0DXgTOoP1JsIebMhnoOQAjCwqwZ3nnbtMsb6L9zcdYz/wLrvtF4L5Bzr/Yz6AeQpn7A59zsBkbKM3LpPcAe4HhzHwJoPn9ruZmS81lW7O8cHwj/AnwO8D/zBurkv9c4GXgr5pDQJ+JiFOok5/MPAT8MfAC8BJwNDO/TKE5NHqZ9411MvM4cBT48XVL/la/SnuP+v9kaQxs/kEt8FV9Tb+fImIIeAD4zcx8ZbmbLjKWy4yvq4i4BjiSmdOrXWWRsb7lp713cxHwF5n5HuBV2i/flzJo+WmOFV9H++X5TwCnRMTNy62yyFhf57CCTvL289/jDuA4cN8KWQYu/6AW+EB/TT8i3ka7vO/LzAeb4cMRsbW5fitwpBlfai4Hm+WF4+vtUuDaiDhA+/8geVlE3Eud/AeBg5m5t7m8h3ahV8kP8H7g+cx8OTNfBx4E3kutOUBv876xTkRsArYA31+35I2IuAW4Brgpm+MfFMo/qAU+sF/Tb951/iywPzM/Oe+qh4BbmuVbaB8bnxu/oXmX+hzgPODrzUvOH0bExc19/vK8ddZNZt6emWdl5gjtx/UrmXlzofzfBV6MiPObocuBp6rkb7wAXBwRJzfbvhzYX2wOc7l6lXf+fV1P++9yXfdgI+JK4HeBazPzv+ddVSI/MJhvYjbzvor2Jzy+BdzR7zzzcv0s7ZdG3wSeaH6uon286zHg2eb3GfPWuaOZxzPM+5QAMAbsa677MzbgTY8Fc2nx5puYZfID7wammn+DvwdOr5S/2fbvA0832/8b2p94GNg5APfTPl7/Ou29zQ/1Mi9wEvAF4Dnan/Q4dwPyP0f7uPXc8/jTg5p/qR+/Si9JRQ3qIRRJ0goscEkqygKXpKIscEkqygKXpKIscEkqygKXpKL+F7PrxFoJSHYxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "corpus_df['len_sentences'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f51a8e34",
   "metadata": {
    "id": "f51a8e34",
    "outputId": "3214c4b1-cfb5-4340-8d41-f79907d46449"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='len_sentences'>"
      ]
     },
     "execution_count": 796,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcZklEQVR4nO3df5Ac5X3n8fdnhcJiCcUgVoKTUBZVVCaIi7HZgC42Kc7EQSE+i9QFSnYcdDEVVShikSOJjeK6812uOHAlRRJdGSqKsRE2ASvYOVQuwCZyXHAXfmSFZYOQOWSQQUZIMiaWwLUgsd/7o5+Vh9VqNTM9PT3d83lVTe3MM/3j2V6pv93f50crIjAzM2vXQNkVMDOzanMgMTOzXBxIzMwsFwcSMzPLxYHEzMxyOa7sCpThlFNOieHh4bKrYWZWKVu2bPlhRAxNLu/LQDI8PMzo6GjZ1TAzqxRJ35+q3KktMzPLxYHEzMxycSAxM7NcHEjMzCwXBxIzM8ulL3ttmVn/GR8Pdr78Gnv2jzF/ziDDc2cxMKCyq1ULDiRmVnvj48H9217i2o1bGTs4zuDMAW66/ByWLz3VwaQDnNoys9rb+fJrh4MIwNjBca7duJWdL79Wcs3qwYHEzGpvz/6xw0FkwtjBcfYeGCupRvXiQGJmtTd/ziCDM996uhucOcC8EwdLqlG9OJCYWe0Nz53FTZefcziYTLSRDM+dVXLN6sGN7WZWewMDYvnSUzlzzQXsPTDGvBPda6uTHEjMrC8MDIjFQ7NZPDS77KrUjlNbZmaWiwOJmZnl4kBiZma5FBpIJH1O0l5JTzaU/bmk70r6jqR/kPT2hu/WStoh6WlJFzeUnyvpifTdOklK5cdL+lIqf1TScJG/j5mZHanoO5LbgOWTyh4Azo6IXwT+H7AWQNJZwEpgaVrnZkkz0jq3AKuBJek1sc0rgVci4ueBvwQ+XdhvYmZmUyo0kETEg8CPJpV9PSIOpY+PAAvT+xXAXRHxekQ8B+wAzpN0GjAnIh6OiABuBy5tWGdDen83cNHE3YqZmXVH2W0kHwXuS+8XAC80fLcrlS1I7yeXv2WdFJx+DMydakeSVksalTS6b9++jv0CZmb9rrRAIumTwCHgjomiKRaLacqnW+fIwoj1ETESESNDQ0OtVtfMzI6ilEAiaRXwAeC3U7oKsjuN0xsWWwi8mMoXTlH+lnUkHQf8LJNSaWZmVqyuBxJJy4FPAB+MiJ80fLUJWJl6Yp1B1qj+WETsBg5IWpbaP64A7mlYZ1V6/1vANxoCk5mZdUGhU6RIuhO4EDhF0i7gU2S9tI4HHkjt4o9ExO9HxDZJG4GnyFJeV0fEm2lTV5H1ADuBrE1lol3lVuALknaQ3YmsLPL3MTOzI6kfL+BHRkZidHS07GqYmVWKpC0RMTK5vOxeW2ZmVnEOJGZmlosDiZmZ5eJAYmZmuTiQmJlZLg4kZmaWiwOJmZnl4kBiZma5OJCYmVkuDiRmZpaLA4mZmeXiQGJmZrkUOvuvmZl1x/h4sPPl19izf4z5cwYZnjuLgYHuPHncgcTMrOLGx4P7t73EtRu3MnZwnMGZA9x0+TksX3pqV4KJU1tmZhW38+XXDgcRgLGD41y7cSs7X36tK/t3IDEzq7g9+8cOB5EJYwfH2XtgrCv7dyAxM6u4+XMGGZz51tP54MwB5p042JX9O5CYmVXc8NxZ3HT5OYeDyUQbyfDcWV3ZvxvbzcwK0M1eVAMDYvnSUzlzzQXsPTDGvBPda8vMrNLK6EU1MCAWD81m8dDsQrY/7b67vkczs5oruxdVtzmQmJl1WNm9qLrNgcTMrMPK7kXVbYUGEkmfk7RX0pMNZSdLekDSM+nnSQ3frZW0Q9LTki5uKD9X0hPpu3WSlMqPl/SlVP6opOEifx8zs2aU3Yuq24q+I7kNWD6p7Dpgc0QsATanz0g6C1gJLE3r3CxpRlrnFmA1sCS9JrZ5JfBKRPw88JfApwv7TczMmjTRi+reNRdw1+rzuXfNBV2brqQMhQaSiHgQ+NGk4hXAhvR+A3BpQ/ldEfF6RDwH7ADOk3QaMCciHo6IAG6ftM7Etu4GLpq4WzEzK9NEL6pli09h8dDs2gYRKKeNZH5E7AZIP+el8gXACw3L7UplC9L7yeVvWSciDgE/BuZOtVNJqyWNShrdt29fh34VMzPrpcb2qcJ1TFM+3TpHFkasj4iRiBgZGhpqs4pmZjZZGYFkT0pXkX7uTeW7gNMbllsIvJjKF05R/pZ1JB0H/CxHptLMzKxAZQSSTcCq9H4VcE9D+crUE+sMskb1x1L664CkZan944pJ60xs67eAb6R2FDMz65JCp0iRdCdwIXCKpF3Ap4AbgY2SrgSeBy4DiIhtkjYCTwGHgKsj4s20qavIeoCdANyXXgC3Al+QtIPsTmRlkb+PmZkdSf14AT8yMhKjo6NlV8PMrFIkbYmIkcnlvdTYbmZmFeRAYmZmuTiQmJlZLg4kZmaWiwOJmZnl4kBiZma5OJCYmVkuDiRmZpaLA4mZmeXiQGJmZrk4kJiZWS4OJGZmlosDiZmZ5eJAYmZmuTiQmJlZLg4kZmaWiwOJmZnl4kBiZma5NB1IJF0jaY4yt0p6XNKvFVk5MzPrfa3ckXw0IvYDvwYMAb8L3FhIrczMrDJaCSRKPy8BPh8R324oMzOzPtVKINki6etkgeRrkk4ExouplpmZVUUrgeRK4DrglyLiJ8DPkKW32iLpP0vaJulJSXdKGpR0sqQHJD2Tfp7UsPxaSTskPS3p4obycyU9kb5bJ8l3SWZmXdRKIAngLGBN+jwLGGxnp5IWpO2MRMTZwAxgJVmg2hwRS4DN6TOSzkrfLwWWAzdLmpE2dwuwGliSXsvbqZOZmbWnlUByM/DvgA+lzweAz+TY93HACZKOA94GvAisADak7zcAl6b3K4C7IuL1iHgO2AGcJ+k0YE5EPBwRAdzesI6ZmXVBK4Hk/Ii4GhgDiIhXyNJbLYuIHwB/ATwP7AZ+HBFfB+ZHxO60zG5gXlplAfBCwyZ2pbIF6f3k8iNIWi1pVNLovn372qm2mZlNoZVAcjClkwJA0hBtNranto8VwBnAvwFmSfrIdKtMURbTlB9ZGLE+IkYiYmRoaKjVKpuZ2VG0EkjWAf8AzJN0PfB/gP/Z5n5/FXguIvZFxEHgK8AvA3tSuor0c29afhdwesP6C8lSYbvS+8nlZmbWJU0Hkoi4A/g4cANZOurSiPj7Nvf7PLBM0ttSL6uLgO3AJmBVWmYVcE96vwlYKel4SWeQNao/ltJfByQtS9u5omEdMzPrguOaXVDSMmBbRHwmfT5R0vkR8WirO42IRyXdDTwOHAK+BawHZgMbJV1JFmwuS8tvk7QReCotf3VEvJk2dxVwG3ACcF96mZlZlyjr7NTEgtK3gHen3lFIGgBGI+LdBdavECMjIzE6Olp2NczMKkXSlogYmVze0hQp0RB1ImKcFu5ozMysnloJJM9KWiNpZnpdAzxbVMXMzKwaWgkkv0/Ws+oHZL2lzicbUW5mZn2s6dRUROwlm6bEzMzssFZ6bQ0BvwcMN64XER/tfLXMzKwqWmksvwd4CPhH4M1jLGtmZn2ilUDytoj4RGE1MTOzSmqlsf2rki4prCZmZlZJrQSSa8iCyZik/ZIOSNpfVMXMzKwaWum1dWKRFTEzs2pq+o5EmY9I+i/p8+mSziuuamZmVgXtPCHxw+nzq+R7QqKZmdVAK722zo+Id6fJG4mIVyS19YREMzOb2vh4sPPl19izf4z5cwYZnjuLgYGpnuHXO1oJJB17QqKZmR1pfDy4f9tLXLtxK2MHxxmcOcBNl5/D8qWn9nQwyfuExBsKqZWZWR/a+fJrh4MIwNjBca7duJWdL79Wcs2m10qvrTskbSF7mqHInpC4vbCamVmtVTGFU7Q9+8cOB5EJYwfH2XtgjMVDs0uq1bG1MtfWFyLid4DvTlFmZta0qqZwijZ/ziCDMwfeEkwGZw4w78TBEmt1bK2ktpY2fkjtJed2tjpm1g+qmsIp2vDcWdx0+TkMzsxOzRMBdnjurJJrNr1j3pFIWgv8KXBCGsk+cbnwBtlz1s3MWlLVFE7RBgbE8qWncuaaC9h7YIx5J3Yu5VdkKvGYgSQibgBukHRDRKztyF7NrK9VNYXTDQMDYvHQ7I4G1KJTiU2ntiJiraQFkn5Z0q9MvHLXwMz6TlVTOFVVdCqxlcb2G8mekPgUP30eSQAPdqQmZtY3ikzh2JGKTiW2MiDxN4F3RMTrufcKSHo78FngbLKA9FHgaeBLZE9h3AlcHhGvpOXXAleSBbE1EfG1VH4ucBtwAnAvcE1ERCfqaGbFKSKFY1MrOpXYSq+tZ4GZHdlr5q+B+yPiTOCdwHbgOmBzRCwBNqfPSDqL7G5oKbAcuDn1GgO4BVgNLEmv5R2so5lZzxgfD57d9yoPf++HPLvvVcbHm7tmLjqV2ModyU+ArZI2A4fvSiJiTas7lTQH+BXgP6VtvAG8IWkFcGFabAPwTeATwArgrnQ39JykHcB5knYCcyLi4bTd24FLgftarZOZWS/L02BedCqxlUCyKb06YTGwD/i8pHcCW8genDU/InYDRMRuSfPS8guARxrW35XKDqb3k8vNzGrlaA3mZ665oKn0YJGpxFamSNkg6QRgUUQ83YH9vhv4WEQ8KumvSWmso5gqbMY05UduQFpNlgJj0aJFrdXWzKxkvTz2ppUHW/0HYCtwf/p8jqR271B2Absi4tH0+W6ywLJH0mlp+6cBexuWP71h/YXAi6l84RTlR4iI9RExEhEjQ0NDbVbbzKwcEw3mjXpl7E0rje3/DTgP+FeAiNgKnNHOTiPiJeAFSe9IRReRdSveBKxKZauAe9L7TcBKScdLOoOsUf2xlAY7IGmZJAFXNKxjZlYbvTz2ppU2kkMR8ePsfH1Ynm62HwPuSA/Hehb4XbLAtlHSlcDzwGUAEbFN0kayYHMIuDoiJsayXMVPu//ehxvazayGennsTSuB5ElJHwZmSFoCrAH+ud0dpzuakSm+uugoy18PXD9F+SjZWBSzvuep2auh3b9Tr469aSWQfAz4JFnX3zuBrwH/o4hKmVnrPDV7NdTx79TKXFs/iYhPRsQvAecDn46IseKqZmat8NTs1VDHv1Mrvbb+TtIcSbOAbcDTkv6kuKqZWSum6x5qxWhnpHkd/06tpLbOioj9kn6bbE6rT5ANJPzzQmpmZi3x1Ozd1W6Kqo5/p1a6/86UNJNsCpJ7IuIg+XptmVkH9XL30DpqN0VVx79TK3ckf0M2I++3gQcl/Rywv4hKmVnrerl7aB21O9K8jn+nVqZIWQesm/gs6Xng3zd8XhURGzpbPTNrRa92D62jPCmquv2dWkltvUVkDjUUXdOB+piZVUIdU1TtaiW1dSzVvS8zM2tRHVNU7epkIHHDu5n1lbqlqNrVdmprCv0Xhs3MrKN3JP+3g9syM2uZ5xorR9OBRNLxwH8EhhvXi4g/Sz//oNOVMzNrVh3nsKqKVlJb95A9O/0Q8FrDy8ysdHWcw6oqWkltLYyI5YXVxMwsh15+FG3dtXJH8s+S/m1hNTEzy6GXH0Vbd60EkvcCWyQ9Lek7kp6Q9J2iKmZm1goPECxPK6mtXy+sFmZmOXmAYHlamWvr+5LeCyyJiM9LGgKceDSznuEBguVo5cFWnyJ7BsnaVDQT+GIRlTIzs+poJbX1m8C7gMcBIuJFSScWUiuzPjDd4DkPrLMqaSWQvBERISkA0iN3zawN0w2eAzywziqllV5bGyX9DfB2Sb8H/CPwt8VUy6zephs854F1VjVNB5KI+AvgbuDLwDuA/xoR/yvPziXNkPQtSV9Nn0+W9ICkZ9LPkxqWXStpR+p+fHFD+bmpK/IOSesk+ZLNet50g+em+86sF7U0+29EPBARfxIRfxwRD3Rg/9cA2xs+XwdsjoglwOb0GUlnASuBpcBy4GZJM9I6twCrgSXp5dH3fWx8PHh236s8/L0f8uy+Vxkf782nG0w3eM4D66xqjhlIJB2QtH+K1wFJbT+zXdJC4DeAzzYUrwAmHte7Abi0ofyuiHg9Ip4DdgDnSToNmBMRD0dEALc3rGN9ZqLd4ZJ1D/Ghv32US9Y9xP3bXurJYDLd4DkPrLOqOWZje0QU1TPrr4CPA43bnx8Ru9N+d0ual8oXAI80LLcrlR1M7yeXH0HSarI7FxYtWtSB6luvOVrbwplrLui5cQXHGjzngXVWJZ18HknTJH0A2BsRWyRd2MwqU5TFNOVHFkasB9YDjIyM9N4lquVWtUn7phs854F1ViWlBBLgPcAHJV0CDAJzJH0R2CPptHQ3chqwNy2/Czi9Yf2FwIupfOEU5X2tX8cgTLQtNAYTty2YFa+Tj9ptWkSsjYiFETFM1oj+jYj4CLAJWJUWW0X2DBRS+UpJx0s6g6xR/bGUBjsgaVnqrXVFwzp9qUrtBJ3mtgWzcpR1R3I0N5KNV7kSeB64DCAitknaCDxF9mCtqyPizbTOVcBtwAnAfenVt6rUTtBpnrSvt/TrnXE/Kj2QRMQ3gW+m9y8DFx1lueuB66coHwXOLq6G1VK1doJOc9tCb/Bjb/tLKaktK47HIFgv8Oj8/uJAUjNuJ7CjKWKw5tG26dH5/aX01JZ1ltsJbCpFpJqm26Z70PUX35HU0EQ7wbLFp7B4aLaDiBWSappum74z7i++IzHrA0V0wjjWNn1n3D8cSMz6QBGppmNt0z3o+odTW2Z9oIhUk9NXNkHZpLn9ZWRkJEZHR8uuhllXTQwQ7GSqqYhtWu+StCUiRiaXO7Vl1ieKSDU5fWXgQGJ2WFWm9KhKPa1/OJCYUZ0pPapSz6pxcM7Hje2WW1UebzudqkzpUVQ96/A3bFc/z5jdKb4jsVzqcoVclckui6hnXf6G7ernGbM7xXcklktVruSPpSqTXRZRz7r8DdvlecHycyCxXKr0n3C69E1VxkQUUc8q/Q2LUJWLiF7m1JblUsbkfO00jB4rfVOVyS6LqGe/T7A4EZwn/9votYuIXuYBiZZLt/Pr7e7v2X2vcsm6h444Wd7rPHjft5GAB1Y2ywMSrRDdvpJvt2G0Ko3pZajK3ViRPLAyHwcSy62b/wnbDQj9nr45Fp9ILQ83tlultNswWpXGdLMq8h2JVUq7DaNO35gVx43tVjluGDUrx9Ea20tJbUk6XdI/SdouaZuka1L5yZIekPRM+nlSwzprJe2Q9LSkixvKz5X0RPpunSSfUWrOjxK2burn6WOaVVYbySHgjyLiF4BlwNWSzgKuAzZHxBJgc/pM+m4lsBRYDtwsaUba1i3AamBJei3v5i9iZvXlebiaU0ogiYjdEfF4en8A2A4sAFYAG9JiG4BL0/sVwF0R8XpEPAfsAM6TdBowJyIejixHd3vDOmYd46vS7uqV493v08c0q/TGdknDwLuAR4H5EbEbsmAjaV5abAHwSMNqu1LZwfR+cvlU+1lNdufCokWLOvgbWN15wF539dLx9vij5pTa/VfSbODLwB9GxP7pFp2iLKYpP7IwYn1EjETEyNDQUOuV7XO9coVYBl+VdlcvHW/Pw9Wc0gKJpJlkQeSOiPhKKt6T0lWkn3tT+S7g9IbVFwIvpvKFU5RbB/V7nrjfJzXstl463h5/1Jyyem0JuBXYHhE3NXy1CViV3q8C7mkoXynpeElnkDWqP5bSYAckLUvbvKJhHeuQXrpCLIOvSrurl473xPije9dcwF2rz+feNRc4pTmFsu5I3gP8DvA+SVvT6xLgRuD9kp4B3p8+ExHbgI3AU8D9wNUR8Wba1lXAZ8ka4L8H3NfV36QP9NIVYhnyXJX2c0qwXb12F+Du5sfmAYl9pp0p2D1zbnuDIHup0bhqPOi0Nx1tQKIDSR9p98RWxgmxnYDXaxyArW48jXwFdfpk2u4U7N2ep6ouV/LuOmr9woGkRxVxMs1zYuvmNOPtBrxe46nrrV94GvkeVURPqV7qDTOdujTu91qjsVlRfEfSo4pIi1Tl2dR1uZKvy9T1dWivsmI5kPSoIk6mVTmxVSXgNaPqTx6sS3uVFcu9tnpUv/8HdvfP3uCeZ9bIvbYqpip3D3lMlzKp+pV8XbjnmTXDgaRgefLLdT6Z9vsdV1XUpb3KiuVeWwUqcrLDqk+90e/zd1WFe55ZM3xH0qR27iyKGg9Rh6t5p0yqoRdTrO5F1nscSJrQ7on7WCfLdv9D1GHAnlMm1dFLKdY6XETVkVNbTWg3DTPdAMA8aa86DNhzysTa4ZRob3IgaUK7J+7pTpZ5/kNUZYT6dPycB2tHHS6i6siprSa0m4aZLr+cp42gLgP2eillYtXglGhvciBpQp4T99FOlnn+Q/RiA6hZN9TlIqpuPLK9SZ0eae1GQ7P2eNaD8vjBVg06PUVKu72v/B/CzKrEU6QUJM+dRT+0EbjPv1n9OZDkVIcxHRM6fdJ3+s6sP7j7b0516Y5YxHQu7vNv1h8cSHKqw5gOKOakX5cgeyxVn/fMLK9aBBJJyyU9LWmHpOu6ue+6jNAu4qRflyA7nSIn5jSrisq3kUiaAXwGeD+wC/gXSZsi4qlu7L8uYzqKGOjVD33+69RGZtauygcS4DxgR0Q8CyDpLmAF0JVAAvXofVXESb8uQXY6nsXYrB6BZAHwQsPnXcD5JdWlsoo66dchyE7HU3aY1aONZKoz3REJakmrJY1KGt23b18XqlU9Eyf9ZYtPYfHQ7FrdORSlLm1kZnnU4Y5kF3B6w+eFwIuTF4qI9cB6yEa2d6dqVnf9kL4zO5Y6BJJ/AZZIOgP4AbAS+HC5VbJ+Uvf0ndmxVD6QRMQhSX8AfA2YAXwuIraVXC0zs75R+UACEBH3AveWXQ8zs35Uh8Z2MzMrkQOJmZnl4kBiZma59OWDrSTtA77f5uqnAD/sYHXqxMfm6Hxsjs7HZmq9eFx+LiKGJhf2ZSDJQ9LoVE8IMx+b6fjYHJ2PzdSqdFyc2jIzs1wcSMzMLBcHktatL7sCPczH5uh8bI7Ox2ZqlTkubiMxM7NcfEdiZma5OJCYmVkuDiQtKPPZ8L1G0uck7ZX0ZEPZyZIekPRM+nlSmXUsg6TTJf2TpO2Stkm6JpX72EiDkh6T9O10bP57Ku/7YwPZY8MlfUvSV9PnyhwXB5ImNTwb/teBs4APSTqr3FqV6jZg+aSy64DNEbEE2Jw+95tDwB9FxC8Ay4Cr078THxt4HXhfRLwTOAdYLmkZPjYTrgG2N3yuzHFxIGne4WfDR8QbwMSz4ftSRDwI/GhS8QpgQ3q/Abi0m3XqBRGxOyIeT+8PkJ0YFuBjQ2ReTR9nplfgY4OkhcBvAJ9tKK7McXEgad5Uz4ZfUFJdetX8iNgN2QkVmFdyfUolaRh4F/AoPjbA4fTNVmAv8EBE+Nhk/gr4ODDeUFaZ4+JA0rymng1vBiBpNvBl4A8jYn/Z9ekVEfFmRJxD9kjs8ySdXXKVSifpA8DeiNhSdl3a5UDSvKaeDd/n9kg6DSD93FtyfUohaSZZELkjIr6Sin1sGkTEvwLfJGtn6/dj8x7gg5J2kqXM3yfpi1TouDiQNO/ws+El/QzZs+E3lVynXrMJWJXerwLuKbEupZAk4FZge0Tc1PCVj400JOnt6f0JwK8C36XPj01ErI2IhRExTHZe+UZEfIQKHRePbG+BpEvIcpkTz4a/vtwalUfSncCFZFNd7wE+BfxvYCOwCHgeuCwiJjfI15qk9wIPAU/w03z3n5K1k/T7sflFskbjGWQXsRsj4s8kzaXPj80ESRcCfxwRH6jScXEgMTOzXJzaMjOzXBxIzMwsFwcSMzPLxYHEzMxycSAxM7NcHEjMzCwXBxIzM8vl/wM3HVmHCZYe0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(data=corpus_df, x=corpus_df.index, y=\"len_sentences\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c19827",
   "metadata": {
    "id": "02c19827"
   },
   "source": [
    "#### 2. No. of Unique Sentences\n",
    "\n",
    "Remove duplicate sentences to save computing time later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab8cfaf3",
   "metadata": {
    "id": "ab8cfaf3"
   },
   "outputs": [],
   "source": [
    "unique_sentences = list(dict.fromkeys(sentences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a4b2c75",
   "metadata": {
    "id": "9a4b2c75",
    "outputId": "7971dae8-1023-4b13-b815-b7bb7b168b36"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6276, 5962)"
      ]
     },
     "execution_count": 798,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sentences),len(unique_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e523f226",
   "metadata": {
    "id": "e523f226",
    "outputId": "706006bd-db95-450d-ab99-77ed57a2fb89"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Chief Justice Roberts, President Carter, President Clinton, President Bush, President Obama, fellow Americans, and people of the world: thank you.',\n",
       " 'We, the citizens of America, are now joined in a great national effort to rebuild our country and to restore its promise for all of our people.',\n",
       " 'Together, we will determine the course of America and the world for years to come.',\n",
       " 'We will face challenges.',\n",
       " 'We will confront hardships.',\n",
       " 'But we will get the job done.',\n",
       " 'Every four years, we gather on these steps to carry out the orderly and peaceful transfer of power, and we are grateful to President Obama and First Lady Michelle Obama for their gracious aid throughout this transition.',\n",
       " 'They have been magnificent.',\n",
       " \"Today's ceremony, however, has very special meaning.\",\n",
       " 'Because today we are not merely transferring power from one Administration to another, or from one party to another – but we are transferring power from Washington, D.C. and giving it back to you, the American People.',\n",
       " \"For too long, a small group in our nation's Capital has reaped the rewards of government while the people have borne the cost.\",\n",
       " 'Washington flourished – but the people did not share in its wealth.',\n",
       " 'Politicians prospered – but the jobs left, and the factories closed.',\n",
       " 'The establishment protected itself, but not the citizens of our country.',\n",
       " \"Their victories have not been your victories; their triumphs have not been your triumphs; and while they celebrated in our nation's Capital, there was little to celebrate for struggling families all across our land.\",\n",
       " 'That all changes – starting right here, and right now, because this moment is your moment: it belongs to you.',\n",
       " 'It belongs to everyone gathered here today and everyone watching all across America.',\n",
       " 'This is your day.',\n",
       " 'This is your celebration.',\n",
       " 'And this, the United States of America, is your country.',\n",
       " 'What truly matters is not which party controls our government, but whether our government is controlled by the people.',\n",
       " 'January 20th 2017, will be remembered as the day the people became the rulers of this nation again.',\n",
       " 'The forgotten men and women of our country will be forgotten no longer.',\n",
       " 'Everyone is listening to you now.',\n",
       " 'You came by the tens of millions to become part of a historic movement the likes of which the world has never seen before.',\n",
       " 'At the center of this movement is a crucial conviction: that a nation exists to serve its citizens.',\n",
       " 'Americans want great schools for their children, safe neighborhoods for their families, and good jobs for themselves.',\n",
       " 'These are the just and reasonable demands of a righteous public.',\n",
       " 'But for too many of our citizens, a different reality exists: Mothers and children trapped in poverty in our inner cities; rusted out factories scattered like tombstones across the landscape of our nation; an education system, flush with cash, but which leaves our young and beautiful students deprived of knowledge; and the crime and gangs and drugs that have stolen too many lives and robbed our country of so much unrealized potential.',\n",
       " 'This American carnage stops right here and stops right now.',\n",
       " 'We are one nation – and their pain is our pain.',\n",
       " 'Their dreams are our dreams; and their success will be our success.',\n",
       " 'We share one heart, one home, and one glorious destiny.',\n",
       " 'The oath of office I take today is an oath of allegiance to all Americans.',\n",
       " \"For many decades, we've enriched foreign industry at the expense of American industry; Subsidized the armies of other countries while allowing for the very sad depletion of our military; We've defended other nation's borders while refusing to defend our own; And spent trillions of dollars overseas while America's infrastructure has fallen into disrepair and decay.\",\n",
       " \"We've made other countries rich while the wealth, strength, and confidence of our country has disappeared over the horizon.\",\n",
       " 'One by one, the factories shuttered and left our shores, with not even a thought about the millions upon millions of American workers left behind.',\n",
       " 'The wealth of our middle class has been ripped from their homes and then redistributed across the entire world.',\n",
       " 'But that is the past.',\n",
       " 'And now we are looking only to the future.',\n",
       " 'We assembled here today are issuing a new decree to be heard in every city, in every foreign capital, and in every hall of power.',\n",
       " 'From this day forward, a new vision will govern our land.',\n",
       " \"From this moment on, it's going to be America First.\",\n",
       " 'Every decision on trade, on taxes, on immigration, on foreign affairs, will be made to benefit American workers and American families.',\n",
       " 'We must protect our borders from the ravages of other countries making our products, stealing our companies, and destroying our jobs.',\n",
       " 'Protection will lead to great prosperity and strength.',\n",
       " 'I will fight for you with every breath in my body – and I will never, ever let you down.',\n",
       " 'America will start winning again, winning like never before.',\n",
       " 'We will bring back our jobs.',\n",
       " 'We will bring back our borders.',\n",
       " 'We will bring back our wealth.',\n",
       " 'And we will bring back our dreams.',\n",
       " 'We will build new roads, and highways, and bridges, and airports, and tunnels, and railways all across our wonderful nation.',\n",
       " 'We will get our people off of welfare and back to work – rebuilding our country with American hands and American labor.',\n",
       " 'We will follow two simple rules: Buy American and Hire American.',\n",
       " 'We will seek friendship and goodwill with the nations of the world – but we do so with the understanding that it is the right of all nations to put their own interests first.',\n",
       " 'We do not seek to impose our way of life on anyone, but rather to let it shine as an example for everyone to follow.',\n",
       " 'We will reinforce old alliances and form new ones – and unite the civilized world against Radical Islamic Terrorism, which we will eradicate completely from the face of the Earth.',\n",
       " 'At the bedrock of our politics will be a total allegiance to the United States of America, and through our loyalty to our country, we will rediscover our loyalty to each other.',\n",
       " 'When you open your heart to patriotism, there is no room for prejudice.',\n",
       " \"The Bible tells us, “how good and pleasant it is when God's people live together in unity.” We must speak our minds openly, debate our disagreements honestly, but always pursue solidarity.\",\n",
       " 'When America is united, America is totally unstoppable.',\n",
       " 'There should be no fear – we are protected, and we will always be protected.',\n",
       " 'We will be protected by the great men and women of our military and law enforcement and, most importantly, we are protected by God.',\n",
       " 'Finally, we must think big and dream even bigger.',\n",
       " 'In America, we understand that a nation is only living as long as it is striving.',\n",
       " 'We will no longer accept politicians who are all talk and no action – constantly complaining but never doing anything about it.',\n",
       " 'The time for empty talk is over.',\n",
       " 'Now arrives the hour of action.',\n",
       " 'Do not let anyone tell you it can not be done.',\n",
       " 'No challenge can match the heart and fight and spirit of America.',\n",
       " 'We will not fail.',\n",
       " 'Our country will thrive and prosper again.',\n",
       " 'We stand at the birth of a new millennium, ready to unlock the mysteries of space, to free the Earth from the miseries of disease, and to harness the energies, industries and technologies of tomorrow.',\n",
       " 'A new national pride will stir our souls, lift our sights, and heal our divisions.',\n",
       " 'It is time to remember that old wisdom our soldiers will never forget: that whether we are black or brown or white, we all bleed the same red blood of patriots, we all enjoy the same glorious freedoms, and we all salute the same great American Flag.',\n",
       " 'And whether a child is born in the urban sprawl of Detroit or the windswept plains of Nebraska, they look up at the same night sky, they fill their heart with the same dreams, and they are infused with the breath of life by the same almighty Creator.',\n",
       " 'So to all Americans, in every city near and far, small and large, from mountain to mountain, and from ocean to ocean, hear these words: You will never be ignored again.',\n",
       " 'Your voice, your hopes, and your dreams, will define our American destiny.',\n",
       " 'And your courage and goodness and love will forever guide us along the way.',\n",
       " 'Together, We Will Make America Strong Again.',\n",
       " 'We Will Make America Wealthy Again.',\n",
       " 'We Will Make America Proud Again.',\n",
       " 'We Will Make America Safe Again.',\n",
       " 'And, Yes, Together, We Will Make America Great Again.',\n",
       " \"Thank you, God Bless You, And God Bless America Mr. Speaker, Mr. Vice President, Members of Congress, the First Lady of the United States, and Citizens of America: Tonight, as we mark the conclusion of our celebration of Black History Month, we are reminded of our Nation's path toward civil rights and the work that still remains.\",\n",
       " \"Recent threats targeting Jewish Community Centers and vandalism of Jewish cemeteries, as well as last week's shooting in Kansas City, remind us that while we may be a Nation divided on policies, we are a country that stands united in condemning hate and evil in all its forms.\",\n",
       " 'Each American generation passes the torch of truth, liberty and justice – - in an unbroken chain all the way down to the present.',\n",
       " 'That torch is now in our hands.',\n",
       " 'And we will use it to light up the world.',\n",
       " 'I am here tonight to deliver a message of unity and strength, and it is a message deeply delivered from my heart.',\n",
       " 'A new chapter of American Greatness is now beginning.',\n",
       " 'A new national pride is sweeping across our Nation.',\n",
       " 'And a new surge of optimism is placing impossible dreams firmly within our grasp.',\n",
       " 'What we are witnessing today is the Renewal of the American Spirit.',\n",
       " 'Our allies will find that America is once again ready to lead.',\n",
       " 'All the nations of the world friend or foe will find that America is strong, America is proud, and America is free.',\n",
       " 'In 9 years, the United States will celebrate the 250th anniversary of our founding 250 years since the day we declared our Independence.',\n",
       " 'It will be one of the great milestones in the history of the world.',\n",
       " 'But what will America look like as we reach our 250th year?',\n",
       " 'What kind of country will we leave for our children?',\n",
       " 'I will not allow the mistakes of recent decades past to define the course of our future.',\n",
       " \"For too long, we've watched our middle class shrink as we've exported our jobs and wealth to foreign countries.\",\n",
       " \"We've financed and built one global project after another, but ignored the fates of our children in the inner cities of Chicago, Baltimore, Detroit and so many other places throughout our land.\",\n",
       " \"We've defended the borders of other nations, while leaving our own borders wide open, for anyone to cross and for drugs to pour in at a now unprecedented rate.\",\n",
       " \"And we've spent trillions of dollars overseas, while our infrastructure at home has so badly crumbled.\",\n",
       " 'Then, in 2016, the earth shifted beneath our feet.',\n",
       " 'The rebellion started as a quiet protest, spoken by families of all colors and creeds - – families who just wanted a fair shot for their children, and a fair hearing for their concerns.',\n",
       " 'But then the quiet voices became a loud chorus as thousands of citizens now spoke out together, from cities small and large, all across our country.',\n",
       " 'Finally, the chorus became an earthquake – and the people turned out by the tens of millions, and they were all united by one very simple, but crucial demand, that America must put its own citizens first.. because only then, can we truly MAKE AMERICA GREAT AGAIN.',\n",
       " 'Dying industries will come roaring back to life.',\n",
       " 'Heroic veterans will get the care they so desperately need.',\n",
       " 'Our military will be given the resources its brave warriors so richly deserve.',\n",
       " 'Crumbling infrastructure will be replaced with new roads, bridges, tunnels, airports and railways gleaming across our beautiful land.',\n",
       " 'Our terrible drug epidemic will slow down and ultimately, stop.',\n",
       " 'And our neglected inner cities will see a rebirth of hope, safety, and opportunity.',\n",
       " 'Above all else, we will keep our promises to the American people.',\n",
       " \"It's been a little over a month since my inauguration, and I want to take this moment to update the Nation on the progress I've made in keeping those promises.\",\n",
       " 'Since my election, Ford, Fiat-Chrysler, General Motors, Sprint, Softbank, Lockheed, Intel, Walmart, and many others, have announced that they will invest billions of dollars in the United States and will create tens of thousands of new American jobs.',\n",
       " 'The stock market has gained almost three trillion dollars in value since the election on November 8th, a record.',\n",
       " \"We've saved taxpayers hundreds of millions of dollars by bringing down the price of the fantastic new F-35 jet fighter, and will be saving billions more dollars on contracts all across our Government.\",\n",
       " 'We have placed a hiring freeze on non military and non essential Federal workers.',\n",
       " 'We have begun to drain the swamp of government corruption by imposing a 5 year ban on lobbying by executive branch officials – - and a lifetime ban on becoming lobbyists for a foreign government.',\n",
       " 'We have undertaken a historic effort to eliminate job‐crushing regulations, creating a deregulation task force inside of every Government agency; imposing a new rule which mandates that for every 1 new regulation, 2 old regulations must be eliminated; and stopping a regulation that threatens the future and livelihoods of our great coal miners.',\n",
       " \"We have cleared the way for the construction of the Keystone and Dakota Access Pipelines thereby creating tens of thousands of jobs and I've issued a new directive that new American pipelines be made with American steel.\",\n",
       " 'We have withdrawn the United States from the job killing Trans Pacific Partnership.',\n",
       " 'With the help of Prime Minister Justin Trudeau, we have formed a Council with our neighbors in Canada to help ensure that women entrepreneurs have access to the networks, markets and capital they need to start a business and live out their financial dreams.',\n",
       " 'To protect our citizens, I have directed the Department of Justice to form a Task Force on Reducing Violent Crime.',\n",
       " 'I have further ordered the Departments of Homeland Security and Justice, along with the Department of State and the Director of National Intelligence, to coordinate an aggressive strategy to dismantle the criminal cartels that have spread across our Nation.',\n",
       " 'We will stop the drugs from pouring into our country and poisoning our youth and we will expand treatment for those who have become so badly addicted.',\n",
       " 'At the same time, my Administration has answered the pleas of the American people for immigration enforcement and border security.',\n",
       " 'By finally enforcing our immigration laws, we will raise wages, help the unemployed, save billions of dollars, and make our communities safer for everyone.',\n",
       " \"We want all Americans to succeed – - but that can't happen in an environment of lawless chaos.\",\n",
       " 'We must restore integrity and the rule of law to our borders.',\n",
       " 'For that reason, we will soon begin the construction of a great wall along our southern border.',\n",
       " 'As we speak, we are removing gang members, drug dealers and criminals that threaten our communities and prey on our citizens.',\n",
       " 'Bad ones are going out as I speak tonight and as I have promised.',\n",
       " 'To any in Congress who do not believe we should enforce our laws, I would ask you this question: what would you say to the American family that loses their jobs, their income, or a loved one, because America refused to uphold its laws and defend its borders?',\n",
       " 'Our obligation is to serve, protect, and defend the citizens of the United States.',\n",
       " 'We are also taking strong measures to protect our Nation from Radical Islamic Terrorism.',\n",
       " 'According to data provided by the Department of Justice, the vast majority of individuals convicted for terrorism related offenses since 9/11 came here from outside of our country.',\n",
       " 'We have seen the attacks at home - – from Boston to San Bernardino to the Pentagon and yes, even the World Trade Center.',\n",
       " 'We have seen the attacks in France, in Belgium, in Germany and all over the world.',\n",
       " 'It is not compassionate, but reckless, to allow uncontrolled entry from places where proper vetting can not occur.',\n",
       " 'Those given the high honor of admission to the United States should support this country and love its people and its values.',\n",
       " 'We can not allow a beachhead of terrorism to form inside America we can not allow our Nation to become a sanctuary for extremists.',\n",
       " 'That is why my Administration has been working on improved vetting procedures, and we will shortly take new steps to keep our Nation safe and to keep out those who would do us harm.',\n",
       " 'As promised, I directed the Department of Defense to develop a plan to demolish and destroy ISIS a network of lawless savages that have slaughtered Muslims and Christians, and men, women, and children of all faiths and beliefs.',\n",
       " 'We will work with our allies, including our friends and allies in the Muslim world, to extinguish this vile enemy from our planet.',\n",
       " \"I have also imposed new sanctions on entities and individuals who support Iran's ballistic missile program, and reaffirmed our unbreakable alliance with the State of Israel.\",\n",
       " 'Finally, I have kept my promise to appoint a Justice to the United States Supreme Court from my list of 20 judges who will defend our Constitution.',\n",
       " 'I am honored to have Maureen Scalia with us in the gallery tonight.',\n",
       " 'Her late, great husband, Antonin Scalia, will forever be a symbol of American justice.',\n",
       " 'To fill his seat, we have chosen Judge Neil Gorsuch, a man of incredible skill, and deep devotion to the law.',\n",
       " 'He was confirmed unanimously to the Court of Appeals, and I am asking the Senate to swiftly approve his nomination.',\n",
       " 'Tonight, as I outline the next steps we must take as a country, we must honestly acknowledge the circumstances we inherited.',\n",
       " 'Ninety-four million Americans are out of the labor force.',\n",
       " 'Over 43 million people are now living in poverty, and over 43 million Americans are on food stamps.',\n",
       " 'More than 1 in 5 people in their prime working years are not working.',\n",
       " 'We have the worst financial recovery in 65 years.',\n",
       " 'In the last 8 years, the past Administration has put on more new debt than nearly all other Presidents combined.',\n",
       " \"We've lost more than one-fourth of our manufacturing jobs since NAFTA was approved, and we've lost 60,000 factories since China joined the World Trade Organization in 2001.\",\n",
       " 'Our trade deficit in goods with the world last year was nearly $ 800 billion dollars.',\n",
       " 'And overseas, we have inherited a series of tragic foreign policy disasters.',\n",
       " 'Solving these, and so many other pressing problems, will require us to work past the differences of party.',\n",
       " 'It will require us to tap into the American spirit that has overcome every challenge throughout our long and storied history.',\n",
       " 'But to accomplish our goals at home and abroad, we must restart the engine of the American economy making it easier for companies to do business in the United States, and much harder for companies to leave.',\n",
       " 'Right now, American companies are taxed at one of the highest rates anywhere in the world.',\n",
       " 'My economic team is developing historic tax reform that will reduce the tax rate on our companies so they can compete and thrive anywhere and with anyone.',\n",
       " 'At the same time, we will provide massive tax relief for the middle class.',\n",
       " 'We must create a level playing field for American companies and workers.',\n",
       " 'Currently, when we ship products out of America, many other countries make us pay very high tariffs and taxes but when foreign companies ship their products into America, we charge them almost nothing.',\n",
       " 'I just met with officials and workers from a great American company, Harley-Davidson.',\n",
       " 'In fact, they proudly displayed five of their magnificent motorcycles, made in the USA, on the front lawn of the White House.',\n",
       " 'At our meeting, I asked them, how are you doing, how is business?',\n",
       " \"They said that it's good.\",\n",
       " 'I asked them further how they are doing with other countries, mainly international sales.',\n",
       " 'They told me without even complaining because they have been mistreated for so long that they have become used to it that it is very hard to do business with other countries because they tax our goods at such a high rate.',\n",
       " 'They said that in one case another country taxed their motorcycles at 100 percent.',\n",
       " \"They weren't even asking for change.\",\n",
       " 'But I am.',\n",
       " 'I believe strongly in free trade but it also has to be FAIR TRADE.',\n",
       " 'The first Republican President, Abraham Lincoln, warned that the “abandonment of the protective policy by the American Government [ will ] produce want and ruin among our people.” Lincoln was right and it is time we heeded his words.',\n",
       " 'I am not going to let America and its great companies and workers, be taken advantage of anymore.',\n",
       " 'I am going to bring back millions of jobs.',\n",
       " 'Protecting our workers also means reforming our system of legal immigration.',\n",
       " 'The current, outdated system depresses wages for our poorest workers, and puts great pressure on taxpayers.',\n",
       " 'Nations around the world, like Canada, Australia and many others – - have a merit based immigration system.',\n",
       " 'It is a basic principle that those seeking to enter a country ought to be able to support themselves financially.',\n",
       " 'Yet, in America, we do not enforce this rule, straining the very public resources that our poorest citizens rely upon.',\n",
       " \"According to the National Academy of Sciences, our current immigration system costs America's taxpayers many billions of dollars a year.\",\n",
       " \"Switching away from this current system of lower skilled immigration, and instead adopting a merit based system, will have many benefits: it will save countless dollars, raise workers ' wages, and help struggling families – - including immigrant families – - enter the middle class.\",\n",
       " 'Another Republican President, Dwight D. Eisenhower, initiated the last truly great national infrastructure program – - the building of the interstate highway system.',\n",
       " 'The time has come for a new program of national rebuilding.',\n",
       " 'America has spent approximately six trillion dollars in the Middle East, all this while our infrastructure at home is crumbling.',\n",
       " 'With this six trillion dollars we could have rebuilt our country – - twice.',\n",
       " 'And maybe even three times if we had people who had the ability to negotiate.',\n",
       " 'To launch our national rebuilding, I will be asking the Congress to approve legislation that produces a $ 1 trillion investment in the infrastructure of the United States financed through both public and private capital – - creating millions of new jobs.',\n",
       " 'This effort will be guided by two core principles: Buy American, and Hire American.',\n",
       " 'Tonight, I am also calling on this Congress to repeal and replace Obamacare with reforms that expand choice, increase access, lower costs, and at the same time, provide better Healthcare.',\n",
       " 'Mandating every American to buy government approved health insurance was never the right solution for America.',\n",
       " 'The way to make health insurance available to everyone is to lower the cost of health insurance, and that is what we will do.',\n",
       " 'Obamacare premiums nationwide have increased by double and triple digits.',\n",
       " 'As an example, Arizona went up 116 percent last year alone.',\n",
       " 'Governor Matt Bevin of Kentucky just said Obamacare is failing in his State it is unsustainable and collapsing.',\n",
       " 'One third of counties have only one insurer on the exchanges – - leaving many Americans with no choice at all.',\n",
       " 'Remember when you were told that you could keep your doctor, and keep your plan?',\n",
       " 'We now know that all of those promises have been broken.',\n",
       " 'Obamacare is collapsing – - and we must act decisively to protect all Americans.',\n",
       " 'Action is not a choice – - it is a necessity.',\n",
       " 'So I am calling on all Democrats and Republicans in the Congress to work with us to save Americans from this imploding Obamacare disaster.',\n",
       " 'Here are the principles that should guide the Congress as we move to create a better healthcare system for all Americans: First, we should ensure that Americans with pre existing conditions have access to coverage, and that we have a stable transition for Americans currently enrolled in the healthcare exchanges.',\n",
       " 'Secondly, we should help Americans purchase their own coverage, through the use of tax credits and expanded Health Savings Accounts – - but it must be the plan they want, not the plan forced on them by the Government.',\n",
       " 'Thirdly, we should give our great State Governors the resources and flexibility they need with Medicaid to make sure no one is left out.',\n",
       " 'Fourthly, we should implement legal reforms that protect patients and doctors from unnecessary costs that drive up the price of insurance – and work to bring down the artificially high price of drugs and bring them down immediately.',\n",
       " 'Finally, the time has come to give Americans the freedom to purchase health insurance across State lines – - creating a truly competitive national marketplace that will bring cost way down and provide far better care.',\n",
       " 'Everything that is broken in our country can be fixed.',\n",
       " 'Every problem can be solved.',\n",
       " 'And every hurting family can find healing, and hope.',\n",
       " 'Our citizens deserve this, and so much more – - so why not join forces to finally get it done?',\n",
       " 'On this and so many other things, Democrats and Republicans should get together and unite for the good of our country, and for the good of the American people.',\n",
       " \"My administration wants to work with members in both parties to make childcare accessible and affordable, to help ensure new parents have paid family leave, to invest in women's health, and to promote clean air and clear water, and to rebuild our military and our infrastructure.\",\n",
       " 'True love for our people requires us to find common ground, to advance the common good, and to cooperate on behalf of every American child who deserves a brighter future.',\n",
       " 'An incredible young woman is with us this evening who should serve as an inspiration to us all.',\n",
       " 'Today is Rare Disease day, and joining us in the gallery is a Rare Disease Survivor, Megan Crowley.',\n",
       " 'Megan was diagnosed with Pompe Disease, a rare and serious illness, when she was 15 months old.',\n",
       " 'She was not expected to live past 5.',\n",
       " \"On receiving this news, Megan's dad, John, fought with everything he had to save the life of his precious child.\",\n",
       " \"He founded a company to look for a cure, and helped develop the drug that saved Megan's life.\",\n",
       " 'Today she is 20 years old and a sophomore at Notre Dame.',\n",
       " \"Megan's story is about the unbounded power of a father's love for a daughter.\",\n",
       " \"But our slow and burdensome approval process at the Food and Drug Administration keeps too many advances, like the one that saved Megan's life, from reaching those in need.\",\n",
       " 'If we slash the restraints, not just at the FDA but across our Government, then we will be blessed with far more miracles like Megan.',\n",
       " 'In fact, our children will grow up in a Nation of miracles.',\n",
       " 'But to achieve this future, we must enrich the mind – - and the souls – - of every American child.',\n",
       " 'Education is the civil rights issue of our time.',\n",
       " 'I am calling upon Members of both parties to pass an education bill that funds school choice for disadvantaged youth, including millions of African-American and Latino children.',\n",
       " 'These families should be free to choose the public, private, charter, magnet, religious or home school that is right for them.',\n",
       " 'Joining us tonight in the gallery is a remarkable woman, Denisha Merriweather.',\n",
       " 'As a young girl, Denisha struggled in school and failed third grade twice.',\n",
       " 'But then she was able to enroll in a private center for learning, with the help of a tax credit scholarship program.',\n",
       " 'Today, she is the first in her family to graduate, not just from high school, but from college.',\n",
       " 'Later this year she will get her masters degree in social work.',\n",
       " 'We want all children to be able to break the cycle of poverty just like Denisha.',\n",
       " 'But to break the cycle of poverty, we must also break the cycle of violence.',\n",
       " 'The murder rate in 2015 experienced its largest single-year increase in nearly half a century.',\n",
       " 'In Chicago, more than 4,000 people were shot last year alone – - and the murder rate so far this year has been even higher.',\n",
       " 'This is not acceptable in our society.',\n",
       " 'Every American child should be able to grow up in a safe community, to attend a great school, and to have access to a high-paying job.',\n",
       " 'But to create this future, we must work with – - not against - – the men and women of law enforcement.',\n",
       " 'We must build bridges of cooperation and trust – - not drive the wedge of disunity and division.',\n",
       " 'Police and sheriffs are members of our community.',\n",
       " \"They are friends and neighbors, they are mothers and fathers, sons and daughters – and they leave behind loved ones every day who worry whether or not they'll come home safe and sound.\",\n",
       " 'We must support the incredible men and women of law enforcement.',\n",
       " 'And we must support the victims of crime.',\n",
       " 'I have ordered the Department of Homeland Security to create an office to serve American Victims.',\n",
       " 'The office is called VOICE – - Victims Of Immigration Crime Engagement.',\n",
       " 'We are providing a voice to those who have been ignored by our media, and silenced by special interests.',\n",
       " 'Joining us in the audience tonight are four very brave Americans whose government failed them.',\n",
       " 'Their names are Jamiel Shaw, Susan Oliver, Jenna Oliver, and Jessica Davis.',\n",
       " \"Jamiel's 17-year-old son was viciously murdered by an illegal immigrant gang member, who had just been released from prison.\",\n",
       " 'Jamiel Shaw Jr. was an incredible young man, with unlimited potential who was getting ready to go to college where he would have excelled as a great quarterback.',\n",
       " 'But he never got the chance.',\n",
       " 'His father, who is in the audience tonight, has become a good friend of mine.',\n",
       " 'Also with us are Susan Oliver and Jessica Davis.',\n",
       " 'Their husbands – - Deputy Sheriff Danny Oliver and Detective Michael Davis – - were slain in the line of duty in California.',\n",
       " 'They were pillars of their community.',\n",
       " 'These brave men were viciously gunned down by an illegal immigrant with a criminal record and two prior deportations.',\n",
       " 'Sitting with Susan is her daughter, Jenna.',\n",
       " 'Jenna: I want you to know that your father was a hero, and that tonight you have the love of an entire country supporting you and praying for you.',\n",
       " 'To Jamiel, Jenna, Susan and Jessica: I want you to know – - we will never stop fighting for justice.',\n",
       " 'Your loved ones will never be forgotten, we will always honor their memory.',\n",
       " 'Finally, to keep America Safe we must provide the men and women of the United States military with the tools they need to prevent war and – - if they must – - to fight and to win.',\n",
       " 'I am sending the Congress a budget that rebuilds the military, eliminates the Defense sequester, and calls for one of the largest increases in national defense spending in American history.',\n",
       " 'My budget will also increase funding for our veterans.',\n",
       " 'Our veterans have delivered for this Nation – - and now we must deliver for them.',\n",
       " 'The challenges we face as a Nation are great.',\n",
       " 'But our people are even greater.',\n",
       " 'And none are greater or braver than those who fight for America in uniform.',\n",
       " 'We are blessed to be joined tonight by Carryn Owens, the widow of a in 1881.',\n",
       " 'Navy Special Operator, Senior Chief William “Ryan” Owens.',\n",
       " 'Ryan died as he lived: a warrior, and a hero – - battling against terrorism and securing our Nation.',\n",
       " \"I just spoke to General Mattis, who reconfirmed that, and I quote, “Ryan was a part of a highly successful raid that generated large amounts of vital intelligence that will lead to many more victories in the future against our enemies.” Ryan's legacy is etched into eternity.\",\n",
       " \"For as the Bible teaches us, there is no greater act of love than to lay down one's life for one's friends.\",\n",
       " 'Ryan laid down his life for his friends, for his country, and for our freedom – - we will never forget him.',\n",
       " 'To those allies who wonder what kind of friend America will be, look no further than the heroes who wear our uniform.',\n",
       " 'Our foreign policy calls for a direct, robust and meaningful engagement with the world.',\n",
       " 'It is American leadership based on vital security interests that we share with our allies across the globe.',\n",
       " 'We strongly support NATO, an alliance forged through the bonds of two World Wars that dethroned fascism, and a Cold War that defeated communism.',\n",
       " 'But our partners must meet their financial obligations.',\n",
       " 'And now, based on our very strong and frank discussions, they are beginning to do just that.',\n",
       " 'We expect our partners, whether in NATO, in the Middle East, or the Pacific – - to take a direct and meaningful role in both strategic and military operations, and pay their fair share of the cost.',\n",
       " 'We will respect historic institutions, but we will also respect the sovereign rights of nations.',\n",
       " 'Free nations are the best vehicle for expressing the will of the people – - and America respects the right of all nations to chart their own path.',\n",
       " 'My job is not to represent the world.',\n",
       " 'My job is to represent the United States of America.',\n",
       " 'But we know that America is better off, when there is less conflict not more.',\n",
       " 'We must learn from the mistakes of the past – - we have seen the war and destruction that have raged across our world.',\n",
       " 'The only long term solution for these humanitarian disasters is to create the conditions where displaced persons can safely return home and begin the long process of rebuilding.',\n",
       " 'America is willing to find new friends, and to forge new partnerships, where shared interests align.',\n",
       " 'We want harmony and stability, not war and conflict.',\n",
       " 'We want peace, wherever peace can be found.',\n",
       " 'America is friends today with former enemies.',\n",
       " 'Some of our closest allies, decades ago, fought on the opposite side of these World Wars.',\n",
       " 'This history should give us all faith in the possibilities for a better world.',\n",
       " 'Hopefully, the 250th year for America will see a world that is more peaceful, more just and more free.',\n",
       " \"On our 100th anniversary, in 1876, citizens from across our Nation came to Philadelphia to celebrate America's centennial.\",\n",
       " \"At that celebration, the country's builders and artists and inventors showed off their creations.\",\n",
       " 'Alexander Graham Bell displayed his telephone for the first time.',\n",
       " 'Remington unveiled the first typewriter.',\n",
       " 'An early attempt was made at electric light.',\n",
       " 'Thomas Edison showed an automatic telegraph and an electric pen.',\n",
       " \"Imagine the wonders our country could know in America's 250th year.\",\n",
       " 'Think of the marvels we can achieve if we simply set free the dreams of our people.',\n",
       " 'Cures to illnesses that have always plagued us are not too much to hope.',\n",
       " 'American footprints on distant worlds are not too big a dream.',\n",
       " 'Millions lifted from welfare to work is not too much to expect.',\n",
       " 'And streets where mothers are safe from fear schools where children learn in peace and jobs where Americans prosper and grow are not too much to ask.',\n",
       " 'When we have all of this, we will have made America greater than ever before.',\n",
       " 'For all Americans.',\n",
       " 'This is our vision.',\n",
       " 'This is our mission.',\n",
       " 'But we can only get there together.',\n",
       " 'We are one people, with one destiny.',\n",
       " 'We all bleed the same blood.',\n",
       " 'We all salute the same flag.',\n",
       " 'And we are all made by the same God.',\n",
       " 'And when we fulfill this vision; when we celebrate our 250 years of glorious freedom, we will look back on tonight as when this new chapter of American Greatness began.',\n",
       " 'The time for small thinking is over.',\n",
       " 'The time for trivial fights is behind us.',\n",
       " 'We just need the courage to share the dreams that fill our hearts.',\n",
       " 'The bravery to express the hopes that stir our souls.',\n",
       " 'And the confidence to turn those hopes and dreams to action.',\n",
       " 'From now on, America will be empowered by our aspirations, not burdened by our fears – - inspired by the future, not bound by the failures of the past – - and guided by our vision, not blinded by our doubts.',\n",
       " 'I am asking all citizens to embrace this Renewal of the American Spirit.',\n",
       " 'I am asking all members of Congress to join me in dreaming big, and bold and daring things for our country.',\n",
       " 'And I am asking everyone watching tonight to seize this moment and Believe in yourselves.',\n",
       " 'Believe in your future.',\n",
       " 'And believe, once more, in America.',\n",
       " 'Thank you, God bless you, and God Bless these United States Thank you, everybody.',\n",
       " 'Thank you very much.',\n",
       " 'How are you?',\n",
       " 'Well I want to thank everybody on stage.',\n",
       " 'They are really a terrific team.',\n",
       " 'We have some of the real winners in the audience, too, that I can tell you, some great, great people.',\n",
       " 'I want to thank Vice President Pence.',\n",
       " \"As always, he's right there and he has really been a help to this administration.\",\n",
       " 'We have some big things happening today, and we have some very big things happening over the next month, and I guess probably I can say over the next eight years.',\n",
       " 'I suspect I can say that.',\n",
       " \"It's wonderful to be here with so many pioneers and visionaries from America's energy industry, great industry.\",\n",
       " 'I want to thank the leaders of our great energy companies for joining us today and for supporting our efforts to bring true wealth and prosperity to our people.',\n",
       " \"That's true.\",\n",
       " 'Come on, give yourself a hand.',\n",
       " 'You deserve it.',\n",
       " \"You've gone through eight years of hell, and actually I could say even a little bit more than that.\",\n",
       " 'I also want to express our sincere gratitude to the labor union leaders and members who have joined us today.',\n",
       " 'Thank you, fellas.',\n",
       " 'Thank you, thank you.',\n",
       " 'Your workers embody the skill, grit and courage that has always been the true source of American strength.',\n",
       " 'They are great people.',\n",
       " 'They break through rock walls, mine the depths of the earth, and reach through the ocean floor, to bring every ounce of energy into our homes and commerce and into our lives.',\n",
       " 'Our nation salutes you.',\n",
       " \"You're brave and you are great workers.\",\n",
       " 'Before turning to the topic at hand, I want to provide a brief update on two crucial votes taking place this afternoon on the House Floor, very important.',\n",
       " 'These bills are vital to public safety and national security, and I want to thank Chairman Bob Goodlatte for his efforts.',\n",
       " 'Bob has been working very hard and really for a long time, but we got it going.',\n",
       " \"First, the House will be voting on Kate's Law, legislation named for Kate Steinle, who was killed by an illegal immigrant with five prior deportations and lots of bad things on his record.\",\n",
       " 'The second is the No Sanctuary for Criminals Act, which blocks federal grants to cities that release dangerous criminal aliens back into the streets, including the vicious and disgusting and horrible MS-13 gang members.',\n",
       " \"And we're getting them out.\",\n",
       " 'We are getting them out.',\n",
       " \"They're going, fast.\",\n",
       " \"General Kelly and his whole group, they've gotten rid of 6,000 so far.\",\n",
       " \"We're about 50 percent there, and we're actually liberating towns.\",\n",
       " \"Like on long Island, where I grew up, we're liberating towns.\",\n",
       " 'Those people are so happy to see our guys, and our guys are a lot tougher than the MS-13 characters.',\n",
       " 'That I can tell you.',\n",
       " 'Liberation.',\n",
       " 'be: ( 1 calling on all lawmakers to put the safety of American families first.',\n",
       " \"And let's pass these bills through the House, through the Senate, and send them to my desk.\",\n",
       " 'I will give you the fastest approval, the fastest signature that you have ever seen.',\n",
       " 'Right, Mike?',\n",
       " 'We will get that signed so fast.',\n",
       " 'So when we get back, the first thing be: ( 1 going to do is how did we do on the vote?',\n",
       " \"I expect good things; otherwise I probably wouldn't be talking about it to be honest with you.\",\n",
       " \"I'd just sort of low key it a little bit.\",\n",
       " '( Laughter. )',\n",
       " \"I don't like losing.\",\n",
       " 'Neither do you folks.',\n",
       " 'Every member of Congress should vote to save American lives.',\n",
       " 'Many great members of Congress are here with us this afternoon, some great people, some great, great people.',\n",
       " 'Not only are they working with us on border security, but they share our desire to unleash American energy.',\n",
       " 'I especially want to thank Secretary Perry for his tremendous leadership in this department.',\n",
       " \"He has really done a terrific job, and he's also a cheerleader, he's really a cheerleader.\",\n",
       " 'I watched that in Texas.',\n",
       " 'The thing that I loved about him, he was always saying how great Texas was.',\n",
       " \"And if you don't say it, I don't know.\",\n",
       " 'You got to say it, right?',\n",
       " \"And you're doing it right now with energy.\",\n",
       " 'Along with Secretary of the Interior Ryan Zinke and EPA Administrator Scott Pruitt.',\n",
       " \"And we've been through our battles, Scott.\",\n",
       " 'Not with each other, with the world.',\n",
       " \"And now the world is starting to say I think they're right.\",\n",
       " \"They'll find out.\",\n",
       " 'We have no doubt.',\n",
       " 'But all three of them strongly believe in putting America first, which is what I believe in, which is why I got elected.',\n",
       " \"It's called Make America Great Again, that's what we're doing, Make America Great Again.\",\n",
       " \"We're here today to usher in a new American energy policy, one that unlocks million and millions of jobs and trillions of dollars in wealth.\",\n",
       " 'For over 40 years, America was vulnerable to foreign regimes that used energy as an economic weapon.',\n",
       " \"Americans ' quality of life was diminished by the idea that energy resources were too scarce to support our people.\",\n",
       " 'We always thought that, and actually at the time it was right to think.',\n",
       " \"We didn't think we had this tremendous wealth under our feet.\",\n",
       " 'Many of us remember the long gas lines and the constant claims that the world was running out of oil and natural gas.',\n",
       " 'Americans were told that our nation could only solve this energy crisis by imposing draconian restrictions on energy production.',\n",
       " 'But we now know that was all a big, beautiful myth.',\n",
       " 'It was fake.',\n",
       " \"Don't we love that term, “fake ”?\",\n",
       " \"What we've learned about fake over the last little while, fake news, CNN.\",\n",
       " 'Fake.',\n",
       " '( Laughter and Applause. )',\n",
       " 'Whoops, their camera just went off.',\n",
       " 'Okay, you can come back.',\n",
       " \"I won't say, I promise I won't say anything more about you.\",\n",
       " 'I see that red light go off, I say, whoa.',\n",
       " 'The truth is that we have near-limitless supplies of energy in our country.',\n",
       " 'Powered by new innovation and technology, we are now on the cusp of a true energy revolution.',\n",
       " \"Our country is blessed with extraordinary energy abundance, which we didn't know of, even five years ago and certainly ten years ago.\",\n",
       " \"We have nearly 100 years ' worth of natural gas and more than 250 years ' worth of clean, beautiful coal.\",\n",
       " 'We are a top producer of petroleum and the number one producer of natural gas.',\n",
       " 'We have so much more than we ever thought possible.',\n",
       " 'We are really in the driving seat.',\n",
       " 'And you know what?',\n",
       " \"We don't want to let other countries take away our sovereignty and tell us what to do and how to do it.\",\n",
       " \"That's not going to happen.\",\n",
       " \"With these incredible resources, my administration will seek not only American energy independence that we've been looking for so long, but American energy dominance.\",\n",
       " \"And we're going to be an exporter, exporter.\",\n",
       " 'We will be dominant.',\n",
       " 'We will export American energy all over the world, all around the globe.',\n",
       " 'These energy exports will create countless jobs for our people, and provide true energy security to our friends, partners, and allies all across the globe.',\n",
       " \"But this full potential can only be realized when government promotes energy development, that's this guy right here, and he'll do it better than anybody, instead of obstructing it like the Democrats.\",\n",
       " 'They obstruct it.',\n",
       " 'But we get through it.',\n",
       " 'We can not have obstruction.',\n",
       " 'We have to get out and do our job better and faster than anybody in the world, certainly when it comes to one of our great assets, energy.',\n",
       " 'This vast energy wealth does not belong to the government.',\n",
       " 'It belongs to the people of the United States of America.',\n",
       " 'Yet, for the past eight years, the federal government imposed massive job killing barriers to American energy development.',\n",
       " 'Since my very first day in office, I have been moving at record pace to cancel these regulations and to eliminate the barriers to domestic energy production, like never before.',\n",
       " \"Job killing regulations are being removed and vital infrastructure projects are being approved at a level that they've never seen before.\",\n",
       " 'As you all know, I approved the Keystone XL Pipeline and the Dakota Access Pipeline in my first week.',\n",
       " 'Thousands of jobs, tremendous things are happening.',\n",
       " \"And, by the way, I thought I'd take a lot of heat.\",\n",
       " \"I didn't take any heat.\",\n",
       " 'I approved them and that was it.',\n",
       " \"I figured we'd have all sorts of protests.\",\n",
       " \"We didn't have anything.\",\n",
       " \"But I have to do, whether it's protesting or not, I have to do what's right.\",\n",
       " 'But people celebrate those two transactions, as opposed to protesting.',\n",
       " 'But sometimes you have to go out and just do it, and you find out.',\n",
       " 'Whatever happens, happens.',\n",
       " 'But you have to be right for the American people.',\n",
       " 'Thank you.',\n",
       " 'be: ( 1 dramatically reducing restrictions on the development of natural gas.',\n",
       " 'I cancelled the moratorium on a new coal leasing, and you know what was happening, the new coal leasing on federal lands, it was being so terribly restricted.',\n",
       " \"And now with Ryan and with a group, it's going to be open, and the land will be left in better shape than it is right now.\",\n",
       " 'Is that right?',\n",
       " 'Better shape.',\n",
       " 'We have finally ended the war on coal.',\n",
       " 'And I am proud to report that Corsa Coal, here with us today, just opened a brand new coal mine in the state of Pennsylvania, the first one in many, many, many years.',\n",
       " 'Corsa, stand up.',\n",
       " 'Come on.',\n",
       " 'Congratulations.',\n",
       " 'Employing a lot of people, and we are putting the coal miners back to work just like I promised, just like I promised when I went through Ohio and West Virginia, Wyoming and all of the different places.',\n",
       " 'And I see Bob back there.',\n",
       " 'Congratulations, Bob.',\n",
       " \"He's in great shape, right?\",\n",
       " 'You in good shape, Bob?',\n",
       " 'Right from the beginning.',\n",
       " 'Good.',\n",
       " 'You just take care of yourself, all right?',\n",
       " \"We're ending intrusive EPA regulations that kill jobs, hurt family farmers and ranchers, and raise the price of energy so quickly and so substantially.\",\n",
       " \"In order to protect American jobs, companies and workers, we've withdrawn the United States from the one-sided Paris Climate Accord.\",\n",
       " \"And I won't get into it, but believe me, that really put this country at a disadvantage.\",\n",
       " \"Number one, we weren't playing on the same field.\",\n",
       " \"It kicked in for us, and it doesn't kick in for others.\",\n",
       " 'The money that we had to pay was enormous.',\n",
       " 'It was not even close.',\n",
       " \"And maybe we'll be back into it someday, but it will be on better terms.\",\n",
       " \"It will be on fair terms, not on terms where we're the people that don't know what we're doing.\",\n",
       " \"So we'll see what happens.\",\n",
       " \"But I will tell you we're proud of it.\",\n",
       " 'And when I go around, there are so many people that say thank you.',\n",
       " 'You saved the sovereignty of our country.',\n",
       " 'You saved our wealth because we would have a hard time getting to this newfound wealth.',\n",
       " \"And that's not going to happen with our country.\",\n",
       " 'Today, I am proudly announcing six brand new initiatives to propel this new era of American energy dominance.',\n",
       " 'First, we will begin to revive and expand our nuclear energy sector, which be: ( 1 so happy about, which produces clean, renewable and emissions free energy.',\n",
       " 'A complete review of in 1881. nuclear energy policy will help us find new ways to revitalize this crucial energy resource.',\n",
       " \"And I know you're very excited about that, Rick.\",\n",
       " 'Second, the Department of the Treasury will address barriers to the financing of highly efficient, overseas coal energy plants.',\n",
       " 'Ukraine already tells us they need millions and millions of metric tons right now.',\n",
       " 'There are many other places that need it, too.',\n",
       " 'And we want to sell it to them, and to everyone else all over the globe who need it.',\n",
       " 'Third, my administration has just approved the construction of a new petroleum pipeline to Mexico, which will further boost American energy exports, and that will go right under the wall, right?',\n",
       " \"It's going under, right?\",\n",
       " '( Laughter and applause. )',\n",
       " 'Have it go down a little deeper in that one section.',\n",
       " 'You know, a little like this.',\n",
       " 'Right under the wall.',\n",
       " 'Fourth, just today, a major in 1881. company, Sempra Energy, signed an agreement to begin negotiations for the sale of more American natural gas to South Korea.',\n",
       " \"And, as you know, the leaders of South Korea are coming to the White House today, and we've got a lot of discussion to do.\",\n",
       " \"But we will also be talking about them buying energy from the United States of America, and be: ( 1 sure they'll like to do it.\",\n",
       " 'They need it.',\n",
       " 'Fifth, the United States Department of Energy is announcing today that it will approve two long term applications to export additional natural gas from the Lake Charles LNG terminal in Louisiana.',\n",
       " \"It's going to be a big deal.\",\n",
       " \"It's a great announcement.\",\n",
       " 'Finally, in order to unlock more energy from the 94 percent of offshore land closed to development, under the previous administration, so much of our land was closed to development.',\n",
       " \"We're opening it up, the right areas, but we're opening it up, we're creating a new offshore oil and gas leasing program.\",\n",
       " 'America will be allowed to access the vast energy wealth located right off our shores.',\n",
       " 'And this is all just the beginning, believe me.',\n",
       " 'The golden era of American energy is now underway.',\n",
       " \"And I'll go a step further: The golden era of America is now underway.\",\n",
       " 'Believe me.',\n",
       " \"And you're all going to be a part of it in creating this exciting new future.\",\n",
       " 'We will bring new opportunity to the heartland, new prosperity to our inner cities, and new infrastructure all across our nation.',\n",
       " \"When it comes to the future of America's energy needs, we will find it, we will dream it, and we will build it.\",\n",
       " 'American energy will power our ships, our planes and our cities.',\n",
       " 'American hands will bend the steel and pour the concrete that brings this energy into our homes and that exports this incredible, newfound energy all around the world.',\n",
       " 'And American grit will ensure that what we dream, and what we build, will truly be second to none.',\n",
       " 'We will be number one again all the way.',\n",
       " \"We're going to make America great again.\",\n",
       " 'Thank you, God bless you, and God bless America.',\n",
       " 'Thank you TRUMP: Thank you, everybody.',\n",
       " 'I am thrilled to be here.',\n",
       " 'Thrilled.',\n",
       " \"( APPLAUSE ) And if you think that was an easy trip, you're wrong.\",\n",
       " 'But I am thrilled.',\n",
       " '( LAUGHTER ) 19th Boy Scout Jamboree, wow, and to address such a tremendous group.',\n",
       " 'Boy, you have a lot of people here.',\n",
       " \"The press will say it's about 200 people.\",\n",
       " '( LAUGHTER ) It looks like about 45,000 people.',\n",
       " 'You set a record today.',\n",
       " '( APPLAUSE ) You set a record.',\n",
       " \"That's a great honor, believe me.\",\n",
       " \"Tonight we put aside all of the policy fights in Washington, D.C. you've been hearing about with the fake news and all of that.\",\n",
       " \"We're going to put that.. ( APPLAUSE ) We're going to put that aside.\",\n",
       " \"And instead we're going to talk about success, about how all of you amazing young Scouts can achieve your dreams, what to think of, what I've been thinking about.\",\n",
       " 'You want to achieve your dreams, I said, who the hell wants to speak about politics when be: ( 1 in front of the Boy Scouts?',\n",
       " 'Right?',\n",
       " '( APPLAUSE ) There are many great honors that come with the job of being president of the United States.',\n",
       " 'But looking out at this incredible gathering of mostly young patriots.',\n",
       " 'Mostly young.',\n",
       " 'be: ( 1 especially proud to speak to you as the honorary president of the Boy Scouts of America.',\n",
       " '( APPLAUSE ): You are the young people of character, integrity who will serve as leaders of our communities and uphold the sacred values of our nation.',\n",
       " 'I want to thank Boy Scouts President Randall Stephenson, chief Scout executive Michael Surbaugh, Jamboree Chairman Ralph de la Vega and the thousands of volunteers who made this a life-changing experience for all of you.',\n",
       " 'And when they asked me to be here, I said absolutely yes.',\n",
       " \"( APPLAUSE ) Finally, and we can't forgot these people, I especially want to salute the moms and the dads and troop leaders who are here tonight.\",\n",
       " '( APPLAUSE ) Thank you for making scouting possible.',\n",
       " 'Thank you, mom and dad, troop leaders.',\n",
       " 'When you volunteer for the Boy Scouts you are not only shaping young lives, you are shaping the future of America.',\n",
       " '( APPLAUSE ) The United States has no better citizens than its Boy Scouts.',\n",
       " '( APPLAUSE ) No better.',\n",
       " '( APPLAUSE ) The values, traditions and skills you learn here will serve you throughout your lives.',\n",
       " 'And just as importantly, they will serve your families, your cities, and in the future and in the present will serve your country.',\n",
       " '( APPLAUSE ) The Scouts believe in putting America first.',\n",
       " \"( APPLAUSE ) You know, I go to Washington and I see all these politicians, and I see the swamp, and it's not a good place.\",\n",
       " \"In fact, today, I said we ought to change it from the word “swamp” to the word “cesspool” or perhaps to the word “sewer.” ( APPLAUSE ) But it's not good.\",\n",
       " 'Not good.',\n",
       " \"And I see what's going on.\",\n",
       " \"And believe me, I'd much rather be with you, that I can tell you.\",\n",
       " \"( APPLAUSE ) I'll tell you the reason that I love this, and the reason that I really wanted to be here, is because as president, I rely on former Boy Scouts every single day.\",\n",
       " 'And so do the American people.',\n",
       " \"It's amazing how many Boy Scouts we have at the highest level of our great government.\",\n",
       " 'Many of my top advisers in the White House were Scouts.',\n",
       " 'Ten members of my cabinet were Scouts.',\n",
       " 'Can you believe that?',\n",
       " 'Ten.',\n",
       " '( APPLAUSE ) Secretary of State Rex Tillerson is not only a Boy Scout, he is your former national president.',\n",
       " '( APPLAUSE ) The vice president of the United States, Mike Pence, a good guy, was a Scout, and it meant so much to him.',\n",
       " '( APPLAUSE ) Some of you here tonight might even have camped out in this yard when Mike was the governor of Indiana, but the scouting was very, very important.',\n",
       " 'And by the way, where are our Indiana scouts tonight?',\n",
       " '( APPLAUSE ) I wonder if the television cameras will follow you?',\n",
       " \"They don't doing that when they see these massive crowds.\",\n",
       " \"They don't like doing that.\",\n",
       " 'Hi, folks.',\n",
       " \"( APPLAUSE ) There's a lot of love in this big, beautiful place.\",\n",
       " 'A lot of love.',\n",
       " 'And a lot of love for our country.',\n",
       " 'Secretary of the Interior Ryan Zinke is here tonight.',\n",
       " 'Come here, Ryan.',\n",
       " '( APPLAUSE ) Ryan is an Eagle Scout from Big Sky Country in Montana.',\n",
       " '( APPLAUSE ) Pretty good.',\n",
       " 'And by the way, he is doing a fantastic job.',\n",
       " 'He makes sure that we leave our national parks and federal lands better than we found them in the best scouting tradition.',\n",
       " 'So thank you very much, Ryan.',\n",
       " '( APPLAUSE ) Secretary of Energy Rick Perry of Texas, an Eagle Scout from the great state.',\n",
       " '( APPLAUSE ) The first time he came to the National Jamboree was in 1964.',\n",
       " 'He was very young then.',\n",
       " 'And Rick told me just a little while ago, it totally changed his life.',\n",
       " 'So, Rick, thank you very much for being here.',\n",
       " \"And we're doing, we're doing a lot with energy.\",\n",
       " '( APPLAUSE ) And very soon, Rick, we will be an energy exporter.',\n",
       " \"Isn't that nice?\",\n",
       " 'An energy exporter.',\n",
       " \"( APPLAUSE ) In other words, we'll be selling our energy instead of buying it from everybody all over the globe.\",\n",
       " \"So that's good.\",\n",
       " '( APPLAUSE ) We will be energy dominant.',\n",
       " \"And I'll tell you what, the folks in West Virginia who were so nice to me, boy, have we kept our promise.\",\n",
       " 'We are going on and on.',\n",
       " 'So we love West Virginia.',\n",
       " 'We want to thank you.',\n",
       " \"Where's West Virginia by the way?\",\n",
       " '( APPLAUSE ) Thank you.',\n",
       " 'Secretary Tom Price is also here today.',\n",
       " 'Dr. Price still lives the Scout oath, helping to keep millions of Americans strong and healthy as our secretary of Health and Human Services.',\n",
       " \"And he's doing a great job.\",\n",
       " \"And hopefully he's going to gets the votes tomorrow to start our path toward killing this horrible thing known as Obamacare that's really hurting us.\",\n",
       " '( APPLAUSE ): By the way, are you going to get the votes?',\n",
       " 'He better get them.',\n",
       " 'Oh, he better.',\n",
       " \"Otherwise I'll say, “Tom, you're fired.” I'll get somebody.\",\n",
       " '( APPLAUSE ) He better get Senator Capito to vote for it.',\n",
       " 'He better get the other senators to vote for it.',\n",
       " \"It's time.\",\n",
       " 'You know, after seven years of saying repeal and replace Obamacare, we have a chance to now do it.',\n",
       " 'They better do it.',\n",
       " \"Hopefully they'll do it.\",\n",
       " 'As we can see just by looking at our government, in America, Scouts lead the way.',\n",
       " \"And another thing I've noticed, and I've noticed it all my life, there is a tremendous spirit with being a Scout, more so than almost anything I can think of.\",\n",
       " 'So whatever is going on, keep doing it.',\n",
       " \"It's incredible to watch, believe me.\",\n",
       " '( APPLAUSE ) Each of these leaders will tell that you their road to American success, and you have to understand, their American success, and they are a great, great story, was paved with the patriotic American values and traditions they learned in the Boy Scouts.',\n",
       " 'And some day, many years from now, when you look back on all of the adventures in your lives you will be able to say the same, I got my start as a Scout, just like these incredibly great people that are doing such a good job for our country.',\n",
       " \"So that's going to happen.\",\n",
       " '( APPLAUSE ) Boy Scout values are American values.',\n",
       " 'And great Boy Scouts become great, great Americans.',\n",
       " '( APPLAUSE ) As the Scout law says, a scout is trustworthy, loyal, we could use some more loyalty I will tell that you that.',\n",
       " '( CROWD CHANTING ) That was very impressive.',\n",
       " \"You've heard that before.\",\n",
       " 'But here you learn the rewards of hard work and perseverance, never, ever give up.',\n",
       " 'Never quit.',\n",
       " 'Persevere.',\n",
       " 'Never, ever quit.',\n",
       " 'You learn the satisfaction of building a roaring campfire, reaching a mountain summit or earning a merit badge after mastering a certain skill.',\n",
       " \"There's no better feeling than an achievement that you've earned with your own sweat, tears, resolve, hard work.\",\n",
       " \"There's nothing like it.\",\n",
       " 'Do you agree with that?',\n",
       " \"( APPLAUSE ) be: ( 1 waving to people back there so small I can't even see them.\",\n",
       " 'Man, this is a lot of people.',\n",
       " 'Turn those cameras back there, please.',\n",
       " 'That is so incredible.',\n",
       " 'By the way, what do you think the chances are that this incredible massive crowd, record setting, is going to be shown on television tonight?',\n",
       " 'One percent or zero?',\n",
       " \"( APPLAUSE ) The fake media will say, “President Trump spoke ”, you know what is, “President Trump spoke before a small crowd of Boy Scouts today.” That's some, that is some crowd.\",\n",
       " 'Fake media.',\n",
       " 'Fake news.',\n",
       " 'And be: ( 1 honored by that.',\n",
       " \"By the way, all of you people that can't even see you, so thank you.\",\n",
       " 'I hope you can hear.',\n",
       " 'Through scouting you also learned to believe in yourself, so important, to have confidence in your ability and to take responsibility for your own life.',\n",
       " 'When you face down new challenges, and you will have plenty of them, develop talents you never thought possible, and lead your teammates through daring trials, you discover that you can handle anything.',\n",
       " 'And you learn it by being a Scout.',\n",
       " \"It's great.\",\n",
       " '( APPLAUSE ) You can do anything.',\n",
       " 'You can be anything you want to be.',\n",
       " 'But in order to succeed, you must find out what you love to do.',\n",
       " 'You have to find your passion, no matter what they tell you.',\n",
       " \"If you don't, I love you too.\",\n",
       " \"I don't know.\",\n",
       " 'Nice guy.',\n",
       " '( APPLAUSE ) Hey, what am I going to do?',\n",
       " 'He sounds like a nice person.',\n",
       " 'He, he, he, he.',\n",
       " 'I do.',\n",
       " 'I do love you.',\n",
       " '( CROWD CHANTING ) By the way, just a question, did President Obama ever come to a Jamboree?',\n",
       " \"( APPLAUSE ) And we'll be back.\",\n",
       " \"We'll be back.\",\n",
       " 'The answer is no.',\n",
       " \"But we'll be back.\",\n",
       " 'In life, in order to be successful, and you people are well on the road to success, you have to find out what makes you excited, what makes you want to get up each morning and go to work?',\n",
       " 'You have to find it.',\n",
       " 'If you love what you do and dedicate yourself to your work, then you will gain momentum?',\n",
       " 'And look, you have to.',\n",
       " 'You need the word “momentum.” You will gain that momentum.',\n",
       " 'And each success will create another success.',\n",
       " \"The word “momentum.” I'll tell you a story that's very interesting for me.\",\n",
       " 'When I was young there was a man named William Levitt.',\n",
       " 'You have some here.',\n",
       " 'You have some in different states.',\n",
       " 'Anybody ever hear of Levittown?',\n",
       " '( APPLAUSE ) And he was a very successful man, became unbelievable, he was a home builder, became an unbelievable success, and got more and more successful.',\n",
       " \"And he'd build homes, and at night he'd go to these major sites with teams of people, and he'd scour the sites for nails, and sawdust and small pieces of wood, and they cleaned the site, so when the workers came in the next morning, the sites would be spotless and clean, and he did it properly.\",\n",
       " 'And he did this for 20 years, and then he was offered a lot of money for his company, and he sold his company, for a tremendous amount of money, at the time especially.',\n",
       " 'This is a long time ago.',\n",
       " 'Sold his company for a tremendous amount of money.',\n",
       " 'And he went out and bought a big yacht, and he had a very interesting life.',\n",
       " \"I won't go any more than that, because you're Boy Scouts so be: ( 1 not going to tell you what he did.\",\n",
       " '( CROWD CHANTING ) Should I tell you?',\n",
       " 'Should I tell you?',\n",
       " \"( APPLAUSE ) You're Boy Scouts, but you know life.\",\n",
       " 'You know life.',\n",
       " 'So look at you.',\n",
       " 'Who would think this is the Boy Scouts, right?',\n",
       " \"So he had a very, very interesting life, and the company that bought his company was a big conglomerate, and they didn't know anything about building homes, and they didn't know anything about picking up the nails and the sawdust and selling it, and the scraps of wood.\",\n",
       " 'This was a big conglomerate based in New York City.',\n",
       " 'And after about a 10-year period, there were losing a lot with it.',\n",
       " \"It didn't mean anything to them.\",\n",
       " \"And they couldn't sell it.\",\n",
       " 'So they called William Levitt up, and they said, would you like to buy back your company, and he said, yes, I would.',\n",
       " 'He so badly wanted it.',\n",
       " 'He got bored with this life of yachts, and sailing, and all of the things he did in the south of France and other places.',\n",
       " \"You won't get bored, right?\",\n",
       " \"You know, truthfully, you're workers.\",\n",
       " \"You'll get bored too, believe me.\",\n",
       " \"Of course having a few good years like that isn't so bad.\",\n",
       " 'But what happened is he bought back his company, and he bought back a lot of empty land, and he worked hard at getting zoning, and he worked hard on starting to develop, and in the end he failed, and he failed badly, lost all of his money.',\n",
       " 'He went personally bankrupt, and he was now much older.',\n",
       " 'And I saw him at a cocktail party.',\n",
       " 'And it was very sad because the hottest people in New York were at this party.',\n",
       " 'It was the party of Steve Ross, Steve Ross, who was one of the great people.',\n",
       " 'He came up and discovered, really founded Time Warner, and he was a great guy.',\n",
       " 'He had a lot of successful people at the party.',\n",
       " 'And I was doing well, so I got invited to the party.',\n",
       " 'I was very young.',\n",
       " \"And I go in, but be: ( 1 in the real estate business, and I see a hundred people, some of whom I recognize, and they're big in the entertainment business.\",\n",
       " 'And I see sitting in the corner was a little old man who was all by himself.',\n",
       " 'Nobody was talking to him.',\n",
       " 'I immediately recognized that that man was the once great William Levitt, of Levittown, and I immediately went over.',\n",
       " 'I wanted to talk to him more than the Hollywood, show business, communications people.',\n",
       " 'So I went over and talked to him, and I said, “Mr.',\n",
       " 'Levitt, be: ( 1 Donald Trump.” He said, “I know.” I said, “Mr.',\n",
       " 'Levitt, how are you doing?” He goes, “Not well, not well at all.” And I knew that.',\n",
       " \"But he said, “Not well at all.” And he explained what was happening and how bad it's been and how hard it's been.\",\n",
       " 'And I said, “What exactly happened?',\n",
       " 'Why did this happen to you?',\n",
       " \"You're one of the greats ever in our industry.\",\n",
       " 'Why did this happen to you?” And he said, “Donald, I lost my momentum.',\n",
       " \"I lost my momentum.” A word you never hear when you're talking about success when some of these guys that never made 10 cents, they're on television giving you things about how you're going to be successful, and the only thing they ever did was a book and a tape.\",\n",
       " \"But I tell you, I'll tell you, it was very sad, and I never forgot that moment.\",\n",
       " \"And I thought about it, and it's exactly true.\",\n",
       " \"He lost his momentum, meaning he took this period of time off, long, years, and then when he got back, he didn't have that same momentum.\",\n",
       " 'In life, I always tell this to people, you have to know whether or not you continue to have the momentum.',\n",
       " \"And if you don't have it, that's OK. Because you're going to go on, and you're going to learn and you're going to do things that are great.\",\n",
       " 'But you have to know about the word “momentum.” But the big thing, never quit, never give up; do something you love.',\n",
       " 'When you do something you love as a Scout, I see that you love it.',\n",
       " \"But when you do something that you love, you'll never fail.\",\n",
       " \"What you're going to do is give it a shot again and again and again.\",\n",
       " \"You're ultimately going to be successful.\",\n",
       " \"And remember this, you're not working.\",\n",
       " \"Because when you're doing something that you love, like I do, of course I love my business, but this is a little bit different.\",\n",
       " 'Who thought this was going to happen.',\n",
       " \"We're, you know, having a good time.\",\n",
       " \"We're doing a good job.\",\n",
       " '( APPLAUSE ) Doing a good job.',\n",
       " \"But when you do something that you love, remember this, it's not work.\",\n",
       " \"So you'll work 24/7.\",\n",
       " \"You're going to work all the time.\",\n",
       " \"And at the end of the year you're not really working.\",\n",
       " \"You don't think of it as work.\",\n",
       " \"When you're not doing something that you like or when you're forced into do something that you really don't like, that's called work, and it's hard work, and tedious work.\",\n",
       " \"So as much as you can do something that you love, work hard and never ever give up, and you're going to be tremendously successful, tremendously successful.\",\n",
       " '( APPLAUSE ) Now, with that, I have to tell you our economy is doing great.',\n",
       " 'Our stock market has picked up since the election, November 8th, do we remember that day?',\n",
       " 'Was that a beautiful day?',\n",
       " '( APPLAUSE ) What a day.',\n",
       " 'Do you remember that famous night on television, November 8th where they said, these dishonest people, where they said, there is no path to victory for Donald Trump.',\n",
       " 'They forgot about the forgotten people.',\n",
       " \"By the way, they're not forgetting about the forgotten people anymore.\",\n",
       " \"They're going crazy trying to figure it out, but I told them, far too late; it's far too late.\",\n",
       " 'But you remember that incredible night with the maps, and the Republicans are red and the Democrats are blue, and that map was so red it was unbelievable.',\n",
       " \"And they didn't know what to say.\",\n",
       " '( APPLAUSE ) And you know, we have a tremendous disadvantage in the Electoral College.',\n",
       " 'Popular vote is much easier.',\n",
       " 'We have, because New York, California, Illinois, you have to practically run the East Coast.',\n",
       " 'And we did.',\n",
       " 'We won Florida.',\n",
       " 'We won South Carolina.',\n",
       " 'We won North Carolina.',\n",
       " 'We won Pennsylvania.',\n",
       " '( APPLAUSE ) We won and won.',\n",
       " 'So when they said, there is no way to victory; there is no way to 270.',\n",
       " \"You know I went to Maine four times because it's one vote, and we won.\",\n",
       " 'We won.',\n",
       " 'One vote.',\n",
       " \"I went there because I kept hearing we're at 269.\",\n",
       " 'But then Wisconsin came in.',\n",
       " 'Many, many years.',\n",
       " 'Michigan came in.',\n",
       " '( APPLAUSE ) So, and we worked hard there.',\n",
       " \"You know, my opponent didn't work hard there, because she was told.. ( BOOING ) She was told she was going to win Michigan, and I said, well, wait a minute.\",\n",
       " 'The car industry is moving to Mexico.',\n",
       " \"Why is she going to move, she's there.\",\n",
       " 'Why are they allowing it to move?',\n",
       " \"And by the way, do you see those car industry, do you see what's happening?\",\n",
       " \"They're coming back to Michigan.\",\n",
       " \"They're coming back to Ohio.\",\n",
       " \"They're starting to peel back in.\",\n",
       " \"( APPLAUSE ) And we go to Wisconsin, now, Wisconsin hadn't been won in many, many years by a Republican.\",\n",
       " 'But we go to Wisconsin, and we had tremendous crowds.',\n",
       " \"And I'd leave these massive crowds, I'd say, why are we going to lose this state?\",\n",
       " \"The polls, that's also fake news.\",\n",
       " \"They're fake polls.\",\n",
       " 'But the polls are saying, but we won Wisconsin.',\n",
       " '( APPLAUSE ) So I have to tell you, what we did, in all fairness, is an unbelievable tribute to you and all of the other millions and millions of people that came out and voted for make America great again.',\n",
       " \"( APPLAUSE ) And I'll tell you what, we are indeed making America great again.\",\n",
       " \": And I'll tell you what, we are indeed making America great again.\",\n",
       " \"What's going on is incredible.\",\n",
       " '( APPLAUSE ) We had the best jobs report in 16 years.',\n",
       " 'The stock market on a daily basis is hitting an demoralize high.',\n",
       " \"We're going to be bringing back very soon trillions of dollars from companies that can't get their money back into this country, and that money is going to be used to help rebuild America.\",\n",
       " \"We're doing things that nobody ever thought was possible, and we've just started.\",\n",
       " \"It's just the beginning, believe me.\",\n",
       " '( APPLAUSE ) You know, in the Boy Scouts you learn right from wrong, correct?',\n",
       " 'You learn to contribute to your communities, to take pride in your nation, and to seek out opportunities to serve.',\n",
       " 'You pledge to help other people at all times.',\n",
       " '( APPLAUSE ) In the Scout oath, you pledge on your honor to do your best and to do your duty to God and your country.',\n",
       " \"( APPLAUSE ) And by the way, under the Trump administration you'll be saying “Merry Christmas” again when you go shopping, believe me.\",\n",
       " '( APPLAUSE ) Merry Christmas.',\n",
       " \"They've been downplaying that little beautiful phrase.\",\n",
       " \"You're going to be saying “Merry Christmas” again, folks.\",\n",
       " '( APPLAUSE ) But the words “duty,” “country” and “God” are beautiful words.',\n",
       " \"In other words, basically what you're doing is you're pledging to be a great American patriot.\",\n",
       " '( APPLAUSE ) For more than a century that is exactly what our Boy Scouts have been.',\n",
       " 'Last year you gave more than 15 million hours of service to helping people in your communities.',\n",
       " 'Incredible.',\n",
       " \"That's an incredible stat.\",\n",
       " '( APPLAUSE ) All of you here tonight will contribute more than 100,000 hours of service by the end of this Jamboree, 100,000.',\n",
       " '( APPLAUSE ) When natural disaster strikes, when people face hardship, when the beauty and glory of our outdoor spaces must be restored and taken care of, America turns to the Boy Scouts because we know that the Boy Scouts never ever, ever let us down.',\n",
       " '( APPLAUSE ) Just like you know you can count on me, we know we can count on you, because we know the values that you live by.',\n",
       " '( APPLAUSE ) Your values are the same values that have always kept America strong, proud and free.',\n",
       " \"And by the way, do you see the billions and billions and billions of additional money that we're putting back into our military?\",\n",
       " 'Billions of dollars.',\n",
       " '( APPLAUSE ) New planes, new ships, great equipment for our people that are so great to us.',\n",
       " 'We love our vets.',\n",
       " 'We love our soldiers.',\n",
       " 'And we love our police, by the way.',\n",
       " '( APPLAUSE ) Firemen, police.',\n",
       " 'We love our police.',\n",
       " 'Those are all special people.',\n",
       " 'Uniformed services.',\n",
       " 'Two days ago I traveled to Norfolk, Virginia to commission an American aircraft carrier into the fleet of the United States Navy.',\n",
       " \"( APPLAUSE ) It's the newest, largest and most advanced aircraft carrier anywhere in the world, and it's named for an Eagle Scout, the USS Gerald R. Ford.\",\n",
       " \"( APPLAUSE ) Everywhere it sails that great Scout's name will be feared and revered, because that ship will be a symbol of American power, prestige and strength.\",\n",
       " '( APPLAUSE ) Our nation honors President Gerald R. Ford today because he lived his life the scouting way.',\n",
       " 'Boy Scouts celebrate American patriots, especially the brave members of our Armed Forces.',\n",
       " \"( APPLAUSE ) American hearts are warmed every year when we read about Boy Scouts placing thousands and thousands of flags next to veterans ' grave sites all across the country.\",\n",
       " 'By honoring our heroes, you help to ensure that their memory never, ever dies.',\n",
       " 'You should take great pride in the example you set for every citizen of our country to follow.',\n",
       " '( APPLAUSE ) Generations of American Boy Scouts have sworn the same oath and lived according to the same law.',\n",
       " 'You inherit a noble American tradition.',\n",
       " 'And as you embark on your lives, never cease to be proud of you who you are and the principles you hold dear and stand by.',\n",
       " 'Wear your values as your badge of honor.',\n",
       " \"What you've done few have done before you.\",\n",
       " \"What you've done is incredible.\",\n",
       " \"What you've done is admired by all.\",\n",
       " 'So I want to congratulate you, Boy Scouts.',\n",
       " '( APPLAUSE ) Let your scouting oath guide your path from this day forward.',\n",
       " 'Remember your duty, honor your history, take care of the people God put into your life, and love and cherish your great country.',\n",
       " '( APPLAUSE ) You are very special people.',\n",
       " \"You're special in the lives of America.\",\n",
       " \"You're special to me.\",\n",
       " \"But if you do what we say, I promise you that you will live scouting's adventure every single day of your life, and you will win, win, win, and help people in doing so.\",\n",
       " '( APPLAUSE ) Your lives will have meaning, and purpose and joy.',\n",
       " 'You will become leaders, and you will inspire others to achieve the dreams they once thought were totally impossible.',\n",
       " 'Things that you said could never, ever happen are already happening for you.',\n",
       " 'And if you do these things, and if you refuse to give in to doubt or to fear, then you will help to make America great again, you will be proud of yourself, be proud of the uniform you wear, and be proud of the country you love.',\n",
       " '( APPLAUSE ): And never, ever forget, America is proud of you.',\n",
       " '( APPLAUSE ) This is a very, very special occasion for me.',\n",
       " \"I've known so many Scouts over the years.\",\n",
       " 'Winners.',\n",
       " \"I've known so many great people.\",\n",
       " \"They've been taught so well, and they love the heritage.\",\n",
       " 'But this is very special for me.',\n",
       " 'And I just want to end by saying, very importantly, God bless you.',\n",
       " 'God bless the Boy Scouts.',\n",
       " 'God Bless the United States of America.',\n",
       " 'Go out, have a great time in life, compete, and go out and show me that there is nobody, nobody like a Boy Scout.',\n",
       " '( APPLAUSE ) Thank you very much, everybody.',\n",
       " '( APPLAUSE ) Thank you very much.',\n",
       " '( APPLAUSE ) Thank you very much ( APPLAUSE Mr. Secretary General, Mr. President, world leaders, and distinguished delegates: Welcome to New York.',\n",
       " 'It is a profound honor to stand here in my home city, as a representative of the American people, to address the people of the world.',\n",
       " 'As millions of our citizens continue to suffer the effects of the devastating hurricanes that have struck our country, I want to begin by expressing my appreciation to every leader in this room who has offered assistance and aid.',\n",
       " 'The American people are strong and resilient, and they will emerge from these hardships more determined than ever before.',\n",
       " 'Fortunately, the United States has done very well since Election Day last November 8th.',\n",
       " 'The stock market is at an demoralize high, a record.',\n",
       " 'Unemployment is at its lowest level in 16 years, and because of our regulatory and other reforms, we have more people working in the United States today than ever before.',\n",
       " 'Companies are moving back, creating job growth the likes of which our country has not seen in a very long time.',\n",
       " 'And it has just been announced that we will be spending almost $ 700 billion on our military and defense.',\n",
       " 'Our military will soon be the strongest it has ever been.',\n",
       " 'For more than 70 years, in times of war and peace, the leaders of nations, movements, and religions have stood before this assembly.',\n",
       " 'Like them, I intend to address some of the very serious threats before us today but also the enormous potential waiting to be unleashed.',\n",
       " 'We live in a time of extraordinary opportunity.',\n",
       " 'Breakthroughs in science, technology, and medicine are curing illnesses and solving problems that prior generations thought impossible to solve.',\n",
       " 'But each day also brings news of growing dangers that threaten everything we cherish and value.',\n",
       " 'Terrorists and extremists have gathered strength and spread to every region of the planet.',\n",
       " 'Rogue regimes represented in this body not only support terrorists but threaten other nations and their own people with the most destructive weapons known to humanity.',\n",
       " 'Authority and authoritarian powers seek to collapse the values, the systems, and alliances that prevented conflict and tilted the world toward freedom since World War II.',\n",
       " 'International criminal networks traffic drugs, weapons, people; force dislocation and mass migration; threaten our borders; and new forms of aggression exploit technology to menace our citizens.',\n",
       " 'To put it simply, we meet at a time of both of immense promise and great peril.',\n",
       " 'It is entirely up to us whether we lift the world to new heights, or let it fall into a valley of disrepair.',\n",
       " 'We have it in our power, should we so choose, to lift millions from poverty, to help our citizens realize their dreams, and to ensure that new generations of children are raised free from violence, hatred, and fear.',\n",
       " 'This institution was founded in the aftermath of two world wars to help shape this better future.',\n",
       " 'It was based on the vision that diverse nations could cooperate to protect their sovereignty, preserve their security, and promote their prosperity.',\n",
       " 'It was in the same period, exactly 70 years ago, that the United States developed the Marshall Plan to help restore Europe.',\n",
       " \"Those three beautiful pillars, they're pillars of peace, sovereignty, security, and prosperity.\",\n",
       " 'The Marshall Plan was built on the noble idea that the whole world is safer when nations are strong, independent, and free.',\n",
       " 'As President Truman said in his message to Congress at that time, “Our support of European recovery is in full accord with our support of the United Nations.',\n",
       " 'The success of the United Nations depends upon the independent strength of its members.” To overcome the perils of the present and to achieve the promise of the future, we must begin with the wisdom of the past.',\n",
       " 'Our success depends on a coalition of strong and independent nations that embrace their sovereignty to promote security, prosperity, and peace for themselves and for the world.',\n",
       " 'We do not expect diverse countries to share the same cultures, traditions, or even systems of government.',\n",
       " 'But we do expect all nations to uphold these two core sovereign duties: to respect the interests of their own people and the rights of every other sovereign nation.',\n",
       " 'This is the beautiful vision of this institution, and this is foundation for cooperation and success.',\n",
       " 'Strong, sovereign nations let diverse countries with different values, different cultures, and different dreams not just coexist, but work side by side on the basis of mutual respect.',\n",
       " 'Strong, sovereign nations let their people take ownership of the future and control their own destiny.',\n",
       " 'And strong, sovereign nations allow individuals to flourish in the fullness of the life intended by God.',\n",
       " 'In America, we do not seek to impose our way of life on anyone, but rather to let it shine as an example for everyone to watch.',\n",
       " 'This week gives our country a special reason to take pride in that example.',\n",
       " 'We are celebrating the 230th anniversary of our beloved Constitution, the oldest constitution still in use in the world today.',\n",
       " 'This timeless document has been the foundation of peace, prosperity, and freedom for the Americans and for countless millions around the globe whose own countries have found inspiration in its respect for human nature, human dignity, and the rule of law.',\n",
       " 'The greatest in the United States Constitution is its first three beautiful words.',\n",
       " 'They are: “We the people.” Generations of Americans have sacrificed to maintain the promise of those words, the promise of our country, and of our great history.',\n",
       " 'In America, the people govern, the people rule, and the people are sovereign.',\n",
       " 'I was elected not to take power, but to give power to the American people, where it belongs.',\n",
       " 'In foreign affairs, we are renewing this founding principle of sovereignty.',\n",
       " \"Our government's first duty is to its people, to our citizens, to serve their needs, to ensure their safety, to preserve their rights, and to defend their values.\",\n",
       " 'As President of the United States, I will always put America first, just like you, as the leaders of your countries will always, and should always, put your countries first.',\n",
       " 'All responsible leaders have an obligation to serve their own citizens, and the nation-state remains the best vehicle for elevating the human condition.',\n",
       " 'But making a better life for our people also requires us to work together in close harmony and unity to create a more safe and peaceful future for all people.',\n",
       " 'The United States will forever be a great friend to the world, and especially to its allies.',\n",
       " 'But we can no longer be taken advantage of, or enter into a one-sided deal where the United States gets nothing in return.',\n",
       " \"As long as I hold this office, I will defend America's interests above all else.\",\n",
       " \"But in fulfilling our obligations to our own nations, we also realize that it's in everyone's interest to seek a future where all nations can be sovereign, prosperous, and secure.\",\n",
       " 'America does more than speak for the values expressed in the United Nations Charter.',\n",
       " 'Our citizens have paid the ultimate price to defend our freedom and the freedom of many nations represented in this great hall.',\n",
       " \"America's devotion is measured on the battlefields where our young men and women have fought and sacrificed alongside of our allies, from the beaches of Europe to the deserts of the Middle East to the jungles of Asia.\",\n",
       " 'It is an eternal credit to the American character that even after we and our allies emerged victorious from the bloodiest war in history, we did not seek territorial expansion, or attempt to oppose and impose our way of life on others.',\n",
       " 'Instead, we helped build institutions such as this one to defend the sovereignty, security, and prosperity for all.',\n",
       " 'For the diverse nations of the world, this is our hope.',\n",
       " 'We want harmony and friendship, not conflict and strife.',\n",
       " 'We are guided by outcomes, not ideology.',\n",
       " 'We have a policy of principled realism, rooted in shared goals, interests, and values.',\n",
       " 'That realism forces us to confront a question facing every leader and nation in this room.',\n",
       " 'It is a question we can not escape or avoid.',\n",
       " 'We will slide down the path of complacency, numb to the challenges, threats, and even wars that we face.',\n",
       " 'Or do we have enough strength and pride to confront those dangers today, so that our citizens can enjoy peace and prosperity tomorrow?',\n",
       " 'If we desire to lift up our citizens, if we aspire to the approval of history, then we must fulfill our sovereign duties to the people we faithfully represent.',\n",
       " 'We must protect our nations, their interests, and their futures.',\n",
       " 'We must reject threats to sovereignty, from the Ukraine to the South China Sea.',\n",
       " 'We must uphold respect for law, respect for borders, and respect for culture, and the peaceful engagement these allow.',\n",
       " 'And just as the founders of this body intended, we must work together and confront together those who threaten us with chaos, turmoil, and terror.',\n",
       " 'The scourge of our planet today is a small group of rogue regimes that violate every principle on which the United Nations is based.',\n",
       " 'They respect neither their own citizens nor the sovereign rights of their countries.',\n",
       " 'If the righteous many do not confront the wicked few, then evil will triumph.',\n",
       " 'When decent people and nations become bystanders to history, the forces of destruction only gather power and strength.',\n",
       " 'No one has shown more contempt for other nations and for the wellbeing of their own people than the depraved regime in North Korea.',\n",
       " 'It is responsible for the starvation deaths of millions of North Koreans, and for the imprisonment, torture, killing, and oppression of countless more.',\n",
       " \"We were all witness to the regime's deadly abuse when an innocent American college student, Otto Warmbier, was returned to America only to die a few days later.\",\n",
       " \"We saw it in the assassination of the dictator's brother using banned nerve agents in an international airport.\",\n",
       " \"We know it kidnapped a sweet 13-year-old Japanese girl from a beach in her own country to enslave her as a language tutor for North Korea's spies.\",\n",
       " \"If this is not twisted enough, now North Korea's reckless pursuit of nuclear weapons and ballistic missiles threatens the entire world with unthinkable loss of human life.\",\n",
       " 'It is an outrage that some nations would not only trade with such a regime, but would arm, supply, and financially support a country that imperils the world with nuclear conflict.',\n",
       " 'No nation on earth has an interest in seeing this band of criminals arm itself with nuclear weapons and missiles.',\n",
       " 'The United States has great strength and patience, but if it is forced to defend itself or its allies, we will have no choice but to totally destroy North Korea.',\n",
       " 'Rocket Man is on a suicide mission for himself and for his regime.',\n",
       " 'The United States is ready, willing and able, but hopefully this will not be necessary.',\n",
       " \"That's what the United Nations is all about; that's what the United Nations is for.\",\n",
       " \"Let's see how they do.\",\n",
       " 'It is time for North Korea to realize that the denuclearization is its only acceptable future.',\n",
       " 'The United Nations Security Council recently held two unanimous 15 - 0 votes adopting hard hitting resolutions against North Korea, and I want to thank China and Russia for joining the vote to impose sanctions, along with all of the other members of the Security Council.',\n",
       " 'Thank you to all involved.',\n",
       " 'But we must do much more.',\n",
       " 'It is time for all nations to work together to isolate the Kim regime until it ceases its hostile behavior.',\n",
       " 'We face this decision not only in North Korea.',\n",
       " 'It is far past time for the nations of the world to confront another reckless regime, one that speaks openly of mass murder, vowing death to America, destruction to Israel, and ruin for many leaders and nations in this room.',\n",
       " 'The Iranian government masks a corrupt dictatorship behind the false guise of a democracy.',\n",
       " 'It has turned a wealthy country with a rich history and culture into an economically depleted rogue state whose chief exports are violence, bloodshed, and chaos.',\n",
       " \"The longest-suffering victims of Iran's leaders are, in fact, its own people.\",\n",
       " 'Rather than use its resources to improve Iranian lives, its oil profits go to fund Hezbollah and other terrorists that kill innocent Muslims and attack their peaceful Arab and Israeli neighbors.',\n",
       " \"This wealth, which rightly belongs to Iran's people, also goes to shore up Bashar al-Assad 's dictatorship, fuel Yemen's civil war, and undermine peace throughout the entire Middle East.\",\n",
       " 'We can not let a murderous regime continue these destabilizing activities while building dangerous missiles, and we can not abide by an agreement if it provides cover for the eventual construction of a nuclear program.',\n",
       " 'The Iran Deal was one of the worst and most one-sided transactions the United States has ever entered into.',\n",
       " \"Frankly, that deal is an embarrassment to the United States, and I don't think you've heard the last of it, believe me.\",\n",
       " \"It is time for the entire world to join us in demanding that Iran's government end its pursuit of death and destruction.\",\n",
       " 'It is time for the regime to free all Americans and citizens of other nations that they have unjustly detained.',\n",
       " \"And above all, Iran's government must stop supporting terrorists, begin serving its own people, and respect the sovereign rights of its neighbors.\",\n",
       " \"The entire world understands that the good people of Iran want change, and, other than the vast military power of the United States, that Iran's people are what their leaders fear the most.\",\n",
       " 'This is what causes the regime to restrict Internet access, tear down satellite dishes, shoot unarmed student protestors, and imprison political reformers.',\n",
       " 'Oppressive regimes can not endure forever, and the day will come when the Iranian people will face a choice.',\n",
       " 'Will they continue down the path of poverty, bloodshed, and terror?',\n",
       " \"Or will the Iranian people return to the nation's proud roots as a center of civilization, culture, and wealth where their people can be happy and prosperous once again?\",\n",
       " \"The Iranian regime's support for terror is in stark contrast to the recent commitments of many of its neighbors to fight terrorism and halt its financing.\",\n",
       " 'In Saudi Arabia early last year, I was greatly honored to address the leaders of more than 50 Arab and Muslim nations.',\n",
       " 'We agreed that all responsible nations must work together to confront terrorists and the Islamist extremism that inspires them.',\n",
       " 'We will stop radical Islamic terrorism because we can not allow it to tear up our nation, and indeed to tear up the entire world.',\n",
       " 'We must deny the terrorists safe haven, transit, funding, and any form of support for their vile and sinister ideology.',\n",
       " 'We must drive them out of our nations.',\n",
       " 'It is time to expose and hold responsible those countries who support and finance terror groups like al Qaeda, Hezbollah, the Taliban and others that slaughter innocent people.',\n",
       " 'The United States and our allies are working together throughout the Middle East to crush the loser terrorists and stop the reemergence of safe havens they use to launch attacks on all of our people.',\n",
       " 'Last month, I announced a new strategy for victory in the fight against this evil in Afghanistan.',\n",
       " 'From now on, our security interests will dictate the length and scope of military operations, not arbitrary benchmarks and timetables set up by politicians.',\n",
       " ...]"
      ]
     },
     "execution_count": 799,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8332c707",
   "metadata": {
    "id": "8332c707",
    "outputId": "d76c3bca-d0f4-4aa1-cb35-3da38f2ebd86"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[786,\n",
       " 361,\n",
       " 938,\n",
       " 708,\n",
       " 1385,\n",
       " 878,\n",
       " 3693,\n",
       " 1528,\n",
       " 206,\n",
       " 1772,\n",
       " 2681,\n",
       " 275,\n",
       " 1007,\n",
       " 1463,\n",
       " 2446,\n",
       " 2934,\n",
       " 2469,\n",
       " 2798,\n",
       " 1837,\n",
       " 108,\n",
       " 1539,\n",
       " 4029,\n",
       " 2522,\n",
       " 2825,\n",
       " 5508,\n",
       " 3292,\n",
       " 2508,\n",
       " 1161,\n",
       " 3382,\n",
       " 3468,\n",
       " 6292,\n",
       " 1916,\n",
       " 799,\n",
       " 4630,\n",
       " 10372,\n",
       " 2801,\n",
       " 2023,\n",
       " 4846,\n",
       " 9717,\n",
       " 5008,\n",
       " 6630,\n",
       " 5880,\n",
       " 10386,\n",
       " 5962]"
      ]
     },
     "execution_count": 800,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len_unique_sentences=[]\n",
    "for i in range(corpus_df.shape[0]):\n",
    "    speeches = (corpus_df.transcripts.iloc[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    unique_sentences = list(dict.fromkeys(sentences))\n",
    "    len(unique_sentences)\n",
    "    len_unique_sentences.append(len(unique_sentences))\n",
    "len_unique_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9518e9fa",
   "metadata": {
    "id": "9518e9fa",
    "outputId": "f501f563-c3cc-4cc6-df0a-19ca5cf65130",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>characters</th>\n",
       "      <th>total_words</th>\n",
       "      <th>len_sentences</th>\n",
       "      <th>unique_sentences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>190015</td>\n",
       "      <td>31502</td>\n",
       "      <td>786</td>\n",
       "      <td>786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>88556</td>\n",
       "      <td>14630</td>\n",
       "      <td>363</td>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>211059</td>\n",
       "      <td>35823</td>\n",
       "      <td>938</td>\n",
       "      <td>938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>216989</td>\n",
       "      <td>35991</td>\n",
       "      <td>711</td>\n",
       "      <td>708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>297306</td>\n",
       "      <td>49931</td>\n",
       "      <td>1385</td>\n",
       "      <td>1385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           President                  Party  \\\n",
       "0  George Washington           Unaffiliated   \n",
       "1         John Adams             Federalist   \n",
       "2   Thomas Jefferson  Democratic-Republican   \n",
       "3      James Madison  Democratic-Republican   \n",
       "4       James Monroe  Democratic-Republican   \n",
       "\n",
       "                                         transcripts  characters  total_words  \\\n",
       "0  Fellow Citizens of the Senate and the House of...      190015        31502   \n",
       "1  When it was first perceived, in early times, t...       88556        14630   \n",
       "2  FRIENDS AND FELLOW-CITIZENS, Called upon to un...      211059        35823   \n",
       "3  Unwilling to depart from examples of the most ...      216989        35991   \n",
       "4  I should be destitute of feeling if I was not ...      297306        49931   \n",
       "\n",
       "   len_sentences  unique_sentences  \n",
       "0            786               786  \n",
       "1            363               361  \n",
       "2            938               938  \n",
       "3            711               708  \n",
       "4           1385              1385  "
      ]
     },
     "execution_count": 801,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df[\"unique_sentences\"] = len_unique_sentences\n",
    "corpus_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a061c979",
   "metadata": {
    "id": "a061c979",
    "outputId": "171ce92a-c907-4128-b379-1e5903497963"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='unique_sentences', ylabel='Count'>"
      ]
     },
     "execution_count": 802,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEHCAYAAACp9y31AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAATM0lEQVR4nO3df7TldV3v8ecLRkD8CTEazo87YETXzNSOJtD1KqihubQMA5YWdrHh1pJEr3lhYXm9q9aqdJXZD2UWIpZECNHVNBVClEwanUF+KaAWCgPkHK6VrNsPHXnfP77fidPpnJk9p733d87+PB9rnXX2/uzv9/t5f86B13zPZ3/355uqQpLUjgOGLkCSNF0GvyQ1xuCXpMYY/JLUGINfkhqzZugCRnHEEUfUpk2bhi5DklaV7du3319Vaxe3r4rg37RpE9u2bRu6DElaVZJ8dal2p3okqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxMx/86zZsJMmq+Vq3YePQPzJJM25VLNnwH3Hvjrs59YJPD13GyC476/ihS5A042b+jF+S9G8Z/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNWZiwZ/koiQ7k9y6xGtvSFJJjphU/5KkpU3yjP9i4OTFjUk2AM8H7ppg35KkZUws+KvqOuDrS7z0m8AbgZpU35Kk5U11jj/JS4B7quqmafYrSXrI1BZpS3IocD7wghG33wxsBti40RUrJWlcpnnG/0TgKOCmJF8B1gM3JPnOpTauqi1VNVdVc2vXrp1imZI026Z2xl9VtwCP2/28D/+5qrp/WjVIkiZ7OeelwPXAsUl2JDlzUn1JkkY3sTP+qjp9L69vmlTfkqTl+cldSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmOmtmSDRnTAGpIMXcVInrB+A/fc7W0VpNXG4N/fPLiLUy/49NBVjOSys44fugRJK+BUjyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNWaSN1u/KMnOJLcuaHtrktuT3JzkT5I8dlL9S5KWNskz/ouBkxe1XQ08uaqeAnwROG+C/UuSljCx4K+q64CvL2q7qqp29U//Clg/qf4lSUsbco7/vwEfWe7FJJuTbEuybX5+foplSdJsGyT4k5wP7AIuWW6bqtpSVXNVNbd27drpFSdJM27qyzInOQN4MXBSVdW0+5ek1k01+JOcDPxP4L9W1T9Os29JUmeSl3NeClwPHJtkR5Izgd8BHgVcneTGJO+aVP+SpKVN7Iy/qk5fovndk+pPkjQaP7krSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNWaSN1u/KMnOJLcuaDs8ydVJvtR/P2xS/UuSljbJM/6LgZMXtZ0LXFNVxwDX9M8lSVM0seCvquuAry9qfinw3v7xe4EfnVT/kqSlTXuO//FVdR9A//1xy22YZHOSbUm2zc/PT61ASZp1++2bu1W1parmqmpu7dq1Q5cjSTNj2sH/tSRHAvTfd065f0lq3rSD/4PAGf3jM4APTLl/SWreJC/nvBS4Hjg2yY4kZwK/Cjw/yZeA5/fPJUlTtGZSB66q05d56aRJ9SlJ2rv99s1dSdJkGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGjNS8Cc5YZQ2SdL+b9Qz/t8esU2StJ/b4+qcSY4DjgfWJnn9gpceDRw4ycIkSZOxt2WZDwIe2W/3qAXt3wBOmVRRkqTJ2WPwV9UngU8mubiqvjqlmiRJEzTqjVgOTrIF2LRwn6o6cRJFSZImZ9Tgvxx4F3Ah8O3JlSNJmrRRg39XVb1zXJ0meR3waqCAW4Cfrqp/HtfxJUnLG/Vyzj9N8nNJjkxy+O6vlXSYZB3w88BcVT2Z7uqg01ZyLEnSvhv1jP+M/vsvLGgr4Oj/QL8PT/It4FDg3hUeR5K0j0YK/qo6alwdVtU9Sd4G3AX8E3BVVV21eLskm4HNABs3bhxX95LUvJGCP8lPLdVeVb+/rx0mOQx4KXAU8PfA5UleWVXvW3TsLcAWgLm5udrXfiRJSxt1qucZCx4fApwE3ADsc/ADzwPurKp5gCRX0n06+H173EuSNBajTvWcvfB5kscAf7DCPu8CnpXkULqpnpOAbSs8liRpH610WeZ/BI5ZyY5VtRW4gu4vhlv6GrassA5J0j4adY7/T+mu4oHu8sv/DLx/pZ1W1ZuBN690f0nSyo06x/+2BY93AV+tqh0TqEeSNGEjTfX0i7XdTrdC52HANydZlCRpcka9A9dPAJ8BXg78BLA1icsyS9IqNOpUz/nAM6pqJ0CStcCf071JK0laRUa9queA3aHf+7/7sK8kaT8y6hn/R5N8DLi0f34q8GeTKUmSNEl7u+fudwGPr6pfSPIy4IeAANcDl0yhPknSmO1tuubtwAMAVXVlVb2+ql5Hd7b/9smWJkmahL0F/6aqunlxY1Vto7sNoyRpldlb8B+yh9cePs5CJEnTsbfg/2ySn1ncmORMYPtkSpIkTdLeruo5B/iTJK/goaCfAw4CfmyCdUmSJmSPwV9VXwOOT/Jc4Ml984er6uMTr0ySNBGjrsd/LXDthGuRJE2Bn76VpMYY/JLUGINfkhpj8EtSYwx+SWrMIMGf5LFJrkhye5Lbkhw3RB2S1KJRl2Uet98CPlpVpyQ5CDh0oDokqTlTD/4kjwaeDbwKoKq+iffwlaSpGWKq52hgHnhPks8luTDJIxZvlGRzkm1Jts3Pz0+/Ss2cdRs2kmRVfK3bsHHoH5dm2BBTPWuApwNnV9XWJL8FnAv84sKNqmoLsAVgbm6upl6lZs69O+7m1As+PXQZI7nsrOOHLkEzbIgz/h3Ajqra2j+/gu4fAknSFEw9+Kvqb4G7kxzbN50EfGHadUhSq4a6quds4JL+ip6/AX56oDokqTmDBH9V3Ui3rr8kacr85K4kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMUN9clez4IA1JBm6Ckn7yODXyj24a9WsdgmueCnt5lSPJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTGDBX+SA5N8LsmHhqpBklo05Bn/a4HbBuxfkpo0SPAnWQ/8CHDhEP1LUsuGOuN/O/BG4MGB+pekZk09+JO8GNhZVdv3st3mJNuSbJufn59SdZI0+4Y44z8BeEmSrwB/BJyY5H2LN6qqLVU1V1Vza9eunXaNkjSzph78VXVeVa2vqk3AacDHq+qV065DklrldfyS1JhBb8RSVZ8APjFkDZLUGs/4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CU1Zd2GjSRZNV/rNmwc+89g0E/uStK03bvjbk694NNDlzGyy846fuzH9Ixfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaM/XgT7IhybVJbkvy+SSvnXYNktSyIZZs2AX8j6q6IcmjgO1Jrq6qLwxQiyQ1Z+pn/FV1X1Xd0D9+ALgNWDftOiSpVYMu0pZkE/A0YOsSr20GNgNs3Dj+1emk/doBa0gydBUje8L6Ddxz911Dl6ERDRb8SR4J/DFwTlV9Y/HrVbUF2AIwNzdXUy5PGtaDu5pfQVKTM8hVPUkeRhf6l1TVlUPUIEmtGuKqngDvBm6rqt+Ydv+S1LohzvhPAH4SODHJjf3XiwaoQ5KaNPU5/qr6FLB63rWSpBnjJ3clqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4JekxgwS/ElOTnJHki8nOXeIGiSpVVMP/iQHAr8LvBB4EnB6kidNuw5JatUQZ/zPBL5cVX9TVd8E/gh46QB1SFKTUlXT7TA5BTi5ql7dP/9J4Aer6jWLttsMbO6fHgvcsQ/dHAHcP4ZyVwPHOrtaGq9jnYz/VFVrFzeumVLnC2WJtn/3r09VbQG2rKiDZFtVza1k39XGsc6ulsbrWKdriKmeHcCGBc/XA/cOUIckNWmI4P8scEySo5IcBJwGfHCAOiSpSVOf6qmqXUleA3wMOBC4qKo+P+ZuVjRFtEo51tnV0ngd6xRN/c1dSdKw/OSuJDXG4JekxsxU8M/CUhBJNiS5NsltST6f5LV9++FJrk7ypf77YQv2Oa8f8x1JfnhB+w8kuaV/7R1JlrqUdnBJDkzyuSQf6p/P8lgfm+SKJLf3v+PjZnW8SV7X/zd8a5JLkxwyK2NNclGSnUluXdA2trElOTjJZX371iSbxjqAqpqJL7o3iv8aOBo4CLgJeNLQda1gHEcCT+8fPwr4It3SFr8OnNu3nwv8Wv/4Sf1YDwaO6n8GB/avfQY4ju6zEx8BXjj0+JYZ8+uBPwQ+1D+f5bG+F3h1//gg4LGzOF5gHXAn8PD++fuBV83KWIFnA08Hbl3QNraxAT8HvKt/fBpw2VjrH/oHOMZfxHHAxxY8Pw84b+i6xjCuDwDPp/vk8pF925HAHUuNk+5qqeP6bW5f0H46cMHQ41lifOuBa4ATeSj4Z3Wsj+7DMIvaZ268ffDfDRxOd/Xgh4AXzNJYgU2Lgn9sY9u9Tf94Dd0nfTOu2mdpqmf3f2i77ejbVq3+z7unAVuBx1fVfQD998f1my037nX948Xt+5u3A28EHlzQNqtjPRqYB97TT21dmOQRzOB4q+oe4G3AXcB9wD9U1VXM4FgXGOfY/nWfqtoF/APwHeMqdJaCf6SlIFaLJI8E/hg4p6q+sadNl2irPbTvN5K8GNhZVdtH3WWJtlUx1t4auumBd1bV04D/RzclsJxVO95+fvuldFMbTwAekeSVe9plibZVMdYRrGRsEx33LAX/zCwFkeRhdKF/SVVd2Td/LcmR/etHAjv79uXGvaN/vLh9f3IC8JIkX6FbpfXEJO9jNscKXZ07qmpr//wKun8IZnG8zwPurKr5qvoWcCVwPLM51t3GObZ/3SfJGuAxwNfHVegsBf9MLAXRv6v/buC2qvqNBS99EDijf3wG3dz/7vbT+qsAjgKOAT7T/6n5QJJn9cf8qQX77Beq6ryqWl9Vm+h+Xx+vqlcyg2MFqKq/Be5OcmzfdBLwBWZzvHcBz0pyaF/jScBtzOZYdxvn2BYe6xS6/zfG95fO0G+QjPnNlhfRXQXz18D5Q9ezwjH8EN2fdDcDN/ZfL6Kb37sG+FL//fAF+5zfj/kOFlzxAMwBt/av/Q5jfHNoAuN+Dg+9uTuzYwWeCmzrf7//BzhsVscLvAW4va/zD+iuapmJsQKX0r138S26s/Mzxzk24BDgcuDLdFf+HD3O+l2yQZIaM0tTPZKkERj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfi1qiWZS/KOoevYF0nOSXLo0HWoXV7HL01Zv0TFXFXdP3QtapNn/NqvJNm06OYWb0jyv5J8IsmvJflMki8m+S/968/JQzdw+Y4kV/UrX16Q5KtJjljumP3jJyb5aJLtSf4iyffsobaXp7upyE1JruvbDkzy1iSfTXJzkrMW1PWJPHTTlUvS+Xm6RcuuTXJtv+0Lklyf5IYkl/cL9JHkK0ne0rffsru2JI9M8p6+7eYkP76X4/xqki/0275tbL8srVoGv1aTNVX1TOAc4M1LvP5m4FPVrXz5QWDjCMfcApxdVT8AvAH4vT1s+0vAD1fV9wMv6dvOpFty+BnAM4Cf6ddjgW5J7XPobsRxNHBCVb2DbiGu51bVc5McAbwJeF5VPZ1uOYfXL+jz/r79nX19AL/Y9/l9VfUU4OPLHSfJ4cCPAd/bb/vLI/xMNOPWDF2AtA92r1S6ne4mGIs9G3gZQFV9OMnf7elg/Rnx8cDleehufgfvYZe/BC5O8v4FtbwAeEqSU/rnj6FbhOubdAtx7ej7urGv+VOLjvksun8Y/rKv4SDg+gWvLxzzy/rHz6Nb1I5+rH+XbonrpY7zDeCfgQuTfJjuhihqnMGv/c0u/u1foocsePwv/fdvs/x/u0u9abXcMQ8A/r6qnjpKYVX135P8IPAjwI1Jnkq3bvrZVfWxhdsmec6CevdUc4Crq+r0Zbpdaszh349z2eMkeSbd6pinAa+hu9uZGuZUj/Y3XwMe18/XHwy8eB/2vQ54BUCSF9KtfLnsMau7wc2dSV7e75Mk37/cwZM8saq2VtUv0d0KbwPdLfJ+Nt09FEjy3enuqrUnD9DdTxngr4ATknxXv/+hSb57L/tfRRfgu+s6bLnj9H/VPKaq/oxu2umpezm2GuAZv/YrVfWtJP+b7naTd9It6zuqtwCXJrkB+CTdmvB7O+YrgHcmeRPwMLobwty0zPHfmuQYurPra/rtbqabwrmhX1N9HvjRvdS5BfhIkvv6ef5X9XXvnmZ6E93y4sv5ZeB3+zesvw28paquXOY4DwAfSHJIX/fr9lKbGuDlnJpZ8bJJaUlO9UhSYzzjlxZJcj7w8kXNl1fVrwxRjzRuBr8kNcapHklqjMEvSY0x+CWpMQa/JDXm/wOxjUz0oyD1nwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.histplot(data=corpus_df, x='unique_sentences')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4c00676",
   "metadata": {
    "id": "d4c00676",
    "outputId": "0313cb16-b043-42ce-af3f-d5fe330f634e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='unique_sentences'>"
      ]
     },
     "execution_count": 803,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbqElEQVR4nO3df7Bc5X3f8ffnCsXil2IEVwJLkIuICEFMissdqoTS8fAjyAqxaGsozLimHabqEBVwcGILx1OnntLSxCGO3JhUYBuROCbCcYLsgmIi/widYsEVpgMCU8lCBhkhXeTECGphifvtH+dcWK7ujz179uzZc87nNXNnd5/ds/vsAZ3vPs/zfZ5HEYGZmVmnBsqugJmZVZsDiZmZ5eJAYmZmuTiQmJlZLg4kZmaWy1FlV6AMJ510UgwNDZVdDTOzStm6devLETE4sbyRgWRoaIiRkZGyq2FmVimSfjBZubu2zMwsFwcSMzPLxYHEzMxycSAxM7NcHEjMzCyXRmZtmZnVzdhYsGv/a+x95SAL5s5h6MRjGRhQTz7bgcTMGqHMC23RxsaCTdte4uYNT3Dw0BhzZg9w+1XnsnzpyT35ju7aMrPaG7/Qrlj7MNfcuYUVax9m07aXGBurxzYau/a/9mYQATh4aIybNzzBrv2v9eTzHUjMrPbKvtAWbe8rB9/8buMOHhpj34GDPfl8BxIzq72yL7RFWzB3DnNmv/1yPmf2APOPn9OTz3cgMbPaK/tCW7ShE4/l9qvOffM7jo+RDJ14bE8+34PtZlZ74xfaiYPRvbrQFm1gQCxfejJn3Xgh+w4cZP7xvU0mUBP3bB8eHg4v2mjWLONZW2VcaOtC0taIGJ5YXmjXlqTPS9on6amWsnmSHpK0Pb09oeW5WyTtkPSspMtays+T9GT63FpJSsvfIekv0vItkoaK/D5mVl0DA2Lx4HEsW3wSiwePcxDpoqLHSO4Glk8oWwNsjoglwOb0MZLOBq4GlqbHfFbSrPSYO4BVwJL0b/w9rwP+PiJ+HvhD4L8V9k3MzGxShQaSiPg74EcTilcC69P764ErWsrvjYjXI+I5YAdwvqRTgLkR8Ugk/XD3TDhm/L2+DFw83loxM7PeKCNra0FE7AFIb+en5QuBF1petzstW5jen1j+tmMi4jDwY+DEyT5U0ipJI5JGRkdHu/RVzMysn9J/J2tJxDTl0x1zZGHEuogYjojhwcEjdoo0M7MOlRFI9qbdVaS3+9Ly3cCpLa9bBLyYli+apPxtx0g6CvhZjuxKMzPrubGxYOfoqzzy/ZfZOfpqbZZjmUwZgWQjcG16/1rg/pbyq9NMrNNJBtUfTbu/Dkhalo5/fHDCMePv9X7gG9HEfGYz6yt1X9troqLTf78EPAL8gqTdkq4DbgMulbQduDR9TERsAzYATwObgNUR8Ub6VtcDd5EMwH8feDAt/xxwoqQdwM2kGWBmZmWq+9peExU6sz0irpniqYuneP2twK2TlI8A50xSfhC4Mk8dzcy6bbq1vRYPHldSrYrTT4PtZma1UPe1vSZyIDEz67KyF1HsNS/aaGbWZWUvothrDiRmZgUYX9urjmMiE7lry8zMcnEgMTOzXBxIzMwsFwcSMzPLxYHEzMxycSAxM7NcHEjMzCwXBxIzM8vFgcTMzHJxIDEzs1wcSMzMLBcHEjMzy8WBxMzMcnEgMTOzXBxIzMwsFwcSMzPLxYHEzMxycSAxM7NcHEjMzCwXBxIzM8vFgcTMzHJxIDEzs1wcSMzMLBcHEjMzy8WBxMzMciktkEj6TUnbJD0l6UuS5kiaJ+khSdvT2xNaXn+LpB2SnpV0WUv5eZKeTJ9bK0nlfCMzs2YqJZBIWgjcCAxHxDnALOBqYA2wOSKWAJvTx0g6O31+KbAc+KykWenb3QGsApakf8t7+FXMzBqvzK6to4CjJR0FHAO8CKwE1qfPrweuSO+vBO6NiNcj4jlgB3C+pFOAuRHxSEQEcE/LMWZm1gOlBJKI+CHwKeB5YA/w44j4OrAgIvakr9kDzE8PWQi80PIWu9Oyhen9ieVHkLRK0oikkdHR0W5+HTOzRiura+sEklbG6cC7gGMlfWC6QyYpi2nKjyyMWBcRwxExPDg4mLXKZmY2hbK6ti4BnouI0Yg4BHwF+BVgb9pdRXq7L339buDUluMXkXSF7U7vTyw3M7MeKSuQPA8sk3RMmmV1MfAMsBG4Nn3NtcD96f2NwNWS3iHpdJJB9UfT7q8Dkpal7/PBlmPMzKwHjirjQyNii6QvA48Dh4HvAuuA44ANkq4jCTZXpq/fJmkD8HT6+tUR8Ub6dtcDdwNHAw+mf2Zm1iNKkp2aZXh4OEZGRsquhplZpUjaGhHDE8s9s93MzHJxIDEzs1wcSMzMLBcHEjMzy8WBxMzMcmk7kEj6PUlzJc2WtFnSyzPMRjczswbI0iL51Yh4BbicZEb5mcBvF1IrMzOrjCyBZHZ6uwL4UkT8qID6mJlZxWSZ2f5VSd8DfgL8hqRB4GAx1TIzs6pou0USEWuAXybZjOoQ8P9IVvA1M7MGyzLYfgywmmRHQkiWfz9iqryZmTVLljGSLwA/JVnuHZIB9//c9RqZmVmlZAkkZ0TE7wGHACLiJ0y+sZSZmTVIlkDyU0lHk+5AKOkM4PVCamVmZpWRJWvrE8Am4FRJXwQuAP5NEZUyM7PqaDuQRMRDkh4HlpF0ad0UES8XVjMzM6uELFlb/xw4HBH/MyK+BhyWdEVhNTMzs0rIMkbyiYj48fiDiPgHku4uMzPrkrGxYOfoqzzy/ZfZOfoqY2P9v4ttljGSyYJOKXu+m5nV0dhYsGnbS9y84QkOHhpjzuwBbr/qXJYvPZmBgf5Nks3SIhmRdLukMyQtlvSHwNaiKmZm1jS79r/2ZhABOHhojJs3PMGu/a+VXLPpZQkkN5BMSPwL4D6SdbZWF1EpM7Mm2vvKwTeDyLiDh8bYd6C/lzXMkrX1GrCmwLqYmTXagrlzmDN74G3BZM7sAeYfP6fEWs0sS9bWmZLWSfq6pG+M/xVZOTOzJhk68Vhuv+pc5sxOLs3jYyRDJx5bcs2ml2Ww/D7gT4C7gDeKqY6ZWXMNDIjlS0/mrBsvZN+Bg8w/fg5DJx7b1wPtkC2QHI6IO2Z+mZnZzMbGgl37X2PvKwdZMLcaF8xeGBgQiwePY/HgcWVXpW1ZN7b6DeCvaFljyzslmllWVU1ztcllydq6lmSP9v9Nkva7FRgpolJmVm9VTXO1yWXJ2jq9yIqYWXNMl+ZapS4dS2TaIVHSxyWtSx8vkXR5cVUzs7oaT3NtVYU0V5tcaTskSnqnpC9L+p6kZyT9sqR5kh6StD29PaHl9bdI2iHpWUmXtZSfJ+nJ9Lm1ktzBatbnqprmapPLMth+RkT8K0nXQLJDYs6L9h8BmyLi/ZJ+BjgG+BiwOSJuk7SGZALkRyWdDVwNLCXZK/5vJZ0ZEW+Q7CG/CvgO8ACwHHgwR73MrGBVTXOtsiKz5LIEkq7tkChpLvDPSDfGioifpu+/EnhP+rL1wLeAjwIrgXsj4nXgOUk7gPMl7QLmRsQj6fveA1yBA4lZ36timmtVFZ0ll6Vr63d5+w6Jm0ku8p1YDIwCX5D0XUl3SToWWBARewDS2/np6xcCL7QcvzstW5jen1huZlY7nS4xX3SWXJasra9L2kp3dkg8CvjHwA0RsUXSHzH9Ol6ThcyYpvzIN5BWkXSBcdppp2WrrZlZyfK0KorOksuStbU5IvaP75AYES9L2tzh5+4GdkfElvTxl0kCy15Jp6Sfdwqwr+X1p7Ycvwh4MS1fNEn5ESJiXUQMR8Tw4OBgh9U2MytHnlZF0VlyMwYSSXMkzQNOknRCmlk1T9IQycB3ZhHxEvCCpF9Iiy4GngY2kkx8JL29P72/Ebha0jsknQ4sAR5Nu78OSFqWDvx/sOUYM7PayLPEfNFZcu10bf174EMkQWMrb3UnvQL8cY7PvgH4YpqxtRP4tySBbYOk64DngSsBImKbpA0kweYwsDrN2AK4HrgbOJpkkN0D7WZWO3mWmC86S04R7Q3WSLohIj7TlU8t2fDwcIyMeHUXM6uOflifTNLWiBieWJ5lsP0zkn4FGGo9LiLu6UoNzcxsSv0896btQCLpT4EzgCd4az+SABxIzPqEl2avt36de5NlQuIwcHa02xdmZj3VD10f1kxZJiQ+BZxcVEXMLB8vzW5lydIiOQl4WtKjvH1jq/d1vVZmlpmXZreyZAkkv1tUJcwsvzzpoWZ5tN21FRHfBnYBs9P7jwGPF1QvM8vIS7NbWbJkbf07krWq5pFkby0E/oRkVrqZlayf00Ot3rJ0ba0Gzge2AETEdknzpz/EzHqpX9NDrd6yZG29nu4bAoCko5hipV0zM2uOLIHk25I+Bhwt6VLgPuCrxVTLzMyqIksgWUOyGdWTJAs5PgB8vIhKmZnVWacbVPWrLGttjQF3Anemy8ov8ix3M7Ns6rgCQZaNrb4laW4aRJ4g2Sb39sJqZmZWQ3VcgSBL19bPRsQrwL8AvhAR5wGXFFMtM7P+10kXVZ4NqvpVlvTfo9Ltb68Cfqeg+piZVUKnXVR1XIEgS4vkk8DfADsi4jFJi4HtxVTLzKy/ddpFVccVCLIMtt9HkvI7/ngn8C/HH0u6JSL+a3erZ2bWnzpdJLOOKxBk6dqayZWAA4mZlaaXG3vl3UO9TisQZOnamkl1w6mZVd74mMWKtQ9zzZ1bWLH2YTZte6mwORp17KLqVDdbJJ5TYmalmWrM4qwbLyzkl38du6g61c1A0ryzZ2Z9o4yNverWRdWpbnZt3TfzS8zMijE+ZtGq6mm1VZFlZvuZkjZLeip9/EuS3lxrKyL+SxEVNDNrh8csyqN2l8uS9G3gt4H/ERHvTsueiohzCqxfIYaHh2NkZKTsaphZl41nbTV9zKIokrZGxPDE8ixjJMdExKPS2/6jHM5dMzOzLvGYRTmyjJG8LOkM0uwsSe8H9hRSKzMzq4ysW+2uA86S9EPgOeADhdTKzMwqI8sSKTuBSyQdCwxExIHiqmVmZlXRdiCR9B8nPAYgIj7Z5TqZmVmFZBkjea3l7w3gvcBQng+XNEvSdyV9LX08T9JDkrantye0vPYWSTskPSvpspby8yQ9mT63VhOyAcz6Vd22W7XmytK19QetjyV9CtiY8/NvAp4B5qaP1wCbI+I2SWvSxx+VdDZwNbAUeBfwt5LOjIg3gDuAVcB3SPaRXw48mLNeZoWq43ar1lx5ZrYfAyzu9GBJi4BfA+5qKV4JrE/vrweuaCm/NyJej4jngB3A+elGW3Mj4pF0//h7Wo4x61t13G7VmivLGMmTvLUw4yxgkGSzq059GvgIcHxL2YKI2AMQEXskzU/LF5K0OMbtTssOpfcnlk9W/1UkLRdOO+20HNU2y6+MdaHMipIl/ffylvuHgb0R0dGEREmXA/siYquk97RzyCRlMU35kYUR60jSlxkeHnZntJWqjtutWnNl6do60PL3E2BuOjg+T9K8jJ97AfA+SbuAe4GLJP0ZsDftriK93Ze+fjdwasvxi4AX0/JFk5Sb9TWvC2V1kmWtrV0kF/O/J2kJvBN4Pn06IqKj8ZK0RfJbEXG5pN8H9rcMts+LiI9IWgr8OXA+yWD7ZmBJRLwh6THgBmALyWD7ZyLigek+s+5rbfVylzjrnNeFsqrpxlpbm4CN4xdpSe8FLomID3epjgC3ARskXUcSpK4EiIhtkjYAT5N0q61OM7YArgfuBo4mydZqdMaWs4Gqw+tCWV1kaZFsjYjzJpSNTBad+l2dWyQ7R19lxdqHj+h7f6CgXeLMrDmmapFkXbTx45KGJP2cpN8B9nevitYN02UDmZkVIUsguYYk5fevgL8G5qdl1ke8S5z1C8/cb44sM9t/RDIT3frYeDbQxDESZwNZL3msrllmHCOR9OmI+JCkrzLJHI2IeF9RlStKncdIwNlA1jtTZQh6rK6e8mRt/Wl6+6nuVsmK4mwg64XpWh2eud8sMwaSiNia3n67+OqYWVG6Pb9oqvXCzrrxQs/cb5i2B9slXZAu7f5/Je2U9JyknUVWzsy6Y7z1sGLtw1xz5xZWrH2YTdteyjUAPl2rwzP3myXLhMTPAb8JbCXZj8TMKmK61kOnXU3TtToGBsTypSdz1o0XeqyuAbKk//44Ih6MiH0RsX/8r7CamVnXFDG/aKZWx/hY3bLFJ7F48DgHkRrL0iL5ZroW1leA18cLI+LxrtfKzLqqiDELtzpsXJZA8k/S2/FlUkSSDnxRV2tkZl1X1PwiZwgaZAsk35qkzFNVzSrArQcrUpZA8mrL/TkkG109093qmOXjJfSn5taDFSXLEil/0PpY0qeAjV2vkVmHvCyHWTmyZG1NdAzQ0WZWZkWYKsV11/7X2jq+KosMVqWeVeJzmk/bLRJJT/LWmMgskpWAP1lEpcw6kWdZjqq0ZqpSzyrxOc0vS4vkcuDX079fBd4VEf+9kFpZpfTLr7k8S+jnbc30SlXqWSU+p/m1HUgi4gctfz+MiMNFVsyqoYilNzqVZ1mOqmwIVpV6VonPaX5ZsrbMjlDE0hudypPiWpVFBqtSzyrxOc0vz2C7Wd/9mut0WY5+W2Rwqu7CfqtnHfic5ucWieVSl19z/TRhb6bB3yLq2eT5N/30376qZtwhsY7qvkNiL5WR8dLpRa8qF8te7y7orCVrV54dEs2m1Otfc51e9Kp0sez17oL9NM5l1eQxEsutl8uFd5qqWaUUzzxpzJ3ot3Euqx4HEquUTi96VbpY9nrwt9eBy+rHXVtWKZ0O7lcpKaDX3YVFLTFvzeHBdquUJoyRlGE8EcFZSzadqQbbHUiscjq96PliaZaPs7asNjrdV8P7cZgVo5TBdkmnSvqmpGckbZN0U1o+T9JDkrantye0HHOLpB2SnpV0WUv5eZKeTJ9bK8k/Mc2sa/plUdJ+VlbW1mHgwxHxi8AyYLWks4E1wOaIWAJsTh+TPnc1sBRYDnxW0qz0ve4AVgFL0r/lvfwiZlZf/bQoaT8rJZBExJ6IeDy9f4Bky96FwEpgffqy9cAV6f2VwL0R8XpEPAfsAM6XdAowNyIeiWSw556WY8ysovqlFVCl+UdlKn2MRNIQ8G5gC7AgIvZAEmwkzU9fthD4Tsthu9OyQ+n9ieVmVlH9lGHX61UGqqrUCYmSjgP+EvhQRLwy3UsnKYtpyif7rFWSRiSNjI6OZq+sNVq//EJugn5qBXiyZntKCySSZpMEkS9GxFfS4r1pdxXp7b60fDdwasvhi4AX0/JFk5QfISLWRcRwRAwPDg5274s0RJMvpO4n761+WoXAS8y3p5SurTSz6nPAMxFxe8tTG4FrgdvS2/tbyv9c0u3Au0gG1R+NiDckHZC0jKRr7IPAZ3r0NRqjn7oayuBFDXurn1Yh8BLz7SmrRXIB8K+BiyQ9kf6tIAkgl0raDlyaPiYitgEbgKeBTcDqiHgjfa/rgbtIBuC/DzzY02/SAP3U1VCGfvqF3AT91gro5aKkVVVKiyQi/heTj28AXDzFMbcCt05SPgKc073a2URNH3Dsp1/ITeBWQPV49V+bUdMHHPP8Qm7y2FIebgVUS+npv9ZbnewS2PTVYTv9hdz0sSVrDi/a2CB5Lmy9XvCwKtviTqfXW+aaFc2LNlZQty+mebKPerngYV1+yTd9bMmaw2MkfaqIuQtVyT6qS5ZY08eWrDkcSPpUERfTqlzYqhLwZtJvaaxmRXHXVp8qolukKoPmdUm3dRqrNYUDSZ8q4mJalQtbVQJeO7yZljWBs7b6VF0GnDvlbXH7Rx0y6Kw7vGd7iyoEEvDF1MrX9B809nZTBRIPtvcxz+61stUlg86K5UBipfHyIf2vLhl0ViwPthfM/cuTc5dJNdQlg86K5RZJgYrcEKnqv+bdZVINngtj7XCLpEBFbYhUh1/zXj6kGqqSMm7lcoukTZ20AIrqX67Dr/mqzLK3/kv6qHprvI7cImlDpy2AmfqXOx0/qcOv+TpNOrTeqUNrvI4cSNrQaRfVdBfLPP8g6jAA6i4T60RR3cWWjwNJGzptAUx3sdw5+mrH/yDq8mvey4dYVnVojdeRA0kb8rQAprpY5vkH4V/z1lR1aI3XkQfb21BECmTeweZ+GwA16wWnI/cnr7XVpm6ve+VBQ7POeA268njRxhb9smij/0GYWZV4z/YCdZrG68FmM6sDB5Kc3EVlZk3nwfac6jDLvEiehWxWf26R5FSnvPZur1Ts1ppZM7hFklNd1owqYqVit9bMmsGBJKe65LUXcdFvyqZI7r6zpnPXVk51mWVeRBddE2Yhu/vOrCYtEknLJT0raYekNb3+/DrMMi+ii64urbXpuPvOrAYtEkmzgD8GLgV2A49J2hgRT5dbs2opYiHIurTWplOnZAuzTlU+kADnAzsiYieApHuBlYADSQZFXfTrPumyCd13ZjOpQ9fWQuCFlse707K3kbRK0oikkdHR0Z5Vrkrq0EXXa03ovjObSR1aJJNd7Y5Im4mIdcA6SNbaKrpS1gxN6L4zm0kdAslu4NSWx4uAF0uqizVQ3bvvzGZSh66tx4Alkk6X9DPA1cDGkutkZtYYlW+RRMRhSf8B+BtgFvD5iNhWcrXMzBqj8oEEICIeAB4oux5mZk1Uh64tMzMrkQOJmZnl0sitdiWNAj/o8PCTgJe7WJ068bmZms/N1HxuJteP5+XnImJwYmEjA0kekkYm27PYfG6m43MzNZ+byVXpvLhry8zMcnEgMTOzXBxIsltXdgX6mM/N1HxupuZzM7nKnBePkZiZWS5ukZiZWS4OJGZmlosDSQZlb+nbTyR9XtI+SU+1lM2T9JCk7entCWXWsQySTpX0TUnPSNom6aa03OdGmiPpUUn/Jz03/yktb/y5gWS3V0nflfS19HFlzosDSZtatvR9L3A2cI2ks8utVanuBpZPKFsDbI6IJcDm9HHTHAY+HBG/CCwDVqf/n/jcwOvARRHxj4BzgeWSluFzM+4m4JmWx5U5Lw4k7XtzS9+I+CkwvqVvI0XE3wE/mlC8Elif3l8PXNHLOvWDiNgTEY+n9w+QXBgW4nNDJF5NH85O/wKfGyQtAn4NuKuluDLnxYGkfW1t6dtwCyJiDyQXVGB+yfUplaQh4N3AFnxugDe7b54A9gEPRYTPTeLTwEeAsZayypwXB5L2tbWlrxmApOOAvwQ+FBGvlF2ffhERb0TEuSQ7mZ4v6ZySq1Q6SZcD+yJia9l16ZQDSfu8pe/M9ko6BSC93VdyfUohaTZJEPliRHwlLfa5aRER/wB8i2Screnn5gLgfZJ2kXSZXyTpz6jQeXEgaZ+39J3ZRuDa9P61wP0l1qUUkgR8DngmIm5vecrnRhqU9M70/tHAJcD3aPi5iYhbImJRRAyRXFe+EREfoELnxTPbM5C0gqQvc3xL31vLrVF5JH0JeA/JUtd7gU8Afw1sAE4DngeujIiJA/K1JumfAg8DT/JWf/fHSMZJmn5ufolk0HgWyY/YDRHxSUkn0vBzM07Se4DfiojLq3ReHEjMzCwXd22ZmVkuDiRmZpaLA4mZmeXiQGJmZrk4kJiZWS4OJGZmlosDiZmZ5fL/AYNqpdDfGB2UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(data=corpus_df, x=corpus_df.index, y=\"unique_sentences\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c646328f",
   "metadata": {
    "id": "c646328f"
   },
   "source": [
    "#### 3. Most used sentences by each president \n",
    "\n",
    "This was included as an interesting output. Instead it revealed the data included irrelevant data like crowd response:\n",
    "- Laughter, Applause\n",
    "- The names of the Presidents att the end of their speech\n",
    "- number indices (bullet) points. e.g. 2.\n",
    "- The President / THE PRESIDENT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42ec2998",
   "metadata": {
    "id": "42ec2998",
    "outputId": "4a7532d5-8754-4523-8567-210ad4ab8381"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('A', 2)"
      ]
     },
     "execution_count": 804,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "def max_frequency(a_list):\n",
    "    frequency = collections.Counter(a_list)\n",
    "    return max(dict(frequency), key=dict(frequency).get), max(dict(frequency).values())\n",
    "\n",
    "\n",
    "random_list = ['A', 'A', 'B', 'C', 'B', 'D', 'D']\n",
    "max_frequency(random_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e286744",
   "metadata": {
    "id": "9e286744",
    "outputId": "f2f8a643-d355-4aca-9856-1f477057c44d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Thank you.', 65)"
      ]
     },
     "execution_count": 805,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_frequency(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc650780",
   "metadata": {
    "id": "cc650780",
    "outputId": "a8f10662-714b-4b46-a943-23129bc450a2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 806,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_frequency(sentences)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3642eaa",
   "metadata": {
    "id": "a3642eaa",
    "outputId": "1da11114-039c-4850-da09-79442f92f7fb",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Fellow Citizens of the Senate and the House of Representatives: Among the vicissitudes incident to life, no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the fourteenth day of the present month.',\n",
       "  'JOHN ADAMS.',\n",
       "  'FRIENDS AND FELLOW-CITIZENS, Called upon to undertake the duties of the first executive office of our country, I avail myself of the presence of that portion of my fellow citizens which is here assembled to express my grateful thanks for the favor with which they have been pleased to look toward me, to declare a sincere consciousness that the task is above my talents, and that I approach it with those anxious and awful presentiments which the greatness of the charge and the weakness of my powers so justly inspire.',\n",
       "  'JAMES MADISON.',\n",
       "  'I should be destitute of feeling if I was not deeply affected by the strong proof which my fellow citizens have given me of their confidence in calling me to the high office whose functions I am about to assume.',\n",
       "  'AND NOW, FRIENDS AND COUNTRYMEN, if the wise and learned philosophers of the elder world, the first observers of nutation and aberration, the discoverers of maddening ether and invisible planets, the inventors of Congreve rockets and Shrapnel shells, should find their hearts disposed to enquire what has America done for the benefit of mankind?',\n",
       "  'As, then, the period approaches when the application of the revenue to the payment of debt will cease, the disposition of the surplus will present a subject for the serious deliberation of Congress; and it may be fortunate for the country that it is yet to be decided.',\n",
       "  'M. VAN BUREN.',\n",
       "  'Called from a retirement which I had supposed was to continue for the residue of my life to fill the chief executive office of this great and free nation, I appear before you, fellow citizens, to take the oaths which the Constitution prescribes as a necessary qualification for the performance of its duties; and in obedience to a custom coeval with our Government and what I believe to be your expectations I proceed to present to you a summary of the principles which will govern me in the discharge of the duties which I shall be called upon to perform.',\n",
       "  'No.',\n",
       "  'JAMES K. POLK.',\n",
       "  'Elected by the American people to the highest office known to our laws, I appear here to take the oath prescribed by the Constitution, and, in compliance with a time honored custom, to address those who are now assembled.',\n",
       "  'Second.',\n",
       "  'FRANKLIN PIERCE.',\n",
       "  'When this fails it can proceed no further.',\n",
       "  'We deny it.',\n",
       "  'ANDREW JOHNSON.',\n",
       "  'U. S. GRANT.',\n",
       "  'SEC.',\n",
       "  'Fellow Citizens: We stand to-day upon an eminence which overlooks a hundred years of national life- a century crowded with perils, but crowned with the triumphs of liberty and law.',\n",
       "  'Second.',\n",
       "  '2.',\n",
       "  'BENJ.',\n",
       "  'This is an obvious duty.',\n",
       "  'No.',\n",
       "  'Second.',\n",
       "  'I shall seek, as I have always sought, to justify the extraordinary confidence thus reposed in me by striving to purge my heart and purpose of every personal and of every misleading party motive and devoting every energy I have to the service of the nation as a whole, praying that I may continue to have the counsel and support of all forward looking men at every turn of the difficult business.',\n",
       "  'Republics have risen and fallen, and a transition from party to personal government has preceded every failure since the world began.',\n",
       "  'Since the close of the last Congress the Nation has lost President Harding.',\n",
       "  '2.',\n",
       "  'Mr. Ehrman to Mr. Hay.',\n",
       "  'Think of that!',\n",
       "  '2.',\n",
       "  'Let them come to Berlin.',\n",
       "  'THE PRESIDENT.',\n",
       "  'I shall keep that promise.',\n",
       "  'Mr. Chief Justice, my dear friends, my fellow Americans: The oath that I have taken is the same oath that was taken by George Washington and by every President under the Constitution.',\n",
       "  'THE PRESIDENT.',\n",
       "  'The President.',\n",
       "  'Mr. Lehrer.',\n",
       "  '[ Applause ] Thank you.',\n",
       "  '( Laughter. )',\n",
       "  '( Laughter. )',\n",
       "  'Thank you.'],\n",
       " [1,\n",
       "  3,\n",
       "  1,\n",
       "  3,\n",
       "  1,\n",
       "  1,\n",
       "  2,\n",
       "  3,\n",
       "  1,\n",
       "  17,\n",
       "  2,\n",
       "  1,\n",
       "  2,\n",
       "  2,\n",
       "  2,\n",
       "  4,\n",
       "  10,\n",
       "  11,\n",
       "  11,\n",
       "  1,\n",
       "  2,\n",
       "  22,\n",
       "  3,\n",
       "  2,\n",
       "  4,\n",
       "  3,\n",
       "  2,\n",
       "  2,\n",
       "  1,\n",
       "  4,\n",
       "  11,\n",
       "  2,\n",
       "  3,\n",
       "  4,\n",
       "  413,\n",
       "  2,\n",
       "  1,\n",
       "  112,\n",
       "  49,\n",
       "  54,\n",
       "  15,\n",
       "  32,\n",
       "  102,\n",
       "  65])"
      ]
     },
     "execution_count": 807,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_common_sent=[]\n",
    "frequency_sent = []\n",
    "for i in range(corpus_df.shape[0]):\n",
    "    speeches = (corpus_df.transcripts.iloc[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    most_common_sent.append(max_frequency(sentences)[0])\n",
    "    frequency_sent.append(max_frequency(sentences)[1])\n",
    "most_common_sent, frequency_sent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32fc690f",
   "metadata": {
    "id": "32fc690f",
    "outputId": "278f8312-b17a-4711-8a20-736a2be06969"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>characters</th>\n",
       "      <th>total_words</th>\n",
       "      <th>len_sentences</th>\n",
       "      <th>unique_sentences</th>\n",
       "      <th>most_spoken_sentence</th>\n",
       "      <th>freqency_max_sent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>190015</td>\n",
       "      <td>31502</td>\n",
       "      <td>786</td>\n",
       "      <td>786</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>88556</td>\n",
       "      <td>14630</td>\n",
       "      <td>363</td>\n",
       "      <td>361</td>\n",
       "      <td>JOHN ADAMS.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>211059</td>\n",
       "      <td>35823</td>\n",
       "      <td>938</td>\n",
       "      <td>938</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>216989</td>\n",
       "      <td>35991</td>\n",
       "      <td>711</td>\n",
       "      <td>708</td>\n",
       "      <td>JAMES MADISON.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>297306</td>\n",
       "      <td>49931</td>\n",
       "      <td>1385</td>\n",
       "      <td>1385</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>John Quincy Adams</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...</td>\n",
       "      <td>222338</td>\n",
       "      <td>36522</td>\n",
       "      <td>878</td>\n",
       "      <td>878</td>\n",
       "      <td>AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Andrew Jackson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: About to undertake the arduou...</td>\n",
       "      <td>933812</td>\n",
       "      <td>157443</td>\n",
       "      <td>3700</td>\n",
       "      <td>3693</td>\n",
       "      <td>As, then, the period approaches when the appli...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Martin Van Buren</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: The practice of all my predec...</td>\n",
       "      <td>389851</td>\n",
       "      <td>64664</td>\n",
       "      <td>1530</td>\n",
       "      <td>1528</td>\n",
       "      <td>M. VAN BUREN.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>William Harrison</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Called from a retirement which I had supposed ...</td>\n",
       "      <td>49669</td>\n",
       "      <td>8461</td>\n",
       "      <td>206</td>\n",
       "      <td>206</td>\n",
       "      <td>Called from a retirement which I had supposed ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>John Tyler</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>To the People of the United States Before my a...</td>\n",
       "      <td>411510</td>\n",
       "      <td>69398</td>\n",
       "      <td>1792</td>\n",
       "      <td>1772</td>\n",
       "      <td>No.</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>James K. Polk</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow citizens: Without solicitation on my pa...</td>\n",
       "      <td>620263</td>\n",
       "      <td>104139</td>\n",
       "      <td>2692</td>\n",
       "      <td>2681</td>\n",
       "      <td>JAMES K. POLK.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Zachary Taylor</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Elected by the American people to the highest ...</td>\n",
       "      <td>68280</td>\n",
       "      <td>11350</td>\n",
       "      <td>275</td>\n",
       "      <td>275</td>\n",
       "      <td>Elected by the American people to the highest ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Millard Fillmore</td>\n",
       "      <td>Whig</td>\n",
       "      <td>To the Senate and House of Representatives: I ...</td>\n",
       "      <td>234451</td>\n",
       "      <td>38775</td>\n",
       "      <td>1008</td>\n",
       "      <td>1007</td>\n",
       "      <td>Second.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Franklin Pierce</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My Countrymen: It a relief to feel that no hea...</td>\n",
       "      <td>385312</td>\n",
       "      <td>63219</td>\n",
       "      <td>1464</td>\n",
       "      <td>1463</td>\n",
       "      <td>FRANKLIN PIERCE.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>James Buchanan</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: I appear before you this day ...</td>\n",
       "      <td>485593</td>\n",
       "      <td>80553</td>\n",
       "      <td>2459</td>\n",
       "      <td>2446</td>\n",
       "      <td>When this fails it can proceed no further.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Abraham Lincoln</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Honors To Henry ClayOn the fourth day of July,...</td>\n",
       "      <td>597666</td>\n",
       "      <td>102775</td>\n",
       "      <td>3951</td>\n",
       "      <td>2934</td>\n",
       "      <td>We deny it.</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Andrew Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I must be permitted to say that I have been al...</td>\n",
       "      <td>569655</td>\n",
       "      <td>94785</td>\n",
       "      <td>2532</td>\n",
       "      <td>2469</td>\n",
       "      <td>ANDREW JOHNSON.</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ulysses S. Grant</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Citizens of the United States: Your suffrages ...</td>\n",
       "      <td>621306</td>\n",
       "      <td>102702</td>\n",
       "      <td>2870</td>\n",
       "      <td>2798</td>\n",
       "      <td>U. S. GRANT.</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Rutherford B. Hayes</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We have assembled to repeat t...</td>\n",
       "      <td>410483</td>\n",
       "      <td>67144</td>\n",
       "      <td>1865</td>\n",
       "      <td>1837</td>\n",
       "      <td>SEC.</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>James A. Garfield</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We stand to-day upon an emine...</td>\n",
       "      <td>17722</td>\n",
       "      <td>2975</td>\n",
       "      <td>108</td>\n",
       "      <td>108</td>\n",
       "      <td>Fellow Citizens: We stand to-day upon an emine...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chester A. Arthur</td>\n",
       "      <td>Republican</td>\n",
       "      <td>For the fourth time in the history of the Repu...</td>\n",
       "      <td>305870</td>\n",
       "      <td>49331</td>\n",
       "      <td>1540</td>\n",
       "      <td>1539</td>\n",
       "      <td>Second.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Grover Cleveland</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: In the presence of this vast ...</td>\n",
       "      <td>919540</td>\n",
       "      <td>148480</td>\n",
       "      <td>4202</td>\n",
       "      <td>4029</td>\n",
       "      <td>2.</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Benjamin Harrison</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: There is no constitutional or...</td>\n",
       "      <td>836204</td>\n",
       "      <td>138208</td>\n",
       "      <td>4189</td>\n",
       "      <td>2522</td>\n",
       "      <td>BENJ.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>William McKinley</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: In obedience to the will of t...</td>\n",
       "      <td>566308</td>\n",
       "      <td>91657</td>\n",
       "      <td>2841</td>\n",
       "      <td>2825</td>\n",
       "      <td>This is an obvious duty.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Franklin D. Roosevelt</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>President Hoover, Mr. Chief Justice, my friend...</td>\n",
       "      <td>756190</td>\n",
       "      <td>131859</td>\n",
       "      <td>5519</td>\n",
       "      <td>5508</td>\n",
       "      <td>No.</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>William Taft</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Citizens: Anyone who has taken the o...</td>\n",
       "      <td>709967</td>\n",
       "      <td>115340</td>\n",
       "      <td>3384</td>\n",
       "      <td>3292</td>\n",
       "      <td>Second.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Woodrow Wilson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>There has been a change of government. It bega...</td>\n",
       "      <td>490304</td>\n",
       "      <td>85367</td>\n",
       "      <td>2691</td>\n",
       "      <td>2508</td>\n",
       "      <td>I shall seek, as I have always sought, to just...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Warren G. Harding</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My countrymen, the surpassing war of all times...</td>\n",
       "      <td>174762</td>\n",
       "      <td>28863</td>\n",
       "      <td>1189</td>\n",
       "      <td>1161</td>\n",
       "      <td>Republics have risen and fallen, and a transit...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Calvin Coolidge</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Since the close of the last Congress the Natio...</td>\n",
       "      <td>452057</td>\n",
       "      <td>73692</td>\n",
       "      <td>3382</td>\n",
       "      <td>3382</td>\n",
       "      <td>Since the close of the last Congress the Natio...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Herbert Hoover</td>\n",
       "      <td>Republican</td>\n",
       "      <td>This campaign now draws near a close. The plat...</td>\n",
       "      <td>546456</td>\n",
       "      <td>88822</td>\n",
       "      <td>3521</td>\n",
       "      <td>3468</td>\n",
       "      <td>2.</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Theodore Roosevelt</td>\n",
       "      <td>Republican</td>\n",
       "      <td>By the President of the United States of Ameri...</td>\n",
       "      <td>1166892</td>\n",
       "      <td>196292</td>\n",
       "      <td>6457</td>\n",
       "      <td>6292</td>\n",
       "      <td>Mr. Ehrman to Mr. Hay.</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Harry S. Truman</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Mr. Speaker, Mr. President, Members of the Con...</td>\n",
       "      <td>207143</td>\n",
       "      <td>36885</td>\n",
       "      <td>1917</td>\n",
       "      <td>1916</td>\n",
       "      <td>Think of that!</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Dwight D. Eisenhower</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My friends, before I begin the expression of t...</td>\n",
       "      <td>105898</td>\n",
       "      <td>18045</td>\n",
       "      <td>809</td>\n",
       "      <td>799</td>\n",
       "      <td>2.</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>John F. Kennedy</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Governor Stevenson, Senator Johnson, Mr. Butle...</td>\n",
       "      <td>698750</td>\n",
       "      <td>121563</td>\n",
       "      <td>4768</td>\n",
       "      <td>4630</td>\n",
       "      <td>Let them come to Berlin.</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Lyndon B. Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>On this hallowed ground, heroic deeds were per...</td>\n",
       "      <td>1421446</td>\n",
       "      <td>249732</td>\n",
       "      <td>12715</td>\n",
       "      <td>10372</td>\n",
       "      <td>THE PRESIDENT.</td>\n",
       "      <td>413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Richard M. Nixon</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Americans: I come before you tonight...</td>\n",
       "      <td>382082</td>\n",
       "      <td>67768</td>\n",
       "      <td>2802</td>\n",
       "      <td>2801</td>\n",
       "      <td>I shall keep that promise.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Gerald Ford</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Mr. Chief Justice, my dear friends, my fellow ...</td>\n",
       "      <td>240004</td>\n",
       "      <td>40397</td>\n",
       "      <td>2023</td>\n",
       "      <td>2023</td>\n",
       "      <td>Mr. Chief Justice, my dear friends, my fellow ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I am Edwin Newman, moderator of this first deb...</td>\n",
       "      <td>588246</td>\n",
       "      <td>101421</td>\n",
       "      <td>5345</td>\n",
       "      <td>4846</td>\n",
       "      <td>THE PRESIDENT.</td>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Ronald Reagan</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Thank you. Thank you very much. Thank you and ...</td>\n",
       "      <td>1188847</td>\n",
       "      <td>208007</td>\n",
       "      <td>10300</td>\n",
       "      <td>9717</td>\n",
       "      <td>The President.</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>I have many friends to thank tonight. I thank ...</td>\n",
       "      <td>512239</td>\n",
       "      <td>91350</td>\n",
       "      <td>5536</td>\n",
       "      <td>5008</td>\n",
       "      <td>Mr. Lehrer.</td>\n",
       "      <td>54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Bill Clinton</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My fellow citizens, today we celebrate the mys...</td>\n",
       "      <td>768815</td>\n",
       "      <td>136804</td>\n",
       "      <td>6695</td>\n",
       "      <td>6630</td>\n",
       "      <td>[ Applause ] Thank you.</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>President Clinton, distinguished guests and my...</td>\n",
       "      <td>630362</td>\n",
       "      <td>108260</td>\n",
       "      <td>5939</td>\n",
       "      <td>5880</td>\n",
       "      <td>( Laughter. )</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>To Chairman Dean and my great friend Dick Durb...</td>\n",
       "      <td>1155148</td>\n",
       "      <td>206788</td>\n",
       "      <td>10672</td>\n",
       "      <td>10386</td>\n",
       "      <td>( Laughter. )</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Chief Justice Roberts, President Carter, Presi...</td>\n",
       "      <td>475677</td>\n",
       "      <td>84848</td>\n",
       "      <td>6276</td>\n",
       "      <td>5962</td>\n",
       "      <td>Thank you.</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                President                  Party  \\\n",
       "0       George Washington           Unaffiliated   \n",
       "1              John Adams             Federalist   \n",
       "2        Thomas Jefferson  Democratic-Republican   \n",
       "3           James Madison  Democratic-Republican   \n",
       "4            James Monroe  Democratic-Republican   \n",
       "5       John Quincy Adams  Democratic-Republican   \n",
       "6          Andrew Jackson             Democratic   \n",
       "7        Martin Van Buren             Democratic   \n",
       "8        William Harrison                   Whig   \n",
       "9              John Tyler           Unaffiliated   \n",
       "10          James K. Polk             Democratic   \n",
       "11         Zachary Taylor                   Whig   \n",
       "12       Millard Fillmore                   Whig   \n",
       "13        Franklin Pierce             Democratic   \n",
       "14         James Buchanan             Democratic   \n",
       "15        Abraham Lincoln             Republican   \n",
       "16         Andrew Johnson             Democratic   \n",
       "17       Ulysses S. Grant             Republican   \n",
       "18    Rutherford B. Hayes             Republican   \n",
       "19      James A. Garfield             Republican   \n",
       "20      Chester A. Arthur             Republican   \n",
       "21       Grover Cleveland             Democratic   \n",
       "22      Benjamin Harrison             Republican   \n",
       "23       William McKinley             Republican   \n",
       "24  Franklin D. Roosevelt             Democratic   \n",
       "25           William Taft             Republican   \n",
       "26         Woodrow Wilson             Democratic   \n",
       "27      Warren G. Harding             Republican   \n",
       "28        Calvin Coolidge             Republican   \n",
       "29         Herbert Hoover             Republican   \n",
       "30     Theodore Roosevelt             Republican   \n",
       "31        Harry S. Truman             Democratic   \n",
       "32   Dwight D. Eisenhower             Republican   \n",
       "33        John F. Kennedy             Democratic   \n",
       "34      Lyndon B. Johnson             Democratic   \n",
       "35       Richard M. Nixon             Republican   \n",
       "36            Gerald Ford             Republican   \n",
       "37           Jimmy Carter             Democratic   \n",
       "38          Ronald Reagan             Republican   \n",
       "39      George H. W. Bush             Republican   \n",
       "40           Bill Clinton             Democratic   \n",
       "41         George W. Bush             Republican   \n",
       "42           Barack Obama             Democratic   \n",
       "43           Donald Trump             Republican   \n",
       "\n",
       "                                          transcripts  characters  \\\n",
       "0   Fellow Citizens of the Senate and the House of...      190015   \n",
       "1   When it was first perceived, in early times, t...       88556   \n",
       "2   FRIENDS AND FELLOW-CITIZENS, Called upon to un...      211059   \n",
       "3   Unwilling to depart from examples of the most ...      216989   \n",
       "4   I should be destitute of feeling if I was not ...      297306   \n",
       "5   AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...      222338   \n",
       "6   Fellow Citizens: About to undertake the arduou...      933812   \n",
       "7   Fellow Citizens: The practice of all my predec...      389851   \n",
       "8   Called from a retirement which I had supposed ...       49669   \n",
       "9   To the People of the United States Before my a...      411510   \n",
       "10  Fellow citizens: Without solicitation on my pa...      620263   \n",
       "11  Elected by the American people to the highest ...       68280   \n",
       "12  To the Senate and House of Representatives: I ...      234451   \n",
       "13  My Countrymen: It a relief to feel that no hea...      385312   \n",
       "14  Fellow Citizens: I appear before you this day ...      485593   \n",
       "15  Honors To Henry ClayOn the fourth day of July,...      597666   \n",
       "16  I must be permitted to say that I have been al...      569655   \n",
       "17  Citizens of the United States: Your suffrages ...      621306   \n",
       "18  Fellow Citizens: We have assembled to repeat t...      410483   \n",
       "19  Fellow Citizens: We stand to-day upon an emine...       17722   \n",
       "20  For the fourth time in the history of the Repu...      305870   \n",
       "21  Fellow Citizens: In the presence of this vast ...      919540   \n",
       "22  Fellow Citizens: There is no constitutional or...      836204   \n",
       "23  Fellow Citizens: In obedience to the will of t...      566308   \n",
       "24  President Hoover, Mr. Chief Justice, my friend...      756190   \n",
       "25  My Fellow Citizens: Anyone who has taken the o...      709967   \n",
       "26  There has been a change of government. It bega...      490304   \n",
       "27  My countrymen, the surpassing war of all times...      174762   \n",
       "28  Since the close of the last Congress the Natio...      452057   \n",
       "29  This campaign now draws near a close. The plat...      546456   \n",
       "30  By the President of the United States of Ameri...     1166892   \n",
       "31  Mr. Speaker, Mr. President, Members of the Con...      207143   \n",
       "32  My friends, before I begin the expression of t...      105898   \n",
       "33  Governor Stevenson, Senator Johnson, Mr. Butle...      698750   \n",
       "34  On this hallowed ground, heroic deeds were per...     1421446   \n",
       "35  My Fellow Americans: I come before you tonight...      382082   \n",
       "36  Mr. Chief Justice, my dear friends, my fellow ...      240004   \n",
       "37  I am Edwin Newman, moderator of this first deb...      588246   \n",
       "38  Thank you. Thank you very much. Thank you and ...     1188847   \n",
       "39  I have many friends to thank tonight. I thank ...      512239   \n",
       "40  My fellow citizens, today we celebrate the mys...      768815   \n",
       "41  President Clinton, distinguished guests and my...      630362   \n",
       "42  To Chairman Dean and my great friend Dick Durb...     1155148   \n",
       "43  Chief Justice Roberts, President Carter, Presi...      475677   \n",
       "\n",
       "    total_words  len_sentences  unique_sentences  \\\n",
       "0         31502            786               786   \n",
       "1         14630            363               361   \n",
       "2         35823            938               938   \n",
       "3         35991            711               708   \n",
       "4         49931           1385              1385   \n",
       "5         36522            878               878   \n",
       "6        157443           3700              3693   \n",
       "7         64664           1530              1528   \n",
       "8          8461            206               206   \n",
       "9         69398           1792              1772   \n",
       "10       104139           2692              2681   \n",
       "11        11350            275               275   \n",
       "12        38775           1008              1007   \n",
       "13        63219           1464              1463   \n",
       "14        80553           2459              2446   \n",
       "15       102775           3951              2934   \n",
       "16        94785           2532              2469   \n",
       "17       102702           2870              2798   \n",
       "18        67144           1865              1837   \n",
       "19         2975            108               108   \n",
       "20        49331           1540              1539   \n",
       "21       148480           4202              4029   \n",
       "22       138208           4189              2522   \n",
       "23        91657           2841              2825   \n",
       "24       131859           5519              5508   \n",
       "25       115340           3384              3292   \n",
       "26        85367           2691              2508   \n",
       "27        28863           1189              1161   \n",
       "28        73692           3382              3382   \n",
       "29        88822           3521              3468   \n",
       "30       196292           6457              6292   \n",
       "31        36885           1917              1916   \n",
       "32        18045            809               799   \n",
       "33       121563           4768              4630   \n",
       "34       249732          12715             10372   \n",
       "35        67768           2802              2801   \n",
       "36        40397           2023              2023   \n",
       "37       101421           5345              4846   \n",
       "38       208007          10300              9717   \n",
       "39        91350           5536              5008   \n",
       "40       136804           6695              6630   \n",
       "41       108260           5939              5880   \n",
       "42       206788          10672             10386   \n",
       "43        84848           6276              5962   \n",
       "\n",
       "                                 most_spoken_sentence  freqency_max_sent  \n",
       "0   Fellow Citizens of the Senate and the House of...                  1  \n",
       "1                                         JOHN ADAMS.                  3  \n",
       "2   FRIENDS AND FELLOW-CITIZENS, Called upon to un...                  1  \n",
       "3                                      JAMES MADISON.                  3  \n",
       "4   I should be destitute of feeling if I was not ...                  1  \n",
       "5   AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...                  1  \n",
       "6   As, then, the period approaches when the appli...                  2  \n",
       "7                                       M. VAN BUREN.                  3  \n",
       "8   Called from a retirement which I had supposed ...                  1  \n",
       "9                                                 No.                 17  \n",
       "10                                     JAMES K. POLK.                  2  \n",
       "11  Elected by the American people to the highest ...                  1  \n",
       "12                                            Second.                  2  \n",
       "13                                   FRANKLIN PIERCE.                  2  \n",
       "14         When this fails it can proceed no further.                  2  \n",
       "15                                        We deny it.                  4  \n",
       "16                                    ANDREW JOHNSON.                 10  \n",
       "17                                       U. S. GRANT.                 11  \n",
       "18                                               SEC.                 11  \n",
       "19  Fellow Citizens: We stand to-day upon an emine...                  1  \n",
       "20                                            Second.                  2  \n",
       "21                                                 2.                 22  \n",
       "22                                              BENJ.                  3  \n",
       "23                           This is an obvious duty.                  2  \n",
       "24                                                No.                  4  \n",
       "25                                            Second.                  3  \n",
       "26  I shall seek, as I have always sought, to just...                  2  \n",
       "27  Republics have risen and fallen, and a transit...                  2  \n",
       "28  Since the close of the last Congress the Natio...                  1  \n",
       "29                                                 2.                  4  \n",
       "30                             Mr. Ehrman to Mr. Hay.                 11  \n",
       "31                                     Think of that!                  2  \n",
       "32                                                 2.                  3  \n",
       "33                           Let them come to Berlin.                  4  \n",
       "34                                     THE PRESIDENT.                413  \n",
       "35                         I shall keep that promise.                  2  \n",
       "36  Mr. Chief Justice, my dear friends, my fellow ...                  1  \n",
       "37                                     THE PRESIDENT.                112  \n",
       "38                                     The President.                 49  \n",
       "39                                        Mr. Lehrer.                 54  \n",
       "40                            [ Applause ] Thank you.                 15  \n",
       "41                                      ( Laughter. )                 32  \n",
       "42                                      ( Laughter. )                102  \n",
       "43                                         Thank you.                 65  "
      ]
     },
     "execution_count": 808,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df[\"most_spoken_sentence\"] = most_common_sent \n",
    "corpus_df[\"freqency_max_sent\"] = frequency_sent\n",
    "corpus_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7137e32b",
   "metadata": {
    "id": "7137e32b"
   },
   "source": [
    "<a name =stepiv></a>\n",
    "    \n",
    "# Step iv:  Data Cleaning\n",
    "Based on findings in the Data Exploration we will remove:\n",
    "\n",
    "- Laughter, Applause\n",
    "- The names of the Presidents at the end of their speech\n",
    "- number indices (bullet) points. e.g. 2.\n",
    "- The President / THE PRESIDENT\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ca1131e",
   "metadata": {
    "id": "8ca1131e"
   },
   "source": [
    "<a name = \"further_cleaning\"></a>\n",
    "### The following will be removed from the Speaches: \n",
    "\n",
    "- Square brackets and Parenthsis and everything inside them.\n",
    "    - [ Laughter ], \n",
    "    - [ Laughter. ] \n",
    "    - [ Applause ] \n",
    "    - [ applause ]\n",
    "\n",
    "\n",
    "- Removing ALL WORDS IN ALL CAPS:\n",
    "    - The following names: written in ALL CAPS \"JOHN ADAMS\", \"JAMES MADISON\" etc.\n",
    "    - will lose some relevant data like: \"FRIENDS AND FELLOW-CITIZENS\" by Thomas Jefferson and AND NOW, FRIENDS AND COUNTRYMEN by John Quincy Adams\n",
    "    \n",
    "\n",
    "- Will **not** remove:\n",
    "   - \"Second.\" is valid language usage by Taft and Chester A Arthur\n",
    "   - Grover Clevland used \"2.\" a lot - in itemizing and references to money. A lot of his figures ended in 20cents! The simple_tokenization by gensim being used removes numbers.\n",
    "   \n",
    "\n",
    "### Further Cleaning is required: \n",
    "   - Stating the date and location speech was given. Often preceeded by \"JOHN ADAMS\" -type terms e.g.\n",
    "        - \"Given at Washington, this 23d day of July, A. D. 1813. \"\n",
    "        - \"Done at the city of Washington, the 1st day of September, A.D. 1814, and of the Independence of the United States the thirty ninth.\"\n",
    "   - There are telegrams within Theodore Rooselvelt's Speech that need to be removed.\n",
    "   - There may be similar correspondence like this in other copora. Further review required. \n",
    "   - I didn't take out proper nouns because of nature of text. But may need to reconsider."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "OI6x6StJmEd1",
   "metadata": {
    "id": "OI6x6StJmEd1"
   },
   "source": [
    "The following preprocessing function was used to create the clean function in the [Number of words](#words) section and the preprocessing in the [normalize text](#normalize) section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "XlsrFtLElo0C",
   "metadata": {
    "id": "XlsrFtLElo0C"
   },
   "outputs": [],
   "source": [
    "def clean_sent(doc):\n",
    "    # Tokenize, clean up input document string\n",
    "    doc = re.sub(r'\\[[^()]*\\]',\"\", doc)#removes [Applause][\"Laughter\"]\n",
    "    doc = re.sub(r'\\([^()]*\\)',\"\", doc)#removes (Applause)(\"Laughter\")\n",
    "    doc = re.sub(r'[^\\w\\s]', \"\", doc)# removes punctuation\n",
    "    doc = re.sub(r'\\b[A-Z][A-Z0-9]+\\b',\"\",doc) #remove ALL CAPS\n",
    "    doc=\" \".join(doc.split()) #remove double whitespace\n",
    "    return doc.lower()\n",
    "    #return [token for token in simple_preprocess(doc, min_len=0, max_len=float(\"inf\"))] #used for Gensim model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "TYIgY7YDsLwe",
   "metadata": {
    "id": "TYIgY7YDsLwe"
   },
   "outputs": [],
   "source": [
    "#remove stop words\n",
    "stopwords = stops_used(8500)\n",
    "\n",
    "def remove_stop_words(doc):\n",
    "    removed_stop_words = []\n",
    "    for tok_sent in doc:\n",
    "        removed_stop_words.append(\n",
    "            ' '.join([word for word in tok_sent.split() \n",
    "                      if word not in stopwords])\n",
    "        )\n",
    "    return removed_stop_words\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "QVe2yd1jEUpD",
   "metadata": {
    "id": "QVe2yd1jEUpD"
   },
   "outputs": [],
   "source": [
    "def get_stemmed_text(doc):\n",
    "    from nltk.stem.porter import PorterStemmer\n",
    "    stemmer = PorterStemmer()\n",
    "    return [' '.join([stemmer.stem(word) for word in tok_sent.split()]) for tok_sent in doc]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "AxVQtaoZD2zl",
   "metadata": {
    "id": "AxVQtaoZD2zl"
   },
   "outputs": [],
   "source": [
    "def get_lemmatized_text(doc):\n",
    "    from nltk.stem import WordNetLemmatizer\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    return [' '.join([lemmatizer.lemmatize(word) for word in tok_sent.split()]) for tok_sent in doc]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "NsXNmpbcD6GX",
   "metadata": {
    "id": "NsXNmpbcD6GX"
   },
   "source": [
    "#### Testing Text Processing and Normalization\n",
    "- did not do stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0k4ZGX3TPFjF",
   "metadata": {
    "id": "0k4ZGX3TPFjF"
   },
   "outputs": [],
   "source": [
    "# sentence tokenize and remove empty tokens\n",
    "speeches=(subset_df.transcripts.iloc[0])\n",
    "sentences = sent_tokenize(speeches)\n",
    "cleaned_speeches = [clean_sent(speech) for speech in sentences]\n",
    "cleanup_sentences = [ sent for sent in cleaned_speeches if sent ]\n",
    "empty_sentences = [ sent for sent in cleaned_speeches if not sent ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 383,
   "id": "ITTvh43hxOHd",
   "metadata": {
    "id": "ITTvh43hxOHd"
   },
   "outputs": [],
   "source": [
    "#stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "id": "sn9E1TzivRHn",
   "metadata": {
    "id": "sn9E1TzivRHn"
   },
   "outputs": [],
   "source": [
    "#\"stops_used(6000)\"\n",
    "#len(cleanup_sentences[7]), cleanup_sentences[7] ,len(cleanup_sentences[55]), cleanup_sentences[55] ,len(cleanup_sentences[100]), cleanup_sentences[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 385,
   "id": "FUmiEOQ8-tq5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FUmiEOQ8-tq5",
    "outputId": "750ef81a-a7a7-4cb7-e76f-47a59d796d42"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(142,\n",
       " 'no people can be bound to acknowledge and adore the invisible hand which conducts the affairs of men more than the people of the united states',\n",
       " 339,\n",
       " 'since your last sessions i have received communications by which it appears that the district of kentucky at present a part of virginia has concurred in certain propositions contained in a law of that state in consequence of which the district is to become a distinct member of the union in case the requisite sanction of congress be added',\n",
       " 111,\n",
       " 'but it does not appear from any proofs yet in the possession of government that oliver phelps has defrauded you')"
      ]
     },
     "execution_count": 385,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\"stops_used(10000)\"\n",
    "len(cleanup_sentences[7]), cleanup_sentences[7] ,len(cleanup_sentences[55]), cleanup_sentences[55] ,len(cleanup_sentences[100]), cleanup_sentences[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 386,
   "id": "r1lkyNs6uk2V",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "r1lkyNs6uk2V",
    "outputId": "861b03a6-cae9-43be-c41f-b5aac9e54f8c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(64,\n",
       " 'bound acknowledge adore invisible hand conducts affairs men than',\n",
       " 236,\n",
       " 'since your last sessions received communications appears district kentucky present part virginia concurred certain propositions contained law state consequence district become distinct member union case requisite sanction congress added',\n",
       " 61,\n",
       " 'does appear any proofs yet possession oliver phelps defrauded')"
      ]
     },
     "execution_count": 386,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\"stops_used(6000)\"\n",
    "#no_stop_words = remove_stop_words(cleanup_sentences)\n",
    "len(no_stop_words[7]), no_stop_words[7],len(no_stop_words[55]), no_stop_words[55],len(no_stop_words[100]), no_stop_words[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 387,
   "id": "pGAW2shq-9R1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pGAW2shq-9R1",
    "outputId": "40070b67-b489-4160-94cc-6fb8546ecf53"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(64,\n",
       " 'bound acknowledge adore invisible hand conducts affairs men than',\n",
       " 236,\n",
       " 'since your last sessions received communications appears district kentucky present part virginia concurred certain propositions contained law state consequence district become distinct member union case requisite sanction congress added',\n",
       " 61,\n",
       " 'does appear any proofs yet possession oliver phelps defrauded')"
      ]
     },
     "execution_count": 387,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\"stops_used(10000)\"\n",
    "no_stop_words = remove_stop_words(cleanup_sentences)\n",
    "len(no_stop_words[7]), no_stop_words[7],len(no_stop_words[55]), no_stop_words[55],len(no_stop_words[100]), no_stop_words[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 388,
   "id": "c4pAeJXwtDw5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c4pAeJXwtDw5",
    "outputId": "5fb046af-a3a3-4f6c-f26a-2a0f889ffa76"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56,\n",
       " 'bound acknowledg ador invis hand conduct affair men than',\n",
       " 205,\n",
       " 'sinc your last session receiv commun appear district kentucki present part virginia concur certain proposit contain law state consequ district becom distinct member union case requisit sanction congress ad',\n",
       " 51,\n",
       " 'doe appear ani proof yet possess oliv phelp defraud')"
      ]
     },
     "execution_count": 388,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#8000\n",
    "stemmed_speech = get_stemmed_text(no_stop_words)\n",
    "len(stemmed_speech[7]), stemmed_speech[7],len(stemmed_speech[55]), stemmed_speech[55],len(stemmed_speech[100]), stemmed_speech[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 389,
   "id": "HoWANHQY2RAO",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HoWANHQY2RAO",
    "outputId": "834d5d17-19ac-4efb-b21a-d5b7b34cb99a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(62,\n",
       " 'bound acknowledge adore invisible hand conduct affair men than',\n",
       " 233,\n",
       " 'since your last session received communication appears district kentucky present part virginia concurred certain proposition contained law state consequence district become distinct member union case requisite sanction congress added',\n",
       " 59,\n",
       " 'doe appear any proof yet possession oliver phelps defrauded')"
      ]
     },
     "execution_count": 389,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\"stops_used(6000)\"\n",
    "#lemmatized_speech = get_lemmatized_text(no_stop_words)\n",
    "len(lemmatized_speech[7]),lemmatized_speech[7],len(lemmatized_speech[55]),lemmatized_speech[55],len(lemmatized_speech[100]),lemmatized_speech[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 390,
   "id": "o6LXzXlW_cBV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "o6LXzXlW_cBV",
    "outputId": "65f91a90-26fc-451a-fe6d-cf3145fac62a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56,\n",
       " 'bound acknowledg ador invis hand conduct affair men than',\n",
       " 205,\n",
       " 'sinc your last session receiv commun appear district kentucki present part virginia concur certain proposit contain law state consequ district becom distinct member union case requisit sanction congress ad',\n",
       " 49,\n",
       " 'doe appear ani proof yet posse oliv phelp defraud')"
      ]
     },
     "execution_count": 390,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\"stops_used(8000)\"\n",
    "lemmatized_speech = get_lemmatized_text(stemmed_speech)\n",
    "len(lemmatized_speech[7]),lemmatized_speech[7],len(lemmatized_speech[55]),lemmatized_speech[55],len(lemmatized_speech[100]),lemmatized_speech[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 391,
   "id": "-jV8JxWTmTtg",
   "metadata": {
    "id": "-jV8JxWTmTtg"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "id": "j8CtAPRin8JE",
   "metadata": {
    "id": "j8CtAPRin8JE"
   },
   "outputs": [],
   "source": [
    "#cleanup_sentences\n",
    "#empty_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 394,
   "id": "EeneMIQPtvHR",
   "metadata": {
    "id": "EeneMIQPtvHR"
   },
   "outputs": [],
   "source": [
    "# remove stop words\n",
    "#no_stop_words = remove_stop_words(cleanup_sentences)\n",
    "# lemmatization\n",
    "#lemmatized_speech = get_lemmatized_text(no_stop_words)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fOGR4adEDQrJ",
   "metadata": {
    "id": "fOGR4adEDQrJ"
   },
   "source": [
    "#### Choosing initial stopwords for sentenceBERT as those appearing >8000. times in the corpora. Will apply lemmatzation (not stemming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "M8SsC_vXm5tg",
   "metadata": {
    "id": "M8SsC_vXm5tg"
   },
   "outputs": [],
   "source": [
    "transcripts_new=[]\n",
    "for i in range(corpus_df.shape[0]): #corpus_df.shape[0]\n",
    "    speeches=(corpus_df.transcripts.iloc[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    unique_sentences = list(dict.fromkeys(sentences))\n",
    "    cleaned_speeches = [clean_sent(speech) for speech in unique_sentences]\n",
    "    cleanup_sentences = [sent for sent in cleaned_speeches if sent] #remove empty sentences\n",
    "    no_stop_words=remove_stop_words(cleanup_sentences)\n",
    "    lemmatized_speech = get_lemmatized_text(no_stop_words)\n",
    "    transcripts_new.append(lemmatized_speech)\n",
    "transcripts_new\n",
    "corpus_df[\"transcripts_new\"]=transcripts_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c88e96",
   "metadata": {
    "id": "b4c88e96"
   },
   "outputs": [],
   "source": [
    "#corpus_df.transcripts.iloc[1] #JOHN ADAMS\n",
    "#corpus_df.transcripts.iloc[21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "5f7d330d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "5f7d330d",
    "outputId": "2129ff6e-2471-4a59-bc3c-d2fc96c73633",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-92a7c28a-7b2e-4895-ad89-d823952702c4\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Party</th>\n",
       "      <th>transcripts</th>\n",
       "      <th>Cleaned Words</th>\n",
       "      <th>transcripts_new</th>\n",
       "      <th>Unique Sentences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Fellow Citizens of the Senate and the House of...</td>\n",
       "      <td>{'the': 2698, 'of': 1883, 'to': 1328, 'and': 1...</td>\n",
       "      <td>[fellow citizen senate house representative am...</td>\n",
       "      <td>[Fellow Citizens of the Senate and the House o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>Federalist</td>\n",
       "      <td>When it was first perceived, in early times, t...</td>\n",
       "      <td>{'the': 1244, 'of': 914, 'and': 640, 'to': 553...</td>\n",
       "      <td>[when first perceived early time middle course...</td>\n",
       "      <td>[When it was first perceived, in early times, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>FRIENDS AND FELLOW-CITIZENS, Called upon to un...</td>\n",
       "      <td>{'the': 2663, 'of': 1940, 'to': 1389, 'and': 1...</td>\n",
       "      <td>[called upon undertake duty first executive of...</td>\n",
       "      <td>[FRIENDS AND FELLOW-CITIZENS, Called upon to u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>Unwilling to depart from examples of the most ...</td>\n",
       "      <td>{'the': 3613, 'of': 2308, 'and': 1275, 'to': 1...</td>\n",
       "      <td>[unwilling depart example most revered authori...</td>\n",
       "      <td>[Unwilling to depart from examples of the most...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>I should be destitute of feeling if I was not ...</td>\n",
       "      <td>{'the': 4688, 'of': 2709, 'to': 1890, 'and': 1...</td>\n",
       "      <td>[should destitute feeling if deeply affected s...</td>\n",
       "      <td>[I should be destitute of feeling if I was not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>John Quincy Adams</td>\n",
       "      <td>Democratic-Republican</td>\n",
       "      <td>AND NOW, FRIENDS AND COUNTRYMEN, if the wise a...</td>\n",
       "      <td>{'the': 4010, 'of': 3109, 'to': 1244, 'and': 1...</td>\n",
       "      <td>[if wise learned philosopher elder world first...</td>\n",
       "      <td>[AND NOW, FRIENDS AND COUNTRYMEN, if the wise ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Andrew Jackson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: About to undertake the arduou...</td>\n",
       "      <td>{'the': 15572, 'of': 9439, 'to': 6253, 'and': ...</td>\n",
       "      <td>[fellow citizen about undertake arduous duty a...</td>\n",
       "      <td>[Fellow Citizens: About to undertake the arduo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Martin Van Buren</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: The practice of all my predec...</td>\n",
       "      <td>{'the': 6129, 'of': 3962, 'to': 2444, 'and': 2...</td>\n",
       "      <td>[fellow citizen practice my predecessor impose...</td>\n",
       "      <td>[Fellow Citizens: The practice of all my prede...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>William Harrison</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Called from a retirement which I had supposed ...</td>\n",
       "      <td>{'the': 829, 'of': 604, 'to': 318, 'and': 231,...</td>\n",
       "      <td>[called retirement had supposed continue resid...</td>\n",
       "      <td>[Called from a retirement which I had supposed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>John Tyler</td>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>To the People of the United States Before my a...</td>\n",
       "      <td>{'the': 6870, 'of': 4452, 'to': 2789, 'and': 2...</td>\n",
       "      <td>[before my arrival seat painful communication ...</td>\n",
       "      <td>[To the People of the United States Before my ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>James K. Polk</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow citizens: Without solicitation on my pa...</td>\n",
       "      <td>{'the': 9968, 'of': 6482, 'and': 3778, 'to': 3...</td>\n",
       "      <td>[fellow citizen without solicitation my part c...</td>\n",
       "      <td>[Fellow citizens: Without solicitation on my p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Zachary Taylor</td>\n",
       "      <td>Whig</td>\n",
       "      <td>Elected by the American people to the highest ...</td>\n",
       "      <td>{'the': 1093, 'of': 757, 'to': 483, 'and': 388...</td>\n",
       "      <td>[elected american highest office known law app...</td>\n",
       "      <td>[Elected by the American people to the highest...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Millard Fillmore</td>\n",
       "      <td>Whig</td>\n",
       "      <td>To the Senate and House of Representatives: I ...</td>\n",
       "      <td>{'the': 3830, 'of': 2534, 'to': 1406, 'and': 1...</td>\n",
       "      <td>[senate house representative herewith transmit...</td>\n",
       "      <td>[To the Senate and House of Representatives: I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Franklin Pierce</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My Countrymen: It a relief to feel that no hea...</td>\n",
       "      <td>{'the': 6284, 'of': 4600, 'to': 2430, 'and': 2...</td>\n",
       "      <td>[my countryman relief feel heart my own know p...</td>\n",
       "      <td>[My Countrymen: It a relief to feel that no he...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>James Buchanan</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: I appear before you this day ...</td>\n",
       "      <td>{'the': 7481, 'of': 4373, 'to': 3077, 'and': 2...</td>\n",
       "      <td>[fellow citizen appear before day take solemn ...</td>\n",
       "      <td>[Fellow Citizens: I appear before you this day...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Abraham Lincoln</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Honors To Henry ClayOn the fourth day of July,...</td>\n",
       "      <td>{'the': 7991, 'of': 4671, 'and': 3282, 'to': 3...</td>\n",
       "      <td>[honor henry clayon fourth day july 1776 few f...</td>\n",
       "      <td>[Honors To Henry ClayOn the fourth day of July...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Andrew Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I must be permitted to say that I have been al...</td>\n",
       "      <td>{'the': 9266, 'of': 5769, 'to': 3287, 'and': 3...</td>\n",
       "      <td>[must permitted say almost overwhelmed announc...</td>\n",
       "      <td>[I must be permitted to say that I have been a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ulysses S. Grant</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Citizens of the United States: Your suffrages ...</td>\n",
       "      <td>{'the': 10135, 'of': 7030, 'to': 3779, 'and': ...</td>\n",
       "      <td>[citizen your suffrage having elected me offic...</td>\n",
       "      <td>[Citizens of the United States: Your suffrages...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Rutherford B. Hayes</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We have assembled to repeat t...</td>\n",
       "      <td>{'the': 7187, 'of': 4703, 'and': 2362, 'to': 2...</td>\n",
       "      <td>[fellow citizen assembled repeat public ceremo...</td>\n",
       "      <td>[Fellow Citizens: We have assembled to repeat ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>James A. Garfield</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: We stand to-day upon an emine...</td>\n",
       "      <td>{'the': 317, 'of': 182, 'and': 119, 'to': 80, ...</td>\n",
       "      <td>[fellow citizen stand today upon eminence over...</td>\n",
       "      <td>[Fellow Citizens: We stand to-day upon an emin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chester A. Arthur</td>\n",
       "      <td>Republican</td>\n",
       "      <td>For the fourth time in the history of the Repu...</td>\n",
       "      <td>{'the': 4885, 'of': 3297, 'to': 1518, 'and': 1...</td>\n",
       "      <td>[fourth time history republic chief magistrate...</td>\n",
       "      <td>[For the fourth time in the history of the Rep...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Grover Cleveland</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Fellow Citizens: In the presence of this vast ...</td>\n",
       "      <td>{'the': 13661, 'of': 9427, 'and': 5332, 'to': ...</td>\n",
       "      <td>[fellow citizen presence vast assemblage my co...</td>\n",
       "      <td>[Fellow Citizens: In the presence of this vast...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Benjamin Harrison</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: There is no constitutional or...</td>\n",
       "      <td>{'the': 13697, 'of': 8696, 'and': 4733, 'to': ...</td>\n",
       "      <td>[fellow citizen there constitutional legal req...</td>\n",
       "      <td>[Fellow Citizens: There is no constitutional o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>William McKinley</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Fellow Citizens: In obedience to the will of t...</td>\n",
       "      <td>{'the': 9383, 'of': 5940, 'and': 3405, 'to': 2...</td>\n",
       "      <td>[fellow citizen obedience presence theauthorit...</td>\n",
       "      <td>[Fellow Citizens: In obedience to the will of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Franklin D. Roosevelt</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>President Hoover, Mr. Chief Justice, my friend...</td>\n",
       "      <td>{'the': 9602, 'of': 6444, 'and': 4752, 'to': 4...</td>\n",
       "      <td>[president hoover mr chief justice my friend d...</td>\n",
       "      <td>[President Hoover, Mr. Chief Justice, my frien...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>William Taft</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Citizens: Anyone who has taken the o...</td>\n",
       "      <td>{'the': 11517, 'of': 7154, 'to': 4046, 'and': ...</td>\n",
       "      <td>[my fellow citizen anyone taken oath just take...</td>\n",
       "      <td>[My Fellow Citizens: Anyone who has taken the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Woodrow Wilson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>There has been a change of government. It bega...</td>\n",
       "      <td>{'the': 6396, 'of': 4667, 'and': 3503, 'to': 2...</td>\n",
       "      <td>[there change, began two year ago whenthe hous...</td>\n",
       "      <td>[There has been a change of government., It be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Warren G. Harding</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My countrymen, the surpassing war of all times...</td>\n",
       "      <td>{'the': 2208, 'of': 1457, 'and': 1074, 'to': 1...</td>\n",
       "      <td>[my countryman surpassing war time involved u ...</td>\n",
       "      <td>[My countrymen, the surpassing war of all time...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Calvin Coolidge</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Since the close of the last Congress the Natio...</td>\n",
       "      <td>{'the': 5594, 'of': 3913, 'to': 2346, 'and': 2...</td>\n",
       "      <td>[since close last congress nation lost preside...</td>\n",
       "      <td>[Since the close of the last Congress the Nati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Herbert Hoover</td>\n",
       "      <td>Republican</td>\n",
       "      <td>This campaign now draws near a close. The plat...</td>\n",
       "      <td>{'the': 7127, 'of': 5149, 'and': 2906, 'to': 2...</td>\n",
       "      <td>[campaign now draw near close, platform two pa...</td>\n",
       "      <td>[This campaign now draws near a close., The pl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Theodore Roosevelt</td>\n",
       "      <td>Republican</td>\n",
       "      <td>By the President of the United States of Ameri...</td>\n",
       "      <td>{'the': 17315, 'of': 10495, 'to': 6570, 'and':...</td>\n",
       "      <td>[president america proclamation terrible berea...</td>\n",
       "      <td>[By the President of the United States of Amer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Harry S. Truman</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Mr. Speaker, Mr. President, Members of the Con...</td>\n",
       "      <td>{'the': 2705, 'of': 1429, 'and': 1269, 'to': 1...</td>\n",
       "      <td>[mr speaker mr president member congress heavy...</td>\n",
       "      <td>[Mr. Speaker, Mr. President, Members of the Co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Dwight D. Eisenhower</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My friends, before I begin the expression of t...</td>\n",
       "      <td>{'the': 1325, 'of': 937, 'and': 653, 'to': 565...</td>\n",
       "      <td>[my friend before begin expression those thoug...</td>\n",
       "      <td>[My friends, before I begin the expression of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>John F. Kennedy</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>Governor Stevenson, Senator Johnson, Mr. Butle...</td>\n",
       "      <td>{'the': 7353, 'of': 4895, 'and': 4547, 'to': 3...</td>\n",
       "      <td>[governor stevenson senator johnson mr butler ...</td>\n",
       "      <td>[Governor Stevenson, Senator Johnson, Mr. Butl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Lyndon B. Johnson</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>On this hallowed ground, heroic deeds were per...</td>\n",
       "      <td>{'the': 15291, 'of': 8801, 'to': 8711, 'and': ...</td>\n",
       "      <td>[hallowed ground heroic deed were performed el...</td>\n",
       "      <td>[On this hallowed ground, heroic deeds were pe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Richard M. Nixon</td>\n",
       "      <td>Republican</td>\n",
       "      <td>My Fellow Americans: I come before you tonight...</td>\n",
       "      <td>{'the': 4361, 'of': 2381, 'to': 2370, 'and': 1...</td>\n",
       "      <td>[my fellow american come before tonight candid...</td>\n",
       "      <td>[My Fellow Americans: I come before you tonigh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Gerald Ford</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Mr. Chief Justice, my dear friends, my fellow ...</td>\n",
       "      <td>{'the': 2417, 'of': 1538, 'and': 1427, 'to': 1...</td>\n",
       "      <td>[mr chief justice my dear friend my fellow ame...</td>\n",
       "      <td>[Mr. Chief Justice, my dear friends, my fellow...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>I am Edwin Newman, moderator of this first deb...</td>\n",
       "      <td>{'the': 6239, 'of': 3397, 'to': 3263, 'and': 3...</td>\n",
       "      <td>[am edwin newman moderator first debate 1976 c...</td>\n",
       "      <td>[I am Edwin Newman, moderator of this first de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Ronald Reagan</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Thank you. Thank you very much. Thank you and ...</td>\n",
       "      <td>{'the': 11873, 'and': 7250, 'of': 7040, 'to': ...</td>\n",
       "      <td>[thank, thank very much, thank good evening, s...</td>\n",
       "      <td>[Thank you., Thank you very much., Thank you a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>I have many friends to thank tonight. I thank ...</td>\n",
       "      <td>{'the': 5191, 'and': 3387, 'to': 3049, 'of': 2...</td>\n",
       "      <td>[many friend thank tonight, thank voter suppor...</td>\n",
       "      <td>[I have many friends to thank tonight., I than...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Bill Clinton</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>My fellow citizens, today we celebrate the mys...</td>\n",
       "      <td>{'the': 6917, 'to': 5534, 'and': 4843, 'of': 3...</td>\n",
       "      <td>[my fellow citizen today celebrate mystery ame...</td>\n",
       "      <td>[My fellow citizens, today we celebrate the my...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>Republican</td>\n",
       "      <td>President Clinton, distinguished guests and my...</td>\n",
       "      <td>{'the': 5838, 'and': 4835, 'to': 3589, 'of': 3...</td>\n",
       "      <td>[president clinton distinguished guest my fell...</td>\n",
       "      <td>[President Clinton, distinguished guests and m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>Democratic</td>\n",
       "      <td>To Chairman Dean and my great friend Dick Durb...</td>\n",
       "      <td>{'the': 9491, 'and': 7277, 'to': 6763, 'of': 5...</td>\n",
       "      <td>[chairman dean my great friend dick durbin my ...</td>\n",
       "      <td>[To Chairman Dean and my great friend Dick Dur...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Republican</td>\n",
       "      <td>Chief Justice Roberts, President Carter, Presi...</td>\n",
       "      <td>{'the': 3456, 'and': 3309, 'to': 2543, 'we': 2...</td>\n",
       "      <td>[chief justice robert president carter preside...</td>\n",
       "      <td>[Chief Justice Roberts, President Carter, Pres...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-92a7c28a-7b2e-4895-ad89-d823952702c4')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-92a7c28a-7b2e-4895-ad89-d823952702c4 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-92a7c28a-7b2e-4895-ad89-d823952702c4');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                President  ...                                   Unique Sentences\n",
       "0       George Washington  ...  [Fellow Citizens of the Senate and the House o...\n",
       "1              John Adams  ...  [When it was first perceived, in early times, ...\n",
       "2        Thomas Jefferson  ...  [FRIENDS AND FELLOW-CITIZENS, Called upon to u...\n",
       "3           James Madison  ...  [Unwilling to depart from examples of the most...\n",
       "4            James Monroe  ...  [I should be destitute of feeling if I was not...\n",
       "5       John Quincy Adams  ...  [AND NOW, FRIENDS AND COUNTRYMEN, if the wise ...\n",
       "6          Andrew Jackson  ...  [Fellow Citizens: About to undertake the arduo...\n",
       "7        Martin Van Buren  ...  [Fellow Citizens: The practice of all my prede...\n",
       "8        William Harrison  ...  [Called from a retirement which I had supposed...\n",
       "9              John Tyler  ...  [To the People of the United States Before my ...\n",
       "10          James K. Polk  ...  [Fellow citizens: Without solicitation on my p...\n",
       "11         Zachary Taylor  ...  [Elected by the American people to the highest...\n",
       "12       Millard Fillmore  ...  [To the Senate and House of Representatives: I...\n",
       "13        Franklin Pierce  ...  [My Countrymen: It a relief to feel that no he...\n",
       "14         James Buchanan  ...  [Fellow Citizens: I appear before you this day...\n",
       "15        Abraham Lincoln  ...  [Honors To Henry ClayOn the fourth day of July...\n",
       "16         Andrew Johnson  ...  [I must be permitted to say that I have been a...\n",
       "17       Ulysses S. Grant  ...  [Citizens of the United States: Your suffrages...\n",
       "18    Rutherford B. Hayes  ...  [Fellow Citizens: We have assembled to repeat ...\n",
       "19      James A. Garfield  ...  [Fellow Citizens: We stand to-day upon an emin...\n",
       "20      Chester A. Arthur  ...  [For the fourth time in the history of the Rep...\n",
       "21       Grover Cleveland  ...  [Fellow Citizens: In the presence of this vast...\n",
       "22      Benjamin Harrison  ...  [Fellow Citizens: There is no constitutional o...\n",
       "23       William McKinley  ...  [Fellow Citizens: In obedience to the will of ...\n",
       "24  Franklin D. Roosevelt  ...  [President Hoover, Mr. Chief Justice, my frien...\n",
       "25           William Taft  ...  [My Fellow Citizens: Anyone who has taken the ...\n",
       "26         Woodrow Wilson  ...  [There has been a change of government., It be...\n",
       "27      Warren G. Harding  ...  [My countrymen, the surpassing war of all time...\n",
       "28        Calvin Coolidge  ...  [Since the close of the last Congress the Nati...\n",
       "29         Herbert Hoover  ...  [This campaign now draws near a close., The pl...\n",
       "30     Theodore Roosevelt  ...  [By the President of the United States of Amer...\n",
       "31        Harry S. Truman  ...  [Mr. Speaker, Mr. President, Members of the Co...\n",
       "32   Dwight D. Eisenhower  ...  [My friends, before I begin the expression of ...\n",
       "33        John F. Kennedy  ...  [Governor Stevenson, Senator Johnson, Mr. Butl...\n",
       "34      Lyndon B. Johnson  ...  [On this hallowed ground, heroic deeds were pe...\n",
       "35       Richard M. Nixon  ...  [My Fellow Americans: I come before you tonigh...\n",
       "36            Gerald Ford  ...  [Mr. Chief Justice, my dear friends, my fellow...\n",
       "37           Jimmy Carter  ...  [I am Edwin Newman, moderator of this first de...\n",
       "38          Ronald Reagan  ...  [Thank you., Thank you very much., Thank you a...\n",
       "39      George H. W. Bush  ...  [I have many friends to thank tonight., I than...\n",
       "40           Bill Clinton  ...  [My fellow citizens, today we celebrate the my...\n",
       "41         George W. Bush  ...  [President Clinton, distinguished guests and m...\n",
       "42           Barack Obama  ...  [To Chairman Dean and my great friend Dick Dur...\n",
       "43           Donald Trump  ...  [Chief Justice Roberts, President Carter, Pres...\n",
       "\n",
       "[44 rows x 6 columns]"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fee309e",
   "metadata": {
    "id": "4fee309e"
   },
   "source": [
    "<a name =stepb></a>\n",
    "    \n",
    "# PART B: SENTENCE EMBEDDING\n",
    "\n",
    "[Go to top](#top)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92c8ecc8",
   "metadata": {
    "id": "92c8ecc8"
   },
   "source": [
    "<a name =step1></a>\n",
    "    \n",
    "# 1. Define Cosine Similarity Function \n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d383d3",
   "metadata": {
    "id": "91d383d3"
   },
   "source": [
    "Source: https://www.analyticsvidhya.com/blog/2020/08/top-4-sentence-embedding-techniques-using-python/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "addbb07e",
   "metadata": {
    "id": "addbb07e"
   },
   "outputs": [],
   "source": [
    "def cosine(u, v):\n",
    "    return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba59bdf6",
   "metadata": {
    "id": "ba59bdf6"
   },
   "source": [
    "<a name =step2></a>\n",
    "    \n",
    "# 2. SentenceBERT \n",
    "\n",
    "[Go to top](#top)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "xpPYPo04vmxG",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xpPYPo04vmxG",
    "outputId": "972ab9ed-87a2-4b97-f6be-8edeebd1043a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting sentence-transformers\n",
      "  Downloading sentence-transformers-2.2.0.tar.gz (79 kB)\n",
      "\u001b[K     |████████████████████████████████| 79 kB 7.0 MB/s \n",
      "\u001b[?25hCollecting transformers<5.0.0,>=4.6.0\n",
      "  Downloading transformers-4.16.2-py3-none-any.whl (3.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 3.5 MB 33.7 MB/s \n",
      "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (4.62.3)\n",
      "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.10.0+cu111)\n",
      "Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (0.11.1+cu111)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.21.5)\n",
      "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.0.2)\n",
      "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (1.4.1)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from sentence-transformers) (3.2.5)\n",
      "Collecting sentencepiece\n",
      "  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.2 MB 50.0 MB/s \n",
      "\u001b[?25hCollecting huggingface-hub\n",
      "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
      "\u001b[K     |████████████████████████████████| 67 kB 5.6 MB/s \n",
      "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.6.0->sentence-transformers) (3.10.0.2)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (4.11.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (21.3)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2.23.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2019.12.20)\n",
      "Collecting sacremoses\n",
      "  Downloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB)\n",
      "\u001b[K     |████████████████████████████████| 895 kB 48.6 MB/s \n",
      "\u001b[?25hCollecting pyyaml>=5.1\n",
      "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
      "\u001b[K     |████████████████████████████████| 596 kB 49.7 MB/s \n",
      "\u001b[?25hCollecting tokenizers!=0.11.3,>=0.10.1\n",
      "  Downloading tokenizers-0.11.4-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB)\n",
      "\u001b[K     |████████████████████████████████| 6.8 MB 47.9 MB/s \n",
      "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (3.4.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers<5.0.0,>=4.6.0->sentence-transformers) (3.0.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers<5.0.0,>=4.6.0->sentence-transformers) (3.7.0)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nltk->sentence-transformers) (1.15.0)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers<5.0.0,>=4.6.0->sentence-transformers) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers<5.0.0,>=4.6.0->sentence-transformers) (1.24.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers<5.0.0,>=4.6.0->sentence-transformers) (2021.10.8)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers<5.0.0,>=4.6.0->sentence-transformers) (2.10)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers<5.0.0,>=4.6.0->sentence-transformers) (1.1.0)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers<5.0.0,>=4.6.0->sentence-transformers) (7.1.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sentence-transformers) (3.1.0)\n",
      "Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision->sentence-transformers) (7.1.2)\n",
      "Building wheels for collected packages: sentence-transformers\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.0-py3-none-any.whl size=120747 sha256=b637901228274e7f88b493b98a4750b5f4f9230c0cbbda755853a3c0c8f3eeee\n",
      "  Stored in directory: /root/.cache/pip/wheels/83/c0/df/b6873ab7aac3f2465aa9144b6b4c41c4391cfecc027c8b07e7\n",
      "Successfully built sentence-transformers\n",
      "Installing collected packages: pyyaml, tokenizers, sacremoses, huggingface-hub, transformers, sentencepiece, sentence-transformers\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 3.13\n",
      "    Uninstalling PyYAML-3.13:\n",
      "      Successfully uninstalled PyYAML-3.13\n",
      "Successfully installed huggingface-hub-0.4.0 pyyaml-6.0 sacremoses-0.0.47 sentence-transformers-2.2.0 sentencepiece-0.1.96 tokenizers-0.11.4 transformers-4.16.2\n"
     ]
    }
   ],
   "source": [
    "!pip install sentence-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ea4ad8",
   "metadata": {
    "id": "d6ea4ad8"
   },
   "source": [
    "### Step 1 - Load the pre-trained BERT model. \n",
    "\n",
    "There are many other pre-trained models available. You can find the full list of models here. https://github.com/UKPLab/sentence-transformers/blob/master/docs/pretrained-models/sts-models.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3f145a48",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 433,
     "referenced_widgets": [
      "7e341e85b25b4bf3bcdce21f5ce084f6",
      "932972e1361040fd86e865c24fe5c5ca",
      "f1d8507a1ec94139b0c54d60d16fc5d0",
      "590705d7d3034148884aba5540536a98",
      "7ffd047eccf643c6b6e854be1df58a75",
      "b57fce2473f542a78b1783822b613674",
      "fd15208e042246d8ae9fc58419321e2a",
      "97b071710fbf4f9b8e9c0aecaaca6889",
      "e9e3bd02a5914ee385f64867b1f6b416",
      "45b05c167b124fa49f11a6850093e592",
      "8ec1cdc9d10e4062863b680187e72ffb",
      "8b287ae1fb374bc39ed4c3b460b4b6bb",
      "1aca5121809d43d1b152f566c705d14d",
      "2a4651d1cfed4ae39bfac35f108f7589",
      "f6ec7ba67e5c431d84d4dfa1a543c365",
      "7826efa8cd7a465c870fbf93fd51aa2c",
      "c2c0992fea9c460f8814ace1ab628a85",
      "3dab40316a9a410183acf7a80e1e5388",
      "29e91b06de6948bca0ca591c324215eb",
      "b2df9c1f2b4148bcbe178431df4adc17",
      "83062e62da9d4d1e9921a1c9e5681eb0",
      "d6b631f2f6244d159c8b735a2930f925",
      "090987a261b642118a70b024ec3a48ac",
      "10699f7ce6624826bd823928d67b01f0",
      "be000e6db11841919affefb01d1cc69d",
      "b5222921b8de47bfaac8ab419f7787be",
      "81919ee6fc804539992831f0ac409dda",
      "179b808a4a0f4243a51c6f4296b59c1b",
      "f23617143c7e47148d4a6c5c3dde9736",
      "2c42f95bd2d74c88a2f9c2b4f006a1ce",
      "1607642289d7410ea6c904d091e731c0",
      "fdd7db36651b497dbbc6ef6d88e6c05b",
      "19819014ba7546e4ae5a1c9ff14d7cba",
      "2ee032c317884769973cb2a0d13af814",
      "a88b0b897977487d940feffef5371323",
      "db5f8a1a9a1b4bdd930a055d92ad300d",
      "9738ee92ddfd4162b8c79866e5cc5041",
      "546210b1798f41559fad6d407b1b4247",
      "ec0a069a87254c889f99024c38a15149",
      "25f7bbe46548478eacab81df6da5074e",
      "f315861665dd4ab38830b3cfdeb20349",
      "7c4ce9dd083240c7b18a63a4c5e28802",
      "6ec60e17b6b4411fb5c2552d42d3e5ac",
      "cd21491a987a4291b5a8580107442a37",
      "13f6ba81685a48518d9950be82896516",
      "6a25cb1d80fb4ede8f999897cbb5b42d",
      "d8c54e246abc4761b5164d882784f257",
      "b0c231d06a0f46f4b69fe5aa7d6a2fab",
      "185f06562daa4f10a53ea6af2608732d",
      "b742c522693646e99372cf3bfa6a766b",
      "69bbe045804841829a390f3ba10e6cf2",
      "a5951beafd7a497387d98f3bda548595",
      "6be0fcc03c8f438e9f12821476178232",
      "d2d127b1606c40b499ff33b1c66081d4",
      "22972150eca4497ca17fa53245566449",
      "78ee08698822442faae1d6bc690bda60",
      "e485622c18d34d4fac700952a32c3601",
      "8327629fe72a4f9981d8147d33d52ba2",
      "47e8b0b6433247429764d0eb698e3db4",
      "ae70eb389fbc49dca15ce17501dff227",
      "7d100f3f328444fabad03b4b3a247eb5",
      "dafc6ae8f7b54941b93f04ca8bbebbfe",
      "7b9f857c7812443691e542ac6b6d9844",
      "c95b134a4f78454b9859e8b6f39dd9c0",
      "cbf6480ef1c74a12887ad74e5a44caf6",
      "b8a1009a648c49988c42d983790ca086",
      "c8fb2a66efdf4893b5765dd762352d76",
      "44f162d6bb6849c5b42606b15509812e",
      "773fddfe7a264255bf3f4cdba5866514",
      "0414337c97d3499cbebcd3432c2c7336",
      "46b5a0c3c06949d595ff53606d6da2e0",
      "f3688e55e852453ebe06cf2490eb6028",
      "0039b9fe0eea47c2a5e122dfb032655b",
      "a44dcf8f47c64a309cad9183a0ed9fd9",
      "17683c1f4ced4d80856c1a4ad6e07ffd",
      "0a6b13bedd834c5290f438c2a1546728",
      "5ef62e01def84e1f9651bf9729fda3eb",
      "c3ebf933cd75477390628363425415f5",
      "ad80cb459d574e6f8e45ce1545576f40",
      "9c21b793c5a9448cb719ef0923556c11",
      "e695c0c27bdc4169b1703d28b420cd36",
      "947629a6421747c0b0a7d06e08756668",
      "ef5a0f74fe0242c98b01f6875921d3e1",
      "e72764e4e95b4c8e82f908acc7d587dc",
      "fc9df5c3d7cc47cc9ce9f7be3e4dc2c2",
      "3e8d762412e64a719ecbd93b5d307712",
      "6c557dea102b495f9cdab0b6cd766d1e",
      "bc1c0147edae484981187292aa4cb03c",
      "62bd80b93dee4d0e9d3faa0ba61b037b",
      "d320e5eb040d4ab393bcbca3af7be3a4",
      "b7a2e7c07f2847d5a76a0ffde37753f8",
      "c99bf972ca0741f5a8e88a58aa8f012b",
      "63985c8d09174c75bc31de679fc9fda2",
      "39ad7d29e62f43b7b87d0d7f8f22a017",
      "544e60ad7326423a83cd2fb81f3a5913",
      "4b3d04e9727b430d8e036c527d873922",
      "6547c0a3bfe846e5881d0554ab1e0887",
      "b05cb01cb0f74c09af6a4238ef0006b5",
      "53e7152513ec4414883043d56966cf4b",
      "b1391ab4aa424f3fa82814e6a0a6ef72",
      "fb44d8006b2a479c9f3f87446c6eec9d",
      "a6400ac3c5364743be7454651d84a15f",
      "19c73a8fc9b648638f94551887ff27f6",
      "1fb3012c041c4b28b2ab2aa67c1ba79c",
      "3dfe3a3fcbf24b0f899277955e14e6e1",
      "6fb98517050646e0871fdd1ec43e77df",
      "f8afbf5129034ccd98cfa70327280dee",
      "018247cdd6534bbfade9a7fa01a4799b",
      "09308f7931c34718a93889f1f5cf2d05",
      "37dcc496d656434d86498bf8a56e0df5",
      "f3ef6fb56e5743e58e5f26446b1c9980",
      "4f67e02829e14b1f8359a783facb625d",
      "65677cba08fa40239c029b3b51bb525c",
      "4d2a1d01e7a140a0887eab486157eef2",
      "227032656b4b4a8285e0ea6fd07a8295",
      "ca99509a03aa437fa3d3b75f7887c2d0",
      "fa0d243ca9b346b4b801f70326572f91",
      "634a6270b47d4176a8fffbd98863c48b",
      "951440988f45421ea90ca35893a9a925",
      "2027c851b89e459d84c494ec409c1f83",
      "a7d6446f467048e4ac53c3baaddd0c15",
      "afe48b5c2cf9482a9fce0fdd6259e0a3",
      "0b214a559fdb4630b76abd9f76105b42",
      "6c57e099739343e0bb8370d3d2bbf23b",
      "8e48d02c0c5047d0a21c8773cc0a2825",
      "f950c954adee46a591f53e848cf66362",
      "16333771bfac437a9027862235d6d761",
      "4945e30fc1124c728b3ad41b944671d5",
      "1e66e45cbc094cccb8c4d868d6353624",
      "da9c7a1951a946dda4c18e040f0809a0",
      "1477347ea4754213a4a213d4a7753201",
      "bc6a0e5e9e9c411cb693fde0347c79b4",
      "a1afdfb299264ad29e5deb056a944d5b",
      "ac50490027214989b8ec2537bedbe719",
      "fd396c2481b94d7fa88ef0e3460955a7",
      "7b2a84f00c7b4d6083262c3e5c541755",
      "9425e3a5a1aa49bb8bd0ca73b619a51d",
      "8f624553120842ceb2b171e82bddb2ec",
      "7715f17b919e4016b2a36123aa30d44c",
      "a32f82578cff4f1ab0b8a460334ea95a",
      "e56ac064e79842f390c1a52c0d8a4476",
      "f501b021e29b415eb6bf9543f41c7610",
      "4154336b17d7499ba83c5bfc1cb5ee5f"
     ]
    },
    "id": "3f145a48",
    "outputId": "95b10d90-67aa-4741-d8dc-5c8bda9823e0"
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "sbert_model = SentenceTransformer('bert-base-nli-mean-tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fd7f63a",
   "metadata": {
    "id": "4fd7f63a"
   },
   "source": [
    "### Step 2 - Encode sentences from existing corpus. \n",
    "\n",
    "We can also display the sentence vectors(just uncomment the code below)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "oQoyG_yYy7DA",
   "metadata": {
    "id": "oQoyG_yYy7DA"
   },
   "source": [
    "my cleaned/pre-tokenized data was not accepted by sentenceBERT:  \n",
    "cleaned_transcripts = corpus_df[\"transcripts_new\"]\n",
    "\n",
    "\n",
    "Dataset to be used:  \n",
    "for i in range(corpus_df.shape[0]):\n",
    "    speeches = corpus_df.transcripts.iloc[i]\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    unique_sentences = list(dict.fromkeys(sentences))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7397f9",
   "metadata": {
    "id": "9c7397f9"
   },
   "source": [
    "#### Let's try for 1 president"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "hc17bTiG0g2r",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hc17bTiG0g2r",
    "outputId": "1d59b402-2066-4d2f-8543-051b5da02802"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "gs_w_NFlu49S",
   "metadata": {
    "id": "gs_w_NFlu49S"
   },
   "outputs": [],
   "source": [
    "cleaned_transcripts=corpus_df[\"transcripts_new\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b5bcbc18",
   "metadata": {
    "id": "b5bcbc18"
   },
   "outputs": [],
   "source": [
    "sentence_embeddings = sbert_model.encode(cleaned_transcripts[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4a25e85",
   "metadata": {
    "id": "e4a25e85"
   },
   "outputs": [],
   "source": [
    "print('Sample BERT embedding vector - length', len(sentence_embeddings[0]))\n",
    "print('Sample BERT embedding vector - note includes negative values', sentence_embeddings[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c42c02e5",
   "metadata": {
    "id": "c42c02e5"
   },
   "source": [
    "### Step 3 - Define a test query and encode it as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9a3419db",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9a3419db",
    "outputId": "28fec66f-cb93-49d1-e360-eea19c1627fe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['believe child future']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"I believe I our children are our future.\"\n",
    "query_clean=clean_sent(query)\n",
    "query_ls=[]\n",
    "query_ls.append(query_clean)                 \n",
    "query_stop_words=remove_stop_words(query_ls)\n",
    "query_lemmatized=get_lemmatized_text(query_stop_words)\n",
    "query_lemmatized\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "WYjKPzu5yle-",
   "metadata": {
    "id": "WYjKPzu5yle-"
   },
   "outputs": [],
   "source": [
    "query_vec = sbert_model.encode([query_lemmatized[0]])[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7468f5a2",
   "metadata": {
    "id": "7468f5a2"
   },
   "source": [
    "### Step 4 - Compute the cosine similarity using scipy. \n",
    "\n",
    "We will retrieve the similarity values between the sentences and our test query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e9d6af42",
   "metadata": {
    "id": "e9d6af42",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sim_dict={}\n",
    "for sent in cleaned_transcripts[0]:\n",
    "    sim = cosine(query_vec, sbert_model.encode([sent])[0])\n",
    "    sim_dict[sent]=sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1aa1b1ea",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1aa1b1ea",
    "outputId": "fe307b17-a9e2-444f-b397-ddea42d50652"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('event answered hope', 0.7147625)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_key = max(sim_dict, key=sim_dict.get)\n",
    "max_value = max(sim_dict.values())\n",
    "max_key, max_value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "317197ad",
   "metadata": {
    "id": "317197ad"
   },
   "source": [
    "### Step 5 - Let's do it for ALL Presidents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f58Ypx5MGdsr",
   "metadata": {
    "id": "f58Ypx5MGdsr"
   },
   "outputs": [],
   "source": [
    "unique_sent=[]\n",
    "for i in range(corpus_df.shape[0]): #corpus_df.shape[0]\n",
    "    speeches=(corpus_df.transcripts.iloc[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    unique_sentences = list(dict.fromkeys(sentences))\n",
    "    unique_sent.append(unique_sentences)\n",
    "unique_sent\n",
    "corpus_df[\"Unique Sentences\"]=unique_sent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "aa7TL9SiJNR4",
   "metadata": {
    "id": "aa7TL9SiJNR4"
   },
   "outputs": [],
   "source": [
    "matching_sent=corpus_df[\"Unique Sentences\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "e67rR63zKRcw",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "id": "e67rR63zKRcw",
    "outputId": "7a399cfd-13e2-4223-94d7-7ed26c29d4bb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'Negligence of its regulations, inattention to its recommendations, if not disobedience to its authority, not only in individuals but in States, soon appeared with their melancholy consequences?'"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matching_sent[1][5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "9PK3sEpmS3qe",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9PK3sEpmS3qe",
    "outputId": "892a27a6-5316-4c4e-87f5-6a54c52bfdc9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4749758",
   "metadata": {},
   "source": [
    "<a name =results_bert></a>\n",
    "### Step 6: Results SentenceBERT\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3FV7kA-kLHzZ",
   "metadata": {
    "id": "3FV7kA-kLHzZ"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Matching Sentence</th>\n",
       "      <th>Normalized Text</th>\n",
       "      <th>Cosine Similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Bill Clinton</td>\n",
       "      <td>We will give these children a future.</td>\n",
       "      <td>give child future</td>\n",
       "      <td>0.955503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>Each day, we send nearly $ 1 billion of our we...</td>\n",
       "      <td>consign child future</td>\n",
       "      <td>0.952316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>John F. Kennedy</td>\n",
       "      <td>For we are both devoting massive sums of money...</td>\n",
       "      <td>cherish childrens future</td>\n",
       "      <td>0.889300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Ronald Reagan</td>\n",
       "      <td>It is also the story of how her son secretly v...</td>\n",
       "      <td>let u remember child future want them</td>\n",
       "      <td>0.886800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>Believe in your future.</td>\n",
       "      <td>believe your future</td>\n",
       "      <td>0.869568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Franklin D. Roosevelt</td>\n",
       "      <td>That is our hope for the future.</td>\n",
       "      <td>hope future</td>\n",
       "      <td>0.845189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>And they remind us that we're tied together in...</td>\n",
       "      <td>future come</td>\n",
       "      <td>0.839028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>They talked about posterity.</td>\n",
       "      <td>talked about future</td>\n",
       "      <td>0.830629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Herbert Hoover</td>\n",
       "      <td>Much of it is of interest only to history.</td>\n",
       "      <td>interest now future</td>\n",
       "      <td>0.821938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>Doesn't this raise a serious question as to wh...</td>\n",
       "      <td>believe good indication kind choose future</td>\n",
       "      <td>0.821646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Harry S. Truman</td>\n",
       "      <td>They can foresee what it will do in the future.</td>\n",
       "      <td>foresee what do future</td>\n",
       "      <td>0.806893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Richard M. Nixon</td>\n",
       "      <td>Now, however, we look to the future.</td>\n",
       "      <td>now however look future</td>\n",
       "      <td>0.806078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Lyndon B. Johnson</td>\n",
       "      <td>What we are here today to do is to salute the ...</td>\n",
       "      <td>go learn progress problem hope concern child f...</td>\n",
       "      <td>0.802219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Abraham Lincoln</td>\n",
       "      <td>The progress of our arms, upon which all else ...</td>\n",
       "      <td>high hope future prediction regard ventured</td>\n",
       "      <td>0.791315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Gerald Ford</td>\n",
       "      <td>We are the embodiment of hope for progress.</td>\n",
       "      <td>embodiment hope progress</td>\n",
       "      <td>0.789917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Dwight D. Eisenhower</td>\n",
       "      <td>That topic is: the future.</td>\n",
       "      <td>topic future</td>\n",
       "      <td>0.782410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Martin Van Buren</td>\n",
       "      <td>There is no reason to suppose that future sale...</td>\n",
       "      <td>there reason suppose future sale productive</td>\n",
       "      <td>0.778534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Calvin Coolidge</td>\n",
       "      <td>We hope the time is near at hand when we can act.</td>\n",
       "      <td>hope time near hand when act</td>\n",
       "      <td>0.767748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Thomas Jefferson</td>\n",
       "      <td>My children, this is what I wished to say to you.</td>\n",
       "      <td>my child what wished say</td>\n",
       "      <td>0.761502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>William Taft</td>\n",
       "      <td>The sum was received in full of the amount due...</td>\n",
       "      <td>beg express hope belief such result need antic...</td>\n",
       "      <td>0.755799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Warren G. Harding</td>\n",
       "      <td>With these I amunafraid, and confidently face ...</td>\n",
       "      <td>amunafraid confidently face future</td>\n",
       "      <td>0.748882</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Benjamin Harrison</td>\n",
       "      <td>We carry the great impulse and increase of the...</td>\n",
       "      <td>there near frontier possible development</td>\n",
       "      <td>0.742493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>John Tyler</td>\n",
       "      <td>Taking the message as his letter of instructio...</td>\n",
       "      <td>may permitted hope example thus set followed s...</td>\n",
       "      <td>0.740487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chester A. Arthur</td>\n",
       "      <td>It is to provide for the interests of all in t...</td>\n",
       "      <td>provide interest future settle individual diff...</td>\n",
       "      <td>0.739048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Theodore Roosevelt</td>\n",
       "      <td>Timid depositors have withdrawn their savings ...</td>\n",
       "      <td>done executive order already accomplished put ...</td>\n",
       "      <td>0.738992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Grover Cleveland</td>\n",
       "      <td>More than fifteen years ago this Government pr...</td>\n",
       "      <td>additional advice soon expected</td>\n",
       "      <td>0.724869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>James A. Garfield</td>\n",
       "      <td>Let our people find a new meaning in the divin...</td>\n",
       "      <td>let find new meaning divine oracle declares li...</td>\n",
       "      <td>0.718752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Millard Fillmore</td>\n",
       "      <td>It is hopedthat those duties may be completed ...</td>\n",
       "      <td>hopedthat those duty may completed within time...</td>\n",
       "      <td>0.716062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>George Washington</td>\n",
       "      <td>It was hoped that the treaty of Holstin, made ...</td>\n",
       "      <td>event answered hope</td>\n",
       "      <td>0.714763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Woodrow Wilson</td>\n",
       "      <td>The people of the United States do not need to...</td>\n",
       "      <td>do need assured now platform definite pledge p...</td>\n",
       "      <td>0.714627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Franklin Pierce</td>\n",
       "      <td>Over this great interest, in which not only ou...</td>\n",
       "      <td>policy had view regard interest embrace future...</td>\n",
       "      <td>0.712023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ulysses S. Grant</td>\n",
       "      <td>Flour shipped from the Mississippi River to Ha...</td>\n",
       "      <td>bill may perfect provision may such best appli...</td>\n",
       "      <td>0.709644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Andrew Jackson</td>\n",
       "      <td>The unexpended balances of appropriation on th...</td>\n",
       "      <td>if therefore future receipt should sufficient ...</td>\n",
       "      <td>0.706464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>James K. Polk</td>\n",
       "      <td>Steps will be taken to procure this informatio...</td>\n",
       "      <td>step taken procure information earliest practi...</td>\n",
       "      <td>0.705038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>William McKinley</td>\n",
       "      <td>We are maintaining the parity of all the money...</td>\n",
       "      <td>done past however may done future</td>\n",
       "      <td>0.701777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Rutherford B. Hayes</td>\n",
       "      <td>The retirement from circulation of United Stat...</td>\n",
       "      <td>anticipated redemption withdrawal note earlies...</td>\n",
       "      <td>0.697649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>James Monroe</td>\n",
       "      <td>This, however, ought not to be calculated on a...</td>\n",
       "      <td>however ought calculated certain</td>\n",
       "      <td>0.696889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Andrew Johnson</td>\n",
       "      <td>The grief of the nation is still fresh.</td>\n",
       "      <td>there one way hope gain necessaryaid</td>\n",
       "      <td>0.695950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Zachary Taylor</td>\n",
       "      <td>I recommend this subject to the early and favo...</td>\n",
       "      <td>recommend subject early favorable consideratio...</td>\n",
       "      <td>0.690188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>James Buchanan</td>\n",
       "      <td>We have been exposed to many threatening and a...</td>\n",
       "      <td>may ever under divine guidance protection</td>\n",
       "      <td>0.690170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>John Quincy Adams</td>\n",
       "      <td>From the experience of the past we derive inst...</td>\n",
       "      <td>experience past derive instructive lesson future</td>\n",
       "      <td>0.680009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>James Madison</td>\n",
       "      <td>By the President: JAMES MONROE, Secretary of S...</td>\n",
       "      <td>result yet known</td>\n",
       "      <td>0.656467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>William Harrison</td>\n",
       "      <td>Until an amendment of the Constitution can be ...</td>\n",
       "      <td>until amendment constitution effected public o...</td>\n",
       "      <td>0.654502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>It would be unbecoming the representatives of ...</td>\n",
       "      <td>may territory residence virtue happiness</td>\n",
       "      <td>0.642435</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                President                                  Matching Sentence  \\\n",
       "40           Bill Clinton              We will give these children a future.   \n",
       "42           Barack Obama  Each day, we send nearly $ 1 billion of our we...   \n",
       "33        John F. Kennedy  For we are both devoting massive sums of money...   \n",
       "38          Ronald Reagan  It is also the story of how her son secretly v...   \n",
       "43           Donald Trump                            Believe in your future.   \n",
       "24  Franklin D. Roosevelt                   That is our hope for the future.   \n",
       "41         George W. Bush  And they remind us that we're tied together in...   \n",
       "39      George H. W. Bush                       They talked about posterity.   \n",
       "29         Herbert Hoover         Much of it is of interest only to history.   \n",
       "37           Jimmy Carter  Doesn't this raise a serious question as to wh...   \n",
       "31        Harry S. Truman    They can foresee what it will do in the future.   \n",
       "35       Richard M. Nixon               Now, however, we look to the future.   \n",
       "34      Lyndon B. Johnson  What we are here today to do is to salute the ...   \n",
       "15        Abraham Lincoln  The progress of our arms, upon which all else ...   \n",
       "36            Gerald Ford        We are the embodiment of hope for progress.   \n",
       "32   Dwight D. Eisenhower                         That topic is: the future.   \n",
       "7        Martin Van Buren  There is no reason to suppose that future sale...   \n",
       "28        Calvin Coolidge  We hope the time is near at hand when we can act.   \n",
       "2        Thomas Jefferson  My children, this is what I wished to say to you.   \n",
       "25           William Taft  The sum was received in full of the amount due...   \n",
       "27      Warren G. Harding  With these I amunafraid, and confidently face ...   \n",
       "22      Benjamin Harrison  We carry the great impulse and increase of the...   \n",
       "9              John Tyler  Taking the message as his letter of instructio...   \n",
       "20      Chester A. Arthur  It is to provide for the interests of all in t...   \n",
       "30     Theodore Roosevelt  Timid depositors have withdrawn their savings ...   \n",
       "21       Grover Cleveland  More than fifteen years ago this Government pr...   \n",
       "19      James A. Garfield  Let our people find a new meaning in the divin...   \n",
       "12       Millard Fillmore  It is hopedthat those duties may be completed ...   \n",
       "0       George Washington  It was hoped that the treaty of Holstin, made ...   \n",
       "26         Woodrow Wilson  The people of the United States do not need to...   \n",
       "13        Franklin Pierce  Over this great interest, in which not only ou...   \n",
       "17       Ulysses S. Grant  Flour shipped from the Mississippi River to Ha...   \n",
       "6          Andrew Jackson  The unexpended balances of appropriation on th...   \n",
       "10          James K. Polk  Steps will be taken to procure this informatio...   \n",
       "23       William McKinley  We are maintaining the parity of all the money...   \n",
       "18    Rutherford B. Hayes  The retirement from circulation of United Stat...   \n",
       "4            James Monroe  This, however, ought not to be calculated on a...   \n",
       "16         Andrew Johnson            The grief of the nation is still fresh.   \n",
       "11         Zachary Taylor  I recommend this subject to the early and favo...   \n",
       "14         James Buchanan  We have been exposed to many threatening and a...   \n",
       "5       John Quincy Adams  From the experience of the past we derive inst...   \n",
       "3           James Madison  By the President: JAMES MONROE, Secretary of S...   \n",
       "8        William Harrison  Until an amendment of the Constitution can be ...   \n",
       "1              John Adams  It would be unbecoming the representatives of ...   \n",
       "\n",
       "                                      Normalized Text  Cosine Similarity  \n",
       "40                                  give child future           0.955503  \n",
       "42                               consign child future           0.952316  \n",
       "33                           cherish childrens future           0.889300  \n",
       "38              let u remember child future want them           0.886800  \n",
       "43                                believe your future           0.869568  \n",
       "24                                        hope future           0.845189  \n",
       "41                                        future come           0.839028  \n",
       "39                                talked about future           0.830629  \n",
       "29                                interest now future           0.821938  \n",
       "37         believe good indication kind choose future           0.821646  \n",
       "31                             foresee what do future           0.806893  \n",
       "35                            now however look future           0.806078  \n",
       "34  go learn progress problem hope concern child f...           0.802219  \n",
       "15        high hope future prediction regard ventured           0.791315  \n",
       "36                           embodiment hope progress           0.789917  \n",
       "32                                       topic future           0.782410  \n",
       "7         there reason suppose future sale productive           0.778534  \n",
       "28                       hope time near hand when act           0.767748  \n",
       "2                            my child what wished say           0.761502  \n",
       "25  beg express hope belief such result need antic...           0.755799  \n",
       "27                 amunafraid confidently face future           0.748882  \n",
       "22           there near frontier possible development           0.742493  \n",
       "9   may permitted hope example thus set followed s...           0.740487  \n",
       "20  provide interest future settle individual diff...           0.739048  \n",
       "30  done executive order already accomplished put ...           0.738992  \n",
       "21                    additional advice soon expected           0.724869  \n",
       "19  let find new meaning divine oracle declares li...           0.718752  \n",
       "12  hopedthat those duty may completed within time...           0.716062  \n",
       "0                                 event answered hope           0.714763  \n",
       "26  do need assured now platform definite pledge p...           0.714627  \n",
       "13  policy had view regard interest embrace future...           0.712023  \n",
       "17  bill may perfect provision may such best appli...           0.709644  \n",
       "6   if therefore future receipt should sufficient ...           0.706464  \n",
       "10  step taken procure information earliest practi...           0.705038  \n",
       "23                  done past however may done future           0.701777  \n",
       "18  anticipated redemption withdrawal note earlies...           0.697649  \n",
       "4                    however ought calculated certain           0.696889  \n",
       "16               there one way hope gain necessaryaid           0.695950  \n",
       "11  recommend subject early favorable consideratio...           0.690188  \n",
       "14          may ever under divine guidance protection           0.690170  \n",
       "5    experience past derive instructive lesson future           0.680009  \n",
       "3                                    result yet known           0.656467  \n",
       "8   until amendment constitution effected public o...           0.654502  \n",
       "1            may territory residence virtue happiness           0.642435  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_matches = pd.DataFrame()\n",
    "similar_dict = {}\n",
    "\n",
    "#For each president:\n",
    "for i in range(corpus_df.shape[0]):\n",
    "    sentence_embeddings = sbert_model.encode(cleaned_transcripts[i])\n",
    "    sim_dict={}\n",
    "    match_dict={}\n",
    "    count=0\n",
    "    for sent in cleaned_transcripts[i]:\n",
    "      count+=1\n",
    "      matching_sentence = matching_sent[i][count-1]\n",
    "      match_dict[sent]= matching_sentence\n",
    "      sim = cosine(query_vec, sbert_model.encode([sent])[0])\n",
    "      sim_dict[sent]=sim\n",
    "    max_key = max(sim_dict, key=sim_dict.get)\n",
    "    max_value = max(sim_dict.values())\n",
    "    max_match = match_dict.get(max_key)\n",
    "    president = corpus_df.President.iloc[i]\n",
    "    similar_dict = {'President': president, 'Matching Sentence': max_match, 'Normalized Text': max_key, 'Cosine Similarity': max_value}\n",
    "    top_matches = top_matches.append(similar_dict, ignore_index = True)\n",
    "\n",
    "top=top_matches.sort_values(by=['Cosine Similarity'], ascending=False)\n",
    "top"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbc22aff",
   "metadata": {
    "id": "fbc22aff"
   },
   "source": [
    "<a name =stepc></a>\n",
    "    \n",
    "# PART C: WORD EMBEDDING\n",
    "\n",
    "[Go to top](#top)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "646092bc",
   "metadata": {
    "id": "646092bc"
   },
   "source": [
    "<a name =step3></a>\n",
    "    \n",
    "# 1. GLoVe\n",
    "\n",
    "[Go to top](#top)\n",
    "\n",
    "### Step 1: Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b738d54",
   "metadata": {
    "id": "6b738d54"
   },
   "outputs": [],
   "source": [
    "import gensim.downloader as api\n",
    "glove = api.load(\"glove-wiki-gigaword-50\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a02b6a2",
   "metadata": {
    "id": "5a02b6a2"
   },
   "outputs": [],
   "source": [
    "from gensim.corpora import Dictionary\n",
    "from gensim.models import TfidfModel\n",
    "from gensim.models import WordEmbeddingSimilarityIndex\n",
    "from gensim.similarities import SparseTermSimilarityMatrix\n",
    "from gensim.similarities import SoftCosineSimilarity\n",
    "\n",
    "  \n",
    "similarity_index = WordEmbeddingSimilarityIndex(glove)\n",
    "\n",
    "# Build the term dictionary, TF-idf model\n",
    "dictionary = Dictionary(corpus+[query])\n",
    "tfidf = TfidfModel(dictionary=dictionary)\n",
    "\n",
    "\n",
    "# Create the term similarity matrix.  \n",
    "similarity_matrix = SparseTermSimilarityMatrix(similarity_index, dictionary, tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9bb1a3",
   "metadata": {
    "id": "4c9bb1a3"
   },
   "source": [
    "### Step 2: Soft Cosine Similarity \n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c1e16d9",
   "metadata": {
    "id": "1c1e16d9",
    "outputId": "150b043e-dcc3-4458-b75c-96b0e2b2d41e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6 \t 0.804 \t In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own; nor those of my fellow citizens at large, less than either.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Compute Soft Cosine Measure between the query and the documents.\n",
    "# From: https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/soft_cosine_tutorial.ipynb\n",
    "query_tf = tfidf[dictionary.doc2bow(query)]\n",
    "\n",
    "index = SoftCosineSimilarity(\n",
    "            tfidf[[dictionary.doc2bow(sentence) for sentence in corpus]],\n",
    "            similarity_matrix)\n",
    "\n",
    "doc_similarity_scores = index[query_tf]\n",
    "\n",
    "# Output the sorted similarity scores and documents\n",
    "sorted_indexes = np.argsort(doc_similarity_scores)[::-1] #[::-1] Start at the end, end when it ends, end when nothing's left and walk backwards by 1. \n",
    "idx = sorted_indexes[0]\n",
    "print(f'{idx} \\t {doc_similarity_scores[idx]:0.3f} \\t {sentences[idx]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2919f4d",
   "metadata": {
    "id": "a2919f4d"
   },
   "source": [
    "### Step 3: Apply to entire Dataframe \n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4361289f",
   "metadata": {
    "id": "4361289f"
   },
   "source": [
    "#### Trial 3 b - Added stop words and more descriptive query "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c06102e",
   "metadata": {
    "id": "2c06102e"
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopwords = stopwords.words('english')\n",
    "\n",
    "# From: https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/soft_cosine_tutorial.ipynb\n",
    "def preprocess(doc):\n",
    "    # Tokenize, clean up input document string\n",
    "    doc = sub(r'<img[^<>]+(>|$)', \" image_token \", doc)\n",
    "    doc = sub(r'<[^<>]+(>|$)', \" \", doc)\n",
    "    #doc = sub(r'\\[img_assist[^]]*?\\]', \" \", doc)\n",
    "    doc = sub(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', \" url_token \", doc)\n",
    "    return [token for token in simple_preprocess(doc, min_len=0, max_len=float(\"inf\")) if token not in stopwords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "141969ee",
   "metadata": {
    "id": "141969ee",
    "outputId": "592b27d8-958f-41c4-ae2c-9bdc07665d82",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sarabernard/opt/anaconda3/lib/python3.9/site-packages/gensim/similarities/termsim.py:358: RuntimeWarning: divide by zero encountered in true_divide\n",
      "  Y = np.multiply(Y, 1 / np.sqrt(Y_norm))\n",
      "/Users/sarabernard/opt/anaconda3/lib/python3.9/site-packages/gensim/similarities/termsim.py:358: RuntimeWarning: invalid value encountered in multiply\n",
      "  Y = np.multiply(Y, 1 / np.sqrt(Y_norm))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>President</th>\n",
       "      <th>Matching Sentence</th>\n",
       "      <th>Index</th>\n",
       "      <th>Similarity Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>John Adams</td>\n",
       "      <td>On this subject it might become me better to b...</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Barack Obama</td>\n",
       "      <td>In order to save our children from a future of...</td>\n",
       "      <td>769.0</td>\n",
       "      <td>0.999880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Lyndon B. Johnson</td>\n",
       "      <td>It does say the only limit to a man's hope for...</td>\n",
       "      <td>2383.0</td>\n",
       "      <td>0.982136</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            President                                  Matching Sentence  \\\n",
       "1          John Adams  On this subject it might become me better to b...   \n",
       "42       Barack Obama  In order to save our children from a future of...   \n",
       "34  Lyndon B. Johnson  It does say the only limit to a man's hope for...   \n",
       "\n",
       "     Index  Similarity Score  \n",
       "1     35.0          1.000000  \n",
       "42   769.0          0.999880  \n",
       "34  2383.0          0.982136  "
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#terms not loop dependent\n",
    "query = preprocess('I believe that the young people in America are our future.')\n",
    "similarity_index = WordEmbeddingSimilarityIndex(glove)\n",
    "top_matches = pd.DataFrame()\n",
    "for i in range(0,corpus_df.shape[0]): #\n",
    "    speeches=(corpus_df.transcripts[i])\n",
    "    sentences=sent_tokenize(speeches)\n",
    "    president =corpus_df.President[i]\n",
    "    # STEP 2 -TEXT NORMALIZATION\n",
    "    # Preprocess the documents, including the query string\n",
    "    corpus = [preprocess(sentence) for sentence in sentences] \n",
    "    # STEP 3 - FEATURE ENGINEERING\n",
    "    # Build the term dictionary, TF-idf model\n",
    "    dictionary = Dictionary(corpus+[query])\n",
    "    tfidf = TfidfModel(dictionary=dictionary)\n",
    "    # Create the term similarity matrix.  \n",
    "    similarity_matrix = SparseTermSimilarityMatrix(similarity_index, dictionary, tfidf)\n",
    "    # STEP 4 - SOFT COSINE SIMILARITY\n",
    "    # Compute Soft Cosine Measure between the query and the documents.\n",
    "    query_tf = tfidf[dictionary.doc2bow(query)]\n",
    "\n",
    "    index = SoftCosineSimilarity(\n",
    "                tfidf[[dictionary.doc2bow(sentence) for sentence in corpus]],\n",
    "                similarity_matrix)\n",
    "    doc_similarity_scores = index[query_tf]\n",
    "    # Output the sorted similarity scores and documents\n",
    "    sorted_indexes = np.argsort(doc_similarity_scores)\n",
    "    ranked=sorted_indexes[::-1]\n",
    "    # runtime warning with index[0] only\n",
    "    idx = ranked[0]\n",
    "    # The most relevant sentence from each president\n",
    "    df = {'President': president, 'Matching Sentence': sentences[idx], 'Index': idx, 'Similarity Score': (doc_similarity_scores[idx])}\n",
    "    top_matches = top_matches.append(df, ignore_index = True)\n",
    "    top=top_matches.sort_values(by=['Similarity Score'], ascending=False)\n",
    "\n",
    "top.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58568971",
   "metadata": {
    "id": "58568971",
    "outputId": "84f6ec28-58c0-45b8-ccdd-abaf39df37c1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 4345, 12, 127, 17, 119)"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#length of top 3\n",
    "len(top.iloc[0][0]), len(top.iloc[0][1]), len(top.iloc[1][0]), len(top.iloc[1][1]), len(top.iloc[2][0]), len(top.iloc[2][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcb13bb",
   "metadata": {
    "id": "8bcb13bb"
   },
   "source": [
    "<a name =results_glove></a>\n",
    "### Step 6: Results GloVe\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e17d03",
   "metadata": {
    "id": "f5e17d03"
   },
   "source": [
    "### Result using \"I believe that young people are our future\"\n",
    "- The number 1 match was by John Adams, the 2nd President of the United States.\n",
    "- His sentence was matching sentence had 4345 words compared to ~120 words for the next top 2 presidents; 44th and 36th respectively. This may have been due to typos, an anomoly, perhaps he is known for being wordy or neing wordy was normal back in those day.\n",
    "- For the more readable sentences at only 120words, by the 2nd and 3rd ranked it is easier to see why they matched well.\n",
    "- Way forward: Inspect the data to determine if:\n",
    "    - length of sentences trend with time.\n",
    "    - length of sentences favour/don't favour being selected in the top 3. \n",
    "    - the performance of the algorithm is independent of era of corpus \n",
    "- Note: The quality of the input from the user affects the ability to detect semantics.\n",
    "    - Consider criteria for query. \n",
    "    - for e.g at least 2 verbs, 2 nouns, 2 adjectives/adverbs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c051d66",
   "metadata": {
    "id": "8c051d66",
    "outputId": "8123a2b7-b5b2-499a-e417-effe8e19af05"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Instead of saying: \"I believe that young people are our future\" ...\n",
      "\n",
      "President John Adams would say \"On this subject it might become me better to be silent or to speak with diffidence; but as something may be expected, the occasion, I hope, will be admitted as an apology if I venture to say that if a preference, upon principle, of a free republican government, formed upon long and serious reflection, after a diligent and impartial inquiry after truth; if an attachment to the Constitution of the United States, and a conscientious determination to support it until it shall be altered by the judgments and wishes of the people, expressed in the mode prescribed in it; if a respectful attention to the constitutions of the individual States and a constant caution and delicacy toward the State governments; if an equal and impartial regard to the rights, interest, honor, and happiness of all the States in the Union, without preference or regard to a northern or southern, an eastern or western, position, their various political opinions on unessential points or their personal attachments; if a love of virtuous men of all parties and denominations; if a love of science and letters and a wish to patronize every rational effort to encourage schools, colleges, universities, academies, and every institution for propagating knowledge, virtue, and religion among all classes of the people, not only for their benign influence on the happiness of life in all its stages and classes, and of society in all its forms, but as the only means of preserving our Constitution from its natural enemies, the spirit of sophistry, the spirit of party, the spirit of intrigue, the profligacy of corruption, and the pestilence of foreign influence, which is the angel of destruction to elective governments; if a love of equal laws, of justice, and humanity in the interior administration; if an inclination to improve agriculture, commerce, and manufacturers for necessity, convenience, and defense; if a spirit of equity and humanity toward the aboriginal nations of America, and a disposition to meliorate their condition by inclining them to be more friendly to us, and our citizens to be more friendly to them; if an inflexible determination to maintain peace and inviolable faith with all nations, and that system of neutrality and impartiality among the belligerent powers of Europe which has been adopted by this Government and so solemnly sanctioned by both Houses of Congress and applauded by the legislatures of the States and the public opinion, until it shall be otherwise ordained by Congress; if a personal esteem for the French nation, formed in a residence of seven years chiefly among them, and a sincere desire to preserve the friendship which has been so much for the honor and interest of both nations; if, while the conscious honor and integrity of the people of America and the internal sentiment of their own power and energies must be preserved, an earnest endeavor to investigate every just cause and remove every colorable pretense of complaint; if an intention to pursue by amicable negotiation a reparation for the injuries that have been committed on the commerce of our fellow citizens by whatever nation, and if success can not be obtained, to lay the facts before the Legislature, that they may consider what further measures the honor and interest of the Government and its constituents demand; if a resolution to do justice as far as may depend upon me, at all times and to all nations, and maintain peace, friendship, and benevolence with all the world; if an unshaken confidence in the honor, spirit, and resources of the American people, on which I have so often hazarded my all and never been deceived; if elevated ideas of the high destinies of this country and of my own duties toward it, founded on a knowledge of the moral principles and intellectual improvements of the people deeply engraven on my mind in early life, and not obscured but exalted by experience and age; and, with humble reverence, I feel it to be my duty to add, if a veneration for the religion of a people who profess and call themselves Christians, and a fixed resolution to consider a decent respect for Christianity among the best recommendations for the public service, can enable me in any degree to comply with your wishes, it shall be my strenuous endeavor that this sagacious injunction of the two Houses shall not be without effect.\"\tSoft Cosine Similarity 1.0\n",
      "\n",
      "President Barack Obama would say \"In order to save our children from a future of debt, we will also end the tax breaks for the wealthiest 2 percent of Americans.\"\tSoft Cosine Similarity 0.99987984\n",
      "\n",
      "President Lyndon B. Johnson would say \"It does say the only limit to a man's hope for happiness, and for the future of his children, shall be his own ability.\"\tSoft Cosine Similarity 0.98213553\n"
     ]
    }
   ],
   "source": [
    "print('Instead of saying: \"I believe that young people are our future\" ...')\n",
    "\n",
    "for i in range(0,3):\n",
    "    print(str('\\n' + 'President ' + top.iloc[i][0] + ' would say ' + '\"' + top.iloc[i][1] + '\"' + '\\t' +'Soft Cosine Similarity'), top.iloc[i][3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77b53b6e",
   "metadata": {
    "id": "77b53b6e"
   },
   "source": [
    "<a name =limitations></a>\n",
    "    \n",
    "# Limitations & Observations\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "544c7507",
   "metadata": {
    "id": "544c7507"
   },
   "source": [
    "### General\n",
    "\n",
    "The dataset needs to be cleaned for e.g. in addition to President Speeches, the corpus included:\n",
    "- correspondence, notes about the speeches including the presidents' names, the dates and location of the speeches etc., audience response.\n",
    "- some of these could be removed easily, others may take considerable time.\n",
    "- I removed duplicate sentences in preparation for the Sentence Embedding Model.\n",
    "- This was not done for Word Embedding. So did not keep all variables controlled in interest of time.\n",
    "- There is an increase in the length of presidential speeches over time.\n",
    "- The John Adams inaugural address is an outlier.\n",
    "    - E.g. when given an idiom it was hard to interpret model performance.\n",
    "- The model may have performed better with more stop words. (Need to validate with more model runs.)\n",
    "- The quality (qualification) of the query matters with semantics.\n",
    "    - E.g. when given an idiom it was hard to interpret model performance.\n",
    "- To better interpret results need to explore the data more. For e.g.\n",
    "    - Is there a preference for longer corpora?\n",
    "    - Is their a bias for older or newer corpora?\n",
    "    - If yes, is that ok.\n",
    "\n",
    "### Observations with SentenceBERT\n",
    "\n",
    "1. As observed in the Data Exploration section, Presidents' speeches are on average longer today that they were decades ago. So the sentence from the inaugural address by John Adam's is an outlier, having many run-on sentences.\n",
    "2. Specific areas for preprocessing improvements have been identified including:\n",
    "    - Removing telegrams and other alternative transcripts and correspondence, introductory and explanatory notes, the president's name and audience response etc\n",
    "    - In some cases this work is significant.\n",
    "3. In the time alotted I was able to limit the corpora analysed by the SentenceBERT model to Unique Sentences i.e. with no duplicates (as sometimes President's/people say somethine more than once.)\n",
    "4. The corpus was not prepared identically both the GLoVe and SentenceBERT.\n",
    "5. SentenceBERT did not accept pre-tokenized words. \n",
    "\n",
    "### Observations with GloVe\n",
    "\n",
    "1. I used GloVe, an unsupervised learning algorithm, \"glove-wiki-gigaword-50\". \n",
    "    - There is a newer release and different versions of GloVe based on from different sources (corpus)\n",
    "    - Perhaps there is a range of corpora better suited to presidential speeches, which are more formally worded.\n",
    "2. The data dates back to 1789 as such, how language is used has changed. This may impact semantic-consistency across the corpora. \n",
    "    - And the results may be biased toward corpora from more modern time periods.\n",
    "    - Or the results may be biased to longer corpora\n",
    "3. Preprocessing should be reviewed so Regex and Tokenizaton can be optimized. \n",
    "    - Increasing stop words may have helped the performance (ability to detect meaning)\n",
    "4. This type of semantic search is limited by the quality of the user input. \n",
    "    - The more meaning conveyed in the query, the better the model can perform.\n",
    "5. Consider comparing the performance against Okapi BM25 Ranking or similar model that is tailored to Information Retrieval.\n",
    "6. Optimize code:\n",
    "    - I get a runtime warning with index[0] only for the sorted indices of similarity.\n",
    "7. Improve how I get the data from the user. There are no checks or error messages.\n",
    "8. I need to investigate how to prove the validity/performance of Text Similarity Models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a260e3",
   "metadata": {
    "id": "f5a260e3"
   },
   "source": [
    "<a name =references></a>\n",
    "    \n",
    "# References\n",
    "\n",
    "[Go to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73bea353",
   "metadata": {
    "id": "73bea353"
   },
   "source": [
    "1. Text Analytics with Python A practical real-world approach to gaining actional insights from your data - Sarkar, Dipanjan\n",
    "2. https://towardsdatascience.com/how-to-rank-text-content-by-semantic-similarity-4d2419a84c32\n",
    "3. GloVe https://nlp.stanford.edu/projects/glove/\n",
    "4. https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/soft_cosine_tutorial.ipynb\n",
    "5. https://towardsdatascience.com/sentiment-analysis-with-python-part-2-4f71e7bde59a#:~:text=library%20for%20Python.-,Removing%20Stop%20Words,the%20performance%20of%20a%20model.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "aa86df4d",
    "dRnp04YmbfR5",
    "02c19827",
    "NsXNmpbcD6GX",
    "0e8cfabf",
    "4361289f",
    "f7c9faae",
    "b6b17094",
    "d47c7d29",
    "544c7507",
    "d27b5d67"
   ],
   "name": "wwps-sentenceBERT.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0039b9fe0eea47c2a5e122dfb032655b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "018247cdd6534bbfade9a7fa01a4799b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0414337c97d3499cbebcd3432c2c7336": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_17683c1f4ced4d80856c1a4ad6e07ffd",
      "max": 438007537,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a44dcf8f47c64a309cad9183a0ed9fd9",
      "value": 438007537
     }
    },
    "090987a261b642118a70b024ec3a48ac": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_be000e6db11841919affefb01d1cc69d",
       "IPY_MODEL_b5222921b8de47bfaac8ab419f7787be",
       "IPY_MODEL_81919ee6fc804539992831f0ac409dda"
      ],
      "layout": "IPY_MODEL_10699f7ce6624826bd823928d67b01f0"
     }
    },
    "09308f7931c34718a93889f1f5cf2d05": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0a6b13bedd834c5290f438c2a1546728": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0b214a559fdb4630b76abd9f76105b42": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "10699f7ce6624826bd823928d67b01f0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "13f6ba81685a48518d9950be82896516": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d8c54e246abc4761b5164d882784f257",
       "IPY_MODEL_b0c231d06a0f46f4b69fe5aa7d6a2fab",
       "IPY_MODEL_185f06562daa4f10a53ea6af2608732d"
      ],
      "layout": "IPY_MODEL_6a25cb1d80fb4ede8f999897cbb5b42d"
     }
    },
    "1477347ea4754213a4a213d4a7753201": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1607642289d7410ea6c904d091e731c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "16333771bfac437a9027862235d6d761": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "17683c1f4ced4d80856c1a4ad6e07ffd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "179b808a4a0f4243a51c6f4296b59c1b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "185f06562daa4f10a53ea6af2608732d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_22972150eca4497ca17fa53245566449",
      "placeholder": "​",
      "style": "IPY_MODEL_d2d127b1606c40b499ff33b1c66081d4",
      "value": " 122/122 [00:00&lt;00:00, 3.37kB/s]"
     }
    },
    "19819014ba7546e4ae5a1c9ff14d7cba": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "19c73a8fc9b648638f94551887ff27f6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_018247cdd6534bbfade9a7fa01a4799b",
      "max": 466081,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f8afbf5129034ccd98cfa70327280dee",
      "value": 466081
     }
    },
    "1aca5121809d43d1b152f566c705d14d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1e66e45cbc094cccb8c4d868d6353624": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1fb3012c041c4b28b2ab2aa67c1ba79c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_37dcc496d656434d86498bf8a56e0df5",
      "placeholder": "​",
      "style": "IPY_MODEL_09308f7931c34718a93889f1f5cf2d05",
      "value": " 466k/466k [00:00&lt;00:00, 6.58MB/s]"
     }
    },
    "2027c851b89e459d84c494ec409c1f83": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "227032656b4b4a8285e0ea6fd07a8295": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a7d6446f467048e4ac53c3baaddd0c15",
      "placeholder": "​",
      "style": "IPY_MODEL_2027c851b89e459d84c494ec409c1f83",
      "value": " 399/399 [00:00&lt;00:00, 10.0kB/s]"
     }
    },
    "22972150eca4497ca17fa53245566449": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "25f7bbe46548478eacab81df6da5074e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "29e91b06de6948bca0ca591c324215eb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2a4651d1cfed4ae39bfac35f108f7589": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3dab40316a9a410183acf7a80e1e5388",
      "placeholder": "​",
      "style": "IPY_MODEL_c2c0992fea9c460f8814ace1ab628a85",
      "value": "Downloading: 100%"
     }
    },
    "2c42f95bd2d74c88a2f9c2b4f006a1ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2ee032c317884769973cb2a0d13af814": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_db5f8a1a9a1b4bdd930a055d92ad300d",
       "IPY_MODEL_9738ee92ddfd4162b8c79866e5cc5041",
       "IPY_MODEL_546210b1798f41559fad6d407b1b4247"
      ],
      "layout": "IPY_MODEL_a88b0b897977487d940feffef5371323"
     }
    },
    "37dcc496d656434d86498bf8a56e0df5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39ad7d29e62f43b7b87d0d7f8f22a017": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3dab40316a9a410183acf7a80e1e5388": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3dfe3a3fcbf24b0f899277955e14e6e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3e8d762412e64a719ecbd93b5d307712": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4154336b17d7499ba83c5bfc1cb5ee5f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "44f162d6bb6849c5b42606b15509812e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "45b05c167b124fa49f11a6850093e592": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "46b5a0c3c06949d595ff53606d6da2e0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5ef62e01def84e1f9651bf9729fda3eb",
      "placeholder": "​",
      "style": "IPY_MODEL_0a6b13bedd834c5290f438c2a1546728",
      "value": " 438M/438M [00:11&lt;00:00, 36.4MB/s]"
     }
    },
    "47e8b0b6433247429764d0eb698e3db4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c95b134a4f78454b9859e8b6f39dd9c0",
      "max": 229,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7b9f857c7812443691e542ac6b6d9844",
      "value": 229
     }
    },
    "4945e30fc1124c728b3ad41b944671d5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b3d04e9727b430d8e036c527d873922": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4d2a1d01e7a140a0887eab486157eef2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_951440988f45421ea90ca35893a9a925",
      "max": 399,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_634a6270b47d4176a8fffbd98863c48b",
      "value": 399
     }
    },
    "4f67e02829e14b1f8359a783facb625d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "53e7152513ec4414883043d56966cf4b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "544e60ad7326423a83cd2fb81f3a5913": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "546210b1798f41559fad6d407b1b4247": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cd21491a987a4291b5a8580107442a37",
      "placeholder": "​",
      "style": "IPY_MODEL_6ec60e17b6b4411fb5c2552d42d3e5ac",
      "value": " 625/625 [00:00&lt;00:00, 10.8kB/s]"
     }
    },
    "590705d7d3034148884aba5540536a98": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e9e3bd02a5914ee385f64867b1f6b416",
      "max": 391,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_97b071710fbf4f9b8e9c0aecaaca6889",
      "value": 391
     }
    },
    "5ef62e01def84e1f9651bf9729fda3eb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "62bd80b93dee4d0e9d3faa0ba61b037b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b7a2e7c07f2847d5a76a0ffde37753f8",
       "IPY_MODEL_c99bf972ca0741f5a8e88a58aa8f012b",
       "IPY_MODEL_63985c8d09174c75bc31de679fc9fda2"
      ],
      "layout": "IPY_MODEL_d320e5eb040d4ab393bcbca3af7be3a4"
     }
    },
    "634a6270b47d4176a8fffbd98863c48b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "63985c8d09174c75bc31de679fc9fda2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_53e7152513ec4414883043d56966cf4b",
      "placeholder": "​",
      "style": "IPY_MODEL_b05cb01cb0f74c09af6a4238ef0006b5",
      "value": " 112/112 [00:00&lt;00:00, 2.18kB/s]"
     }
    },
    "6547c0a3bfe846e5881d0554ab1e0887": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "65677cba08fa40239c029b3b51bb525c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fa0d243ca9b346b4b801f70326572f91",
      "placeholder": "​",
      "style": "IPY_MODEL_ca99509a03aa437fa3d3b75f7887c2d0",
      "value": "Downloading: 100%"
     }
    },
    "69bbe045804841829a390f3ba10e6cf2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6a25cb1d80fb4ede8f999897cbb5b42d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6be0fcc03c8f438e9f12821476178232": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c557dea102b495f9cdab0b6cd766d1e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6c57e099739343e0bb8370d3d2bbf23b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4945e30fc1124c728b3ad41b944671d5",
      "placeholder": "​",
      "style": "IPY_MODEL_16333771bfac437a9027862235d6d761",
      "value": "Downloading: 100%"
     }
    },
    "6ec60e17b6b4411fb5c2552d42d3e5ac": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6fb98517050646e0871fdd1ec43e77df": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7715f17b919e4016b2a36123aa30d44c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "773fddfe7a264255bf3f4cdba5866514": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0039b9fe0eea47c2a5e122dfb032655b",
      "placeholder": "​",
      "style": "IPY_MODEL_f3688e55e852453ebe06cf2490eb6028",
      "value": "Downloading: 100%"
     }
    },
    "7826efa8cd7a465c870fbf93fd51aa2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d6b631f2f6244d159c8b735a2930f925",
      "placeholder": "​",
      "style": "IPY_MODEL_83062e62da9d4d1e9921a1c9e5681eb0",
      "value": " 3.95k/3.95k [00:00&lt;00:00, 111kB/s]"
     }
    },
    "78ee08698822442faae1d6bc690bda60": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8327629fe72a4f9981d8147d33d52ba2",
       "IPY_MODEL_47e8b0b6433247429764d0eb698e3db4",
       "IPY_MODEL_ae70eb389fbc49dca15ce17501dff227"
      ],
      "layout": "IPY_MODEL_e485622c18d34d4fac700952a32c3601"
     }
    },
    "7b2a84f00c7b4d6083262c3e5c541755": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e56ac064e79842f390c1a52c0d8a4476",
      "max": 190,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a32f82578cff4f1ab0b8a460334ea95a",
      "value": 190
     }
    },
    "7b9f857c7812443691e542ac6b6d9844": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "7c4ce9dd083240c7b18a63a4c5e28802": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7d100f3f328444fabad03b4b3a247eb5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7e341e85b25b4bf3bcdce21f5ce084f6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f1d8507a1ec94139b0c54d60d16fc5d0",
       "IPY_MODEL_590705d7d3034148884aba5540536a98",
       "IPY_MODEL_7ffd047eccf643c6b6e854be1df58a75"
      ],
      "layout": "IPY_MODEL_932972e1361040fd86e865c24fe5c5ca"
     }
    },
    "7ffd047eccf643c6b6e854be1df58a75": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8ec1cdc9d10e4062863b680187e72ffb",
      "placeholder": "​",
      "style": "IPY_MODEL_45b05c167b124fa49f11a6850093e592",
      "value": " 391/391 [00:00&lt;00:00, 8.82kB/s]"
     }
    },
    "81919ee6fc804539992831f0ac409dda": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_19819014ba7546e4ae5a1c9ff14d7cba",
      "placeholder": "​",
      "style": "IPY_MODEL_fdd7db36651b497dbbc6ef6d88e6c05b",
      "value": " 2.00/2.00 [00:00&lt;00:00, 36.2B/s]"
     }
    },
    "83062e62da9d4d1e9921a1c9e5681eb0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8327629fe72a4f9981d8147d33d52ba2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dafc6ae8f7b54941b93f04ca8bbebbfe",
      "placeholder": "​",
      "style": "IPY_MODEL_7d100f3f328444fabad03b4b3a247eb5",
      "value": "Downloading: 100%"
     }
    },
    "8b287ae1fb374bc39ed4c3b460b4b6bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2a4651d1cfed4ae39bfac35f108f7589",
       "IPY_MODEL_f6ec7ba67e5c431d84d4dfa1a543c365",
       "IPY_MODEL_7826efa8cd7a465c870fbf93fd51aa2c"
      ],
      "layout": "IPY_MODEL_1aca5121809d43d1b152f566c705d14d"
     }
    },
    "8e48d02c0c5047d0a21c8773cc0a2825": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_da9c7a1951a946dda4c18e040f0809a0",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1e66e45cbc094cccb8c4d868d6353624",
      "value": 231508
     }
    },
    "8ec1cdc9d10e4062863b680187e72ffb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8f624553120842ceb2b171e82bddb2ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "932972e1361040fd86e865c24fe5c5ca": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9425e3a5a1aa49bb8bd0ca73b619a51d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4154336b17d7499ba83c5bfc1cb5ee5f",
      "placeholder": "​",
      "style": "IPY_MODEL_f501b021e29b415eb6bf9543f41c7610",
      "value": " 190/190 [00:00&lt;00:00, 4.29kB/s]"
     }
    },
    "947629a6421747c0b0a7d06e08756668": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bc1c0147edae484981187292aa4cb03c",
      "placeholder": "​",
      "style": "IPY_MODEL_6c557dea102b495f9cdab0b6cd766d1e",
      "value": " 53.0/53.0 [00:00&lt;00:00, 1.32kB/s]"
     }
    },
    "951440988f45421ea90ca35893a9a925": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9738ee92ddfd4162b8c79866e5cc5041": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c4ce9dd083240c7b18a63a4c5e28802",
      "max": 625,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f315861665dd4ab38830b3cfdeb20349",
      "value": 625
     }
    },
    "97b071710fbf4f9b8e9c0aecaaca6889": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9c21b793c5a9448cb719ef0923556c11": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e72764e4e95b4c8e82f908acc7d587dc",
      "placeholder": "​",
      "style": "IPY_MODEL_ef5a0f74fe0242c98b01f6875921d3e1",
      "value": "Downloading: 100%"
     }
    },
    "a1afdfb299264ad29e5deb056a944d5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_fd396c2481b94d7fa88ef0e3460955a7",
       "IPY_MODEL_7b2a84f00c7b4d6083262c3e5c541755",
       "IPY_MODEL_9425e3a5a1aa49bb8bd0ca73b619a51d"
      ],
      "layout": "IPY_MODEL_ac50490027214989b8ec2537bedbe719"
     }
    },
    "a32f82578cff4f1ab0b8a460334ea95a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a44dcf8f47c64a309cad9183a0ed9fd9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a5951beafd7a497387d98f3bda548595": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a6400ac3c5364743be7454651d84a15f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6fb98517050646e0871fdd1ec43e77df",
      "placeholder": "​",
      "style": "IPY_MODEL_3dfe3a3fcbf24b0f899277955e14e6e1",
      "value": "Downloading: 100%"
     }
    },
    "a7d6446f467048e4ac53c3baaddd0c15": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a88b0b897977487d940feffef5371323": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ac50490027214989b8ec2537bedbe719": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ad80cb459d574e6f8e45ce1545576f40": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae70eb389fbc49dca15ce17501dff227": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b8a1009a648c49988c42d983790ca086",
      "placeholder": "​",
      "style": "IPY_MODEL_cbf6480ef1c74a12887ad74e5a44caf6",
      "value": " 229/229 [00:00&lt;00:00, 4.44kB/s]"
     }
    },
    "afe48b5c2cf9482a9fce0fdd6259e0a3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6c57e099739343e0bb8370d3d2bbf23b",
       "IPY_MODEL_8e48d02c0c5047d0a21c8773cc0a2825",
       "IPY_MODEL_f950c954adee46a591f53e848cf66362"
      ],
      "layout": "IPY_MODEL_0b214a559fdb4630b76abd9f76105b42"
     }
    },
    "b05cb01cb0f74c09af6a4238ef0006b5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b0c231d06a0f46f4b69fe5aa7d6a2fab": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6be0fcc03c8f438e9f12821476178232",
      "max": 122,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a5951beafd7a497387d98f3bda548595",
      "value": 122
     }
    },
    "b1391ab4aa424f3fa82814e6a0a6ef72": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a6400ac3c5364743be7454651d84a15f",
       "IPY_MODEL_19c73a8fc9b648638f94551887ff27f6",
       "IPY_MODEL_1fb3012c041c4b28b2ab2aa67c1ba79c"
      ],
      "layout": "IPY_MODEL_fb44d8006b2a479c9f3f87446c6eec9d"
     }
    },
    "b2df9c1f2b4148bcbe178431df4adc17": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b5222921b8de47bfaac8ab419f7787be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1607642289d7410ea6c904d091e731c0",
      "max": 2,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2c42f95bd2d74c88a2f9c2b4f006a1ce",
      "value": 2
     }
    },
    "b57fce2473f542a78b1783822b613674": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b742c522693646e99372cf3bfa6a766b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b7a2e7c07f2847d5a76a0ffde37753f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_544e60ad7326423a83cd2fb81f3a5913",
      "placeholder": "​",
      "style": "IPY_MODEL_39ad7d29e62f43b7b87d0d7f8f22a017",
      "value": "Downloading: 100%"
     }
    },
    "b8a1009a648c49988c42d983790ca086": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bc1c0147edae484981187292aa4cb03c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bc6a0e5e9e9c411cb693fde0347c79b4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "be000e6db11841919affefb01d1cc69d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f23617143c7e47148d4a6c5c3dde9736",
      "placeholder": "​",
      "style": "IPY_MODEL_179b808a4a0f4243a51c6f4296b59c1b",
      "value": "Downloading: 100%"
     }
    },
    "c2c0992fea9c460f8814ace1ab628a85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c3ebf933cd75477390628363425415f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9c21b793c5a9448cb719ef0923556c11",
       "IPY_MODEL_e695c0c27bdc4169b1703d28b420cd36",
       "IPY_MODEL_947629a6421747c0b0a7d06e08756668"
      ],
      "layout": "IPY_MODEL_ad80cb459d574e6f8e45ce1545576f40"
     }
    },
    "c8fb2a66efdf4893b5765dd762352d76": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_773fddfe7a264255bf3f4cdba5866514",
       "IPY_MODEL_0414337c97d3499cbebcd3432c2c7336",
       "IPY_MODEL_46b5a0c3c06949d595ff53606d6da2e0"
      ],
      "layout": "IPY_MODEL_44f162d6bb6849c5b42606b15509812e"
     }
    },
    "c95b134a4f78454b9859e8b6f39dd9c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c99bf972ca0741f5a8e88a58aa8f012b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6547c0a3bfe846e5881d0554ab1e0887",
      "max": 112,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4b3d04e9727b430d8e036c527d873922",
      "value": 112
     }
    },
    "ca99509a03aa437fa3d3b75f7887c2d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cbf6480ef1c74a12887ad74e5a44caf6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cd21491a987a4291b5a8580107442a37": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d2d127b1606c40b499ff33b1c66081d4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d320e5eb040d4ab393bcbca3af7be3a4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d6b631f2f6244d159c8b735a2930f925": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d8c54e246abc4761b5164d882784f257": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_69bbe045804841829a390f3ba10e6cf2",
      "placeholder": "​",
      "style": "IPY_MODEL_b742c522693646e99372cf3bfa6a766b",
      "value": "Downloading: 100%"
     }
    },
    "da9c7a1951a946dda4c18e040f0809a0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dafc6ae8f7b54941b93f04ca8bbebbfe": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "db5f8a1a9a1b4bdd930a055d92ad300d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_25f7bbe46548478eacab81df6da5074e",
      "placeholder": "​",
      "style": "IPY_MODEL_ec0a069a87254c889f99024c38a15149",
      "value": "Downloading: 100%"
     }
    },
    "e485622c18d34d4fac700952a32c3601": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e56ac064e79842f390c1a52c0d8a4476": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e695c0c27bdc4169b1703d28b420cd36": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3e8d762412e64a719ecbd93b5d307712",
      "max": 53,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_fc9df5c3d7cc47cc9ce9f7be3e4dc2c2",
      "value": 53
     }
    },
    "e72764e4e95b4c8e82f908acc7d587dc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e9e3bd02a5914ee385f64867b1f6b416": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ec0a069a87254c889f99024c38a15149": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ef5a0f74fe0242c98b01f6875921d3e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f1d8507a1ec94139b0c54d60d16fc5d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fd15208e042246d8ae9fc58419321e2a",
      "placeholder": "​",
      "style": "IPY_MODEL_b57fce2473f542a78b1783822b613674",
      "value": "Downloading: 100%"
     }
    },
    "f23617143c7e47148d4a6c5c3dde9736": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f315861665dd4ab38830b3cfdeb20349": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f3688e55e852453ebe06cf2490eb6028": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f3ef6fb56e5743e58e5f26446b1c9980": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_65677cba08fa40239c029b3b51bb525c",
       "IPY_MODEL_4d2a1d01e7a140a0887eab486157eef2",
       "IPY_MODEL_227032656b4b4a8285e0ea6fd07a8295"
      ],
      "layout": "IPY_MODEL_4f67e02829e14b1f8359a783facb625d"
     }
    },
    "f501b021e29b415eb6bf9543f41c7610": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f6ec7ba67e5c431d84d4dfa1a543c365": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b2df9c1f2b4148bcbe178431df4adc17",
      "max": 3951,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_29e91b06de6948bca0ca591c324215eb",
      "value": 3951
     }
    },
    "f8afbf5129034ccd98cfa70327280dee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f950c954adee46a591f53e848cf66362": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bc6a0e5e9e9c411cb693fde0347c79b4",
      "placeholder": "​",
      "style": "IPY_MODEL_1477347ea4754213a4a213d4a7753201",
      "value": " 232k/232k [00:00&lt;00:00, 3.56MB/s]"
     }
    },
    "fa0d243ca9b346b4b801f70326572f91": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fb44d8006b2a479c9f3f87446c6eec9d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fc9df5c3d7cc47cc9ce9f7be3e4dc2c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "fd15208e042246d8ae9fc58419321e2a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fd396c2481b94d7fa88ef0e3460955a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7715f17b919e4016b2a36123aa30d44c",
      "placeholder": "​",
      "style": "IPY_MODEL_8f624553120842ceb2b171e82bddb2ec",
      "value": "Downloading: 100%"
     }
    },
    "fdd7db36651b497dbbc6ef6d88e6c05b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}