TweetSentimentClassification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiMuzQfQw9xb"
      },
      "source": [
        "#  <center>Tweet Sentiment Classification <br> <small> Martin Sejas\n",
        "About the dataset:\n",
        "\n",
        "List of tweet texts with emotion labels like joy, sadness, fear, anger...\n",
        "Dataset is split into train, test and validation sets for building the machine learning model. At first, you are\n",
        "given only train and test sets. The validation one will be given in the end of the project for you to check\n",
        "the final performance of your algorithm (to make sure there is no overfitting over the test data).\n",
        "You can work on this project on group of one, two or three students. This exercise is mandatory, not\n",
        "giving it back is equivalent to getting to lowest grade.\n",
        "Goal:\n",
        "\n",
        "• Train different kind of models able to classify each text according to the sentiment mainly present\n",
        "in it\n",
        "\n",
        "• Compare the results of your different models and try to analyze and explain the differences\n",
        "\n",
        "Train different classification models relying mainly on\n",
        "\n",
        "1. A Fully Connected Neural Network (see Course 2) 5 points\n",
        "\n",
        "2. A Recurrent Neural Network, based on LSTM or GRU (see Course 3) 5 points\n",
        "\n",
        "3. A fine-tuned Transformer Architecture from a pretrained model that can be found on sites\n",
        "like HuggingFace (see Course 4) 5 points\n",
        "\n",
        "4. Compare the different models to find the best approach and try to duplicate it on a “real life”\n",
        "text classification approach (this new “real life” dataset will be given to you soon) 5 points"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ijX_IQb8w9xd"
      },
      "source": [
        "# Loading and Preprocessing the data sets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H9-Ro1Pcw9xd"
      },
      "outputs": [],
      "source": [
        "import math\n",
        "import spacy\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "from sklearn.metrics import confusion_matrix, accuracy_score, classification_report\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from keras.utils import to_categorical\n",
        "from tensorflow.keras import Sequential\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.metrics import Precision, Recall, AUC\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from keras.layers import Embedding, LSTM, Dense, Dropout, Bidirectional\n",
        "from keras.callbacks import EarlyStopping, LearningRateScheduler, CallbackList, ReduceLROnPlateau\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fc6ncigw9xe"
      },
      "source": [
        "The dataset is a tweets sentiment dataset, it's quite straightforward, there are only 2 columns, the tweet itself, and the sentiment attached to it. We have been provided a train set and a test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "N0tKIWlfw9xe",
        "outputId": "3840602c-f1f0-4307-fdcd-fea8c3c76a79"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-ae55dbb3-de42-425c-977e-591de03f2b82\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>i didnt feel humiliated</td>\n",
              "      <td>sadness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>i can go from feeling so hopeless to so damned...</td>\n",
              "      <td>sadness</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>im grabbing a minute to post i feel greedy wrong</td>\n",
              "      <td>anger</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>i am ever feeling nostalgic about the fireplac...</td>\n",
              "      <td>love</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>i am feeling grouchy</td>\n",
              "      <td>anger</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ae55dbb3-de42-425c-977e-591de03f2b82')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-e030b829-509e-4de0-826d-b3dbebcb6db6\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e030b829-509e-4de0-826d-b3dbebcb6db6')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-e030b829-509e-4de0-826d-b3dbebcb6db6 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ae55dbb3-de42-425c-977e-591de03f2b82 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ae55dbb3-de42-425c-977e-591de03f2b82');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                               tweet sentiment\n",
              "0                            i didnt feel humiliated   sadness\n",
              "1  i can go from feeling so hopeless to so damned...   sadness\n",
              "2   im grabbing a minute to post i feel greedy wrong     anger\n",
              "3  i am ever feeling nostalgic about the fireplac...      love\n",
              "4                               i am feeling grouchy     anger"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train = pd.read_csv('./train.txt', header=None, delimiter=';')\n",
        "df_test = pd.read_csv('./test.txt', header=None, delimiter=';')\n",
        "df_train = df_train.rename(columns={0: 'tweet', 1: 'sentiment'})\n",
        "df_test = df_test.rename(columns={0: 'tweet', 1: 'sentiment'})\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UDZebKhQw9xf"
      },
      "source": [
        "After loading our dataset we can see the class (sentiment) distribution of our tweets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jdmVDmpKw9xf",
        "outputId": "bdf6fff5-b03e-40e7-bab3-d09c21a2b9cd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(joy         5362\n",
              " sadness     4666\n",
              " anger       2159\n",
              " fear        1937\n",
              " love        1304\n",
              " surprise     572\n",
              " Name: sentiment, dtype: int64,\n",
              " (16000, 2))"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train['sentiment'].value_counts(), df_train.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k2KUWQaMw9xf",
        "outputId": "89dcbf75-0115-452f-e0aa-72d24656a97f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(joy         695\n",
              " sadness     581\n",
              " anger       275\n",
              " fear        224\n",
              " love        159\n",
              " surprise     66\n",
              " Name: sentiment, dtype: int64,\n",
              " (2000, 2))"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_test['sentiment'].value_counts(), df_test.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3TbKTKsfw9xf"
      },
      "source": [
        "We can see here that there are class inbalances. Especially regarding joy and surprise.\n",
        "\n",
        "We will also separate each tweet from it's label to train."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LwaSIpB2w9xf"
      },
      "outputs": [],
      "source": [
        "texts_train = df_train['tweet']\n",
        "texts_test = df_test['tweet']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HfsZ_HE4w9xf",
        "outputId": "48b5bc49-1860-4362-daac-7421d37b3010"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([4, 4, 0, ..., 2, 0, 4])"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "le = LabelEncoder()\n",
        "y_train = le.fit_transform(df_train['sentiment'])\n",
        "y_test = le.transform(df_test['sentiment'])\n",
        "y_train"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_xQxJ1ojw9xf"
      },
      "source": [
        "## Pre-Processing the tweets (Lemmetizing) with the Spacy library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KPa9FZlGw9xf"
      },
      "source": [
        "### Lemmetization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaWeFcHEw9xg"
      },
      "source": [
        "We are going to be using the famous [Spacy](https://spacy.io/) Library for tokenization, it's the best performing one in terms of computational cost and performance.\n",
        "\n",
        "We will be loading the english core *medium* dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6d1WH_qw9xg"
      },
      "source": [
        "To have spacy working please make sure to run the following commands.\n",
        "\n",
        "```bash\n",
        "pip install spacy\n",
        "```\n",
        "\n",
        "Once the package has been installed, make sure to download the language model, in our case, *en-core-web-md* will be used, and can be downloaded by the following command.\n",
        "\n",
        "```bash\n",
        "python -m spacy download en_core_web_md\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8ihFJGgBxosS",
        "outputId": "2f7c2f2e-2a3a-4a18-8f92-ff4efc808cdf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (3.5.4)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.4)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.9)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.7)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.8)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy) (8.1.10)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.4.6)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.8)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.10.2)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (6.3.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (4.65.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.22.4)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.27.1)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.10.11)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (23.1)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy) (4.7.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.4)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.1.0)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy) (8.1.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy) (2.1.3)\n"
          ]
        }
      ],
      "source": [
        "!pip install spacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_uTtnJ3NxqVH",
        "outputId": "7a67c090-36b3-4462-ce4e-1d2cee6563d0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-07-20 14:55:34.552592: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "Collecting en-core-web-md==3.5.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.5.0/en_core_web_md-3.5.0-py3-none-any.whl (42.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.8/42.8 MB\u001b[0m \u001b[31m17.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.6.0,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from en-core-web-md==3.5.0) (3.5.4)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.0.4)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.0.9)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.0.7)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (3.0.8)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (8.1.10)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.4.6)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.0.8)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (0.10.2)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (6.3.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (4.65.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.22.4)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.27.1)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.10.11)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (23.1)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (4.7.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (3.4)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (0.7.9)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (0.1.0)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (8.1.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.6.0,>=3.5.0->en-core-web-md==3.5.0) (2.1.3)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_md')\n"
          ]
        }
      ],
      "source": [
        "!python -m spacy download en_core_web_md"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0i2IMpbtw9xg"
      },
      "outputs": [],
      "source": [
        "nlp = spacy.load('en_core_web_md')\n",
        "\n",
        "#Function to tokenize\n",
        "def preprocess_text(text):\n",
        "    #every tweet will become a document\n",
        "    doc = nlp(text)\n",
        "\n",
        "    #Creating a list of tokens by lemmatizing the words, filtering stop words.\n",
        "    tokens = [token.lemma_.lower().strip() for token in doc if not token.is_stop]\n",
        "\n",
        "    #Join all the words in one sentence\n",
        "    return ' '.join(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RNddfp3hw9xg"
      },
      "source": [
        "With the *preprocess_text* function, we will pre-process the tweet text, and with that we can apply it to both of our dataframes as a new series, and see the output.\n",
        "\n",
        "*Warning this cell takes over 2 minutes to run!*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "P234c0Qhw9xg",
        "outputId": "1e2d5c99-10ed-4b50-c3a2-7027d7b3c8f6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-ab4d3f2f-69d4-4ef9-be7f-83edb2934cef\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>processed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>i didnt feel humiliated</td>\n",
              "      <td>sadness</td>\n",
              "      <td>not feel humiliate</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>i can go from feeling so hopeless to so damned...</td>\n",
              "      <td>sadness</td>\n",
              "      <td>feel hopeless damn hopeful care awake</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>im grabbing a minute to post i feel greedy wrong</td>\n",
              "      <td>anger</td>\n",
              "      <td>m grab minute post feel greedy wrong</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>i am ever feeling nostalgic about the fireplac...</td>\n",
              "      <td>love</td>\n",
              "      <td>feel nostalgic fireplace know property</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>i am feeling grouchy</td>\n",
              "      <td>anger</td>\n",
              "      <td>feel grouchy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ab4d3f2f-69d4-4ef9-be7f-83edb2934cef')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-ba66769c-e44d-49e3-9010-a565e12a58de\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ba66769c-e44d-49e3-9010-a565e12a58de')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-ba66769c-e44d-49e3-9010-a565e12a58de button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ab4d3f2f-69d4-4ef9-be7f-83edb2934cef button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ab4d3f2f-69d4-4ef9-be7f-83edb2934cef');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                               tweet sentiment  \\\n",
              "0                            i didnt feel humiliated   sadness   \n",
              "1  i can go from feeling so hopeless to so damned...   sadness   \n",
              "2   im grabbing a minute to post i feel greedy wrong     anger   \n",
              "3  i am ever feeling nostalgic about the fireplac...      love   \n",
              "4                               i am feeling grouchy     anger   \n",
              "\n",
              "                                processed  \n",
              "0                      not feel humiliate  \n",
              "1   feel hopeless damn hopeful care awake  \n",
              "2    m grab minute post feel greedy wrong  \n",
              "3  feel nostalgic fireplace know property  \n",
              "4                            feel grouchy  "
            ]
          },
          "execution_count": 38,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train['processed'] = df_train['tweet'].apply(preprocess_text)\n",
        "df_test['processed'] = df_test['tweet'].apply(preprocess_text)\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4s87w_0cw9xg"
      },
      "source": [
        "## Task 1- Fully Connected Neural Network(5 Points)\n",
        "### TF-IDF embedding for Sequential Neural Network\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHDe3KA6w9xg"
      },
      "source": [
        "We will be using sklearn's TFidfVectorizer to be able to process the data and feed it into our Neural Network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pcFGE1Rqw9xg",
        "outputId": "1259a7ce-4cc3-4e08-8bcd-3b5e82e5c334"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(16000, 5587)"
            ]
          },
          "execution_count": 39,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, stop_words='english')\n",
        "\n",
        "dtm_train = tfidf.fit_transform(df_train['processed'])\n",
        "dtm_test = tfidf.transform(df_test['processed'])\n",
        "dtm_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gB3bbKcCw9xg"
      },
      "source": [
        "Split the training set to training and validation sets for training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9o_L8ExDw9xg"
      },
      "outputs": [],
      "source": [
        "X_train_ = dtm_train.toarray()\n",
        "X_test_ = dtm_test.toarray()\n",
        "y_train_ = y_train\n",
        "y_test_ = y_test\n",
        "# X_train_ = np.vstack((X_train_, X_test_))\n",
        "# y_train_ = np.concatenate((y_train, y_test))\n",
        "\n",
        "\n",
        "class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)\n",
        "\n",
        "\n",
        "# X_train_, X_test_, y_train, y_test_ = train_test_split(X_train_, y_train, test_size=0.2, random_state=42, stratify=y_train)\n",
        "X_train_, X_val_, y_train_, y_val_ = train_test_split(X_train_, y_train, test_size=0.2, random_state=42, stratify=y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YKaeaIKGw9xg"
      },
      "source": [
        "Encode the label in one hot encoding fashion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "93YvHwq3w9xh",
        "outputId": "9cce109a-ffab-4dd9-9255-6f530ba8141e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "((12800, 6), (2000, 6), (3200, 6))"
            ]
          },
          "execution_count": 41,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_train_encoded = to_categorical(y_train_)\n",
        "y_test_encoded = to_categorical(y_test_)\n",
        "y_val_encoded = to_categorical(y_val_)\n",
        "y_train_encoded.shape, y_test_encoded.shape, y_val_encoded.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjAtjjYQw9xh"
      },
      "source": [
        "# Fully connect NN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0y-AK0Y_w9xh"
      },
      "source": [
        "Build the fully connected NN with tensorflow.\n",
        "\n",
        "The first layer has a high number of nerons because the input data set has `>5000` features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r1kFDNv-w9xh",
        "outputId": "58a732df-8674-4a26-fe0e-3f49a64e5d01"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_5 (Dense)             (None, 4096)              22888448  \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 2048)              8390656   \n",
            "                                                                 \n",
            " dense_7 (Dense)             (None, 1024)              2098176   \n",
            "                                                                 \n",
            " dense_8 (Dense)             (None, 64)                65600     \n",
            "                                                                 \n",
            " dense_9 (Dense)             (None, 6)                 390       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 33,443,270\n",
            "Trainable params: 33,443,270\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(4096, activation='selu', kernel_initializer='lecun_normal', input_shape=(X_train_.shape[1],), kernel_regularizer=tf.keras.regularizers.l2(0.01)))\n",
        "model.add(Dense(2048, activation='selu', kernel_initializer='lecun_normal', kernel_regularizer=tf.keras.regularizers.l2(0.01)))\n",
        "model.add(Dense(1024, activation='selu', kernel_initializer='lecun_normal', kernel_regularizer=tf.keras.regularizers.l2(0.1)))\n",
        "# model.add(Dense(512, activation='selu', kernel_initializer='lecun_normal', ))\n",
        "# model.add(Dense(128, activation='selu', kernel_initializer='lecun_normal', ))\n",
        "model.add(Dense(64, activation='selu',))\n",
        "model.add(Dense(6, activation='softmax'))\n",
        "\n",
        "model.layers[-1].bias.assign(class_weights)\n",
        "model.compile(optimizer='Adam', loss=tf.losses.categorical_crossentropy, metrics=['accuracy'])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgKe82i5w9xh"
      },
      "source": [
        "Set up the early stopping condition and use a dynamic learning rate scheduler"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TX_BkxxRw9xh"
      },
      "outputs": [],
      "source": [
        "def step_decay(epoch):\n",
        "    initial_lrate = 0.001\n",
        "    drop = 0.5\n",
        "    epochs_drop = 10.0\n",
        "    lrate = initial_lrate * math.pow(drop, math.floor((1 + epoch) / epochs_drop))\n",
        "    return lrate\n",
        "\n",
        "lr_scheduler = LearningRateScheduler(step_decay)\n",
        "\n",
        "early_stopping = EarlyStopping(monitor='val_loss', restore_best_weights=True, patience=3) #"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKvID1piw9xh"
      },
      "source": [
        "Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KQW_WgcFw9xh",
        "outputId": "34bdf3a3-c66b-47e6-9167-2af4399fd81c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "50/50 [==============================] - 4s 37ms/step - loss: 37.1047 - accuracy: 0.6616 - val_loss: 2.2726 - val_accuracy: 0.7844 - lr: 0.0010\n",
            "Epoch 2/20\n",
            "50/50 [==============================] - 1s 29ms/step - loss: 1.4871 - accuracy: 0.8019 - val_loss: 1.3084 - val_accuracy: 0.7962 - lr: 0.0010\n",
            "Epoch 3/20\n",
            "50/50 [==============================] - 1s 28ms/step - loss: 1.2147 - accuracy: 0.8252 - val_loss: 1.2724 - val_accuracy: 0.7987 - lr: 0.0010\n",
            "Epoch 4/20\n",
            "50/50 [==============================] - 1s 26ms/step - loss: 1.0433 - accuracy: 0.8612 - val_loss: 1.0891 - val_accuracy: 0.8347 - lr: 0.0010\n",
            "Epoch 5/20\n",
            "50/50 [==============================] - 1s 25ms/step - loss: 0.9411 - accuracy: 0.8799 - val_loss: 1.0446 - val_accuracy: 0.8434 - lr: 0.0010\n",
            "Epoch 6/20\n",
            "50/50 [==============================] - 1s 22ms/step - loss: 0.8852 - accuracy: 0.8995 - val_loss: 1.1083 - val_accuracy: 0.8288 - lr: 0.0010\n",
            "Epoch 7/20\n",
            "50/50 [==============================] - 1s 23ms/step - loss: 0.8901 - accuracy: 0.9027 - val_loss: 1.1112 - val_accuracy: 0.8309 - lr: 0.0010\n",
            "Epoch 8/20\n",
            "50/50 [==============================] - 1s 24ms/step - loss: 0.8703 - accuracy: 0.9120 - val_loss: 1.0964 - val_accuracy: 0.8356 - lr: 0.0010\n"
          ]
        }
      ],
      "source": [
        "history = model.fit(X_train_, y_train_encoded, validation_data=(X_val_, y_val_encoded), epochs=20, batch_size=256, callbacks=[early_stopping, lr_scheduler])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vW1RgK5Lw9xh"
      },
      "source": [
        "Plot the training loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "v6KHTstaw9xh",
        "outputId": "66b37ffa-852a-4b42-b41d-8ff549eae6ec"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA65klEQVR4nO3dfXhU9Z3//9eZycwkIZmBcJMbCHfKbSKsBcWIdVWoiC2LlVqrbMV605UrWIG6dfO72qrfbo3tftfadpHWbot1K6XbfsVWW0VFwNYFBVyq3EgBEZCEgAiZJJBJMuf8/pjMJIGAmWRmztw8H9d1LmbOOXPOe4JtXnzujmFZliUAAIAEcdhdAAAAyCyEDwAAkFCEDwAAkFCEDwAAkFCEDwAAkFCEDwAAkFCEDwAAkFCEDwAAkFBZdhdwJtM0VVNTo/z8fBmGYXc5AACgByzLUkNDg0pKSuRwnL9tI+nCR01NjUpLS+0uAwAA9MKhQ4c0bNiw856TdOEjPz9fUqh4r9drczUAAKAn/H6/SktLI7/Hzyfpwke4q8Xr9RI+AABIMT0ZMsGAUwAAkFCEDwAAkFCEDwAAkFBJN+YDAADLstTW1qZgMGh3KejE5XLJ6XT2+TqEDwBAUmlpaVFtba1OnTpldyk4g2EYGjZsmPLy8vp0HcIHACBpmKap/fv3y+l0qqSkRG63mwUnk4RlWTp27Jg+/PBDjRkzpk8tIIQPAEDSaGlpkWmaKi0tVW5urt3l4AyDBw/WBx98oNbW1j6FDwacAgCSzictzw17xKoVir9dAACQUIQPAACQUIQPAABi4KqrrtLixYvtLiMlED4AAEBCZcxsl5qTp7XqrYMKtJmqun6C3eUAAJCxMqbloynQph+9tle/2nRApmnZXQ4AoIcsy9KplraEb5bV+98VJ06c0G233aYBAwYoNzdXs2fP1p49eyLHDxw4oDlz5mjAgAHq16+fysrK9Kc//Sny2fnz52vw4MHKycnRmDFjtGLFij7/HJNJxrR8jB6cp2yXQ00tQX1wvEmjB/dtdTYAQGKcbg1q4rfXJPy+O//PLOW6e/dr8vbbb9eePXv0hz/8QV6vVw888ICuv/567dy5Uy6XS5WVlWppadHrr7+ufv36aefOnZFVQ7/1rW9p586devHFFzVo0CDt3btXp0+fjuVXs13GhA+nw9CEYq/+9+BJba/xEz4AAHERDh1vvPGGLr/8cknSM888o9LSUj333HO66aabdPDgQc2bN08XXXSRJGn06NGRzx88eFAXX3yxpk6dKkkaOXJkwr9DvGVM+JCk8hKf/vfgSe04XK9/mFxidzkAgB7IcTm18//MsuW+vbFr1y5lZWVp2rRpkX0DBw7UuHHjtGvXLknS1772NS1cuFAvv/yyZs6cqXnz5mnSpEmSpIULF2revHl6++23de211+qGG26IhJh0kTFjPiSpfKhXkrS9pt7mSgAAPWUYhnLdWQnf4vlMmbvuukvvv/++vvzlL+vdd9/V1KlT9eMf/1iSNHv2bB04cEBLlixRTU2NZsyYofvvvz9utdgho8JHWYlPkrT9sL9PA4kAADiXCRMmqK2tTW+++WZk3/Hjx7V7925NnDgxsq+0tFT33HOPnn32WX3961/Xz372s8ixwYMHa8GCBfrVr36lxx9/XE8++WRCv0O8ZVS3y9jCfLmchupPt+rDE6dVWsBDiwAAsTVmzBjNnTtXd999t376058qPz9f//Iv/6KhQ4dq7ty5kqTFixdr9uzZGjt2rE6cOKF169ZpwoTQMhDf/va3NWXKFJWVlSkQCOiFF16IHEsXGdXy4c5yaFxRviRpB10vAIA4WbFihaZMmaLPfe5zqqiokGVZ+tOf/iSXyyVJCgaDqqys1IQJE3Tddddp7NixeuKJJyRJbrdbVVVVmjRpkq688ko5nU6tWrXKzq8Tc4aVZP0Pfr9fPp9P9fX18nq9Mb/+v/y/d7Rq8yEtuvpC3T9rXMyvDwDovebmZu3fv1+jRo1Sdna23eXgDOf7+4nm93dGtXxIUtnQ9nEftHwAAGCLjAsf5SXtM14O1zPoFAAAG2Rc+JhQ7JXTYeijxhYdbQjYXQ4AABkn48JHtsupC9tXN91+mK4XAAASLePChySVhRcbO+y3uRIAADJPRoaP8hIGnQIAYJfMDB/tM1521tDyAQBAokUVPpYvX65JkybJ6/XK6/WqoqJCL774YuT4VVddJcMwumz33HNPzIvuqwnFoYXGDp88rY+bWmyuBgCAzBJV+Bg2bJgeffRRbd26VVu2bNE111yjuXPnaseOHZFz7r77btXW1ka273//+zEvuq/ys10aNaifJFY6BQAg0aIKH3PmzNH111+vMWPGaOzYsfrud7+rvLw8bdq0KXJObm6uioqKIls8VimNhbISBp0CAJLHyJEj9fjjj/foXMMw9Nxzz8W1nnjq9ZiPYDCoVatWqampSRUVFZH9zzzzjAYNGqTy8nJVVVXp1KlT571OIBCQ3+/vsiVCOSudAgBgi6ifavvuu++qoqJCzc3NysvL0+rVqyOPCL711ls1YsQIlZSU6J133tEDDzyg3bt369lnnz3n9aqrq/Xwww/3/hv0UnjGyw7W+gAAIKGibvkYN26ctm3bpjfffFMLFy7UggULtHPnTknSV7/6Vc2aNUsXXXSR5s+fr6efflqrV6/Wvn37znm9qqoq1dfXR7ZDhw71/ttEIdzt8sHxU/I3tybkngCAXrAsqaUp8VsUj+B48sknVVJSItM0u+yfO3eu7rjjDu3bt09z585VYWGh8vLydMkll+jVV1+N2Y/o3Xff1TXXXKOcnBwNHDhQX/3qV9XY2Bg5vn79el166aXq16+f+vfvr+nTp+vAgQOSpL/+9a+6+uqrlZ+fL6/XqylTpmjLli0xq607Ubd8uN1uXXjhhZKkKVOmaPPmzfrhD3+on/70p2edO23aNEnS3r17dcEFF3R7PY/HI4/HE20ZfTagn1tD++fo8MnT2lnj12WjBya8BgBAD7Sekh4pSfx9/78ayd2vR6fedNNNuvfee7Vu3TrNmDFDkvTxxx/rpZde0p/+9Cc1Njbq+uuv13e/+115PB49/fTTmjNnjnbv3q3hw4f3qcympibNmjVLFRUV2rx5s44ePaq77rpLixYt0lNPPaW2tjbdcMMNuvvuu/XrX/9aLS0teuutt2QYhiRp/vz5uvjii7V8+XI5nU5t27ZNLperTzV9kqjDx5lM01Qg0P0zUrZt2yZJKi4u7utt4qJ8qFeHT57W9sP1hA8AQK8NGDBAs2fP1sqVKyPh43e/+50GDRqkq6++Wg6HQ5MnT46c/53vfEerV6/WH/7wBy1atKhP9165cqWam5v19NNPq1+/UFj6j//4D82ZM0ff+9735HK5VF9fr8997nORhoAJEyZEPn/w4EH98z//s8aPHy9JGjNmTJ/q6YmowkdVVZVmz56t4cOHq6GhQStXrtT69eu1Zs0a7du3TytXrtT111+vgQMH6p133tGSJUt05ZVXatKkSfGqv0/KS3xas6NOO1hsDACSlys31Aphx32jMH/+fN1999164okn5PF49Mwzz+hLX/qSHA6HGhsb9dBDD+mPf/yjamtr1dbWptOnT+vgwYN9LnPXrl2aPHlyJHhI0vTp02Wapnbv3q0rr7xSt99+u2bNmqXPfOYzmjlzpr74xS9GGgaWLl2qu+66S//1X/+lmTNn6qabbjpnb0WsRDXm4+jRo7rttts0btw4zZgxQ5s3b9aaNWv0mc98Rm63W6+++qquvfZajR8/Xl//+tc1b948Pf/88/Gqvc8iM14YdAoAycswQt0fid7auyV6as6cObIsS3/84x916NAh/fnPf9b8+fMlSffff79Wr16tRx55RH/+85+1bds2XXTRRWppScxClytWrNDGjRt1+eWX6ze/+Y3Gjh0bWSbjoYce0o4dO/TZz35Wr732miZOnKjVq1fHtZ6oWj5+/vOfn/NYaWmpNmzY0OeCEin8gLl9xxp1qqVNue4+90IBADJUdna2brzxRj3zzDPau3evxo0bp0996lOSpDfeeEO33367Pv/5z0uSGhsb9cEHH8TkvhMmTNBTTz2lpqamSOvHG2+8IYfDoXHjxkXOu/jii3XxxRerqqpKFRUVWrlypS677DJJ0tixYzV27FgtWbJEt9xyi1asWBGpNR4y8tkuYUPyszUk3yPTknbVNthdDgAgxc2fP19//OMf9Ytf/CLS6iGFxlE8++yz2rZtm/7617/q1ltvPWtmTF/umZ2drQULFmj79u1at26d7r33Xn35y19WYWGh9u/fr6qqKm3cuFEHDhzQyy+/rD179mjChAk6ffq0Fi1apPXr1+vAgQN64403tHnz5i5jQuIh4/+pXz7Up9feO6odNfWaMmKA3eUAAFLYNddco4KCAu3evVu33nprZP9jjz2mO+64Q5dffrkGDRqkBx54IGaLaubm5mrNmjW67777dMkllyg3N1fz5s3TY489Fjn+3nvv6Ze//KWOHz+u4uJiVVZW6p/+6Z/U1tam48eP67bbblNdXZ0GDRqkG2+8Me7rbxmWFcVE5gTw+/3y+Xyqr69PyNLs//7ybv34tb26eWqpvveF5BwYCwCZorm5Wfv379eoUaOUnZ1tdzk4w/n+fqL5/Z3R3S6SVFbCMusAACRSxoeP8vZBp3+ra1CgLWhzNQCATPfMM88oLy+v262srMzu8mIi48d8DO2fo/65Lp081ao9dY2R6bcAANjhH/7hHyIrhJ8p3iuPJkrGhw/DMFRe4tNf9n6k7YfrCR8AAFvl5+crPz/f7jLiKuO7XaSO9T4Y9wEAySHJ5kKgXaz+XggfCi2zLknbD7PMOgDYKdytcOrUKZsrQXfCK7I6nc4+XSfju12kjmXWd9X61RY0leUkkwGAHZxOp/r376+jR49KCq1RYUS5zDniwzRNHTt2TLm5ucrK6lt8IHxIGlGQqzxPlhoDbdp3rEnjitK7rw0AkllRUZEkRQIIkofD4dDw4cP7HAgJH5IcDkMTS7x6a//H2n64nvABADYyDEPFxcUaMmSIWltb7S4Hnbjdbjkcfe8dIHy0Ky/xhcJHTb3mTRlmdzkAkPGcTmefxxYgOTG4oV14sbEdDDoFACCuCB/twoNOd9TUyzSZ4gUAQLwQPtqNHtRP2S6HmlqC+uB4k93lAACQtggf7bKcDo0vau96qaHrBQCAeCF8dFLOSqcAAMQd4aOT8EqnDDoFACB+CB+dhAedbq+p57kCAADECeGjkzGFeXI5DZ081arDJ0/bXQ4AAGmJ8NGJJ8upsYWh1U15yBwAAPFB+DhDZNwHg04BAIgLwscZIjNeDhM+AACIB8LHGcoig07pdgEAIB4IH2eYUOSVw5CONQR01N9sdzkAAKQdwscZctxOXTgkTxKLjQEAEA+Ej26EB50y4wUAgNgjfHQjMu6DQacAAMQc4aMb5SU8YA4AgHghfHRjYnv4OHzytE40tdhcDQAA6YXw0Y38bJdGDsyVROsHAACxRvg4h7JOD5kDAACxQ/g4h44ZL4QPAABiifBxDuFl1ul2AQAgtqIKH8uXL9ekSZPk9Xrl9XpVUVGhF198MXK8ublZlZWVGjhwoPLy8jRv3jzV1dXFvOhEKGtv+dj/UZMamlttrgYAgPQRVfgYNmyYHn30UW3dulVbtmzRNddco7lz52rHjh2SpCVLluj555/Xb3/7W23YsEE1NTW68cYb41J4vBX0c2to/xxJ0k5aPwAAiBnDsiyrLxcoKCjQv/3bv+kLX/iCBg8erJUrV+oLX/iCJOm9997ThAkTtHHjRl122WU9up7f75fP51N9fb28Xm9fSuuzrz69RS/vrNO3PjdRd14xytZaAABIZtH8/u71mI9gMKhVq1apqalJFRUV2rp1q1pbWzVz5szIOePHj9fw4cO1cePGc14nEAjI7/d32ZJFefuMlx0MOgUAIGaiDh/vvvuu8vLy5PF4dM8992j16tWaOHGijhw5Irfbrf79+3c5v7CwUEeOHDnn9aqrq+Xz+SJbaWlp1F8iXsKDTpluCwBA7EQdPsaNG6dt27bpzTff1MKFC7VgwQLt3Lmz1wVUVVWpvr4+sh06dKjX14q18HTbvUcbdbolaHM1AACkh6xoP+B2u3XhhRdKkqZMmaLNmzfrhz/8oW6++Wa1tLTo5MmTXVo/6urqVFRUdM7reTweeTye6CtPgCHebA3O9+hYQ0C7jvj1qeED7C4JAICU1+d1PkzTVCAQ0JQpU+RyubR27drIsd27d+vgwYOqqKjo621sE3nIHOM+AACIiahaPqqqqjR79mwNHz5cDQ0NWrlypdavX681a9bI5/Ppzjvv1NKlS1VQUCCv16t7771XFRUVPZ7pkozKh/q0bvcxbT+cPANhAQBIZVGFj6NHj+q2225TbW2tfD6fJk2apDVr1ugzn/mMJOkHP/iBHA6H5s2bp0AgoFmzZumJJ56IS+GJUhZu+ail5QMAgFjo8zofsZZM63xI0qGPT+nT318nl9PQjoevkzuLFekBADhTQtb5yBTDBuTIl+NSa9DS3+oa7C4HAICUR/j4BIZhdHrIHF0vAAD0FeGjB8LrfTDoFACAviN89EBZ+zLrrHQKAEDfET56ILzWx65av9qCps3VAACQ2ggfPTByYD/1czvV3Grq/Y+a7C4HAICURvjoAYfDUFlk3AddLwAA9AXho4fKwk+4ZdApAAB9QvjoociMFwadAgDQJ4SPHipvn/Gys8Yv00yqRWEBAEgphI8eumBwP3myHGoMtOnAx6fsLgcAgJRF+OihLKdD44vD4z7oegEAoLcIH1EIr/exo4ZBpwAA9BbhIwrhcR884wUAgN4jfEShvNNaH5bFoFMAAHqD8BGFsUV5ynIYOnGqVTX1zXaXAwBASiJ8RMGT5dTYwnxJDDoFAKC3CB9RKm9f6XQH4QMAgF4hfEQpPOh0OzNeAADoFcJHlHjAHAAAfUP4iNKE4nw5DOloQ0BH/Qw6BQAgWoSPKOW6s3TB4DxJLDYGAEBvED56ITLug64XAACiRvjohbL2Zda3s9IpAABRI3z0QkfLB90uAABEi/DRCxPbWz4OnzytE00tNlcDAEBqIXz0gjfbpREDcyVJO2tp/QAAIBqEj14qZ70PAAB6hfDRS2VDw4NOafkAACAahI9eCrd88IwXAACiQ/jopfB02/c/alJDc6vN1QAAkDoIH700MM+jEl+2JGlXbYPN1QAAkDoIH31QxkqnAABEjfDRB5EZL6x0CgBAjxE++qC8fcbLDlY6BQCgxwgffRBeZn3P0QadbgnaXA0AAKkhqvBRXV2tSy65RPn5+RoyZIhuuOEG7d69u8s5V111lQzD6LLdc889MS06WQzJ92hQnkemJb13hNYPAAB6IqrwsWHDBlVWVmrTpk165ZVX1NraqmuvvVZNTU1dzrv77rtVW1sb2b7//e/HtOhkYRhGpOuFxcYAAOiZrGhOfumll7q8f+qppzRkyBBt3bpVV155ZWR/bm6uioqKYlNhkisv8Wn97mMsNgYAQA/1acxHfX3oF25BQUGX/c8884wGDRqk8vJyVVVV6dSpU+e8RiAQkN/v77KlkvBiY8x4AQCgZ6Jq+ejMNE0tXrxY06dPV3l5eWT/rbfeqhEjRqikpETvvPOOHnjgAe3evVvPPvtst9eprq7Www8/3NsybBcedPq3I41qaTPlzmIMLwAA52NYlmX15oMLFy7Uiy++qL/85S8aNmzYOc977bXXNGPGDO3du1cXXHDBWccDgYACgUDkvd/vV2lpqerr6+X1entTWkJZlqXJD78sf3Ob/vi1K1TWvvYHAACZxO/3y+fz9ej3d6/+mb5o0SK98MILWrdu3XmDhyRNmzZNkrR3795uj3s8Hnm93i5bKgkNOg0/ZC61uowAALBDVOHDsiwtWrRIq1ev1muvvaZRo0Z94me2bdsmSSouLu5VgakgHD4Y9wEAwCeLasxHZWWlVq5cqd///vfKz8/XkSNHJEk+n085OTnat2+fVq5cqeuvv14DBw7UO++8oyVLlujKK6/UpEmT4vIFkkFk0CkzXgAA+ERRhY/ly5dLCi0k1tmKFSt0++23y+1269VXX9Xjjz+upqYmlZaWat68efrmN78Zs4KTUbjlY2etX0HTktNh2FwRAADJK6rw8UljU0tLS7Vhw4Y+FZSKRg3sp35up5pagnr/WKPGFObbXRIAAEmLeaEx4HAYmsh6HwAA9AjhI0bCU2y3M+MFAIDzInzESGTGC4NOAQA4L8JHjIQfMLezxi/T7NW6bQAAZATCR4xcODhPniyHGgJtOvjxuZ9lAwBApiN8xEiW06HxxQw6BQDgkxA+YqhjsTEGnQIAcC6Ejxgqb5/xsoOWDwAAzonwEUPhQac7avyfuCAbAACZivARQ2ML85XlMPRxU4tq65vtLgcAgKRE+IihbJczsrQ6630AANA9wkeMlUeWWWfQKQAA3SF8xFh4pdMdtHwAANAtwkeMhQedstYHAADdI3zE2IRirwxDqvMHdLSBQacAAJyJ8BFjue4sXTA4T1Joyi0AAOiK8BEH4UGnjPsAAOBshI84CA86ZZl1AADORviIg7L2ZdYZdAoAwNkIH3Ewsb3b5cMTp3XyVIvN1QAAkFwIH3Hgy3FpeEGuJAadAgBwJsJHnHQ8ZI6uFwAAOiN8xElk3AeDTgEA6ILwESeRGS+0fAAA0AXhI07K2ged7v+oSY2BNpurAQAgeRA+4mRQnkfFvmxZlrSrlq4XAADCCB9x1DHug64XAADCCB9xFHnCLYNOAQCIIHzEUXl7ywfTbQEA6ED4iKPwjJc9RxvV3Bq0uRoAAJID4SOOCr0eDcpzK2haeu9Ig93lAACQFAgfcWQYBoNOAQA4A+EjzlhmHQCArggfccYy6wAAdEX4iLPwjJfdRxrUGjRtrgYAAPtFFT6qq6t1ySWXKD8/X0OGDNENN9yg3bt3dzmnublZlZWVGjhwoPLy8jRv3jzV1dXFtOhUUlqQo/zsLLUETe2pa7S7HAAAbBdV+NiwYYMqKyu1adMmvfLKK2ptbdW1116rpqamyDlLlizR888/r9/+9rfasGGDampqdOONN8a88FRhGEak9YOHzAEAIGVFc/JLL73U5f1TTz2lIUOGaOvWrbryyitVX1+vn//851q5cqWuueYaSdKKFSs0YcIEbdq0SZdddlnsKk8h5UO92vj+ce04XC9NLbW7HAAAbNWnMR/19aF/yRcUFEiStm7dqtbWVs2cOTNyzvjx4zV8+HBt3Lix22sEAgH5/f4uW7oJLza2vSb9vhsAANHqdfgwTVOLFy/W9OnTVV5eLkk6cuSI3G63+vfv3+XcwsJCHTlypNvrVFdXy+fzRbbS0vRrGQjPeNlZ41fQtGyuBgAAe/U6fFRWVmr79u1atWpVnwqoqqpSfX19ZDt06FCfrpeMRg3qp1y3U6dbg9r/EYNOAQCZrVfhY9GiRXrhhRe0bt06DRs2LLK/qKhILS0tOnnyZJfz6+rqVFRU1O21PB6PvF5vly3dOB2GJhbzhFsAAKQow4dlWVq0aJFWr16t1157TaNGjepyfMqUKXK5XFq7dm1k3+7du3Xw4EFVVFTEpuIUFRn3wTLrAIAMF9Vsl8rKSq1cuVK///3vlZ+fHxnH4fP5lJOTI5/PpzvvvFNLly5VQUGBvF6v7r33XlVUVGTsTJewspL2lg+m2wIAMlxU4WP58uWSpKuuuqrL/hUrVuj222+XJP3gBz+Qw+HQvHnzFAgENGvWLD3xxBMxKTaVhVs+dhz2yzQtORyGzRUBAGCPqMKHZX3yTI3s7GwtW7ZMy5Yt63VR6ejCIXlyZznUEGjToROnNGJgP7tLAgDAFjzbJUFcTocmFOVLYtApACCzET4SaCLLrAMAQPhIpPKh4em2hA8AQOYifCRQeaeVTnsyfgYAgHRE+EigcUX5cjoMHW9q0RF/s93lAABgC8JHAmW7nBozJE8Sg04BAJmL8JFgrHQKAMh0hI8EK29f6XQHM14AABmK8JFgHS0fdLsAADIT4SPBJhR7ZRjSEX+zjjUE7C4HAICEI3wkWD9PlkYPCi2tTtcLACATET5sEHnIXA1dLwCAzEP4sEF4sTFmvAAAMhHhwwZl4WXW6XYBAGQgwocNytpbPg59fFr1p1ptrgYAgMQifNjAl+NSaUGOJAadAgAyD+HDJpFxH4QPAECGIXzYhBkvAIBMRfiwSVn7MuvMeAEAZBrCh03Cg07f/6hJTYE2m6sBACBxCB82GZzvUZE3W5Yl7aql6wUAkDkIHzYqH0rXCwAg8xA+bFQWmfFCywcAIHMQPmwUnvFCywcAIJMQPmwU7nbZc7RRza1Bm6sBACAxCB82KvJma2A/t4Kmpd1HGuwuBwCAhCB82MgwDJUNZaVTAEBmIXzYrDyy2BiDTgEAmYHwYbOOZdZp+QAAZAbCh83Cy6y/V9ug1qBpczUAAMQf4cNmwwtylZ+dpZagqT11jXaXAwBA3BE+bGYYRqT1g64XAEAmIHwkgfKS8LgPBp0CANIf4SMJsNIpACCTED6SQHil0521fgVNy+ZqAACIr6jDx+uvv645c+aopKREhmHoueee63L89ttvl2EYXbbrrrsuVvWmpVGD8pTjcupUS1D7P2qyuxwAAOIq6vDR1NSkyZMna9myZec857rrrlNtbW1k+/Wvf92nItOd02FoIoNOAQAZIivaD8yePVuzZ88+7zkej0dFRUW9LioTlZd4tfXACW0/XK+5fzfU7nIAAIibuIz5WL9+vYYMGaJx48Zp4cKFOn78+DnPDQQC8vv9XbZMFHnGC8usAwDSXMzDx3XXXaenn35aa9eu1fe+9z1t2LBBs2fPVjDY/SPjq6ur5fP5IltpaWmsS0oJ4em222vqZVkMOgUApK+ou10+yZe+9KXI64suukiTJk3SBRdcoPXr12vGjBlnnV9VVaWlS5dG3vv9/owMIGMK8+R2OtTQ3KZDH5/W8IG5dpcEAEBcxH2q7ejRozVo0CDt3bu32+Mej0der7fLlolcTofGF+dLCrV+AACQruIePj788EMdP35cxcXF8b5Vygsvs85iYwCAdBZ1t0tjY2OXVoz9+/dr27ZtKigoUEFBgR5++GHNmzdPRUVF2rdvn77xjW/owgsv1KxZs2JaeDoqK/FJOqTtLLMOAEhjUYePLVu26Oqrr468D4/XWLBggZYvX6533nlHv/zlL3Xy5EmVlJTo2muv1Xe+8x15PJ7YVZ2mwsus7zgcGnRqGIbNFQEAEHtRh4+rrrrqvLMx1qxZ06eCMtn4onw5HYaON7Wozh9QkS/b7pIAAIg5nu2SRLJdTo0ZkieJcR8AgPRF+EgyZZ3W+wAAIB0RPpJM+Am3rHQKAEhXhI8kExl0SssHACBNET6SzIRirwxDqq1v1keNAbvLAQAg5ggfSSbPk6VRg/pJknaw3gcAIA0RPpJQ5CFzzHgBAKQhwkcSCg86ZdwHACAdET6SUEfLB90uAID0Q/hIQuG1Pg5+fEr1p1ptrgYAgNgifCQhX65LwwbkSJJ21NL1AgBIL4SPJBXuetlB1wsAIM0QPpJUZKVTBp0CANIM4SNJlUVWOqXlAwCQXggfSSrc7bLvWKNOtbTZXA0AALFD+EhSg/M9KvR6ZFnSrlpaPwAA6YPwkcRY7wMAkI4IH0ksPO6DZdYBAOmE8JHEykvCM15o+QAApA/CRxIrb2/52FPXoObWoM3VAAAQG4SPJFbsy1ZBP7faTEt/q2uwuxwAAGKC8JHEDMNQWbjrhUGnAIA0QfhIcuGuF1Y6BQCkC8JHkut4xgvhAwCQHggfSS7c7bLrSINag6bN1QAA0HeEjyQ3vCBX+Z4stbSZ2nu00e5yAADoM8JHknM4DE2MDDql6wUAkPoIHymgnCfcAgDSCOEjBZQPDbV87GDGCwAgDRA+UkBkxkuNX6Zp2VwNAAB9Q/hIAaMH5ynb5dCplqD2H2+yuxwAAPqE8JECnA5DE4sZdAoASA+EjxTBoFMAQLogfKSI8LgPWj4AAKmO8JEiyoZ2dLtYFoNOAQCpi/CRIsYMyZfb6ZC/uU0fnjhtdzkAAPRa1OHj9ddf15w5c1RSUiLDMPTcc891OW5Zlr797W+ruLhYOTk5mjlzpvbs2ROrejOWO8uhcUX5kuh6AQCktqjDR1NTkyZPnqxly5Z1e/z73/++fvSjH+knP/mJ3nzzTfXr10+zZs1Sc3Nzn4vNdOHFxraz2BgAIIVlRfuB2bNna/bs2d0esyxLjz/+uL75zW9q7ty5kqSnn35ahYWFeu655/SlL32pb9VmuIklPkmHtP0wM14AAKkrpmM+9u/fryNHjmjmzJmRfT6fT9OmTdPGjRu7/UwgEJDf7++yoXvlJQw6BQCkvpiGjyNHjkiSCgsLu+wvLCyMHDtTdXW1fD5fZCstLY1lSWllQrFXToeh400tqvMH7C4HAIBesX22S1VVlerr6yPboUOH7C4paWW7nLpwcJ4kHjIHAEhdMQ0fRUVFkqS6urou++vq6iLHzuTxeOT1ertsOLeO9T7ongIApKaYho9Ro0apqKhIa9eujezz+/168803VVFREctbZazISqe0fAAAUlTUs10aGxu1d+/eyPv9+/dr27ZtKigo0PDhw7V48WL967/+q8aMGaNRo0bpW9/6lkpKSnTDDTfEsu6MFXnGC2t9AABSVNThY8uWLbr66qsj75cuXSpJWrBggZ566il94xvfUFNTk7761a/q5MmTuuKKK/TSSy8pOzs7dlVnsIntM15q6pt1vDGggXkemysCACA6hpVkczb9fr98Pp/q6+sZ/3EO1/zf9Xr/oyY9fcelunLsYLvLAQAgqt/fts92QfTKhjLuAwCQuggfKSi82NgOZrwAAFIQ4SMFldPyAQBIYYSPFFTW3vJx4Pgp1Z9utbkaAACiQ/hIQf1z3Ro2IEeStLOGrhcAQGohfKSocOsHy6wDAFIN4SNFRVY6ZbExAECKIXykqI5Bp3S7AABSC+EjRYUfMPf+sUadammzuRoAAHqO8JGihuRna0i+R6Yl7aptsLscAAB6jPCRwiIPmWPQKQAghRA+Ulh4pVMGnQIAUgnhI4VFnvHCMusAgBRC+Ehh4W6Xv9U1KNAWtLkaAAB6hvCRwkp82RqQ61KbaelvRxrtLgcAgB4hfKQwwzB4yBwAIOUQPlJcGSudAgBSDOEjxZW3LzbGSqcAgFRB+Ehx4ZaPXbV+tQZNm6sBAOCTET5S3IiCXOV5stTSZmrfMQadAgCSH+EjxTkchiZGFhuj6wUAkPwIH2mgnEGnAIAUQvhIA+FBpzsZdAoASAGEjzTQ+QFzpmnZXA0AAOdH+EgDowf1U7bLoaaWoD443mR3OQAAnBfhIw1kOR2aUMx6HwCA1ED4SBPhQac7GHQKAEhyhI800bHSKeEDAJDcCB9pouMZL35ZFoNOAQDJi/CRJsYW5svlNFR/ulUfnjhtdzkAAJwT4SNNuLMcGleULyk05RYAgGRF+Egj5Z26XgAASFaEjzRSVsKgUwBA8iN8pJGyoR3PeGHQKQAgWRE+0siEIq8chvRRY4uONgTsLgcAgG7FPHw89NBDMgyjyzZ+/PhY3wbdyHE7deGQPEk84RYAkLzi0vJRVlam2trayPaXv/wlHrdBNyIrnbLMOgAgSWXF5aJZWSoqKorHpfEJyob69Oz/HqblAwCQtOLS8rFnzx6VlJRo9OjRmj9/vg4ePHjOcwOBgPx+f5cNvVfePuOFlg8AQLKKefiYNm2annrqKb300ktavny59u/fr09/+tNqaGjo9vzq6mr5fL7IVlpaGuuSMsrE9vBx+ORpfdzUYnM1AACczbDiPCfz5MmTGjFihB577DHdeeedZx0PBAIKBDpmZvj9fpWWlqq+vl5erzeepaWtq//veu3/qEn/deel+vSYwXaXAwDIAH6/Xz6fr0e/v+M+1bZ///4aO3as9u7d2+1xj8cjr9fbZUPfRBYbY6VTAEASinv4aGxs1L59+1RcXBzvW6FdeXixMVY6BQAkoZiHj/vvv18bNmzQBx98oP/5n//R5z//eTmdTt1yyy2xvhXOITLdlhkvAIAkFPOpth9++KFuueUWHT9+XIMHD9YVV1yhTZs2afBgxh4kSrjb5YPjp+RvbpU322VzRQAAdIh5+Fi1alWsL4koDejn1tD+OTp88rR21vh12eiBdpcEAEAEz3ZJUx2DTul6AQAkF8JHmgoPOmWxMQBAsiF8pKnyobR8AACSE+EjTYVnvOw71qjTLUGbqwEAoAPhI00N8WZrcL5HpiXtOkLXCwAgeRA+0ljkIXN0vQAAkgjhI41FVjplmXUAQBIhfKSxshKWWQcAJB/CRxoLz3j5W12DAm0MOgUAJAfCRxob2j9H/XNdag1a2lPXaHc5AABIInykNcMwIlNuWe8DAJAsCB9priy82BjjPgAASYLwkeY6Wj6Y8QIASA6EjzQXnm67q9avtqBpczUAABA+0t6IglzlebIUaDO171iT3eUAAED4SHcOh6GJxTxkDgCQPAgfGYBBpwCAZJJldwEJ03paen6xVDBKKhgd2gaMknILJMOwu7q4Cg863cGgUwBAEsic8HHigPTOqrP3e3xSwciOMFIwuiOg5BVJjtRvHAoPOt1Z65dpWnI40jtsAQCSW+aEj2yfNONB6eP3pY/3Syf2S/7DUqBeqv1raDtTVnZ7IGkPIwNGdrSa+EolZ2r8+C4Y3E+eLIcaA2068PEpjRrUz+6SAAAZLDV+e8aCt1j69NKu+1pPh1pEPn4/FEbCweTj96WTB6W2ZunYrtB2JkeW1H/42a0lA0aFQoorOyFfqyeynA5NKPZq26GT2n64nvABALBV5oSP7rhypCHjQ9uZgq1S/aGOMHLig66tJm3N7e/fl/atPePDhuQtOaO1pFM4yfYm4Mt1VT60PXzU1GvO5JKE3x8AgLDMDh/n43R1dLFoRtdjpik1HukIH5GAsj/0OuAPden4D0sf/Pnsa+cOOjuQhN/nDozLAFgGnQIAkgXhozccjlDLhrdEGnlF12OWJZ36uPuunBP7paZj0qmPQtuHb519bY+3+9aSgtFSfnGvB8CGB51ur6mXZVky0nyGDwAgeRE+Ys0wpH4DQ1vpJWcfb/Z3dOF0CSfhAbB+6cg7oe1MWdkdwSQyEHZUpwGwrnOWNaYwTy6noZOnWvXyzjqV+HKU43Yo2+VUjsupHLdT2VlOZsIAAOLOsCzLsruIzvx+v3w+n+rr6+X1Jn5shK1am6WTB7rpymkfAGu2nfuzhjM0ALa7rpwBIyVXjj77oz9rR835u108WQ7luNsDicsZCifuM187Qu87nZfr7nrumcdzwsddTrmcBi0vAJBmovn9TfhIFcG20ADYM1tLwuNM2k6f//P5JfrYM1R/9ecpYBpqM6VW01KraajNtCQZMmXIat/CryV1eR9+LRmypHPvtxyyOn1Wnc4xDENZToecTqeynI72LSvy2uV0KivLqSynU66s0DFXVmi/K8vZ6XWW3FlOubOccrmccjlDr92uLLmc7a04hkOSEWqRCr92ZklOj5TlkZzu0BZ+neUJtSCFjzuc8fs7BYA0QvjINJEBsPu7784JsKx6b5lyynS4FHS4ZDrcMp1uWQ6XLKdHltMtKxxenG4ZWaHAYjjdMlweObI8ckT+dMvpyg6d0yXwtIedLuHHfe7X4T8JRQCSTDS/vxnzkQ66DICd3vWYZUmnT3SEkYZayTJDm6zQccvq9Nrs4Wt1+pz5ia8t05RpWQoGTQXNYPufpoLBYOhP05QZDCpoWjJNU6bZ+XWnzTJD1zKtTq9NWZYlyzJlmZYMw+rUHmPJEXkdat9xKSi32uRWq1xGW+S1R23yGK1df7QKymEGlWU2x/WvMFqmnAo6XAo63DLDwahLOAqFFssZCkPKCgUXI8sjh9MlORxyGIYMh1OGYchwOOQwnDIcoX0Oh0OGYcjhcMowHB0tR5GWpPA+44x93Z1rnGN/tOc6uh4/5zXOd277+/D/Bkyz47UV7LQ/2PHfb7f7z9i67O/02cj+M69hneeen3RtKwb37Mm/OXtwTrJdp8fX6iTSBWx0fX3msU96f85j3dzH9vu2LwdxXbXsQvhId4YRen5NboE0bKp9ZUhytm/xZJqWAm2mTrcGQ1tLm063dH4fVEvQVGubqdagqZagqZY2U61BK/RnW1DBtlYFW5tltgWktoDM1oCsYIustoDU1iIFW2QEA+1/tsgwW+UIBmSYLXKaofdOs1VOs0UutbaHndZQwDHa5FJH4AkFoKA8au0IRGqT2wgfD517rlDkSrJQBCBFDBorifABxITDYYQGvbqTo1siaFqRkNPaFv7T6hR6QtupNlMng6EQ1Np+rKX9WOdQ1NbSLLOtRVbncNTW2h6MAjKCoaCkYEt7IGqVEWyRI9gih9kih9kqh9Uqq/1fwZZlybBMWVZoZE54pI5DlhwK7zO7tCI5umlVchid90XxufbzFTk/VIPD6NjnOKsF6+zrdXet7msLbcFO1QWt0Guz/VNm+9ikjtfd7G//TLD9Lh2vO/ZHrh/Z75BlGJ1eh1pqgoYj9K2NrvtNwynJCO3vdI4lR/uU+/b9Rvgz7X8ahizDGdqv9tcOh2Q4I8e7e6B5d+0F1rkOdNodHht2viudfU6nY5HTO86xznHTc13nzMaOjvPO3woSaRMwrPb/9o3Q31qn24R/Ug4jPIqt43MOWaHGh07XC4+UC51vtF83dA/JkGGFP2N1uk7og5FzI9fruL4ROda1hlBLb8f5hhH62kbne7Z/tvP1s/r117Xn/enEF+EDiCOnw5DTEZrpk+xM01LQshQ0LbWZloJBS22m2bEvGPqz+/dm5H1beF+w/bVpqbWbc0zLOsd7s9trtJmWzPD128858xrBTlubaUbqi9Ta6V6WJZmW1b5JVvufphW6T8dxdXs+kMpGD+5H+ABgP4fDkEOGUiAn2c6yPjmcdAkznY4Huw02nYPP+a8X+nz31+72epHPWgqaoZB5roaI7nafa1p89+f2bF/o82cfiGYGfnd1nevj3V033FpiKfTz6bzfktVx3OrUymNZXVqDwudZ5/pspxtZ4eNnvu907a51WWfU2NEi1FFbx3vrPJ+VZZ1V44Bc9zl+WolB+ACAKBmGIcOQHOfpTgBwbr1bqxsAAKCX4hY+li1bppEjRyo7O1vTpk3TW2918xwTAACQceISPn7zm99o6dKlevDBB/X2229r8uTJmjVrlo4ePRqP2wEAgBQSl/Dx2GOP6e6779ZXvvIVTZw4UT/5yU+Um5urX/ziF/G4HQAASCExDx8tLS3aunWrZs6c2XETh0MzZ87Uxo0bY307AACQYmI+2+Wjjz5SMBhUYWFhl/2FhYV67733zjo/EAgoEAhE3vv953/qKgAASG22z3aprq6Wz+eLbKWlpXaXBAAA4ijm4WPQoEFyOp2qq6vrsr+urk5FRUVnnV9VVaX6+vrIdujQoViXBAAAkkjMw4fb7daUKVO0du3ayD7TNLV27VpVVFScdb7H45HX6+2yAQCA9BWXFU6XLl2qBQsWaOrUqbr00kv1+OOPq6mpSV/5ylficTsAAJBC4hI+br75Zh07dkzf/va3deTIEf3d3/2dXnrppbMGoQIAgMxjWJ2fqJME/H6/fD6f6uvr6YIBACBFRPP72/bZLgAAILMk3VNtww0xrPcBAEDqCP/e7kmHStKFj4aGBklivQ8AAFJQQ0ODfD7fec9JujEfpmmqpqZG+fn5Mgwjptf2+/0qLS3VoUOHMnI8SaZ/f4mfQaZ/f4mfAd8/s7+/FL+fgWVZamhoUElJiRyO84/qSLqWD4fDoWHDhsX1Hpm+nkimf3+Jn0Gmf3+JnwHfP7O/vxSfn8EntXiEMeAUAAAkFOEDAAAkVEaFD4/HowcffFAej8fuUmyR6d9f4meQ6d9f4mfA98/s7y8lx88g6QacAgCA9JZRLR8AAMB+hA8AAJBQhA8AAJBQhA8AAJBQGRM+li1bppEjRyo7O1vTpk3TW2+9ZXdJCfP6669rzpw5KikpkWEYeu655+wuKaGqq6t1ySWXKD8/X0OGDNENN9yg3bt3211WQi1fvlyTJk2KLCpUUVGhF1980e6ybPPoo4/KMAwtXrzY7lIS5qGHHpJhGF228ePH211WQh0+fFj/+I//qIEDByonJ0cXXXSRtmzZYndZCTNy5Miz/hswDEOVlZUJryUjwsdvfvMbLV26VA8++KDefvttTZ48WbNmzdLRo0ftLi0hmpqaNHnyZC1btszuUmyxYcMGVVZWatOmTXrllVfU2tqqa6+9Vk1NTXaXljDDhg3To48+qq1bt2rLli265pprNHfuXO3YscPu0hJu8+bN+ulPf6pJkybZXUrClZWVqba2NrL95S9/sbukhDlx4oSmT58ul8ulF198UTt37tS///u/a8CAAXaXljCbN2/u8vf/yiuvSJJuuummxBdjZYBLL73UqqysjLwPBoNWSUmJVV1dbWNV9pBkrV692u4ybHX06FFLkrVhwwa7S7HVgAEDrP/8z/+0u4yEamhosMaMGWO98sor1t///d9b9913n90lJcyDDz5oTZ482e4ybPPAAw9YV1xxhd1lJJX77rvPuuCCCyzTNBN+77Rv+WhpadHWrVs1c+bMyD6Hw6GZM2dq48aNNlYGu9TX10uSCgoKbK7EHsFgUKtWrVJTU5MqKirsLiehKisr9dnPfrbL/x9kkj179qikpESjR4/W/PnzdfDgQbtLSpg//OEPmjp1qm666SYNGTJEF198sX72s5/ZXZZtWlpa9Ktf/Up33HFHzB/i2hNpHz4++ugjBYNBFRYWdtlfWFioI0eO2FQV7GKaphYvXqzp06ervLzc7nIS6t1331VeXp48Ho/uuecerV69WhMnTrS7rIRZtWqV3n77bVVXV9tdii2mTZump556Si+99JKWL1+u/fv369Of/rQaGhrsLi0h3n//fS1fvlxjxozRmjVrtHDhQn3ta1/TL3/5S7tLs8Vzzz2nkydP6vbbb7fl/kn3VFsgniorK7V9+/aM6usOGzdunLZt26b6+nr97ne/04IFC7Rhw4aMCCCHDh3Sfffdp1deeUXZ2dl2l2OL2bNnR15PmjRJ06ZN04gRI/Tf//3fuvPOO22sLDFM09TUqVP1yCOPSJIuvvhibd++XT/5yU+0YMECm6tLvJ///OeaPXu2SkpKbLl/2rd8DBo0SE6nU3V1dV3219XVqaioyKaqYIdFixbphRde0Lp16zRs2DC7y0k4t9utCy+8UFOmTFF1dbUmT56sH/7wh3aXlRBbt27V0aNH9alPfUpZWVnKysrShg0b9KMf/UhZWVkKBoN2l5hw/fv319ixY7V37167S0mI4uLis4L2hAkTMqrrKezAgQN69dVXddddd9lWQ9qHD7fbrSlTpmjt2rWRfaZpau3atRnX352pLMvSokWLtHr1ar322msaNWqU3SUlBdM0FQgE7C4jIWbMmKF3331X27Zti2xTp07V/PnztW3bNjmdTrtLTLjGxkbt27dPxcXFdpeSENOnTz9riv3f/vY3jRgxwqaK7LNixQoNGTJEn/3sZ22rISO6XZYuXaoFCxZo6tSpuvTSS/X444+rqalJX/nKV+wuLSEaGxu7/Otm//792rZtmwoKCjR8+HAbK0uMyspKrVy5Ur///e+Vn58fGevj8/mUk5Njc3WJUVVVpdmzZ2v48OFqaGjQypUrtX79eq1Zs8bu0hIiPz//rDE+/fr108CBAzNm7M/999+vOXPmaMSIEaqpqdGDDz4op9OpW265xe7SEmLJkiW6/PLL9cgjj+iLX/yi3nrrLT355JN68skn7S4toUzT1IoVK7RgwQJlZdkYARI+v8YmP/7xj63hw4dbbrfbuvTSS61NmzbZXVLCrFu3zpJ01rZgwQK7S0uI7r67JGvFihV2l5Ywd9xxhzVixAjL7XZbgwcPtmbMmGG9/PLLdpdlq0ybanvzzTdbxcXFltvttoYOHWrdfPPN1t69e+0uK6Gef/55q7y83PJ4PNb48eOtJ5980u6SEm7NmjWWJGv37t221mFYlmXZE3sAAEAmSvsxHwAAILkQPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEIRPgAAQEL9/yvK24MPzsYRAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "loss, val_loss = history.history['loss'], history.history['val_loss']\n",
        "\n",
        "plt.plot(loss, label='loss')\n",
        "plt.plot(val_loss, label='val_loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6L4MTH1Bw9xh"
      },
      "source": [
        "Test the model on test dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "31UYBJnWw9xi",
        "outputId": "12d30afc-e269-460e-fb3f-93f803de1427"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "63/63 [==============================] - 0s 2ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.85      0.83      0.84       275\n",
            "           1       0.79      0.82      0.80       224\n",
            "           2       0.86      0.89      0.88       695\n",
            "           3       0.63      0.76      0.69       159\n",
            "           4       0.92      0.83      0.87       581\n",
            "           5       0.66      0.61      0.63        66\n",
            "\n",
            "    accuracy                           0.84      2000\n",
            "   macro avg       0.78      0.79      0.78      2000\n",
            "weighted avg       0.84      0.84      0.84      2000\n",
            "\n",
            "[[228  12  12   4  19   0]\n",
            " [  5 183  12   0   7  17]\n",
            " [  7   6 620  51  10   1]\n",
            " [  2   0  28 121   6   2]\n",
            " [ 25  23  39  13 480   1]\n",
            " [  1   9  11   3   2  40]]\n",
            "0.836\n"
          ]
        }
      ],
      "source": [
        "predictions = model.predict(X_test_)\n",
        "prediction_labels = predictions.argmax(axis=1)\n",
        "print(classification_report(y_test, prediction_labels))\n",
        "print(confusion_matrix(y_test_, prediction_labels))\n",
        "print(accuracy_score(y_test_, prediction_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wghXruyvw9xi"
      },
      "source": [
        "I have tried several dozens of times on this model: increasing and reducing the layers, increasing and reducing the neurons in each layer, tuning the regularization parameter, changing regularization, tuning the learning rate, batch size, *etc*.\n",
        "\n",
        "I also tried to experiment on the different input, using different representation of original text for model training, such as embedding with spacy. There were several different accuracy ceiling for my model from low to high with increasing the model complexity, `34%` to `56%`, and `86%`. And finally I got this result with the tf_idf matrix representation for training the model while the scale of the model is large with nearly 34 million. And the Accuracy on test was improved from `< 40%` to now `> 80%`.\n",
        "\n",
        "With `> 80%` accuracy, I also run dozens of experimentation to improve the accuracy, however, it is very hard to get a better accuracy, in the meantime I am already overfitting with the training set. I tried to tackle the overfitting problem, but I will get lower accuracy. So I decided to keep this model setup."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6U51XoDnw9xi"
      },
      "source": [
        "Save the model, the folder of saved model is ignored for commit to github"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5A0z5wO5w9xi"
      },
      "outputs": [],
      "source": [
        "model.save('FNN_model')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_DTknQorw9xq"
      },
      "source": [
        "# RNN, LSTM and GRU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qROdD8yVw9xq"
      },
      "outputs": [],
      "source": [
        "def plot_results(history):\n",
        "    hist_df = pd.DataFrame(history.history)\n",
        "    hist_df.columns=[\"loss\", \"accuracy\", \"val_loss\", \"val_accuracy\"]\n",
        "    hist_df.index = np.arange(1, len(hist_df)+1)\n",
        "\n",
        "    fig, axs = plt.subplots(nrows=2, sharex=True, figsize=(16, 10))\n",
        "    axs[0].plot(hist_df.val_accuracy, lw=3, label='Validation Accuracy')\n",
        "    axs[0].plot(hist_df.accuracy, lw=3, label='Training Accuracy')\n",
        "    axs[0].set_ylabel('Accuracy')\n",
        "    axs[0].set_xlabel('Epoch')\n",
        "    axs[0].grid()\n",
        "    axs[0].legend(loc=0)\n",
        "    axs[1].plot(hist_df.val_loss, lw=3, label='Validation Loss')\n",
        "    axs[1].plot(hist_df.loss, lw=3, label='Training Loss')\n",
        "    axs[1].set_ylabel('Loss')\n",
        "    axs[1].set_xlabel('Epoch')\n",
        "    axs[1].grid()\n",
        "    axs[1].legend(loc=0)\n",
        "\n",
        "    plt.show();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "bg4Zfv9xw9xq",
        "outputId": "df325a97-ae41-4e42-816d-e714f96568bc"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-f4488661-d6cc-4199-8400-ee635347c9ac\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>processed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>i didnt feel humiliated</td>\n",
              "      <td>sadness</td>\n",
              "      <td>not feel humiliate</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>i can go from feeling so hopeless to so damned...</td>\n",
              "      <td>sadness</td>\n",
              "      <td>feel hopeless damn hopeful care awake</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>im grabbing a minute to post i feel greedy wrong</td>\n",
              "      <td>anger</td>\n",
              "      <td>m grab minute post feel greedy wrong</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>i am ever feeling nostalgic about the fireplac...</td>\n",
              "      <td>love</td>\n",
              "      <td>feel nostalgic fireplace know property</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>i am feeling grouchy</td>\n",
              "      <td>anger</td>\n",
              "      <td>feel grouchy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f4488661-d6cc-4199-8400-ee635347c9ac')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-32945e0c-0877-450f-9e3e-7b47fe48e16d\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-32945e0c-0877-450f-9e3e-7b47fe48e16d')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-32945e0c-0877-450f-9e3e-7b47fe48e16d button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f4488661-d6cc-4199-8400-ee635347c9ac button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f4488661-d6cc-4199-8400-ee635347c9ac');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                               tweet sentiment  \\\n",
              "0                            i didnt feel humiliated   sadness   \n",
              "1  i can go from feeling so hopeless to so damned...   sadness   \n",
              "2   im grabbing a minute to post i feel greedy wrong     anger   \n",
              "3  i am ever feeling nostalgic about the fireplac...      love   \n",
              "4                               i am feeling grouchy     anger   \n",
              "\n",
              "                                processed  \n",
              "0                      not feel humiliate  \n",
              "1   feel hopeless damn hopeful care awake  \n",
              "2    m grab minute post feel greedy wrong  \n",
              "3  feel nostalgic fireplace know property  \n",
              "4                            feel grouchy  "
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EjAAzhOHw9xq"
      },
      "outputs": [],
      "source": [
        "max_length = 100\n",
        "trunc_type='post'\n",
        "\n",
        "\n",
        "tokenizer = Tokenizer() #(num_words = vocab_size, oov_token=oov_tok)\n",
        "tokenizer.fit_on_texts(df_train['tweet'])\n",
        "word_index = tokenizer.word_index\n",
        "sequences = tokenizer.texts_to_sequences(df_train['tweet'])\n",
        "padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)\n",
        "\n",
        "testing_sequences = tokenizer.texts_to_sequences(df_test['tweet'])\n",
        "testing_padded = pad_sequences(testing_sequences,maxlen=max_length)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "49NbdK_pw9xq"
      },
      "outputs": [],
      "source": [
        "le = LabelEncoder()\n",
        "y_train = le.fit_transform(df_train['sentiment'])\n",
        "y_test = le.transform(df_test['sentiment'])\n",
        "y_train_encoded = to_categorical(y_train)\n",
        "y_test_encoded = to_categorical(y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a73pO3WHw9xr",
        "outputId": "fc3cc65c-132d-4024-90d9-026647be111a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, 100, 100)          1521300   \n",
            "                                                                 \n",
            " bidirectional (Bidirectiona  (None, 128)              84480     \n",
            " l)                                                              \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 128)               0         \n",
            "                                                                 \n",
            " dense_10 (Dense)            (None, 32)                4128      \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 32)                0         \n",
            "                                                                 \n",
            " dense_11 (Dense)            (None, 6)                 198       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,610,106\n",
            "Trainable params: 1,610,106\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "vocab_size = len(tokenizer.word_index) + 1\n",
        "embedding_dim = max_length\n",
        "\n",
        "\n",
        "model = tf.keras.Sequential([\n",
        "    Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    Bidirectional(LSTM(64)), #32\n",
        "    Dropout(0.4),\n",
        "    Dense(32, activation='leaky_relu', kernel_regularizer='l1_l2'),\n",
        "    Dropout(0.4),\n",
        "    Dense(6, activation='softmax')\n",
        "])\n",
        "\n",
        "\n",
        "loss_function = 'categorical_crossentropy'\n",
        "optimizer = 'adam'\n",
        "\n",
        "model.compile(loss=loss_function, optimizer=optimizer, metrics=['accuracy', Precision(), Recall(), AUC()])\n",
        "\n",
        "print(model.summary())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HmxDgZbKw9xr",
        "outputId": "b6aa3323-0ab7-42f7-bad2-f6fcca5e6df3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "400/400 [==============================] - 41s 89ms/step - loss: 2.4036 - accuracy: 0.3648 - precision: 0.6458 - recall: 0.0330 - auc: 0.7186 - val_loss: 1.5023 - val_accuracy: 0.5266 - val_precision: 0.7307 - val_recall: 0.1950 - val_auc: 0.7828\n",
            "Epoch 2/50\n",
            "400/400 [==============================] - 10s 25ms/step - loss: 1.3698 - accuracy: 0.5701 - precision: 0.8441 - recall: 0.2631 - auc: 0.8312 - val_loss: 1.2686 - val_accuracy: 0.5909 - val_precision: 0.8893 - val_recall: 0.2812 - val_auc: 0.8621\n",
            "Epoch 3/50\n",
            "400/400 [==============================] - 7s 18ms/step - loss: 1.2138 - accuracy: 0.6031 - precision: 0.8394 - recall: 0.3397 - auc: 0.8721 - val_loss: 1.2043 - val_accuracy: 0.5944 - val_precision: 0.8868 - val_recall: 0.2766 - val_auc: 0.8778\n",
            "Epoch 4/50\n",
            "400/400 [==============================] - 8s 19ms/step - loss: 1.0482 - accuracy: 0.6361 - precision: 0.8305 - recall: 0.5164 - auc: 0.9163 - val_loss: 0.9813 - val_accuracy: 0.7006 - val_precision: 0.8303 - val_recall: 0.5872 - val_auc: 0.9357\n",
            "Epoch 5/50\n",
            "400/400 [==============================] - 7s 18ms/step - loss: 0.8405 - accuracy: 0.7531 - precision: 0.9182 - recall: 0.5998 - auc: 0.9580 - val_loss: 0.7950 - val_accuracy: 0.8163 - val_precision: 0.9071 - val_recall: 0.6681 - val_auc: 0.9615\n",
            "Epoch 6/50\n",
            "400/400 [==============================] - 7s 18ms/step - loss: 0.6492 - accuracy: 0.8480 - precision: 0.9239 - recall: 0.7448 - auc: 0.9792 - val_loss: 0.6397 - val_accuracy: 0.8700 - val_precision: 0.9058 - val_recall: 0.8388 - val_auc: 0.9748\n",
            "Epoch 7/50\n",
            "400/400 [==============================] - 6s 14ms/step - loss: 0.5241 - accuracy: 0.8946 - precision: 0.9358 - recall: 0.8451 - auc: 0.9879 - val_loss: 0.5635 - val_accuracy: 0.8759 - val_precision: 0.9111 - val_recall: 0.8650 - val_auc: 0.9812\n",
            "Epoch 8/50\n",
            "400/400 [==============================] - 6s 15ms/step - loss: 0.4414 - accuracy: 0.9126 - precision: 0.9461 - recall: 0.8845 - auc: 0.9923 - val_loss: 0.5622 - val_accuracy: 0.8831 - val_precision: 0.9144 - val_recall: 0.8675 - val_auc: 0.9795\n",
            "Epoch 9/50\n",
            "400/400 [==============================] - 6s 14ms/step - loss: 0.4110 - accuracy: 0.9197 - precision: 0.9524 - recall: 0.8973 - auc: 0.9934 - val_loss: 0.5265 - val_accuracy: 0.8909 - val_precision: 0.9189 - val_recall: 0.8816 - val_auc: 0.9812\n",
            "Epoch 10/50\n",
            "400/400 [==============================] - 6s 16ms/step - loss: 0.3607 - accuracy: 0.9339 - precision: 0.9626 - recall: 0.9123 - auc: 0.9955 - val_loss: 0.5106 - val_accuracy: 0.8969 - val_precision: 0.9202 - val_recall: 0.8756 - val_auc: 0.9831\n",
            "Epoch 11/50\n",
            "400/400 [==============================] - 6s 15ms/step - loss: 0.3374 - accuracy: 0.9441 - precision: 0.9644 - recall: 0.9219 - auc: 0.9964 - val_loss: 0.5604 - val_accuracy: 0.8844 - val_precision: 0.9107 - val_recall: 0.8669 - val_auc: 0.9798\n",
            "Epoch 12/50\n",
            "400/400 [==============================] - 7s 17ms/step - loss: 0.3142 - accuracy: 0.9541 - precision: 0.9708 - recall: 0.9317 - auc: 0.9971 - val_loss: 0.5328 - val_accuracy: 0.8994 - val_precision: 0.9160 - val_recall: 0.8863 - val_auc: 0.9793\n",
            "Epoch 13/50\n",
            "400/400 [==============================] - 6s 15ms/step - loss: 0.2931 - accuracy: 0.9597 - precision: 0.9721 - recall: 0.9402 - auc: 0.9976 - val_loss: 0.5165 - val_accuracy: 0.9062 - val_precision: 0.9199 - val_recall: 0.8972 - val_auc: 0.9801\n"
          ]
        }
      ],
      "source": [
        "predictors = np.array(padded)\n",
        "label = np.array(y_train_encoded)\n",
        "epochs_value = 50\n",
        "validation_split_value = 0.2\n",
        "early_stopping = tf.keras.callbacks.EarlyStopping(patience=3)\n",
        "\n",
        "history = model.fit(predictors, label, epochs=epochs_value, verbose=1, validation_split=validation_split_value, callbacks=[early_stopping])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 434
        },
        "id": "4BwoVgP9w9xr",
        "outputId": "ffe29c42-94a1-4b6b-b297-1f270ec6a496"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGhCAYAAACzurT/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOSklEQVR4nO3dd3hUZd7G8e/MpLdJCKRBQu8dAggoiqKIiGDBhoIF1gKu6O6q7KuurqusdV2VIrrC6opdilgAkV6kBulFQhJKCC29z8z7x4RAIIG0yZkk9+e65srMmTPn/GaimZvnPMXkcDgciIiIiBjEbHQBIiIiUr8pjIiIiIihFEZERETEUAojIiIiYiiFERERETGUwoiIiIgYSmFEREREDKUwIiIiIoZSGBERERFDKYyIiIiIoSoURiZPnkyvXr0IDAwkLCyMESNGsGfPnou+ZtasWZhMphI3Hx+fKhUtIiIidUeFwsjy5csZP34869atY/HixRQUFHDdddeRlZV10dcFBQVx9OjR4ltCQkKVihYREZG6w6MiO//0008lHs+aNYuwsDA2bdrEgAEDynydyWQiIiKichUCdrudI0eOEBgYiMlkqvRxREREpOY4HA4yMjKIiorCbC67/aNCYeR8aWlpADRo0OCi+2VmZtK0aVPsdjs9evTglVdeoWPHjmXun5eXR15eXvHjw4cP06FDh6qUKiIiIgZJSkqiSZMmZT5vcjgcjsoc2G63c9NNN5GamsqqVavK3G/t2rXs27ePLl26kJaWxhtvvMGKFSvYsWNHmYW98MILvPjiixdsT0pKIigoqDLlioiISA1LT08nOjqa1NRUrFZrmftVOow88sgj/Pjjj6xateqiaed8BQUFtG/fnrvuuouXXnqp1H3Obxk582bS0tIURkRERGqJ9PR0rFbrJb+/K3WZZsKECSxYsIAVK1ZUKIgAeHp60r17d/bv31/mPt7e3nh7e1emNBEREallKjSaxuFwMGHCBObMmcMvv/xC8+bNK3xCm83Gtm3biIyMrPBrRUREpO6pUMvI+PHjmT17NvPmzSMwMJDk5GQArFYrvr6+AIwePZrGjRszefJkAP7+979z2WWX0apVK1JTU3n99ddJSEhg7Nix1fxWREREpDaqUBiZNm0aAFdddVWJ7TNnzuS+++4DIDExscTwndOnTzNu3DiSk5MJCQmhZ8+erFmzRqNjRESk3BwOB4WFhdhsNqNLkXNYLBY8PDyqPO1GpTuw1qTydoAREZG6Jz8/n6NHj5KdnW10KVIKPz8/IiMj8fLyuuA5l3ZgFRERqQl2u534+HgsFgtRUVF4eXlp8ks34XA4yM/P5/jx48THx9O6deuLTmx2MQojIiLitvLz87Hb7URHR+Pn52d0OXIeX19fPD09SUhIID8/v9Jrz2nVXhERcXuV/Re3uF51/G702xURERFDKYyIiIiIoRRGREREXOCqq65i4sSJRpdRKyiMiIiIiKHqdRj59NcE/vjZFpLTco0uRUREpN6q12Hks/WJzN96hA0HTxldioiIlJPD4SA7v9CQW2XnCT19+jSjR48mJCQEPz8/hgwZwr59+4qfT0hIYNiwYYSEhODv70/Hjh354Ycfil87atQoGjVqhK+vL61bt2bmzJnV8lm6i3o9z0hs0wZsP5zOpoTTDOsaZXQ5IiJSDjkFNjo8v9CQc+/8+2D8vCr+1Xnfffexb98+5s+fT1BQEE8//TQ33HADO3fuxNPTk/Hjx5Ofn8+KFSvw9/dn586dBAQEAPDcc8+xc+dOfvzxRxo2bMj+/fvJycmp7rdmqHodRno0DWHWmoNsSjhtdCkiIlJHnQkhq1evpl+/fgB8+umnREdHM3fuXEaOHEliYiK33nornTt3BqBFixbFr09MTKR79+7ExsYC0KxZsxp/D65Wr8NIbNMQAHYeTSc7v7BSaVdERGqWr6eFnX8fbNi5K2rXrl14eHjQp0+f4m2hoaG0bduWXbt2AfDHP/6RRx55hEWLFjFo0CBuvfVWunTpAsAjjzzCrbfeyubNm7nuuusYMWJEcaipK+p1n5GoYF8irT7Y7A7iklKNLkdERMrBZDLh5+VhyM1V6+KMHTuWAwcOcO+997Jt2zZiY2N59913ARgyZAgJCQk88cQTHDlyhGuuuYY///nPLqnDKPU6jAD0LGod2axLNSIi4gLt27ensLCQX3/9tXjbyZMn2bNnDx06dCjeFh0dzcMPP8y3337Ln/70Jz744IPi5xo1asSYMWP43//+x9tvv82MGTNq9D24msJIURjZqDAiIiIu0Lp1a4YPH864ceNYtWoVW7du5Z577qFx48YMHz4cgIkTJ7Jw4ULi4+PZvHkzS5cupX379gA8//zzzJs3j/3797Njxw4WLFhQ/FxdoTByTsuI3V65IVsiIiIXM3PmTHr27MmNN95I3759cTgc/PDDD3h6egJgs9kYP3487du35/rrr6dNmzZMnToVAC8vLyZNmkSXLl0YMGAAFouFzz//3Mi3U+1MjsoOmq5B6enpWK1W0tLSCAoKqtZjF9jsdHlhETkFNhY/MYDW4YHVenwREam83Nxc4uPjad68eaWXpxfXutjvqLzf3/W+ZcTTYqZrtBXQpRoREREj1PswAs7JzwDNNyIiImIAhRHO9htRGBEREal5CiNAjxhnGIk/kcXJzDyDqxEREalfFEYAq58nrcOcawCodURERKRmKYwUiW1WdKkmUWFERESkJimMFDlzqWbTQYURERGRmqQwUuRMJ9bfDqeRV2gzuBoREZH6Q2GkSPOG/jTw9yK/0M72w+lGlyMiIlJvKIwUMZlMxZdqtGieiIgYrVmzZrz99tvl2tdkMjF37lyX1uNKCiPnKO7EqjAiIiJSYxRGznHuCr61YMkeERGROkFh5BydG1vxtJg4kZlH0qkco8sREZHSOByQn2XMrZz/UJ0xYwZRUVHY7fYS24cPH84DDzzA77//zvDhwwkPDycgIIBevXrx888/V9tHtG3bNq6++mp8fX0JDQ3lD3/4A5mZmcXPL1u2jN69e+Pv709wcDD9+/cnISEBgK1btzJw4EACAwMJCgqiZ8+ebNy4sdpqK42HS49ey/h4WujU2MqWxFQ2JpwiJtTP6JJEROR8BdnwSpQx5/7rEfDyv+RuI0eO5LHHHmPp0qVcc801AJw6dYqffvqJH374gczMTG644QZefvllvL29+fjjjxk2bBh79uwhJiamSiVmZWUxePBg+vbty4YNG0hJSWHs2LFMmDCBWbNmUVhYyIgRIxg3bhyfffYZ+fn5rF+/HpPJBMCoUaPo3r0706ZNw2KxEBcXh6enZ5VquhSFkfPENg1hS2IqmxJOc0uPJkaXIyIitVBISAhDhgxh9uzZxWHk66+/pmHDhgwcOBCz2UzXrl2L93/ppZeYM2cO8+fPZ8KECVU69+zZs8nNzeXjjz/G398ZnN577z2GDRvGq6++iqenJ2lpadx44420bNkSgPbt2xe/PjExkb/85S+0a9cOgNatW1epnvJQGDlPz6YhfLAyXp1YRUTclaefs4XCqHOX06hRoxg3bhxTp07F29ubTz/9lDvvvBOz2UxmZiYvvPAC33//PUePHqWwsJCcnBwSExOrXOKuXbvo2rVrcRAB6N+/P3a7nT179jBgwADuu+8+Bg8ezLXXXsugQYO4/fbbiYyMBODJJ59k7NixfPLJJwwaNIiRI0cWhxZXUZ+R8/Qo6sS651gG6bkFBlcjIiIXMJmcl0qMuBVdyiiPYcOG4XA4+P7770lKSmLlypWMGjUKgD//+c/MmTOHV155hZUrVxIXF0fnzp3Jz8931adWwsyZM1m7di39+vXjiy++oE2bNqxbtw6AF154gR07djB06FB++eUXOnTowJw5c1xaj8LIecICfYhp4IfDAVsSU40uR0REaikfHx9uueUWPv30Uz777DPatm1Ljx49AFi9ejX33XcfN998M507dyYiIoKDBw9Wy3nbt2/P1q1bycrKKt62evVqzGYzbdu2Ld7WvXt3Jk2axJo1a+jUqROzZ88ufq5NmzY88cQTLFq0iFtuuYWZM2dWS21lURgpxZkhvrpUIyIiVTFq1Ci+//57Pvroo+JWEXD2w/j222+Ji4tj69at3H333ReMvKnKOX18fBgzZgzbt29n6dKlPPbYY9x7772Eh4cTHx/PpEmTWLt2LQkJCSxatIh9+/bRvn17cnJymDBhAsuWLSMhIYHVq1ezYcOGEn1KXEF9RkrRs2kIc7Yc1kysIiJSJVdffTUNGjRgz5493H333cXb33rrLR544AH69etHw4YNefrpp0lPr56lSPz8/Fi4cCGPP/44vXr1ws/Pj1tvvZW33nqr+Pndu3fz3//+l5MnTxIZGcn48eN56KGHKCws5OTJk4wePZpjx47RsGFDbrnlFl588cVqqa0sJkctmN0rPT0dq9VKWloaQUFBLj/frqPpDPn3Svy9LGz923V4WNSAJCJihNzcXOLj42nevDk+Pj5GlyOluNjvqLzf3/qWLUWb8EACvT3Iyrex51iG0eWIiIjUaQojpbCYTXSLCQbUb0RERIz16aefEhAQUOqtY8eORpdXLdRnpAyxTRuwct8JNiWcZnTfZkaXIyIi9dRNN91Enz59Sn3O1TOj1hSFkTIUL5p3UC0jIiJinMDAQAIDA40uw6V0maYM3WKCMZvgcGoOyWm5RpcjIlKv1YKxFvVWdfxuFEbKEODtQbsIZ89f9RsRETHGmcsQ2dnZBlciZTnzu6nKJSNdprmInk1D2Hk0nU0JpxnaJdLockRE6h2LxUJwcDApKSmAc44MUwWmZBfXcTgcZGdnk5KSQnBwMBaLpdLHUhi5iNhmIXyyLoFNiWoZERExSkREBEBxIBH3EhwcXPw7qiyFkYvoEePsxLrjcBo5+TZ8vSqf+kREpHJMJhORkZGEhYVRUKAFTN2Jp6dnlVpEzlAYuYgmIb6EB3lzLD2P3w6l0qdFqNEliYjUWxaLpVq++MT9qAPrRZhMprNDfNWJVURExCUURi6hZ9MGAFo0T0RExEUURi7hTMvIpsTT2O0a5y4iIlLdFEYuoWNUED6eZlKzCzhwIsvockREROochZFL8LSY6dIkGIBNCaeMLUZERKQOUhgph9gzl2rUb0RERKTaKYyUQ0+FEREREZdRGCmHM5Of/X48i9NZ+QZXIyIiUrcojJRDiL8XLRv5A7BZU8OLiIhUK4WRctLkZyIiIq6hMFJOsUWTn6nfiIiISPVSGCmnHkUtI1uTUskvtBtcjYiISN2hMFJOLRv5E+znSV6hnZ1H040uR0REpM5QGCknk8lEz6JRNRsPavIzERGR6qIwUgE9mznDiEbUiIiIVB+FkQo40zKyKeE0DocWzRMREakOCiMV0KVJMB5mE8fS8zh0OsfockREROoEhZEK8PWy0LGxFdClGhERkeqiMFJBZzuxKoyIiIhUB4WRCoptpkXzREREqpPCSAWdmRZ+d3I6mXmFBlcjIiJS+1UojEyePJlevXoRGBhIWFgYI0aMYM+ePZd83VdffUW7du3w8fGhc+fO/PDDD5Uu2GjhQT40CfHF7oC4xFSjyxEREan1KhRGli9fzvjx41m3bh2LFy+moKCA6667jqysrDJfs2bNGu666y4efPBBtmzZwogRIxgxYgTbt2+vcvFGObtoniY/ExERqSqTowoTZhw/fpywsDCWL1/OgAEDSt3njjvuICsriwULFhRvu+yyy+jWrRvTp08v13nS09OxWq2kpaURFBRU2XKrzSdrD/LcvB1c0bohnzzYx+hyRERE3FJ5v7+r1GckLS0NgAYNGpS5z9q1axk0aFCJbYMHD2bt2rVlviYvL4/09PQSN3dyZtG8uMRUbHZNfiYiIlIVlQ4jdrudiRMn0r9/fzp16lTmfsnJyYSHh5fYFh4eTnJycpmvmTx5MlartfgWHR1d2TJdom14IP5eFjLyCtl7LMPockRERGq1SoeR8ePHs337dj7//PPqrAeASZMmkZaWVnxLSkqq9nNUhYfFTPcYDfEVERGpDpUKIxMmTGDBggUsXbqUJk2aXHTfiIgIjh07VmLbsWPHiIiIKPM13t7eBAUFlbi5mzOXahRGREREqqZCYcThcDBhwgTmzJnDL7/8QvPmzS/5mr59+7JkyZIS2xYvXkzfvn0rVqmbiVUYERERqRYeFdl5/PjxzJ49m3nz5hEYGFjc78NqteLr6wvA6NGjady4MZMnTwbg8ccf58orr+TNN99k6NChfP7552zcuJEZM2ZU81upWd1igjGZIPFUNikZuYQF+hhdkoiISK1UoZaRadOmkZaWxlVXXUVkZGTx7YsvvijeJzExkaNHjxY/7tevH7Nnz2bGjBl07dqVr7/+mrlz516002ttEOTjSdvwQAA2q3VERESk0irUMlKeKUmWLVt2wbaRI0cycuTIipyqVujZNITdyRlsPHia6ztFGl2OiIhIraS1aaqgeNG8RLWMiIiIVJbCSBX0jHFO9rb9cBq5BTaDqxEREamdFEaqILqBL40CvSmwOdh2OM3ockRERGolhZEqMJlM9NTkZyIiIlWiMFJFxSv4HlQYERERqQyFkSrqWdSJdXPi6XKNNhIREZGSFEaqqGNUEF4eZk5l5RN/IsvockRERGodhZEq8vaw0LWJFVC/ERERkcpQGKkGWjRPRESk8hRGqkFsU+d8IwojIiIiFacwUg16xAQDsC8lk7TsAmOLERERqWUURqpBaIA3LRr6A85RNSIiIlJ+CiPVRP1GREREKkdhpJoUT36WcMrgSkRERGoXhZFqElsURrYmpVFgsxtcjYiISO2hMFJNWjYKIMjHg5wCG7uOphtdjoiISK2hMFJNzGZT8aUa9RsREREpP4WRanS234jCiIiISHkpjFSjnkWTn21WGBERESk3hZFq1DXaisVs4mhaLodTc4wuR0REpFZQGKlGfl4edIwKAtRvREREpLwURqpZjxhnvxFdqhERESkfhZFqFttMk5+JiIhUhMJINTszombX0Qyy8goNrkZERMT9KYxUs0irL1FWH2x2B1uTUo0uR0RExO0pjLhAz2bOIb7qxCoiInJpCiMu0DMmGNDkZyIiIuWhMOICsUUtI5sTT2O3OwyuRkRExL0pjLhAu4hA/LwsZOQWsi8l0+hyRERE3JrCiAt4WMx0iw4G1G9ERETkUhRGXEQr+IqIiJSPwoiLnA0jmvxMRETkYhRGXKR70bTwB09mcyIzz+BqRERE3JfCiItYfT1pEx4A6FKNiIjIxSiMuFDPpkVDfBVGREREyqQw4kJn+o1o8jMREZGyKYy4UGxRGNl2KI28QpvB1YiIiLgnhREXahrqR6i/F/k2O9sPpxldjoiIiFtSGHEhk8mk+UZEREQuQWHExRRGRERELk5hxMVim50NIw6HFs0TERE5n8KIi3WMsuJlMXMiM5/EU9lGlyMiIuJ2FEZczMfTQqfGQQBsPKhLNSIiIudTGKkBsc2ck59tSlQYEREROZ/CSA3oUbROzSa1jIiIiFxAYaQGnBlRszclg7ScAoOrERERcS8KIzWgUaA3TUP9cDhgiy7ViIiIlKAwUkPOtI5o0TwREZGSFEZqSPHkZ2oZERERKUFhpIbENnWOqNmSmEqhzW5wNSIiIu5DYaSGtA4LINDHg+x8G7uTM4wuR0RExG0ojNQQs9l0doiv+o2IiIgUUxipQVo0T0RE5EIKIzVIYURERORCCiM1qFt0MGYTHE7N4WhajtHliIiIuAWFkRrk7+1B+0jnonlqHREREXFSGKlhsbpUIyIiUoLCSA3roZlYRURESlAYqWGxzZyTn+04kk5Ovs3gakRERIynMFLDoqw+RAT5UGh3sPVQqtHliIiIGE5hpIaZTCZ6NlO/ERERkTMURgzQUzOxioiIFFMYMcC5k5/Z7Q6DqxERETGWwogBOkQF4eNpJi2ngAMnMo0uR0RExFAKIwbwtJjp2iQYgI0HdalGRETqN4URg8SqE6uIiAigMGKY4n4jiQojIiJSvymMGKRH0YiaA8ezOJWVb3A1IiIixqlwGFmxYgXDhg0jKioKk8nE3LlzL7r/smXLMJlMF9ySk5MrW3OdEOznRauwAEBTw4uISP1W4TCSlZVF165dmTJlSoVet2fPHo4ePVp8CwsLq+ip65wzi+ZtVBgREZF6zKOiLxgyZAhDhgyp8InCwsIIDg6u8OtcqiAXjm6FmD6GnL5H0xA+35CklhEREanXaqzPSLdu3YiMjOTaa69l9erVF903Ly+P9PT0ErdqV5gHX9wDs4bCnh+r//jlcKZlZOuhVPIL7YbUICIiYjSXh5HIyEimT5/ON998wzfffEN0dDRXXXUVmzdvLvM1kydPxmq1Ft+io6OrvzCTGbz8wV4AX9wLu7+v/nNcQvOG/oT4eZJXaGfHkbQaP7+IiIg7MDkcjkrPR24ymZgzZw4jRoyo0OuuvPJKYmJi+OSTT0p9Pi8vj7y8vOLH6enpREdHk5aWRlBQUGXLvZCtEL4dBzu+BbMHjJwF7YdV3/HLYex/N/DzrhSeHdqesVe0qNFzi4iIuFJ6ejpWq/WS39+GDO3t3bs3+/fvL/N5b29vgoKCStxcwuIBt3wAnW4DeyF8dR/snOeac5WhZ9MGgCY/ExGR+suQMBIXF0dkZKQRp76QxQNufh86314USO6HHXNq7PQ9zxlRU4VGKhERkVqrwqNpMjMzS7RqxMfHExcXR4MGDYiJiWHSpEkcPnyYjz/+GIC3336b5s2b07FjR3Jzc/nwww/55ZdfWLRoUfW9i6qyeMDN0539SH77HL5+EBx26HSry0/dpYkVT4uJ4xl5HDqdQ3QDP5efU0RExJ1UOIxs3LiRgQMHFj9+8sknARgzZgyzZs3i6NGjJCYmFj+fn5/Pn/70Jw4fPoyfnx9dunTh559/LnEMt2C2wIipzp9xn8I3Y8Fuhy4jXXpaH08LHaOsxCWlsinhtMKIiIjUO1XqwFpTytsBplrY7fDdY7Dlf86WkhHToesdLj3lPxbs5MNV8dxzWQz/GNHZpecSERGpKW7dgdWtmc0w7F3oMdp5qWbOQxD3mUtPWbxoXkKqS88jIiLijhRGSmM2w43/hp73Aw6Y+whs+dRlpzsTRvYkp5ORW+Cy84iIiLgjhZGymM0w9C2IfRBwwLzxsLn0eVGqKizIh+gGvtgdEJeU6pJziIiIuCuFkYsxm2Hom9BrHOCA+RNg0yyXnKpnTNEQ34Oab0REROoXhZFLMZnghtehz8POx989Dhs/qvbT9GzmnPxsc6LCiIiI1C8KI+VhMsH1/4TLHnU+XvAEbPiwWk9xpmVkS2IqNrvbD3ASERGpNgoj5WUyweBXoO8E5+Pv/wTrP6i2w7eNCCTA24PMvEL2JGdU23FFRETcncJIRZhMcN0/oN8fnY9/+DOsm14th7aYTXSPCQZgky7ViIhIPaIwUlEmE1z7d+g/0fn4p6dh7dRqOXTxfCMHT1XL8URERGoDhZHKMJlg0AtwxZ+cjxdOgjXvVvmwxWFELSMiIlKPKIxUlskEVz8HA55yPl70LKz+d5UO2S06GLMJkk7lkJKeWw1FioiIuD+FkaowmeDq/4Mrn3E+Xvw8rHyr0ocL9PGkbYRz7v5NCWodERGR+kFhpDoMnARX/dV5f8mLsOKNSh+qZ9NgADYqjIiISD2hMFJdrnoaBj7rvP/LS7D8tUodJrapc/KzxTuPkXAyq7qqExERcVsKI9Xpyr/ANc877y99GZb9s8KHuKJ1Q4L9PEk8lc31b69k5up47JoETURE6jCFkep2xZ+cI20Alk2Gpa+Ao/xhIjTAm/njL6dvi1ByCmy8+N1O7pixlvgTaiUREZG6SWHEFS5/Aq59yXl/+avOVpIKBJKYUD8+HduHf4zohL+XhQ0HT3P92yv4YMUBTRUvIiJ1jsKIq/T/I1z3svP+itdhyd8rFEjMZhP3XNaUhU8M4PJWDckrtPPyD7u4bfoa9qdkuqhoERGRmqcw4kr9JsDgyc77q96Cn1+oUCABaBLixycP9uaft3Qm0NuDLYmp3PDOSqYt+51Cm736axYREalhCiOu1vdRGFI0smb127D4uQoHEpPJxJ29Y1j4xACuatuI/EI7r/60m1unrdGieiIiUuspjNSEPg/BDUVzj6x5Fxb+X4UDCUBUsC8z7+vF67d1IdDHg62H0rjx3ZW8u2QfBWolERGRWkphpKb0HgdDi2ZnXTcFfppUqUBiMpkYGRvNz09eyaD2YRTYHLy5eC8jpqxm55H0ai5aRETE9RRGalKvB+HGt533f50GPz5dqUACEB7kwwejY3n7jm4E+3my40g6N723in8t3kt+oVpJRESk9lAYqWmx98OwdwATrH8ffvhzpQOJyWRiRPfGLHpiAIM7hlNod/DvJfu46b1VbD+cVr11i4iIuIjCiBF6joGb3gVMsOFD+P5JsFe+NSMs0Ifp9/Tk3bu608Dfi93JGQyfspo3Fu4hr9BWfXWLiIi4gMKIUXrcCyOmAibY+BF8/0SVAonJZGJY1ygWPzGAoV0isdkdvLd0P8PeXcXWpNRqK1tERKS6KYwYqdvdcPN0wASbZsGCx6sUSMA5nfyUu3swbVQPGgZ4sfdYJjdPXc3kH3eRW6BWEhERcT8KI0breifcMgNMZtj8Mcx/rMqBBGBI50gWP3Elw7tFYXfA+8sPcMM7K9mUcLoaihYREak+CiPuoMvtcMsHzkAS9z+YNx7sVW/FCPH34t93dueD0bE0CvTmwPEsbpu+hn8s2ElOvlpJRETEPZgcjkoO5ahB6enpWK1W0tLSCAoKMroc19n+DXwzDhw2iO4D7YZCy6shvBOYTFU6dFp2AX9fsJNvNh8CoFmoH6/d1pXezRtUR+UiIiIXKO/3t8KIu9kxB74ZC/bCs9v8w6DFVc5g0nIgBEZU+vBLd6cw6dttJKfnYjLBmL7NeOr6tvh5eVS9dhERkXMojNRmp+Jh70L4/Rc4uBIKsks+H9bhbDCJ6QdefhU6fHpuAS8v2MUXG5MAiG7gy6u3dqFfy4bV9Q5EREQURuqMwjxIWg8HljrDyZE44JxfmcULYvqeDSfhncFcvq5AK/YeZ9K32zicmgPAPZfF8MyQ9gR4q5VERESqTmGkrso+BQeWOYPJ70sh/VDJ5/0aOkNJi4HOn0FRFz1cRm4B//xxN5/+mghA42BnK8nlrdVKIiIiVaMwUh84HHBy/9lgcnAl5GeW3KdRe2coaXk1NO0HXv6lHmrN/hM89c1vHDrtbCW5s1c0fx3aniAfT1e/CxERqaMURuqjwnw4tOHsJZ3Dm7ngkk50n6JLOldDRJcSl3Sy8gp57afd/HdtAgCRVh9euaUzA9uG1fAbERGRukBhRJyXdOKXO1tNfl8KaYkln/cLdY7SOXNJx9oEgF8PnOSpb34j4aSz4+xtPZvw3NAOWP3USiIiIuWnMCIlORxw6kDRJZ1fIH4l5GeU3Kdh2+JLOjlRl/HGssN8tDoehwPCAr155ebODOoQbkz9IiJS6yiMyMXZCuDQRmcwObAUDm8CxznT0Js9IboPh0Mv46VdESw6HYEdM7f1bMLfhnUgUH1JRETkEhRGpGJyTkP8iqJLOksgteQlnRxLEIvyO/FawR2YG8Twr9u7EdtMs7eKiEjZFEak8s5c0jlQ1NckfgXkpQNwjFDuypvEQaJ49KpWPD6oNZ4WLXEkIiIXUhiR6mMrdI7SWTARju8mwxLMHdlPsdPRjC5NrPzrjm60bBRgdJUiIuJmyvv9rX/SyqVZPKBpX7jvB4jsSqAtlXkB/+QKnwP8diiNoe+s5JN1CdSCXCsiIm5IYUTKzz8UxnwHMX3xLEjnv56v8FCTJHIL7Dw3dzsPzNpASkau0VWKiEgtozAiFeNjhXu+hZZXYy7I5pnTz/NBn+N4eZhZuuc417+9kkU7ko2uUkREahGFEak4Lz+463NoPwyTLY9rf3uSZYNP0D4yiFNZ+fzhk008881vZOUVGl2piIjUAgojUjke3nDbLOh6FzhsRC15jPl99/LQgBaYTPD5hiRueGclmxNPG12piIi4OYURqTyLBwyfCr3GAQ48f3iSScE/M3vsZURZfUg4mc3I6Wv51+K9FNjslzyciIjUTwojUjVmM9zwOlz+hPPxomfpm/g+Pz5+BcO7RWGzO/j3kn3cNn0t8SeyjK1VRETcksKIVJ3JBINegGuedz5e/irWFX/j33d04993diPQx4OtSanc8O+VzP41UUOARUSkBIURqT5X/AlueMN5f91UmP8Yw7tEsHDiAPq2CCWnwMZf52xj3McbOZGZZ2ytIiLiNhRGpHr1HgcjpoHJDFs+gW/GEhVg4dOxffi/G9rjZTHz864Urn97BUt2HTO6WhERcQMKI1L9ut0NI2c5V/7d8S18cQ9mWy7jBrRg3oT+tA0P5ERmPg/+dyN/nbON7HwNARYRqc8URsQ1OgyHuz8HD1/YtxA+HQl5GbSPDGLehP6Mvbw5ALN/TWToO6vYmpRqbL0iImIYhRFxnVaD4N5vwTsIDq6Ej4dD9il8PC08e2MHPh3bh4ggH+JPZHHLtDW8s2QfhRoCLCJS7yiMiGs17Qdj5oNvAzi8CWbdCBnOviL9WzXkp4lXMLRLJDa7g7cW7+X299eScFJDgEVE6hOFEXG9qO5w/w8QEAEpO2DmEEhNAiDYz4v37urOv+7oSqC3B5sTnUOAv9yQpCHAIiL1hMKI1Iyw9vDAjxAcA6d+h4+uhxP7ATCZTNzcvQk/TryCPs0bkJVv46lvfuOhTzZxKivf4MJFRMTVFEak5jRoAff/BA3bQPohmHk9JG8vfrpJiB+zx13GM0Pa4WkxsWjnMQa/vYKle1IMLFpERFxNYURqlrUx3PcDRHSGrOMw6wY4tLH4aYvZxMNXtmTu+P60DgvgeEYe98/cwPPztpOTbzOwcBERcRWFEal5AY1gzAKI7gO5ac5RNvErS+zSMcrKd49dzv39mwHw8doEbnx3JdsPpxlQsIiIuJLCiBjDNxjunQMtroL8TPj0Nti7sMQuPp4W/jasI5882JvwIG9+P57FiCmrmbJ0Pza7OreKiNQVCiNiHC9/uOsLaDsUCnPh87th+zcX7HZF60b89PgAbugcQaHdwesL93DnjLUknco2oGgREaluCiNiLE8fuP2/0Hkk2Avh6wdh88cX7Bbi78WUu3vw5siuBHh7sOHgaYb8eyVfbzqkIcAiIrWcwogYz+IJN8+AnvcDDpj/GKydesFuJpOJW3s24cfHryC2aQiZeYX8+autPPbZFnVuFRGpxRRGxD2YzXDjv6DfH52PF06CZa9CKa0e0Q38+OKhvvxlcFs8zCYW/HaUuz9cx8nMvBouWkREqoPCiLgPkwmu/Ttc/azz8bJXYNGzpQYSi9nE+IGt+OwPl2H19WRLYiq3TlvDwROaSl5EpLZRGBH3YjLBgL/A9a86H699D757HOylX4bp1awB3zzSjyYhvhw8mc0t09awJfF0DRYsIiJVpTAi7umyh2H4FDCZYfN/4ds/gK2g1F1bhQXw7aP96NQ4iFNZ+dz1wToW7zxWwwWLiEhlVTiMrFixgmHDhhEVFYXJZGLu3LmXfM2yZcvo0aMH3t7etGrVilmzZlWiVKl3ut8Dt30EZg/Y/jV8cS8U5Ja6a1igD1/8oS9XtW1EboGdhz7ZyCdrD9ZsvSIiUikVDiNZWVl07dqVKVOmlGv/+Ph4hg4dysCBA4mLi2PixImMHTuWhQsXXvrFIh1vhjs/Aw8f2PsjzB4JeZml7urv7cGHo2O5s1c0dgc8N28H//xxN3ZNkCYi4tZMjipM0mAymZgzZw4jRowoc5+nn36a77//nu3bzy6Iduedd5KamspPP/1UrvOkp6djtVpJS0sjKCiosuVKbXZwFcy+wzlba5NeMOor8A0pdVeHw8F7v+znzcV7AbipaxSvj+yCt4elJisWEan3yvv97fI+I2vXrmXQoEEltg0ePJi1a9eW+Zq8vDzS09NL3KSea3Y5jJ4PPsFwaAPMGgaZx0vd1WQy8dg1rXlzZFc8zCbmbz3CmI/Wk5ZTep8TERExlsvDSHJyMuHh4SW2hYeHk56eTk5OTqmvmTx5MlartfgWHR3t6jKlNmjSE+7/AfzD4Ng2mHk9pB0qc/dbezZh5v29CPD2YN2BU4ycvobDqaX/NyciIsZxy9E0kyZNIi0trfiWlJRkdEniLsI7wgM/gTUaTu6Hj66Hk7+XufsVrRvx5UN9CQ/yZu+xTG6ZupodR7Tyr4iIO3F5GImIiODYsZLDLI8dO0ZQUBC+vr6lvsbb25ugoKASN5FioS2dgSS0FaQlwUeDnX1KytAhKog5j/anTXgAx9LzuOP9dazcV/olHhERqXkuDyN9+/ZlyZIlJbYtXryYvn37uvrUUpdZm8D9P0JEF8g6Dv+9ybmeTRn9saOCffnq4X70bRFKZl4h98/cwFcb1eImIuIOKhxGMjMziYuLIy4uDnAO3Y2LiyMxMRFwXmIZPXp08f4PP/wwBw4c4KmnnmL37t1MnTqVL7/8kieeeKJ63oHUXwFh8MBC54q/DptzPZtvx0F+6VPCW309mfVAL0Z0i6LQ7uAvX//Gv3/ep1V/RUQMVuEwsnHjRrp370737t0BePLJJ+nevTvPP/88AEePHi0OJgDNmzfn+++/Z/HixXTt2pU333yTDz/8kMGDB1fTW5B6zcsPbvnAOX28yQLbvoL/XAenDpS6u7eHhbdu78YjV7UE4F8/7+WZb7ZRYLPXZNUiInKOKs0zUlM0z4iUy8HV8NV9kJUCPla45UNoc12Zu3+yLoG/zduO3QFXtmnE1FE98Pf2qLl6RUTqOLeZZ0SkxjTrDw8td06KlpsGs2+H5a+BvfRWj3sva8r798bi42lm+d7j3DFjLSkZpU83LyIirqMwInVLUBTc9z3EPgg4YOnL8PndkJNa6u7Xdgjn8z/0JdTfi+2H07l5yhr2p2TUaMkiIvWdwojUPR7ecONbzlV/Ld7ONW0+uBqO7Sx1927RwXz7aD+ahfpxODWHW6etZX38qRouWkSk/lIYkbqr+z3w4ELnBGmnfocPB8H2b0vdtWmoP9880o/uMcGk5RRwz4e/8v1vR2u4YBGR+klhROq2qO7wh+XQ/EooyIKv74dFz4Kt8IJdQwO8mT32Mq7rEE6+zc742Zv5cOUBDf0VEXExhRGp+/xD4Z5vof9E5+M178InIyDrxAW7+npZmHZPT+7r1wyAf3y/ixe/24nNrkAiIuIqCiNSP1g84NoXYeR/wSsADq6E96+Ew5su3NVs4m/DOvB/N7QHYNaagzz66SZyC2w1XbWISL2gMCL1S8cRMHaJc12b9EPOhfY2f3zBbiaTiXEDWvDuXd3xsphZuOMYd3+wjlNZ+TVfs4hIHacwIvVPWDsY9wu0HQq2fJj/GHz3OBTmXbDrsK5RfPJgb4J8PNicmMqt09aQcLL06eZFRKRyFEakfvKxwh3/g6ufBUywaRbMHAJphy/YtU+LUL55pB+Ng32JP5HFLVPXsDUptaYrFhGpsxRGpP4ym2HAX+Cer8En2Nl/5P0BEL/ygl1bhwcy59F+dIwK4mRWPnfOWMeSXcdqvmYRkTpIYUSk1SD4wzKI6AzZJ+Dj4bB2Cpw3pDcsyIcvHurLlW0akVNgY9zHG/nfugRjahYRqUMURkQAGjSHBxZBlzvAYYOFf4VvHoT8kv1DArw9+HBMLLfHNsHugGfnbufVn3Zj19BfEZFKUxgROcPLD25+H4a8BmYP2P6Nc9bWk7+X2M3TYubVW7swcVBrAKYt+50nv4wjv7D0BflEROTiFEZEzmUyQZ+HYMwCCAiHlJ0wYyDsXXjebiYmDmrDa7d1wcNsYm7cEcZ8tJ60nAKDChcRqb0URkRK07Svcxr56D6Qlwazb4dl/wR7ydaP22Oj+ei+Xvh7WVh74CS3T1/LkdQcg4oWEamdFEZEyhIU6Wwh6TXW+XjZZPj8LshJLbHbgDaN+PLhvoQFerPnWAa3TF3DrqPpNV+viEgtpTAicjEeXjD0TRgxDTx8YO9P8MFAOLajxG4do6zMGd+f1mEBJKfnMnL6WpbvPW5Q0SIitYvCiEh5dLsbHlgI1hg4dcDZsXXb1yV2aRzsy9cP96NP8wZk5hVy/8z1fLBCq/6KiFyKwohIeUV1c85H0uIqKMh2Dv1d+H9gKyzexernyccP9i4e+vvyD7t48sutWmRPROQiFEZEKsI/FO75Fi5/wvl47XvwyQjIPHtJxtvDwqu3duHFmzpiMZuYs+UwI9WxVUSkTAojIhVltsCgF+D2T8ArAA6uhBlXwqFNxbuYTCbG9GvGJw/2JsTPk22H07jpvVVsOHjKuLpFRNyUwohIZXW4ybn6b2hrSD8MM693Lrh3jn4tGzJ/wuW0iwjkRGY+d3+wjtm/JhpTr4iIm1IYEamKRm2dgaTdjWDLh+8eh/mPQWFe8S7RDfz49tF+DO0cSYHNwV/nbOPZuds0Y6uISBGFEZGq8glyXrK55nnABJs/ho+uh9SzLSB+Xh68d3d3/jK4LSYT/G9dIvf851dOZOaVfVwRkXpCYUSkOpjNcMWf4J5vwDcEjmyGqX1hw4fFs7aaTCbGD2zFh6NjCfD2YH38KW56dxXbD6cZXLyIiLEURkSqU6trnMN/Y/pCfiZ8/yf4+Cbn3CRFrmkfztzx/WnR0J8jabncNn0N8+IOG1eziIjBFEZEqltIM7jvB+fqv55+ztE2U/vB2qlgd8430iosgDnj+3NV20bkFth5/PM4Jv+4C5tdE6SJSP2jMCLiCmazc/XfR9ZA8wFQmAMLJ8HMIXB8LwBWX0/+M6YXD1/ZEoD3lx/ggVkbtPKviNQ7CiMirtSgOYyeDze+DV6BkPQrTL8cVv0LbIVYzCaeGdKOf9/ZDR9PM8v3HmfElNXsT8kwunIRkRqjMCLiaiYTxN4Pj66FlteALQ9+fgH+MwiO7QRgeLfGfP1wPxoH+xJ/IosRU9bw885jxtYtIlJDFEZEakpwtHO0zfCp4GOFI1vg/QGw/DWwFdCpsZV5E/rTu2ihvXGfbOS9X/ZpoT0RqfMURkRqkskE3UfBo79C2xvAXgBLX4YZA+FIHA0DvPl0bB/uvawpDge8sWgv42dvJiuv8NLHFhGppRRGRIwQFAl3zoZb/wO+DeDYNvjgaljyEp6OAl4a0YnJt3TG02Lih23J3DptDUmnso2uWkTEJRRGRIxiMkHn22D8r9BhBDhssPIN56WbQxu5q3cMn427jIYB3uxOzuCm91ax5vcTRlctIlLtFEZEjBYQBrf/F27/GPwbwfHd8J9rYdGzxDb25bvH+tOliZXT2QXc+5/1zFodr34kIlKnKIyIuIsOw2H8euhyBzjssOZdmNafyNQ4vnyoLzd3b4zN7uCF73by1Ne/kVdoM7piEZFqoTAi4k78GsAtM+CuLyAwEk79DjOH4LN4Em+NaMWzQ9tjNsFXmw5xx/vrOJaea3TFIiJVpjAi4o7aXg+ProPu9wIOWP8+pmn9GNs4if8+0BurrydxSakMe3cVWxJPG12tiEiVKIyIuCvfYBj+HtzzLVijITUBPr6JK3b/g/njutA6LICUjDzueH8dX21MMrpaEZFKUxgRcXetrnHO3hr7oPPxplk0/fxq5l+fy3Udwsm32fnL17/x4nc7KLTZja1VRKQSFEZEagPvQLjxLRizwLkqcPphfL+8nfcD/8NTV4YDMHP1QUZ/tJ7TWfnG1ioiUkEKIyK1SfMrnCsBX/YoYMK0dTaP7hjFN1en4udlYc3vJ7lpyip2HU03ulIRkXJTGBGpbbz84frJ8MBPENoKMpPpueZR1rWZTeeQQpJO5XDL1DX8uO2o0ZWKiJSLwohIbRVzGTy8CvpPBJOZoP3zmGd6kiejdpJTUMgjn27mrUV7sNs1QZqIuDeFEZHazNMXrn0Rxv4MYR0wZ5/gj6f+wY8RH9CIVN75ZT9/+GQTGbkFRlcqIlImhRGRuqBxT/jDMrjyaTB70D51GasCJ3Gb52p+3pXMzVPXEH8iy+gqRURKpTAiUld4eMPAv8K4pRDRBe+CNN6wTOF/vv8iIyWR4e+tYtmeFKOrFBG5gMKISF0T2QXG/QJXPwsWLy53bOQX36e4vmAx981cz8OfbGJ/SobRVYqIFDM5asHyn+np6VitVtLS0ggKCjK6HJHaI2UXzBsPhzcB8Ku9HV/ZrmSJvSeDerRj4rVtaBzsa3CRIlJXlff7W2FEpK6zFcK6qbD0ZSh0LqxX4LCw1t6BxfQhpMfNjLm2F6EB3gYXKiJ1jcKIiJR0+iDEfQa7voOUHcWb7Q4Tm2lHZosh9LrhPvwbNTWuRhGpUxRGRKRsJ/bj2DWfjC1zCDr1W4mnjgd1JLjnbXh2Gg6hLQ0qUETqAoURESkX++lEdi2djW37PDrZdmE2nf2T4AjrgKnDcGh/E4S1B5PJwEpFpLZRGBGRCim02VmwJo49yz6nb/4a+pl34GE6ZxXg0FbQfpgzmER1VzARkUtSGBGRSsktsPHx2oP8b2kcvfN/5Xrzeq60bMOTwrM7WaPPBpPoPmDWLAEiciGFERGpkvTcAj5YcYAPV8ZjKchgoDmOe61biS3YiLkw5+yOAeHQbqgzmDS7HCyexhUtIm5FYUREqsXxjDymLN3Pp78mUGBz4E0+TzZP5O7AOAITlkBe2tmdfUOg7VBnq0nLgc5ZYUWk3lIYEZFqlXQqm38t3sucuMM4HGAxm7izezh/apNMg4SfYPf3kH3y7Au8AqHNYGcwaX0tePkbV7yIGEJhRERcYndyOm8s3MvPu44B4OVhZkzfpjw6oBkhJzbBrvnOuUwyjp59kYcvtLrGeSmn7fXgYzWoehGpSQojIuJSmxJO8epPe1gffwqAQG8Pxg1owYOXN8ff0+ycgn7XPNg5H1ITzr7Q7AktroION0HbG8C/oTFvQERcTmFERFzO4XCwfO9xXvtpDzuPpgPQMMCLCQNbcVefGLw9LOBwQPI2Z4vJzvlwYs/ZA5jM0LQ/hDQFTM7HJlPJ+yZz0ePz75tK2V7K64vvU87jFt339IEGLZxDmgPCNZS5tsnLBFu+sx+TfneGURgRkRpjtztYsO0oby3aw8GT2QA0DvblyWvbMKJ7Yyzmc74Mju85G0ySfyvjiG7GO8g5G21oa2c4adiq6H5L9YVxB7YCOLbD2Rp3eLPz5/HdgAM8/cDa5JxbtPNnUOOzPz19jH4HdZbCiIjUuAKbnS83JvHvn/eRkpEHQJvwAP58XVuu7RCO6fx/oZ4+CPsWQ166swXF4QCKfjrsl7hvL9qfc+47Sr9f4vXlPEd+JpzcD6mJRc+VIahxUUBpXTKsWKPBbKnOj1fA+fs5He8MHYc2OoNH8m/Fi0BWin/YhWHl3Jt/I7WuVJLCiIgYJiffxn/XHmTq0v2k5zonS+seE8xTg9vRt2WowdVVUGEenIqHk/vgxD44+fvZ+zmnyn6dxbuoNaWoRaVhUVAJbQV+DWqu/tou8zgcKWrtOHPLOX3hfj5WiOoBjXtCk1jnfd9gSD8MaYfOuSWVfFyQfekaLN5gbVxGWIl2BlIvv2p/63WBwoiIGC4tu4D3V/zOR6vjyS1wti4MaNOIpwa3pVPjOjCiJvuUs/XkxL6SYeXU787+CmXxC73wkk/D1hDSHDy8aq5+d5OfBUe3lgweqYkX7mfxgoguzuBx5tagRcVnAnY4nMEmLQnSDpceVjKOAuX4mvRtUHZYsTZx9juqhzMVK4yIiNtISc/l3V/289n6RArtzj85Q7tE8qdr29CiUYDB1bmA3eb8Ej25/7ywsh8yjpT9OpMZgpuec8mn5dn7gRF161KBrdDZr+PwJji80XnZJWVn6ZfEGrYtCh1FLR/hnWoutNkKIP1I6S0r6YchNQnyMy59HLMnBEU5g4lfqLPVxjfEefM55/65270Cav3vXGFERNxOwsks/rV4L/O2HimeOO322CY8fGVLmobWk46geZnOlpMT+84LK/ud/VTK4hXgDCdBjc/58gou5X7RYx+r+0zN73A4w1lxi8dmOBpX+iWSwKizoaNxT4jq5v7z0uSmlX0ZKO2QM8w4bBU/rtmj7KBSYvv5z7nP796lYWTKlCm8/vrrJCcn07VrV95991169+5d6r6zZs3i/vvvL7HN29ub3NzydzZSGBGpW3YeSeeNRXv4ZXdK8bau0cEM7xrFjV0iCQuqh6MbHA7ISHa2oJzc72xFOXPpJzXh4p1oy+IVcM4XV3ApgSW49EDjY61a59vsU2dHtZy5ZZ8opb5AaNy9KHjEOkNIUFTlz+uubIWQmXy2NSX7FOSkOi8R5Rb9zDl9dlvOqYtf5isPr8Ci36m1nAEm2HkpqZpbnFwWRr744gtGjx7N9OnT6dOnD2+//TZfffUVe/bsISws7IL9Z82axeOPP86ePWfnFjCZTISHh1f7mxGR2mXDwVO8+8t+Vu07TtHVG8wm6NsylOFdGzO4UwRWX/f4F56hCvOdI0hO7IOsFOeXVm7qeV9oqWe356VX8YQm8AkqZwtMsPNf8Mm/OUPHoY3OWs9n9oSITiX7eYS2rpf9KC7J4YCCnDKCyumLhJjUkmtFVdSYBdD8imp5C2e4LIz06dOHXr168d577wFgt9uJjo7mscce45lnnrlg/1mzZjFx4kRSU1Mr9g7OoTAiUrcdz8jj+9+OMG/rEbYkphZv97KYuaptI4Z3a8w17cPw8dRQ2XKxFToDyZkvqNzTZQeX4p9F+xRkVU8Noa1KBo/wTprPoyaU+N2fF2IuFm5yTsMfljkDYzUq7/e3R0UOmp+fz6ZNm5g0aVLxNrPZzKBBg1i7dm2Zr8vMzKRp06bY7XZ69OjBK6+8QseOHStyahGpwxoFenNf/+bc1785iSez+e63I8yLO8zeY5ks2nmMRTuP4e9lYXDHCG7qFkX/Vg3xtOhf1GWyeDiHD1dmCHFh/nkhJfW8EHP6whBTkAPhHc/29Yjq7mw1kZpX2d+9wd1HKxRGTpw4gc1mu+ASS3h4OLt37y71NW3btuWjjz6iS5cupKWl8cYbb9CvXz927NhBkyZNSn1NXl4eeXl5xY/T06va5CgitUVMqB/jB7Zi/MBW7E5OZ17cEebHHeFwag7fbjnMt1sOE+rvxQ2dIxneLYoeMSGYz53hVarGwwsCwpw3qT8MHrVToTBSGX379qVv377Fj/v160f79u15//33eemll0p9zeTJk3nxxRddXZqIuLl2EUG0uz6Ipwa3ZXPiaebFHeH7345yMiufT9Yl8Mm6BBoH+zKsaxTDu0XRLiLwwlleRcTtVajPSH5+Pn5+fnz99deMGDGiePuYMWNITU1l3rx55TrOyJEj8fDw4LPPPiv1+dJaRqKjo9VnREQotNlZ/ftJ5sUdZuH2ZLLyzw6ZbB0WwPBuUdzUtTExoZoRU8Ro5e0zUqGLrl5eXvTs2ZMlS5YUb7Pb7SxZsqRE68fF2Gw2tm3bRmRkZJn7eHt7ExQUVOImIgLgYTFzZZtGvHV7NzY9dy1TR/VgcMdwvCxm9qVk8saivQx4fSkjpqxm5up4UjKqsGaJiNSISg3tHTNmDO+//z69e/fm7bff5ssvv2T37t2Eh4czevRoGjduzOTJkwH4+9//zmWXXUarVq1ITU3l9ddfZ+7cuWzatIkOHTqU65waTSMil5KWU8DCHcnMjzvCmt9PlBgq3K9lQ27qFsX1nSII8tFQYZGa4pLRNAB33HEHx48f5/nnnyc5OZlu3brx008/FXdqTUxMxHzOuPHTp08zbtw4kpOTCQkJoWfPnqxZs6bcQUREpDysvp7cHhvN7bHRpGTk8v1vR5kXd4S4pFRW7T/Bqv0neHbudgYWDRW+up2GCou4C00HLyJ1WsLJLL7beoS5cUfYn3J2uvUAbw+u6xjO8G6N6d8yFA8NFRapdlqbRkTkHA6Hg11HM5i/9QjfbXUOFT4j1N+LoV3ODhXWiByR6qEwIiJSBrvdwabE08yPO8L3245yKuvsOiBNQpxDhW/oFEnLMH/8vFw+A4JInaUwIiJSDgU2O6v2n+C7uCMs3FFyqDBAwwAvmoT4Ed3Aj+gQ36KffkQ38CUq2FczwYpchMKIiEgF5eTb+GV3CnPjDrPuwEkycgsvur/ZBJFWX5qcF1LO3A8L9NbssFKvKYyIiFRRWnYBSaezSTqVXfQzp/jxodM55BXaL/p6Lw8zTYJ9aVJKq0p0iB/Bfp7qnyJ1msuG9oqI1BdWP0+sflY6NbZe8Jzd7uBEZt7ZkHJeYDmalkt+oZ0DJ7I4cKL0lXADvD0ubFU5c0moga/6q0i9oZYREREXKLTZOZqWW2qrStLpHI5n5F3yGKH+Xhe0qrQJD6BjlBVfL82RIu5PLSMiIgbysJiLWjhKXyMnt8DGofNDyjn303MLOZmVz8msfLYmpZZ4rcVsol1EIN2ig+kaHUz36GBaNgpQ/xSptdQyIiLihtJyCor6pjj7pySdyibhVDY7jqSX2qoS4O1BlybWEgElLMjHgMpFzlIHVhGROsjhcHA0LZe4pFS2JqWyJSmVbYfSyCmwXbBvpNWHbtHBxQGlc2Mr/t5qEJeaozAiIlJPFNrs7EvJLA4ocUmp7D2WUbxY4BlmE7QJDywRUNqEB2LR5R1xEYUREZF6LCuvkG2H00oElKNpuRfs5+dloXNja4mAEmn10ZBjqRYKIyIiUsKxdOflnTMB5bdDaWTmXTixW1igd4m+J52bWAn08TSgYqntFEZEROSibHYHB45nsuWcgLI7OQPbedd3TCZo1SigOKB0iw6mbUSgpsKXS1IYERGRCsvJt7HjiPPyzpaigHLodM4F+/l4mukU5by807mJlSYhvoQF+tAo0BsfT82BIk4KIyIiUi2OZ+Tx26HU4ks8cUmpF123J8jHg7AgH8ICvQkv+tko0PuCbRrZU/cpjIiIiEvY7Q7iT2YVd4zdeSSdYxm5pKTnXXK9nnP5e1kIC3K2ppwJKGGB3oQFeRMWWPQ4yIcgHw91qK2lFEZERKRGORwO0nMLSUnPJSUjj5SigJKSkcexom3HM/JISc8lK//CeVHK4u1hLhFQwosCTNh5rS0hWnjQ7Wg6eBERqVEmkwmrrydWX09ahwdedN/MvHNDizOgHD8ntJzZlp5bSF6hvWgxwgv7rpzL02KiUYA3jc5pZSlucTkTZoK8CfX31twqbkZhREREalyAtwcBjQJo0SjgovvlFtiKWldyiwOKs6XFue14UXA5lZVPgc3BkbRcjpQyn8q5zCZoGHA2oIQHedMo8NzLRM5tDQO8NWKohiiMiIiI2/LxtBAT6kdMaOkLDp6RX2jneGZeidaW4+m5xaHlzLaTmXnYHRQ/hvQyj2kyQQM/r+LOt+EX9Gfx1giiaqIwIiIitZ6Xh5nGwb40Dva96H6FNjsns/LPa23JK+6AezzjbN+WQrujeOXk3ckZFz2u1dez5Oih8/q4nAkvfl762i2NPhUREak3PCxmwoN8CA/yAaxl7me3OziVXTK0FPdpOS/I5NvspOUUkJZTwL6UzIueP8Dbg4YBXjQMcF4Gahh4zv0Abxqd89jPy1JvOuQqjIiIiJzHbDYVh4IOlD0KxOFwkJZTUBxMUjJKXho6fs62nAIbmXmFZOYVcvBk9iVr8PE0lxlUnDcvQgO8aRTgTZBv7R7+rDAiIiJSSSaTiWA/L4L9vGhzkRFEDofDOYIoI48TGXmcyMznRGZe0e3c+3mcyMgnp8BGboGdQ6dzSp0B93xeFjOhxS0uRT8DvUs+Lrof4ueF2c1GEymMiIiIuJjJZCLQx5NAH09aXmIEEThXXT4TTo5nXBhWTmYVhZiMPDLyCsm32Tmallvqyszns5hNNPA/G1waFQWXu3vH0Kyhf3W83QpTGBEREXEz/t4e+Ht70DT00uEgt8B2toUlI69Ei8vxTGdLzMksZ6BJzS7AZndwvKgPzLkGd4xQGBEREZGK8/G00CTEjyYhFx/+DM4h0KeKgsmZoHLmMlFMg0u/3lUURkREROoJLw8zEVYfIqw+RpdSgqaWExEREUMpjIiIiIihFEZERETEUAojIiIiYiiFERERETGUwoiIiIgYSmFEREREDKUwIiIiIoZSGBERERFDKYyIiIiIoRRGRERExFAKIyIiImIohRERERExVK1YtdfhcACQnp5ucCUiIiJSXme+t898j5elVoSRjIwMAKKjow2uRERERCoqIyMDq9Va5vMmx6Xiihuw2+0cOXKEwMBATCZTtR03PT2d6OhokpKSCAoKqrbj1kX6rCpGn1f56bMqP31W5afPqvxc+Vk5HA4yMjKIiorCbC67Z0itaBkxm800adLEZccPCgrSf6zlpM+qYvR5lZ8+q/LTZ1V++qzKz1Wf1cVaRM5QB1YRERExlMKIiIiIGKpehxFvb2/+9re/4e3tbXQpbk+fVcXo8yo/fVblp8+q/PRZlZ87fFa1ogOriIiI1F31umVEREREjKcwIiIiIoZSGBERERFDKYyIiIiIoep1GJkyZQrNmjXDx8eHPn36sH79eqNLcjuTJ0+mV69eBAYGEhYWxogRI9izZ4/RZdUK//znPzGZTEycONHoUtzS4cOHueeeewgNDcXX15fOnTuzceNGo8tyOzabjeeee47mzZvj6+tLy5Yteemlly651kd9sWLFCoYNG0ZUVBQmk4m5c+eWeN7hcPD8888TGRmJr68vgwYNYt++fcYUa7CLfVYFBQU8/fTTdO7cGX9/f6Kiohg9ejRHjhypkdrqbRj54osvePLJJ/nb3/7G5s2b6dq1K4MHDyYlJcXo0tzK8uXLGT9+POvWrWPx4sUUFBRw3XXXkZWVZXRpbm3Dhg28//77dOnSxehS3NLp06fp378/np6e/Pjjj+zcuZM333yTkJAQo0tzO6+++irTpk3jvffeY9euXbz66qu89tprvPvuu0aX5haysrLo2rUrU6ZMKfX51157jXfeeYfp06fz66+/4u/vz+DBg8nNza3hSo13sc8qOzubzZs389xzz7F582a+/fZb9uzZw0033VQzxTnqqd69ezvGjx9f/NhmszmioqIckydPNrAq95eSkuIAHMuXLze6FLeVkZHhaN26tWPx4sWOK6+80vH4448bXZLbefrppx2XX3650WXUCkOHDnU88MADJbbdcsstjlGjRhlUkfsCHHPmzCl+bLfbHREREY7XX3+9eFtqaqrD29vb8dlnnxlQofs4/7Mqzfr16x2AIyEhweX11MuWkfz8fDZt2sSgQYOKt5nNZgYNGsTatWsNrMz9paWlAdCgQQODK3Ff48ePZ+jQoSX++5KS5s+fT2xsLCNHjiQsLIzu3bvzwQcfGF2WW+rXrx9Llixh7969AGzdupVVq1YxZMgQgytzf/Hx8SQnJ5f4f9FqtdKnTx/9rS+HtLQ0TCYTwcHBLj9XrVgor7qdOHECm81GeHh4ie3h4eHs3r3boKrcn91uZ+LEifTv359OnToZXY5b+vzzz9m8eTMbNmwwuhS3duDAAaZNm8aTTz7JX//6VzZs2MAf//hHvLy8GDNmjNHluZVnnnmG9PR02rVrh8ViwWaz8fLLLzNq1CijS3N7ycnJAKX+rT/znJQuNzeXp59+mrvuuqtGFhqsl2FEKmf8+PFs376dVatWGV2KW0pKSuLxxx9n8eLF+Pj4GF2OW7Pb7cTGxvLKK68A0L17d7Zv38706dMVRs7z5Zdf8umnnzJ79mw6duxIXFwcEydOJCoqSp+VuERBQQG33347DoeDadOm1cg56+VlmoYNG2KxWDh27FiJ7ceOHSMiIsKgqtzbhAkTWLBgAUuXLqVJkyZGl+OWNm3aREpKCj169MDDwwMPDw+WL1/OO++8g4eHBzabzegS3UZkZCQdOnQosa19+/YkJiYaVJH7+stf/sIzzzzDnXfeSefOnbn33nt54oknmDx5stGlub0zf8/1t778zgSRhIQEFi9eXCOtIlBPw4iXlxc9e/ZkyZIlxdvsdjtLliyhb9++BlbmfhwOBxMmTGDOnDn88ssvNG/e3OiS3NY111zDtm3biIuLK77FxsYyatQo4uLisFgsRpfoNvr373/BEPG9e/fStGlTgypyX9nZ2ZjNJf9UWywW7Ha7QRXVHs2bNyciIqLE3/r09HR+/fVX/a0vxZkgsm/fPn7++WdCQ0Nr7Nz19jLNk08+yZgxY4iNjaV37968/fbbZGVlcf/99xtdmlsZP348s2fPZt68eQQGBhZfZ7Varfj6+hpcnXsJDAy8oC+Nv78/oaGh6mNznieeeIJ+/frxyiuvcPvtt7N+/XpmzJjBjBkzjC7N7QwbNoyXX36ZmJgYOnbsyJYtW3jrrbd44IEHjC7NLWRmZrJ///7ix/Hx8cTFxdGgQQNiYmKYOHEi//jHP2jdujXNmzfnueeeIyoqihEjRhhXtEEu9llFRkZy2223sXnzZhYsWIDNZiv+e9+gQQO8vLxcW5zLx+u4sXfffdcRExPj8PLycvTu3duxbt06o0tyO0Cpt5kzZxpdWq2gob1l++677xydOnVyeHt7O9q1a+eYMWOG0SW5pfT0dMfjjz/uiImJcfj4+DhatGjh+L//+z9HXl6e0aW5haVLl5b6N2rMmDEOh8M5vPe5555zhIeHO7y9vR3XXHONY8+ePcYWbZCLfVbx8fFl/r1funSpy2szORyaxk9ERESMUy/7jIiIiIj7UBgRERERQymMiIiIiKEURkRERMRQCiMiIiJiKIURERERMZTCiIiIiBhKYUREREQMpTAiIiIihlIYEREREUMpjIiIiIihFEZERETEUP8P1wX2YAJev/kAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "loss, val_loss = history.history['loss'], history.history['val_loss']\n",
        "plt.plot(loss, label='loss')\n",
        "plt.plot(val_loss, label='val_loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q5FjHU53w9xr",
        "outputId": "fcf6a64c-72e3-444a-fb72-b4b581fa81e8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "63/63 [==============================] - 1s 7ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.94      0.87      0.90       275\n",
            "           1       0.85      0.93      0.89       224\n",
            "           2       0.93      0.92      0.93       695\n",
            "           3       0.76      0.84      0.80       159\n",
            "           4       0.96      0.96      0.96       581\n",
            "           5       0.63      0.55      0.59        66\n",
            "\n",
            "    accuracy                           0.91      2000\n",
            "   macro avg       0.85      0.84      0.84      2000\n",
            "weighted avg       0.91      0.91      0.91      2000\n",
            "\n",
            "[[239  14   3   1  16   2]\n",
            " [  4 208   1   1   4   6]\n",
            " [  3   1 642  36   3  10]\n",
            " [  1   0  21 134   1   2]\n",
            " [  6   4  11   2 557   1]\n",
            " [  2  17   9   2   0  36]]\n",
            "0.908\n"
          ]
        }
      ],
      "source": [
        "predictions = model.predict(testing_padded)\n",
        "prediction_labels = predictions.argmax(axis=1)\n",
        "print(classification_report(y_test, prediction_labels))\n",
        "print(confusion_matrix(y_test, prediction_labels))\n",
        "print(accuracy_score(y_test, prediction_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kn4-Pcc5w9xr"
      },
      "source": [
        "# Transformer based Pretrained model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "6amiAoIjw9xr",
        "outputId": "2a772584-3579-4f74-b359-b12dce055fa6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-c9fefe7e-f6a4-4089-8850-006eab7cc84b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>processed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>i didnt feel humiliated</td>\n",
              "      <td>sadness</td>\n",
              "      <td>not feel humiliate</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>i can go from feeling so hopeless to so damned...</td>\n",
              "      <td>sadness</td>\n",
              "      <td>feel hopeless damn hopeful care awake</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>im grabbing a minute to post i feel greedy wrong</td>\n",
              "      <td>anger</td>\n",
              "      <td>m grab minute post feel greedy wrong</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>i am ever feeling nostalgic about the fireplac...</td>\n",
              "      <td>love</td>\n",
              "      <td>feel nostalgic fireplace know property</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>i am feeling grouchy</td>\n",
              "      <td>anger</td>\n",
              "      <td>feel grouchy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c9fefe7e-f6a4-4089-8850-006eab7cc84b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-03053ac0-fed0-45b6-8528-85dc61f7b841\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-03053ac0-fed0-45b6-8528-85dc61f7b841')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-03053ac0-fed0-45b6-8528-85dc61f7b841 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c9fefe7e-f6a4-4089-8850-006eab7cc84b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c9fefe7e-f6a4-4089-8850-006eab7cc84b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                               tweet sentiment  \\\n",
              "0                            i didnt feel humiliated   sadness   \n",
              "1  i can go from feeling so hopeless to so damned...   sadness   \n",
              "2   im grabbing a minute to post i feel greedy wrong     anger   \n",
              "3  i am ever feeling nostalgic about the fireplac...      love   \n",
              "4                               i am feeling grouchy     anger   \n",
              "\n",
              "                                processed  \n",
              "0                      not feel humiliate  \n",
              "1   feel hopeless damn hopeful care awake  \n",
              "2    m grab minute post feel greedy wrong  \n",
              "3  feel nostalgic fireplace know property  \n",
              "4                            feel grouchy  "
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "texts_train = df_train['tweet']\n",
        "texts_test = df_test['tweet']\n",
        "\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kkLPzGPw9xr"
      },
      "source": [
        "- Importing Transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ioc_Jw_lw9xr"
      },
      "source": [
        "Fast Encode Function Retrieved From the internet."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MBt8deQww9xr"
      },
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n",
        "    \"\"\"\n",
        "    Encoder for encoding the text into sequence of integers for BERT Input\n",
        "    \"\"\"\n",
        "    tokenizer.enable_truncation(max_length=maxlen)\n",
        "    tokenizer.enable_padding(length=maxlen)\n",
        "    all_ids = []\n",
        "\n",
        "    for i in tqdm(range(0, len(texts), chunk_size)):\n",
        "        text_chunk = texts[i:i+chunk_size].tolist()\n",
        "        encs = tokenizer.encode_batch(text_chunk)\n",
        "        all_ids.extend([enc.ids for enc in encs])\n",
        "\n",
        "    return np.array(all_ids)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oBNNOj540_4q",
        "outputId": "b84bea64-269b-42b0-c2d1-809e220bf197"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "  Downloading transformers-4.31.0-py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m20.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.2)\n",
            "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers)\n",
            "  Downloading huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m24.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.27.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers)\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m76.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers)\n",
            "  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m76.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.65.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (4.7.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4)\n",
            "Installing collected packages: tokenizers, safetensors, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.16.4 safetensors-0.3.1 tokenizers-0.13.3 transformers-4.31.0\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 148,
          "referenced_widgets": [
            "694330cda08b453c8ed46d22c463b93c",
            "5337a253cb86488ebe32eb598556413f",
            "dd19cd086a9647a9b5e11c737dae9778",
            "0869185102034822b086f01a7f82c698",
            "9e5b62996fef4c9f858d3137f1f7ddcd",
            "80a1d649f235493092c4222b0483d38b",
            "00f8fee3a784465980e6874ce6c184a6",
            "1cb25ddeb15142d7b15476897f30783f",
            "bd98daaf6f934b23abff82c2fecb3fc1",
            "36fc5ac7e3ee4acaac95266078d0201e",
            "c625e94451a84cdcaa7f17e3346dc0a2",
            "8958269a6e984badad837614e2bc9f12",
            "4868bab11dd14a8abac93b647e723435",
            "af72e62cebb7458e897fd2e76595012b",
            "f13bc3c2724b4b8d8d1ff5c82308f002",
            "b764ed364d9444c89ac118389c8c8e48",
            "92b6625bc97a4d6aa57a8bcb4e85bf70",
            "c443289e3af94bc191d0abf32a2b1147",
            "a5e829830d014b32b6fe08f3b1da0477",
            "42343b32671b4f1193656b01b7efa816",
            "ed24da5ac71c4999a3609e32b809314e",
            "20c1cdce5b4e497a879e1d19880afe6e",
            "3cb2ac723c354105a1326ed751c16dc3",
            "488651f1c2c94891958f936842db07fa",
            "8d33b4b1932f4f64a82c7390e3aeb696",
            "43c9ecffb2264a62a5c56a5670b345b0",
            "ab5c57f5bde2430399ab49a7d7eb4a15",
            "cca7527080ff4144ad92f3b7a9396f83",
            "753ee821bd0844c5abc11575df9ef275",
            "ed4d36090a9a4d6ea01d912edad8dcc8",
            "8393277a5a384980a72222ea03700758",
            "d2d83f468bd24cd68cc38e095f75e04f",
            "59b46d93eeec4d32a70d3432b9a10165"
          ]
        },
        "id": "VTuB7IxFw9xr",
        "outputId": "13dd8157-70ac-401d-cba7-5a2742da7b5c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "694330cda08b453c8ed46d22c463b93c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/996k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8958269a6e984badad837614e2bc9f12",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)okenizer_config.json:   0%|          | 0.00/29.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3cb2ac723c354105a1326ed751c16dc3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/466 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "Tokenizer(vocabulary_size=119547, model=BertWordPiece, unk_token=[UNK], sep_token=[SEP], cls_token=[CLS], pad_token=[PAD], mask_token=[MASK], clean_text=True, handle_chinese_chars=True, strip_accents=None, lowercase=False, wordpieces_prefix=##)"
            ]
          },
          "execution_count": 59,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import transformers\n",
        "from tokenizers import BertWordPieceTokenizer\n",
        "\n",
        "#Loading bert tokenizer\n",
        "bert_tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n",
        "\n",
        "#Saving it on our local folder\n",
        "bert_tokenizer.save_pretrained('.')\n",
        "\n",
        "\n",
        "#Loading a fast tokenizer\n",
        "fast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\n",
        "fast_tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Is2Vn53ew9xs"
      },
      "source": [
        "Creation of validation set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KNV7RZutw9xs"
      },
      "outputs": [],
      "source": [
        "bert_X_train, bert_X_valid, bert_Y_train, bert_Y_valid = train_test_split(df_train['tweet'].values, df_train['sentiment'].values,test_size=0.2,random_state=42,stratify=df_train['sentiment'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fa6mnk6jw9xs"
      },
      "source": [
        "Encoding raw test for training validation and test."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yvgpDXD4w9xs",
        "outputId": "5bfcad80-15f0-4646-f6fd-6f71f039123f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 50/50 [00:02<00:00, 21.83it/s]\n",
            "100%|██████████| 13/13 [00:00<00:00, 18.28it/s]\n",
            "100%|██████████| 8/8 [00:00<00:00, 17.79it/s]\n"
          ]
        }
      ],
      "source": [
        "X_train = fast_encode(bert_X_train, fast_tokenizer, maxlen=max_length)\n",
        "X_valid = fast_encode(bert_X_valid, fast_tokenizer, maxlen=max_length)\n",
        "X_test = fast_encode(df_test['tweet'].to_numpy(), fast_tokenizer, maxlen=max_length)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5UchjnhWw9xs",
        "outputId": "2880911c-2a1a-47f3-dd13-de105e915e67"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(12800, 100)"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HPsiMJpTw9xs"
      },
      "source": [
        "Encoding the labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rBAPwyW0w9xs"
      },
      "outputs": [],
      "source": [
        "le = LabelEncoder()\n",
        "y_train = le.fit_transform(bert_Y_train)\n",
        "y_valid = le.transform(bert_Y_valid)\n",
        "y_test = le.transform(df_test['sentiment'])\n",
        "y_train_encoded = to_categorical(y_train)\n",
        "y_valid_encoded = to_categorical(y_valid)\n",
        "y_test_encoded = to_categorical(y_test)\n",
        "\n",
        "\n",
        "\n",
        "BATCH_SIZE = 16\n",
        "\n",
        "train_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_train, y_train_encoded))\n",
        "    .repeat()\n",
        "    .shuffle(2048)\n",
        "    .batch(BATCH_SIZE)\n",
        "\n",
        ")\n",
        "\n",
        "valid_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_valid, y_valid_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        "    .cache()\n",
        "\n",
        ")\n",
        "\n",
        "test_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_test,y_test_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mf83XyI0w9xs"
      },
      "source": [
        "Creating TensorFlow Datasets with our data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6tx3IteYw9xs"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 16\n",
        "\n",
        "train_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_train, y_train_encoded))\n",
        "    .repeat()\n",
        "    .shuffle(2048)\n",
        "    .batch(BATCH_SIZE)\n",
        "\n",
        ")\n",
        "\n",
        "valid_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_valid, y_valid_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        "    .cache()\n",
        "\n",
        ")\n",
        "\n",
        "test_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_test,y_test_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WE_iRtnIw9xt"
      },
      "source": [
        "Building the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J0lLpLLKw9xt"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.optimizers.experimental import Adam\n",
        "\n",
        "def build_model(transformer, max_len=512):\n",
        "    \"\"\"\n",
        "    function for training the BERT model\n",
        "    \"\"\"\n",
        "    input_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n",
        "    sequence_output = transformer(input_word_ids)[0]\n",
        "    cls_token = sequence_output[:, 0, :]\n",
        "    out = Dense(6, activation='softmax')(cls_token)\n",
        "\n",
        "    model = tf.keras.Model(inputs=input_word_ids, outputs=out)\n",
        "    model.compile(Adam(learning_rate=1e-5), loss='categorical_crossentropy', metrics=['accuracy', Precision(), Recall(), AUC()])\n",
        "\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "js20JCrxw9xt",
        "outputId": "1d5ec961-8474-4b6f-ecfc-e775f0c56a2e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFDistilBertModel: ['vocab_layer_norm.bias', 'vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_projector.bias']\n",
            "- This IS expected if you are initializing TFDistilBertModel from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing TFDistilBertModel from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "All the weights of TFDistilBertModel were initialized from the PyTorch model.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
          ]
        }
      ],
      "source": [
        "transformer_layer = (\n",
        "    transformers.TFDistilBertModel.from_pretrained('distilbert-base-multilingual-cased')\n",
        ")\n",
        "\n",
        "\n",
        "bert_model = build_model(transformer_layer, max_len=max_length)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sN31mI43w9xt",
        "outputId": "889b54fa-7f20-4d5f-c520-794efd9ef756"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_word_ids (InputLayer)  [(None, 100)]            0         \n",
            "                                                                 \n",
            " tf_distil_bert_model_1 (TFD  TFBaseModelOutput(last_h  134734080\n",
            " istilBertModel)             idden_state=(None, 100,             \n",
            "                             768),                               \n",
            "                              hidden_states=None, att            \n",
            "                             entions=None)                       \n",
            "                                                                 \n",
            " tf.__operators__.getitem_1   (None, 768)              0         \n",
            " (SlicingOpLambda)                                               \n",
            "                                                                 \n",
            " dense_13 (Dense)            (None, 6)                 4614      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 134,738,694\n",
            "Trainable params: 134,738,694\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "bert_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83hKNZqhw9xt"
      },
      "source": [
        "Freezing all other layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pYqfjqrHw9xt"
      },
      "outputs": [],
      "source": [
        "bert_model.layers[0].trainable = False\n",
        "bert_model.layers[1].trainable = False\n",
        "bert_model.layers[2].trainable = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2pkrGbU75Sn2"
      },
      "source": [
        "Now that we have frozen the layers, we can see that the trainable params (thankfully) have reduced significantly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1PuHL8T0w9xt",
        "outputId": "b7bbb387-afe3-4d02-9662-04b6ee71b52d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_word_ids (InputLayer)  [(None, 100)]            0         \n",
            "                                                                 \n",
            " tf_distil_bert_model_1 (TFD  TFBaseModelOutput(last_h  134734080\n",
            " istilBertModel)             idden_state=(None, 100,             \n",
            "                             768),                               \n",
            "                              hidden_states=None, att            \n",
            "                             entions=None)                       \n",
            "                                                                 \n",
            " tf.__operators__.getitem_1   (None, 768)              0         \n",
            " (SlicingOpLambda)                                               \n",
            "                                                                 \n",
            " dense_13 (Dense)            (None, 6)                 4614      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 134,738,694\n",
            "Trainable params: 4,614\n",
            "Non-trainable params: 134,734,080\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "bert_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFimjXsL51DI"
      },
      "source": [
        "Now we can train the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "91sXh61kw9xt",
        "outputId": "9fbee141-0aef-4995-8c3e-c37d14b4d857"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "800/800 [==============================] - 215s 236ms/step - loss: 0.9586 - accuracy: 0.6375 - precision_2: 0.8114 - recall_2: 0.4913 - auc_2: 0.9060 - val_loss: 0.3435 - val_accuracy: 0.8803 - val_precision_2: 0.8971 - val_recall_2: 0.8662 - val_auc_2: 0.9856\n",
            "Epoch 2/3\n",
            "800/800 [==============================] - 161s 201ms/step - loss: 0.2651 - accuracy: 0.9032 - precision_2: 0.9162 - recall_2: 0.8921 - auc_2: 0.9917 - val_loss: 0.2081 - val_accuracy: 0.9134 - val_precision_2: 0.9216 - val_recall_2: 0.9075 - val_auc_2: 0.9938\n",
            "Epoch 3/3\n",
            "800/800 [==============================] - 157s 196ms/step - loss: 0.1683 - accuracy: 0.9298 - precision_2: 0.9375 - recall_2: 0.9241 - auc_2: 0.9965 - val_loss: 0.2057 - val_accuracy: 0.9153 - val_precision_2: 0.9197 - val_recall_2: 0.9122 - val_auc_2: 0.9941\n"
          ]
        }
      ],
      "source": [
        "n_steps = X_train.shape[0] // BATCH_SIZE\n",
        "train_history = bert_model.fit(\n",
        "    train_dataset,\n",
        "    steps_per_epoch=n_steps,\n",
        "    validation_data=valid_dataset,\n",
        "    epochs=3\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i-XEgWQF59uz"
      },
      "source": [
        "Plotting losses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "4JM8P3Eu4Ct8",
        "outputId": "b10ced50-7a35-41ef-e1a3-9a88ef2a8adf"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABT70lEQVR4nO3de1xUdf4/8NeZgRmuw0XuOIl38AaCyqpZWhRpmW5lmKTmpm2mdqH2m7SblrbZxcw2UcufVpv3bVFLXS+Rl7zvChgmoiICXripMNwHZs7vD2BwFJBB4MwMr+fjcR7qmc+ZeR+POi8/53M+H0EURRFEREREEpFJXQARERF1bAwjREREJCmGESIiIpIUwwgRERFJimGEiIiIJMUwQkRERJJiGCEiIiJJMYwQERGRpGykLqA59Ho9rl69CmdnZwiCIHU5RERE1AyiKKK4uBh+fn6QyRrv/7CIMHL16lWo1WqpyyAiIqIWyM7ORufOnRt93SLCiLOzM4Cak1GpVBJXQ0RERM2h0WigVqsN3+ONsYgwUndrRqVSMYwQERFZmLsNseAAViIiIpIUwwgRERFJimGEiIiIJGURY0aIiKhjE0UR1dXV0Ol0UpdCt5DL5bCxsbnnaTcYRoiIyKxptVpcu3YNZWVlUpdCDXBwcICvry8UCkWL34NhhIiIzJZer0dGRgbkcjn8/PygUCg4+aWZEEURWq0W+fn5yMjIQM+ePZuc2KwpDCNERGS2tFot9Ho91Go1HBwcpC6HbmNvbw9bW1tkZmZCq9XCzs6uRe/DAaxERGT2Wvo/bmp7rXFteHWJiIhIUgwjREREJCmGESIiojYwcuRIvP7661KXYREYRoiIiEhSHTqMHL5QgKlrTqCiipPoEBERSaXDhpFyrQ6vbUzGgXP5WLD9jNTlEBFRM4miiDJttSSbKIotqvnmzZuYMmUK3Nzc4ODggNGjR+P8+fOG1zMzMzF27Fi4ubnB0dERffv2xc6dOw3HRkdHw9PTE/b29ujZsye++eabVvm9NBcddp4Re4UcS54NxtRvTmD98SwM6uKGp0I7S10WERHdRXmVDn3m7Zbks88siISDwvSvzhdeeAHnz5/Hjz/+CJVKhbfffhtjxozBmTNnYGtri1mzZkGr1eLgwYNwdHTEmTNn4OTkBAB49913cebMGfznP/+Bh4cHLly4gPLy8tY+NUl12DACAA/08sRrD/fE0p/P469bTqOfvwt6eTtLXRYREVmRuhBy+PBhDBs2DACwbt06qNVqbN26FRMmTEBWVhaefvpp9O/fHwDQrVs3w/FZWVkYOHAgBg0aBAAICAho93Noax06jADAnId64mTmTfx6vgAvrz2JH2ffDydlh/9tISIyW/a2cpxZECnZZ5sqNTUVNjY2CA8PN+zr1KkTevfujdTUVADAq6++ipkzZ2LPnj2IiIjA008/jQEDBgAAZs6ciaeffhqJiYl49NFHMX78eEOosRYddsxIHblMwNKoEPio7HAxvxSx8SktvidIRERtTxAEOChsJNnaal2c6dOn4+LFi5g8eTJSUlIwaNAgfPnllwCA0aNHIzMzE2+88QauXr2Khx9+GG+99Vab1CGVDh9GAKCTkxJx0QNhIxPw06mr+P5YptQlERGRlQgKCkJ1dTWOHz9u2Hf9+nWkpaWhT58+hn1qtRovv/wy4uPj8eabb2LVqlWG1zw9PTF16lSsXbsWS5cuxddff92u59DWGEZqhXVxR+yYIADAwu1nkJxdKG1BRERkFXr27Ilx48ZhxowZOHToEE6dOoXnn38e/v7+GDduHADg9ddfx+7du5GRkYHExETs27cPQUE130nz5s3Dtm3bcOHCBfz+++/Yvn274TVrwTByiz8ND8Dofj6o0omYtS4RN0u1UpdERERW4JtvvkFYWBieeOIJDB06FKIoYufOnbC1tQUA6HQ6zJo1C0FBQXjsscfQq1cvLF++HACgUCgQGxuLAQMG4IEHHoBcLsfGjRulPJ1WJ4gWMEBCo9HAxcUFRUVFUKlUbftZFVV48stDuHS9DKN6e2L11MGQydrmHiERETWtoqICGRkZ6Nq1a4uXp6e21dQ1au73N3tGbqOys8Xy6DAobWTYl5aPFQfSpS6JiIjIqjGMNKCPnwoLx/cDAHy2Jw1H0gskroiIiMh6MYw04tlBakwI6wy9CLy6IQm5mgqpSyIiIrJKDCNNWDCuHwJ9nFFQosWc9Umo1umlLomIiMjqMIw0wV4hx4rnw+CktMGJSzfw6Z40qUsiIiKyOgwjd9HVwxGfPlMzJe9XBy5i75lciSsiIiKyLgwjzTC6vy/+NLwrAODNzcnIul4mcUVERETWg2GkmeaODkTofa7QVFTjlfUnUVGlk7okIiIiq8Aw0kwKGxmWTQqFu6MCp69osGD7GalLIiIisgoMIybwc7XH0qgQCAKw/ngWtiRdlrokIiKyUgEBAVi6dGmz2gqCgK1bt7ZpPW2JYcRED/TyxKsP9QQAvBN/GudyiyWuiIiIyLIxjLTAqw/3xIieHiiv0uHltSdRUlktdUlEREQWq0VhJC4uDgEBAbCzs0N4eDhOnDjRaNuqqiosWLAA3bt3h52dHYKDg7Fr164WF2wO5DIBS6NC4KOyw8X8UsTGp8AC1hskIrIOoghoS6XZmvlv/ddffw0/Pz/o9caTZY4bNw5/+tOfkJ6ejnHjxsHb2xtOTk4YPHgwfv7551b7LUpJScFDDz0Ee3t7dOrUCS+99BJKSkoMr+/fvx9DhgyBo6MjXF1dMXz4cGRmZgIATp06hVGjRsHZ2RkqlQphYWH43//+12q1NcTG1AM2bdqEmJgYrFy5EuHh4Vi6dCkiIyORlpYGLy+vO9r/7W9/w9q1a7Fq1SoEBgZi9+7d+OMf/4gjR45g4MCBrXISUujkpERc9EBEfXUMP526isEBbpgyNEDqsoiIrF9VGfChnzSf/c5VQOF412YTJkzAnDlzsG/fPjz88MMAgBs3bmDXrl3YuXMnSkpKMGbMGPz973+HUqnEP//5T4wdOxZpaWm477777qnE0tJSREZGYujQofjvf/+LvLw8TJ8+HbNnz8a3336L6upqjB8/HjNmzMCGDRug1Wpx4sQJCELNCvXR0dEYOHAgVqxYAblcjuTkZNja2t5TTXdjcs/IkiVLMGPGDEybNg19+vTBypUr4eDggDVr1jTY/vvvv8c777yDMWPGoFu3bpg5cybGjBmDzz777J6Ll1pYF3fMHR0IAFi4/QySswulLYiIiMyCm5sbRo8ejfXr1xv2/fDDD/Dw8MCoUaMQHByMP//5z+jXrx969uyJhQsXonv37vjxxx/v+bPXr1+PiooK/POf/0S/fv3w0EMPYdmyZfj++++Rm5sLjUaDoqIiPPHEE+jevTuCgoIwdepUQwjKyspCREQEAgMD0bNnT0yYMAHBwcH3XFdTTOoZ0Wq1OHnyJGJjYw37ZDIZIiIicPTo0QaPqayshJ2dndE+e3t7HDp0qNHPqaysRGVlpeHXGo3GlDLb1Yv3d8X/Lt3Ert9zMGtdIrbPuR9ujgqpyyIisl62DjU9FFJ9djNFR0djxowZWL58OZRKJdatW4eJEydCJpOhpKQE7733Hnbs2IFr166huroa5eXlyMrKuucSU1NTERwcDEfH+h6c4cOHQ6/XIy0tDQ888ABeeOEFREZG4pFHHkFERASeffZZ+Pr6AgBiYmIwffp0fP/994iIiMCECRPQvXv3e66rKSb1jBQUFECn08Hb29tov7e3N3Jycho8JjIyEkuWLMH58+eh1+uxd+9exMfH49q1a41+zqJFi+Di4mLY1Gq1KWW2K0EQ8MmEAQjo5IArheWI2ZwMvZ7jR4iI2owg1NwqkWKrvZXRHGPHjoUoitixYweys7Px66+/Ijo6GgDw1ltvYcuWLfjwww/x66+/Ijk5Gf3794dWq22r3zUj33zzDY4ePYphw4Zh06ZN6NWrF44dOwYAeO+99/D777/j8ccfxy+//II+ffpgy5YtbVpPmz9N88UXX6Bnz54IDAyEQqHA7NmzMW3aNMhkjX90bGwsioqKDFt2dnZbl3lPVHa2WB4dBqWNDPvS8rHiQLrUJRERkcTs7Ozw1FNPYd26ddiwYQN69+6N0NBQAMDhw4fxwgsv4I9//CP69+8PHx8fXLp0qVU+NygoCKdOnUJpaalh3+HDhyGTydC7d2/DvoEDByI2NhZHjhxBv379jG4p9erVC2+88Qb27NmDp556Ct98802r1NYYk8KIh4cH5HI5cnONF4vLzc2Fj49Pg8d4enpi69atKC0tRWZmJs6ePQsnJyd069at0c9RKpVQqVRGm7nr46fCwnH9AACf7UnDkfQCiSsiIiKpRUdHY8eOHVizZo2hVwQAevbsifj4eCQnJ+PUqVOYNGnSHU/e3Mtn2tnZYerUqTh9+jT27duHOXPmYPLkyfD29kZGRgZiY2Nx9OhRZGZmYs+ePTh//jyCgoJQXl6O2bNnY//+/cjMzMThw4fx3//+F0FBQa1SW2NMCiMKhQJhYWFISEgw7NPr9UhISMDQoUObPNbOzg7+/v6orq7Gv//9b4wbN65lFZuxZwerMSGsM/Qi8OqGJORqKqQuiYiIJPTQQw/B3d0daWlpmDRpkmH/kiVL4ObmhmHDhmHs2LGIjIw09JrcKwcHB+zevRs3btzA4MGD8cwzz+Dhhx/GsmXLDK+fPXsWTz/9NHr16oWXXnoJs2bNwp///GfI5XJcv34dU6ZMQa9evfDss89i9OjReP/991ultsYIookTZGzatAlTp07FV199hSFDhmDp0qXYvHkzzp49C29vb0yZMgX+/v5YtGgRAOD48eO4cuUKQkJCcOXKFbz33nvIyMhAYmIiXF1dm/WZGo0GLi4uKCoqMvteknKtDn9cfhhnc4oxJMAd62eEw0bOueWIiFqioqICGRkZ6Nq16x0PQ5B5aOoaNff72+RvyaioKCxevBjz5s1DSEgIkpOTsWvXLsOg1qysLKPBqRUVFfjb3/6GPn364I9//CP8/f1x6NChZgcRS2OvkGPF82FwUtrgxKUb+HRPmtQlERERmTWTe0akYEk9I3X+k3INM9clAgBWTRmER/p43+UIIiK6HXtGgHXr1uHPf/5zg6916dIFv//+eztXZKw1ekZMnoGVmmd0f1/8aXhXrDmcgTc3J2P7nBG4r1Pzn08nIiICgCeffBLh4eENvtbWM6O2F4aRNjR3dCCSs28iMasQr6w/iR9eHgY7W7nUZRERkQVxdnaGs7Oz1GW0KY6sbEMKGxmWTQqFu6MCp69osGD7GalLIiKySBYwoqDDao1rwzDSxvxc7bE0KgSCAKw/noUtSZelLomIyGLU3YYoKyuTuBJqTN21uZdbRrxN0w4e6OWJVx/qiS8SzuOd+NPo6+eCXt7W3eVGRNQa5HI5XF1dkZeXB6BmjgzBhCnZqe2IooiysjLk5eXB1dUVcnnLhyEwjLSTVx/uicSsm/j1fAFeXnsSP86+H05K/vYTEd1N3QzfdYGEzIurq2ujs7A3Fx/tbUfXSyrx+D8OIUdTgbHBfvjHxBAmfCKiZtLpdKiqqpK6DLqFra1tkz0ifLTXDHVyUiIueiCivjqGn05dxZAAN0weGiB1WUREFkEul9/TrQAyXxzA2s7Curhj7uhAAMCC7WeQnF0obUFEREQSYxiRwIv3d8VjfX1QpRMxa10ibpZqpS6JiIhIMgwjEhAEAZ9MGICATg64UliOmM3J0OvNfugOERFRm2AYkYjKzhbLo8OgtJFhX1o+VhxIl7okIiIiSTCMSKiPnwoLx/UDAHy2Jw1H0gskroiIiKj9MYxI7NnBakwI6wy9CLy6IQm5mgqpSyIiImpXDCNmYMG4fgj0cUZBiRZzNiShWqeXuiQiIqJ2wzBiBuwVcqx4PgxOShucyLiBxXvOSV0SERFRu2EYMRNdPRzxyTMDAAArD6Rj75lciSsiIiJqHwwjZmRMf1/8aXhXAMCbm5ORdZ2rVBIRkfVjGDEzc0cHIvQ+V2gqqvHK+pOoqNJJXRIREVGbYhgxMwobGZZNCoWbgy1OX9Fg4fYzUpdERETUphhGzJCfqz2WThwIQQDWHc/ClqTLUpdERETUZhhGzNSDvTzx6kM9AQDvxJ/GudxiiSsiIiJqGwwjZuzVh3tiRE8PlFfpMHPtSZRWVktdEhERUatjGDFjcpmApVEh8FHZIT2/FHPjUyCKXFCPiIisC8OImevkpMSySQNhIxPw06mrWHssU+qSiIiIWhXDiAUYFOCOuaMDAQALtp9BcnahtAURERG1IoYRC/Hi/V3xWF8fVOlEzFqXiMIyrdQlERERtQqGEQshCAI+mTAAAZ0ccKWwHDGbT0Gv5/gRIiKyfAwjFkRlZ4u46FAobWT45WweVhxIl7okIiKie8YwYmH6+rlg4bh+AIDP9qThSHqBxBURERHdG4YRC/TsYDUmhHWGXgRe3ZCMPE2F1CURERG1GMOIhVowrh8CfZxRUFKJ2RuSUK3TS10SERFRi7QojMTFxSEgIAB2dnYIDw/HiRMnmmy/dOlS9O7dG/b29lCr1XjjjTdQUcH/zd8Le4Ucy6ND4aS0wYmMG1i855zUJREREbWIyWFk06ZNiImJwfz585GYmIjg4GBERkYiLy+vwfbr16/H3LlzMX/+fKSmpmL16tXYtGkT3nnnnXsuvqPr5umET54ZAABYeSAde8/kSlwRERGR6UwOI0uWLMGMGTMwbdo09OnTBytXroSDgwPWrFnTYPsjR45g+PDhmDRpEgICAvDoo4/iueeeu2tvCjXPmP6++NPwrgCANzcnI/tGmcQVERERmcakMKLVanHy5ElERETUv4FMhoiICBw9erTBY4YNG4aTJ08awsfFixexc+dOjBkzptHPqayshEajMdqocXNHB2Lgfa7QVFRj5rqTqKjSSV0SERFRs5kURgoKCqDT6eDt7W2039vbGzk5OQ0eM2nSJCxYsAD3338/bG1t0b17d4wcObLJ2zSLFi2Ci4uLYVOr1aaU2eEobGSImxQKNwdbnL6iwcLtZ6QuiYiIqNna/Gma/fv348MPP8Ty5cuRmJiI+Ph47NixAwsXLmz0mNjYWBQVFRm27Ozsti7T4vm52mPpxIEQBGDd8SxsSbosdUlERETNYmNKYw8PD8jlcuTmGg+UzM3NhY+PT4PHvPvuu5g8eTKmT58OAOjfvz9KS0vx0ksv4a9//StksjvzkFKphFKpNKU0AvBgL0+8+lBPfJFwHu/En0ZfPxf08naWuiwiIqImmdQzolAoEBYWhoSEBMM+vV6PhIQEDB06tMFjysrK7ggccrkcACCKXFultb36cE/c38MD5VU6zFx7EqWV1VKXRERE1CSTb9PExMRg1apV+O6775CamoqZM2eitLQU06ZNAwBMmTIFsbGxhvZjx47FihUrsHHjRmRkZGDv3r149913MXbsWEMoodYjlwn4YmIIfFR2SM8vxdz4FIY+IiIyaybdpgGAqKgo5OfnY968ecjJyUFISAh27dplGNSalZVl1BPyt7/9DYIg4G9/+xuuXLkCT09PjB07Fn//+99b7yzISCcnJZZNGoiJXx/DT6euYkiAGyYPDZC6LCIiogYJogX8t1mj0cDFxQVFRUVQqVRSl2Mx/t+vF/HBjlTYygX88PIwBKtdpS6JiIg6kOZ+f3NtGiv24v1d8VhfH1TpRLyyLhGFZVqpSyIiIroDw4gVEwQBn0wYgC6dHHClsBwxm09Brzf7jjAiIupgGEasnMrOFsujQ6G0keGXs3lYcSBd6pKIiIiMMIx0AH39XLBwXD8AwGd70nAkvUDiioiIiOoxjHQQzw5W45mwztCLwKsbkpGnqZC6JCIiIgAMIx3KwnH9EOjjjIKSSszekIRqnV7qkoiIiBhGOhJ7hRzLo0PhpLTBiYwbWLznnNQlERERMYx0NN08nfDJMwMAACsPpGPvmdy7HEFERNS2GEY6oDH9fTFteAAA4M3Nyci+USZtQURE1KExjHRQsaODMPA+V2gqqjFz3UlUVOmkLomIiDoohpEOSmEjQ9ykULg52OL0FQ0Wbj8jdUlERNRBMYx0YH6u9lg6cSAEAVh3PAtbk65IXRIREXVADCMd3IO9PDHnoZ4AgNj4FJzPLZa4IiIi6mgYRgivPdwT9/fwQHmVDi+vPYnSymqpSyIiog6EYYQglwn4YmIIfFR2SM8vxdz4FIgiF9QjIqL2wTBCAIBOTkosmzQQNjIBP526irXHMqUuiYiIOgiGETIYFOCOuaMDAQALt6fiVHahtAUREVGHwDBCRl68vysi+3pDq9PjlXWJKCzTSl0SERFZOYYRMiIIAj6dEIwunRxwpbAcMZtPQa/n+BEiImo7DCN0B5WdLZZHh0JpI8MvZ/Ow4kC61CUREZEVYxihBvX1c8GCcX0BAJ/tScPR9OsSV0RERNaKYYQa9ewgNZ4J6wy9CMzZkIQ8TYXUJRERkRViGKFGCYKAheP6IdDHGQUllZi9IQnVOr3UZRERkZVhGKEm2SvkWB4dCielDU5k3MDiPeekLomIiKwMwwjdVTdPJ3zyzAAAwMoD6fj5TK7EFRERkTVhGKFmGdPfF9OGBwAAYjYnI/tGmbQFERGR1WAYoWaLHR2Egfe5QlNRjZnrTqKiSid1SUREZAUYRqjZFDYyxE0KhZuDLU5f0WDh9jNSl0RERFaAYYRM4udqj6UTB0IQgHXHs7A16YrUJRERkYVjGCGTPdjLE3Me6gkAiI1PwfncYokrIiIiS8YwQi3y2sM9cX8PD5RX6fDy2pMorayWuiQiIrJQLQojcXFxCAgIgJ2dHcLDw3HixIlG244cORKCINyxPf744y0umqQnlwn4YmIIfFR2SM8vxdz4FIgiF9QjIiLTmRxGNm3ahJiYGMyfPx+JiYkIDg5GZGQk8vLyGmwfHx+Pa9euGbbTp09DLpdjwoQJ91w8SauTkxLLJg2EXCbgp1NXsfZYptQlERGRBTI5jCxZsgQzZszAtGnT0KdPH6xcuRIODg5Ys2ZNg+3d3d3h4+Nj2Pbu3QsHBweGESsxKMAdsaMDAQALt6fiVHahtAUREZHFMSmMaLVanDx5EhEREfVvIJMhIiICR48ebdZ7rF69GhMnToSjo2OjbSorK6HRaIw2Ml8v3t8VkX29odXp8cq6RBSWaaUuiYiILIhJYaSgoAA6nQ7e3t5G+729vZGTk3PX40+cOIHTp09j+vTpTbZbtGgRXFxcDJtarTalTGpngiDg0wnB6NLJAVcKyxGz+RT0eo4fISKi5mnXp2lWr16N/v37Y8iQIU22i42NRVFRkWHLzs5upwqppVR2tlgeHQqFjQy/nM3DigPpUpdEREQWwqQw4uHhAblcjtxc44XScnNz4ePj0+SxpaWl2LhxI1588cW7fo5SqYRKpTLayPz19XPBwnF9AQCf7UnD0fTrEldERESWwKQwolAoEBYWhoSEBMM+vV6PhIQEDB06tMlj//Wvf6GyshLPP/98yyoli/DsIDWeCesMvQjM2ZCEPE2F1CUREZGZM/k2TUxMDFatWoXvvvsOqampmDlzJkpLSzFt2jQAwJQpUxAbG3vHcatXr8b48ePRqVOne6+azJYgCFg4rh8CfZxRUFKJ2RuSUK3TS10WERGZMRtTD4iKikJ+fj7mzZuHnJwchISEYNeuXYZBrVlZWZDJjDNOWloaDh06hD179rRO1WTW7BVyLI8OxZPLDuNExg0s3nMOc2sf/yUiIrqdIFrAtJkajQYuLi4oKiri+BELsuO3a5i1PhEA8P+mDEJEH++7HEFERNakud/fXJuG2szjA3wxbXgAACBmczKyb5RJWxAREZklhhFqU7GjgzDwPldoKqoxc91JVFTppC6JiIjMDMMItSmFjQzLJoXCzcEWp69osHD7GalLIiIiM8MwQm3O39Uen0eFQBCAdcezsDXpitQlERGRGWEYoXYxsrcX5jzUEwAQG5+C87nFEldERETmgmGE2s1rD/fE/T08UF6lw8trT6K0slrqkoiIyAwwjFC7kcsELJ0YAm+VEun5pYiNT4EFPFlORERtjGGE2pWHkxJxk0Ihlwn48dRVrD2eJXVJREQkMYYRaneDAtwRWzsj68KfzuBUdqG0BRERkaQYRkgSL97fFZF9vaHV6fHKukQUlmmlLomIiCTCMEKSEAQBnzwTjC6dHHClsBwxm09Br+f4ESKijohhhCTjYm+L5dGhUNjI8MvZPKw8mC51SUREJAGGEZJUXz8XLBzXFwCweHcajqZfl7giIiJqbwwjJLlnB6nxdGhn6EVgzoYk5GkqpC6JiIjaEcMISU4QBHwwvh8CfZxRUFKJ2RuSUK3TS10WERG1E4YRMgv2CjmWR4fCSWmDExk38Nnec1KXRERE7YRhhMxGN08nfPz0AADAiv3p+PlMrsQVERFRe2AYIbPy+ABfvDAsAAAQszkZ2TfKpC2IiIjaHMMImZ13xgQhRO0KTUU1XlmXiIoqndQlERFRG2IYIbOjsJEhLjoUbg62SLlShA92nJG6JCIiakMMI2SW/F3t8XlUCAQBWHssC1uTrkhdEhERtRGGETJbI3t7Yc6oHgCA2PgUnM8tlrgiIiJqCwwjZNZei+iF4T06obxKh5nrElFaWS11SURE1MoYRsisyWUCvpg4EN4qJS7klSA2PgWiyAX1iIisCcMImT0PJyXiJoVCLhPw46mrWHs8S+qSiIioFTGMkEUYFOCOuY8FAgAW/nQGp7ILpS2IiIhaDcMIWYzpI7ri0T7e0Or0eGVdIgrLtFKXRERErYBhhCyGIAj4dEIwunRywJXCcry5+RT0eo4fISKydAwjZFFc7G2xPDoUChsZEs7mYeXBdKlLIiKie8QwQhanr58LFjzZFwCweHcajqZfl7giIiK6FwwjZJGiBqvxdGhn6EVgzoYk5GkqpC6JiIhaiGGELJIgCPhgfD8E+jijoKQSczYkoVqnl7osIiJqgRaFkbi4OAQEBMDOzg7h4eE4ceJEk+0LCwsxa9Ys+Pr6QqlUolevXti5c2eLCiaqY6+QY3l0KJyUNjiecQOf7T0ndUlERNQCJoeRTZs2ISYmBvPnz0diYiKCg4MRGRmJvLy8BttrtVo88sgjuHTpEn744QekpaVh1apV8Pf3v+fiibp5OuHjpwcAAFbsT8fPZ3IlroiIiEwliCbOrR0eHo7Bgwdj2bJlAAC9Xg+1Wo05c+Zg7ty5d7RfuXIlPv30U5w9exa2trYtKlKj0cDFxQVFRUVQqVQteg+ybu/9+Du+PXIJKjsb7Hh1BNTuDlKXRETU4TX3+9uknhGtVouTJ08iIiKi/g1kMkRERODo0aMNHvPjjz9i6NChmDVrFry9vdGvXz98+OGH0Ol0jX5OZWUlNBqN0UbUlHfGBCFE7QpNRTVeWZeIyurG/3wREZF5MSmMFBQUQKfTwdvb22i/t7c3cnJyGjzm4sWL+OGHH6DT6bBz5068++67+Oyzz/DBBx80+jmLFi2Ci4uLYVOr1aaUSR2QwkaGuOhQuDnYIuVKERZuPyN1SURE1Ext/jSNXq+Hl5cXvv76a4SFhSEqKgp//etfsXLlykaPiY2NRVFRkWHLzs5u6zLJCvi72uPzqBAIArD2WBa2Jl2RuiQiImoGk8KIh4cH5HI5cnONBwnm5ubCx8enwWN8fX3Rq1cvyOVyw76goCDk5ORAq214bRGlUgmVSmW0ETXHyN5emDOqBwAgNj4F53OLJa6IiIjuxqQwolAoEBYWhoSEBMM+vV6PhIQEDB06tMFjhg8fjgsXLkCvr58D4ty5c/D19YVCoWhh2USNey2iF4b36ITyKh1mrktEaWW11CUREVETTL5NExMTg1WrVuG7775DamoqZs6cidLSUkybNg0AMGXKFMTGxhraz5w5Ezdu3MBrr72Gc+fOYceOHfjwww8xa9as1jsLolvIZQK+mDgQ3iolLuSVIDY+BSY+NEZERO3IxtQDoqKikJ+fj3nz5iEnJwchISHYtWuXYVBrVlYWZLL6jKNWq7F792688cYbGDBgAPz9/fHaa6/h7bffbr2zILqNh5MSyyaFYuLXx/DjqasY3NUdk//QReqyiIioASbPMyIFzjNCLbXq4EX8fWcqFHIZfpg5FAM6u0pdEhFRh9Em84wQWZrpI7ri0T7e0Or0mLk2EYVlDQ+aJiIi6TCMkFUTBAGfTghGl04OuFJYjjc3n4Jeb/adgUREHQrDCFk9F3tbLI8OhcJGhoSzeVh5MF3qkoiI6BYMI9Qh9PVzwYIn+wIAFu9Ow9H06xJXREREdRhGqMOIGqzG06GdoReBORuSkKepkLokIiICwwh1IIIg4IPx/RDo44yCkkrM2ZCEap3+7gcSEVGbYhihDsVeIUdcdCgcFXIcz7iBz/aek7okIqIOj2GEOpzunk74+JkBAIAV+9ORkJp7lyOIiKgtMYxQh/TEAD+8MCwAAPDGpmRk3yiTtiAiog6MYYQ6rHfGBCFE7QpNRTVeWZeIymqd1CUREXVIDCPUYSlsZIiLDoWrgy1SrhRh4fYzUpdERNQhMYxQh+bvao+lUSEQBGDtsSxsS74idUlERB0Owwh1eCN7e2HOqB4AgNj4FJzPLZa4IiKijoVhhAjAaxG9MLxHJ5RpdZi5LhGlldVSl0RE1GEwjBABkMsEfDFxILxVSlzIK0FsfApEkQvqERG1B4YRoloeTkosmxQKuUzAj6euYu3xLKlLIiLqEBhGiG4xOMAdcx8LBAAs/OkMfrtcKG1BREQdAMMI0W2mj+iKR/t4Q6vTY+baRBSWaaUuiYjIqjGMEN1GEAR8OiEY97k74EphOd7cfAp6PcePEBG1FYYRoga42NtieXQoFDYyJJzNw8qD6VKXRERktRhGiBrRz98FC57sCwBYvDsNR9OvS1wREZF1YhghakLUYDWeDu0MvQjM2ZCEPE2F1CUREVkdhhGiJgiCgA/G90Nvb2cUlFRizoYkVOv0UpdFRGRVGEaI7sJeIcfy50PhqJDjeMYNfLb3nNQlERFZFYYRombo7umEj58ZAABYsT8dCam5EldERGQ9GEaImumJAX54YVgAAOCNTcnIvlEmbUFERFaCYYTIBO+MCUKI2hWaimq8si4RldU6qUsiIrJ4DCNEJlDYyBAXHQpXB1ukXCnCwu1npC6JiMjiMYwQmcjf1R5Lo0IgCMDaY1nYlnxF6pKIiCwawwhRC4zs7YU5o3oAAGLjU3A+t1jiioiILBfDCFELvRbRC8N7dEKZVoeZ6xJRWlktdUlERBapRWEkLi4OAQEBsLOzQ3h4OE6cONFo22+//RaCIBhtdnZ2LS6YyFzIZQK+mDgQ3iolLuSV4J0tKRBFLqhHRGQqk8PIpk2bEBMTg/nz5yMxMRHBwcGIjIxEXl5eo8eoVCpcu3bNsGVmZt5T0UTmwsNJiWWTQiGXCdiWfBXrjmdJXRIRkcUxOYwsWbIEM2bMwLRp09CnTx+sXLkSDg4OWLNmTaPHCIIAHx8fw+bt7X1PRROZk8EB7pj7WCAAYMFPZ/Db5UJpCyIisjAmhRGtVouTJ08iIiKi/g1kMkRERODo0aONHldSUoIuXbpArVZj3Lhx+P3335v8nMrKSmg0GqONyJxNH9EVj/bxhlanx8y1iSgs00pdEhGRxTApjBQUFECn093Rs+Ht7Y2cnJwGj+nduzfWrFmDbdu2Ye3atdDr9Rg2bBguX77c6OcsWrQILi4uhk2tVptSJlG7EwQBn04Ixn3uDrhSWI43N5+CXs/xI0REzdHmT9MMHToUU6ZMQUhICB588EHEx8fD09MTX331VaPHxMbGoqioyLBlZ2e3dZlE98zF3hbLo0OhsJEh4Wwevjp4UeqSiIgsgklhxMPDA3K5HLm5xouE5ebmwsfHp1nvYWtri4EDB+LChQuNtlEqlVCpVEYbkSXo5++CBU/2BQB8uvssjqZfl7giIiLzZ1IYUSgUCAsLQ0JCgmGfXq9HQkIChg4d2qz30Ol0SElJga+vr2mVElmIqMFqPBXqD70IzNmQhDxNhdQlERGZNZNv08TExGDVqlX47rvvkJqaipkzZ6K0tBTTpk0DAEyZMgWxsbGG9gsWLMCePXtw8eJFJCYm4vnnn0dmZiamT5/eemdBZEYEQcAH4/uht7czCkoqMWdDEqp1eqnLIiIyWzamHhAVFYX8/HzMmzcPOTk5CAkJwa5duwyDWrOysiCT1WecmzdvYsaMGcjJyYGbmxvCwsJw5MgR9OnTp/XOgsjMOChssPz5UDz55SEcz7iBJXvP4f9qH/8lIiJjgmgBU0ZqNBq4uLigqKiI40fIomz/7Spmr08CAKyeOggPB3GOHSLqOJr7/c21aYja0BMD/PDCsAAAwBubkpF9o0zagoiIzBDDCFEbe2dMEELUrtBUVGPW+kRUVuukLomIyKwwjBC1MYWNDHHRoXB1sMVvl4vwwfZUqUsiIjIrDCNE7cDf1R5Lo0IgCMD3xzKxLfmK1CUREZkNhhGidjKytxdmj+oBAIiNT8H53GKJKyIiMg8MI0Tt6PWIXhjWvRPKtDrMXJeI0spqqUsiIpIcwwhRO5LLBPzjuYHwVilxIa8Ef92SAgt4up6IqE0xjBC1Mw8nJZZNCoVcJmBr8lWsO54ldUlERJJiGCGSwOAAd7z9WG8AwIKfzuC3y4XSFkREJCGGESKJzBjRDY/28YZWp8cr6xJRVFYldUlERJJgGCGSiCAI+HRCMO5zd8Dlm+WI2ZwMvZ7jR4io42EYIZKQi70tlkeHQmEjQ8LZPHx18KLUJRERtTuGESKJ9fN3wftP9gUAfLr7LI6mX5e4IiKi9sUwQmQGJg5W46lQf+hFYM6GJOQVV0hdEhFRu2EYITIDgiDgg/H90NvbGQUllXh1QxKqdXqpyyIiahcMI0RmwkFhg+XPh8JRIcexizewZO85qUsiImoXDCNEZqS7pxM+fmYAAGD5/nQkpOZKXBERUdtjGCEyM08M8MMLwwIAADGbTyH7Rpm0BRERtTGGESIz9M6YIISoXVFUXoVZ6xNRWa2TuiQiojbDMEJkhhQ2MsRFh8LVwRa/XS7CB9tTpS6JiKjNMIwQmSl/V3t8HhUCAPj+WCa2JV+RtiAiojbCMEJkxkb19sKch3oAAGLjU3Ahr1jiioiIWh/DCJGZez2iF4Z174QyrQ4vr01EaWW11CUREbUqhhEiMyeXCfjHcwPhrVLiQl4J/rolBaLIBfWIyHowjBBZAA8nJZZNCoVcJmBr8lWsO54ldUlERK2GYYTIQgwOcMfbj/UGACz46Qx+u1wobUFERK2EYYTIgswY0Q2P9vGGVqfHK+sSUVRWJXVJRET3jGGEyIIIgoBPJwTjPncHXL5ZjpjNydDrOX6EiCwbwwiRhXGxt8Xy6FAobGRIOJuHrw5elLokIqJ7wjBCZIH6+bvg/Sf7AgA+3X0Wxy5el7giIqKWYxghslATB6vxVKg/9CIwZ0MS8oorpC6JiKhFGEaILJQgCPhgfD/09nZGfnElXt2QhGqdXuqyiIhM1qIwEhcXh4CAANjZ2SE8PBwnTpxo1nEbN26EIAgYP358Sz6WiG7joLDB8udD4aiQ49jFG1iy95zUJRERmczkMLJp0ybExMRg/vz5SExMRHBwMCIjI5GXl9fkcZcuXcJbb72FESNGtLhYIrpTd08nfPT0AADA8v3pSEjNlbgiIiLTmBxGlixZghkzZmDatGno06cPVq5cCQcHB6xZs6bRY3Q6HaKjo/H++++jW7du91QwEd1pbLAfXhgWAACI2XwK2TfKpC2IiMgEJoURrVaLkydPIiIiov4NZDJERETg6NGjjR63YMECeHl54cUXX2zW51RWVkKj0RhtRNS0d8YEIUTtiqLyKsxan4jKap3UJRERNYtJYaSgoAA6nQ7e3t5G+729vZGTk9PgMYcOHcLq1auxatWqZn/OokWL4OLiYtjUarUpZRJ1SAobGeKiQ+HqYIvfLhfhg+2pUpdERNQsbfo0TXFxMSZPnoxVq1bBw8Oj2cfFxsaiqKjIsGVnZ7dhlUTWw9/VHp9HhQAAvj+WiW3JV6QtiIioGWxMaezh4QG5XI7cXOMBcrm5ufDx8bmjfXp6Oi5duoSxY8ca9un1NY8e2tjYIC0tDd27d7/jOKVSCaVSaUppRFRrVG8vzHmoB7785QJi41PQ10+FHl7OUpdFRNQok3pGFAoFwsLCkJCQYNin1+uRkJCAoUOH3tE+MDAQKSkpSE5ONmxPPvkkRo0aheTkZN5+IWojr0f0wrDunVCm1eHltYkorayWuiQiokaZ1DMCADExMZg6dSoGDRqEIUOGYOnSpSgtLcW0adMAAFOmTIG/vz8WLVoEOzs79OvXz+h4V1dXALhjPxG1HrlMwBcTB+Lxf/yKC3kl+OuWFHweFQJBEKQujYjoDiaHkaioKOTn52PevHnIyclBSEgIdu3aZRjUmpWVBZmME7sSSc3TWYllk0Lx3Kpj2Jp8FYMC3PH8H7pIXRYR0R0EURTNfv1xjUYDFxcXFBUVQaVSSV0OkUX5+mA6Ptx5Fgq5DP+eOQz9O7tIXRIRdRDN/f5mFwaRlZsxohse7eMNrU6PmetOoqisSuqSiIiMMIwQWTlBEPDphGDc5+6AyzfLEbM5GXq92XeIElEHwjBC1AG42NtieXQoFDYyJJzNw1cHL0pdEhGRAcMIUQfRz98F7z/ZFwCweE8ajl28LnFFREQ1GEaIOpCJg9V4aqA/dHoRczYkIa+4QuqSiIgYRog6EkEQ8MEf+6GXtxPyiyvx6oYkVOv0UpdFRB0cwwhRB+OgsMGK58PgqJDj2MUbWLL3nNQlEVEHxzBC1AF193TCR08PAAAs35+OX87m3uUIIqK2wzBC1EGNDfbD1KE1M7K+sekUsm+USVwREXVUDCNEHdg7jwchWO2KovIqzFqfiMpqndQlEVEHxDBC1IEpbeSImzQQrg62+O1yET7Ynip1SUTUATGMEHVwnd0c8HlUCADg+2OZ2JZ8RdqCiKjDYRghIozq7YXZo3oAAGLjU3Ahr1jiioioI2EYISIAwBuP9MKw7p1QptXh5bWJKK2slrokIuogGEaICAAglwn4YuJAeDkrcSGvBH/dkgJR5IJ6RNT2OnYY0VwFtHyckaiOp7MSyyaFQi4TsDX5KtafyJK6JCLqADp2GNn5F+DjLsA3Y4B9i4BLh4HqSqmrIpLUkK7u+L/I3gCA9388g5TLRRJXRETWThAtoB9Wo9HAxcUFRUVFUKlUrffGK4YDuaeN99nYA+ohQNcHaja/gYDctvU+k8gCiKKIl74/ib1nctHZzR475oyAiwP/HhCRaZr7/d2xw4goAjcuAhkHa7ZLvwKl+cZtFE7AfUOBriNqwonPAEAmb70aiMxUUXkVnvjyV2TfKEdEkBe+njwIMpkgdVlEZEEYRlpCFIH8tNpgchC4dAgov2ncxs4F6DK8JpgEjAC8+gCyjn23i6zX6StFeGrFEWir9Zg7OhAvP9hd6pKIyIIwjLQGvb7mNs6lX4GMX4HMw0ClxriNQycg4P6aYNL1AcCjFyDwf49kPdYfz8I7W1IglwlYNz0cf+jWSeqSiMhCMIy0BV01kHOq9rbOr0DWUaDqtqdxnLzrg0nXEYBbV4YTsmiiKOLNzacQn3QFns5K7Hj1fng520ldFhFZAIaR9lCtBa4m1gSTSweBrOOA7rancVSd64NJwAjAVS1NrUT3oExbjfFxh3EutwR/6OaOtS+Gw0bO25NE1DSGESlUVQCX/1t7W+cgcPl/gL7KuI1b19pgUhtQnH2kqZXIROn5JXjyy0Mo1eowa1R3/CUyUOqSiMjMMYyYA20pkHWsfszJ1SRAvG2Jdo9e9bd1AkYAjrwfT+brp1NXMWdDEgBgzQuD8FCgt8QVEZE5YxgxRxWamnEmdY8S56QAuO2336tv/W2dLsMBe1cpKiVq1Pxtp/Hd0Uy42Nti+5z7oXZ3kLokIjJTDCOWoOwGkHmkfo6TvDPGrwuymnlNuo4Auj4I3PcHQOksTa1EtSqrdXj2q2M4lV2IAZ1d8K+Xh0Jpw7l3iOhODCOWqCS/JpTUjTm5fsH4dUEO+IfW39JRhwMK/q+U2t/lm2V44stDKCyrwpShXbBgXD+pSyIiM8QwYg00V2smXss4UDPmpDDT+HW5Aug8uHbMyYian9sopamVOpx9aXmY9s1/AQBfTAzBuBB/iSsiInPDMGKNbmbWD4bNOAgUXzV+3caupreE6+pQO1m8Ow3L9l2Ag0KOH2cPRw8v3kYkonoMI9bO1HV1AkYAvsFcV4dalU4vYvLq4ziSfh09vZywbfZwOChspC6LiMxEc7+/WzRrUVxcHAICAmBnZ4fw8HCcOHGi0bbx8fEYNGgQXF1d4ejoiJCQEHz//fct+Vi6lSAAnboDg6YBE74B3joPvHIcGP0pEDQWsHcDtCXAhb3A3nnAqlHAx12BDc8BR5cDOadrprsnugdymYAvJg6El7MS5/NK8E58Cizg/zdEZGZM7hnZtGkTpkyZgpUrVyI8PBxLly7Fv/71L6SlpcHLy+uO9vv378fNmzcRGBgIhUKB7du3480338SOHTsQGRnZrM9kz0gLNGddHXv3+l4TrqtD9+BExg08t+oYdHoRf/9jP0SHd5G6JCIyA212myY8PByDBw/GsmXLAAB6vR5qtRpz5szB3Llzm/UeoaGhePzxx7Fw4cJmtWcYaQWGdXVqx5tkHQOqSo3bGNbVqQ0nXFeHTPDVgXQs+s9ZKOQy/HvmMPTv7CJ1SUQksTYJI1qtFg4ODvjhhx8wfvx4w/6pU6eisLAQ27Zta/J4URTxyy+/4Mknn8TWrVvxyCOPNNiusrISlZX1a7xoNBqo1WqGkdbU7HV1bpkdluvqUBNEUcRL35/E3jO56Oxmjx1zRsDFgQOoiTqy5oYRk0aaFRQUQKfTwdvbeApob29vnD17ttHjioqK4O/vj8rKSsjlcixfvrzRIAIAixYtwvvvv29KaWQqG0XNJGr3/QF48C8Nr6ujuQyc2lCzAYBbQG0w4bo6dCdBELB4QjCe+PJXZN8ox5v/SsbXkwdBJmPvGhE1rV2GvTs7OyM5ORklJSVISEhATEwMunXrhpEjRzbYPjY2FjExMYZf1/WMUBuytavtBRkBjHqnZl2d7OO1T+vUrqtz81LNlvjPmmO4rg7dxsXeFiuiw/DUiiP4OTUPX/96ES8/2F3qsojIzJkURjw8PCCXy5Gbm2u0Pzc3Fz4+jf8vWSaToUePHgCAkJAQpKamYtGiRY2GEaVSCaWSk3dJSuEIdH+oZgOM19W59Ctw7Teg4FzN9r/VNW24rg4B6OfvgvfG9sU7W1Lw6e40hKhd8YduDKpE1DiTwohCoUBYWBgSEhIMY0b0ej0SEhIwe/bsZr+PXq83GhNCFsBOBfSKrNmAhtfVyfu9Zju+AoBQM69J1xE1t3W6DOW6Oh3Ic0PU+N+lG4hPuoI5G5Kw49X74eVsJ3VZRGSmTL5NExMTg6lTp2LQoEEYMmQIli5ditLSUkybNg0AMGXKFPj7+2PRokUAasZ/DBo0CN27d0dlZSV27tyJ77//HitWrGjdM6H25eAOBD1RswG3ravzK3D9PHAtuWY78mX9ujp1t3W4ro5VEwQBH/yxH05fLcK53BK8tiEZ3784BDbyFk1tRERWzuQwEhUVhfz8fMybNw85OTkICQnBrl27DINas7KyIJPV/4NTWlqKV155BZcvX4a9vT0CAwOxdu1aREVFtd5ZkPScPIF+T9VsAKC5Vj8YNuNgzbo6l/9bsx1awnV1OgAHhQ2WR4dh3LJDOHrxOj7/+Rz+EhkodVlEZIY4HTy1j8Ks+jlOLv0KaK4Yv25YV2cE0PVBrqtjRX48dRWvbkgCAKx5YRAeCvS+yxFEZC24Ng2Zr1vX1anrPWlwXZ0/1D+pw3V1LNr8bafx3dFMuNjbYvuc+6F25y06oo6AYYQshygC+Wm1weQAcOkQUH7TuI3SBQgYXn9bx6svIOP4A0tRWa3Ds18dw6nsQgR3dsHml4dCacNwSWTtGEbIcun1NU/l1M1x0ti6OgH31z5KzHV1LMHlm2V4/B+HUFRehSlDu2DBuH5Sl0REbYxhhKyHqevqBIwA3LsxnJihfWfzMO3b/wIAvpgYgnEh/hJXRERtiWGErJeuCriSWDvm5CCQfQKorjBuU7euTt2jxFxXx2ws3p2GZfsuwEEhx4+zh6OHF+efIbJWDCPUcRitq/Nrzc/1VcZtuK6O2dDpRUxefRxH0q+jp5cTts0eDgdFu6xMQUTtjGGEOi7Dujq1t3WuJgGizriNYV2d2t4TRw9pau2g8osr8fg/fkVecSXGh/jh86gQCLytRmR1GEaI6jS0rg5u+2Pv1bd2jpMHgC7DAHs3SUrtSI5fvI5J/+84dHoRTwb7YXCAG4J8Vejt4wxnO84xQ2QNGEaIGlO3rk7dHCd5Z25rIAC+A+pv63BdnTbz1YF0LPrP2Tv2q93tEeSjQpCvCkG+zgjyVUHt5gCZjL0nRJaEYYSouUrygcxD9Y8SXz9v/DrX1WkzoijiwLl8nMi4gdRrGqReK0aOpqLBto4KOQJ9VQj0ca4NKTU/d1RyvAmRuWIYIWqphtbVuZVcAfgPqp3jhOvqtLabpVqk5tQEk7PXNEjN0eBcbgm01foG23fp5HBHL0pnN3uOQSEyAwwjRK2F6+pIrlqnx8WCUkPvSc2PGuQVVzbY3llpg0BfZwTeElJ6+zjzqR2idsYwQtQW7lhX51egNM+4ja1jzTgTrqvT5q6XVOJsTk04OXNNg7PXinEhrwRa3Z29KIIABHRyrOk9qQ0pgb7O8HdlLwpRW2EYIWoPRuvq1AYUrqsjqSqdHun5JTh7rT6kpF4rRkFJw70oKjsbBPqqEHTLWJTePs6ws2WAJLpXDCNEUjCsq1MbTriujtnIL67E2RyN0a2eC3klqNbf+U+gTAACPBwR5KtCn9rbPIE+Kvi62LEXhcgEDCNE5uDWdXUu/QpkHuW6OmZEW63HhbySO0LK9VJtg+1dHWzrn+apvdXT09uJvShEjWAYITJHdevqXKp9Uofr6pgdURSRX1JpNFD27LViXMgvga6BXhS5TEDX2l6Uuqd5gnxU8FYp2YtCHR7DCJElqKoArvyvfo6TxtbVCah9Uofr6kimslqH87klNeEkpz6o3CyrarC9m4Ot0Xwodb0oShv2olDHwTBCZIm0ZUD2Ma6rYyFEUURecWXtINn6uVEuFpQ22ovS3dPREFKCagfOejqzF4WsE8MIkTWo0ABZx4CMA1xXx4JUVNX3oqTeMh6lqLzhXpROjgrDbZ66uVF6eDlBYcOnrsiyMYwQWaPym8Clw/VznOT9flsDrqtjrkRRxLWiitrBssWG3pRLBaVooBMFtnIB3T2djMaiBPqo4OnM2X7JcjCMEHUEpQX1wSTjINfVsUDlWh3O5RYbxqLUhZTiiuoG23s4KRHk64w+tZO2Bfmq0N3TCbZy9qKQ+WEYIeqINNeAS4fqb+vcvGT8usy2Zi0drqtj1kRRxJXCcsPEbXVr9Vy6XoqG/sVWyGXo4eWEwNqQUjdotpMTry1Ji2GEiOrX1ambIbapdXUCHqjpReG6OmarTFuNtJxiw2PHZ3NqHjsurmy4F8XLWXnLYNmaXpRuHo6wYS8KtROGESIyVreujmFF4ibW1am7rcN1dcyeKIq4fLPceBHBHA0yr5c12F5hI0MvbyejRQSDfFRwc1S0c+XUETCMEFHTRBEoOFcbTA7W3N4pv2HchuvqWKzSymqj+VDO5tQ8dlyq1TXY3kdlVz9Q1leFPr7OCOjEXhS6NwwjRGQarqtj9fR6Edk3y4xml03N0SD7RnmD7ZU2MvT2ca6fAr92dlkXB97Ko+ZhGCGie6PXAddO1a9G3Oi6OrXhhOvqWKziiqrasSganLlWjLM5GqTlFKOskV4UPxe7+tlla3tTAjo5Qi7jtSdjDCNE1LqM1tX5Fcg+3sC6Ov71waTrCMD1PmlqpXum14vIvFFWuzZPTUhJvabBlcKGe1HsbeXo5eOMoFt6UQJ9naGyYy9KR8YwQkRty7CuTu1tHa6r0yEUldf3otTc5ilGWo4GFVX6Btv7u9ojqHYMSmBtSOni7gAZe1E6BIYRImpft66rc+nXml6U29fV6dSzfo4TrqtjNXR6EZeulxpWOK4LKleLKhps76CQ145FqQkpQb4q9PZxhjN7UawOwwgRSatuXZ1LtU/rNLWujpMXAKF2vEntj4Lszn1Gr6GJ1xo6TnbbPjTxWlPHCYCAZtTX2GuNfU5zam/onE2pvRnnfNfam6+orOqWtXlqnuhJyylGZXXDvShqd3sE+aiM5kZRu7EXxZK1aRiJi4vDp59+ipycHAQHB+PLL7/EkCFDGmy7atUq/POf/8Tp06cBAGFhYfjwww8bbd8QhhEiK1B+E8g8Uj/HyR3r6pBluLcAJkKAXgSq9UBV3Y86EToRECFARN2PAkSx5hhbGxls5XLY2sihqN1ksgY+z1BDA3U2OzxCmnB7TwGxJeG2gd+jARMBlW+r/mlp7ve3jalvvGnTJsTExGDlypUIDw/H0qVLERkZibS0NHh5ed3Rfv/+/XjuuecwbNgw2NnZ4eOPP8ajjz6K33//Hf7+/qZ+PBFZKns3IPDxmg2oX1cn6xigLantNBFr5j+BCIj6W35++4+3v4a7HyfqG3gvNPFaU8eJNfU2+lpTtTeyr6XHNVX77T1RreLW3zvTjxYAyGs35e0vNNQYAHS1m9b0zyMTdLm/1cNIc5ncMxIeHo7Bgwdj2bJlAAC9Xg+1Wo05c+Zg7ty5dz1ep9PBzc0Ny5Ytw5QpU5r1mewZISJqoZaEGKPX0MKAeOvxLTlORLVOh6s3y3DpeikyC0qReb0EmQWlKCqrrLlTJtT1pQACRDjaynCfux26uDtA7WaP+9zs4O9mDzsbmWWcc5NB9G61w8Tjbq0BNT8+8FbN4/mtqE16RrRaLU6ePInY2FjDPplMhoiICBw9erRZ71FWVoaqqiq4u7s32qayshKVlZWGX2s0mkbbEhFRE1ow1sNc2AC47z7g9gfEr5dUGmaXPVM7Df6FvGJUVYrANdRstQRBj4BO9oZp7+seOfZ3tYdgob8v1sikMFJQUACdTgdvb2+j/d7e3jh79myz3uPtt9+Gn58fIiIiGm2zaNEivP/++6aURkREHUQnJyWG91BieI/6p7GqdHqk55cYnuipCykFJZXIKChFRkEpdqbkGNqr7Gxqp71XGWaY7e3jDDtbrsUkBZPHjNyLjz76CBs3bsT+/fthZ2fXaLvY2FjExMQYfq3RaKBWq9ujRCIiskC2chkCfVQI9FEBA+v35xdX4qzhiZ6a3pQLeSXQVFTjRMYNnMioX49JJgABHo6186LUPM0T6KOCr4sde1HamElhxMPDA3K5HLm5uUb7c3Nz4ePT9GRGixcvxkcffYSff/4ZAwYMaLKtUqmEUqlssg0REdHdeDor4ensiRE9PQ37tNV6XMir7UXJqQ8p10u1uJhfiov5pdjxW/29HlcH2/r1eWpv9fT0dmIvSisyKYwoFAqEhYUhISEB48ePB1AzgDUhIQGzZ89u9LhPPvkEf//737F7924MGjTongomIiK6FwobGfr4qdDHr35ApSiKyC+uROqts8te0yA9vxSFZVU4dvEGjl2s70WRywR083CsnVW2Pqh4q5TsRWkBk2/TxMTEYOrUqRg0aBCGDBmCpUuXorS0FNOmTQMATJkyBf7+/li0aBEA4OOPP8a8efOwfv16BAQEICen5p6dk5MTnJycWvFUiIiIWkYQBHip7OClssODvep7USqrdTifW2K4zVN3y+dmWRXO55XgfF4JfjpV/z5uDrb1a/PU9qb09HaC0oa9KE0xOYxERUUhPz8f8+bNQ05ODkJCQrBr1y7DoNasrKzayWhqrFixAlqtFs8884zR+8yfPx/vvffevVVPRETUhpQ2cvTzd0E/fxfDPlEUkaupvGV22ZrelIv5JbhZVoUj6ddxJP26ob2NTEB3TyfDCsc1vSjO8HRmL0odTgdPRETUCiqq6ntRztwyHqWovKrB9p0cFYZp7wNrx6L08HKCwkbWYHtLxLVpiIiIJCaKIq4VVRjW5jlTOxblUkEp9A18+9rKa3pRgm4di+KrgoeTZT7UwTBCRERkpsq1OpzLvWWwbO3A2eKK6gbbezgpEeTrXDMvSm1I6e7pBFu5efeiMIwQERFZEFEUcaWwvGag7DVN7ZiUYly6XoqGvqkVchl6eNWMRenjW7/asbujov2LbwTDCBERkRUo01YjLafYMFD2bE7NLLPFlQ33ong5K+sHytb2onTzcISNBL0oDCNERERWShRFXL5ZXjNQtjakpOZokHm9rMH2ChsZenk7GQbK1t3ycXVo214UhhEiIqIOpqSyrhdFYxg0e/aaBqVaXYPtfVR2ht6TCYPU6Orh2Kr1tMmqvURERGS+nJQ2COvihrAuboZ9er2I7JtlRnOipOZokH2jHDmaCuRoKrAvLR8PBXq1ehhpLoYRIiIiKyaTCejSyRFdOjnisX6+hv3FFVWGXpQz14rR28dZshoZRoiIiDogZztbDApwx6AAd6lLgXk/oExERERWj2GEiIiIJMUwQkRERJJiGCEiIiJJMYwQERGRpBhGiIiISFIMI0RERCQphhEiIiKSFMMIERERSYphhIiIiCTFMEJERESSYhghIiIiSTGMEBERkaQsYtVeURQBABqNRuJKiIiIqLnqvrfrvscbYxFhpLi4GACgVqslroSIiIhMVVxcDBcXl0ZfF8S7xRUzoNfrcfXqVTg7O0MQhFZ7X41GA7VajezsbKhUqlZ7X3Ni7efI87N81n6OPD/LZ+3n2JbnJ4oiiouL4efnB5ms8ZEhFtEzIpPJ0Llz5zZ7f5VKZZV/wG5l7efI87N81n6OPD/LZ+3n2Fbn11SPSB0OYCUiIiJJMYwQERGRpDp0GFEqlZg/fz6USqXUpbQZaz9Hnp/ls/Zz5PlZPms/R3M4P4sYwEpERETWq0P3jBAREZH0GEaIiIhIUgwjREREJCmGESIiIpKU1YWRuLg4BAQEwM7ODuHh4Thx4kST7f/1r38hMDAQdnZ26N+/P3bu3Gn0uiiKmDdvHnx9fWFvb4+IiAicP3++LU+hSaac36pVqzBixAi4ubnBzc0NERERd7R/4YUXIAiC0fbYY4+19Wk0yZRz/Pbbb++o387OzqiNJV/DkSNH3nF+giDg8ccfN7Qxp2t48OBBjB07Fn5+fhAEAVu3br3rMfv370doaCiUSiV69OiBb7/99o42pv69biumnl98fDweeeQReHp6QqVSYejQodi9e7dRm/fee++O6xcYGNiGZ9E0U89x//79Df4ZzcnJMWpnqdewob9fgiCgb9++hjbmdA0XLVqEwYMHw9nZGV5eXhg/fjzS0tLuepzU34VWFUY2bdqEmJgYzJ8/H4mJiQgODkZkZCTy8vIabH/kyBE899xzePHFF5GUlITx48dj/PjxOH36tKHNJ598gn/84x9YuXIljh8/DkdHR0RGRqKioqK9TsvA1PPbv38/nnvuOezbtw9Hjx6FWq3Go48+iitXrhi1e+yxx3Dt2jXDtmHDhvY4nQaZeo5AzayBt9afmZlp9LolX8P4+Hijczt9+jTkcjkmTJhg1M5crmFpaSmCg4MRFxfXrPYZGRl4/PHHMWrUKCQnJ+P111/H9OnTjb6wW/Jnoq2Yen4HDx7EI488gp07d+LkyZMYNWoUxo4di6SkJKN2ffv2Nbp+hw4daovym8XUc6yTlpZmdA5eXl6G1yz5Gn7xxRdG55WdnQ13d/c7/g6ayzU8cOAAZs2ahWPHjmHv3r2oqqrCo48+itLS0kaPMYvvQtGKDBkyRJw1a5bh1zqdTvTz8xMXLVrUYPtnn31WfPzxx432hYeHi3/+859FURRFvV4v+vj4iJ9++qnh9cLCQlGpVIobNmxogzNomqnnd7vq6mrR2dlZ/O677wz7pk6dKo4bN661S20xU8/xm2++EV1cXBp9P2u7hp9//rno7OwslpSUGPaZ2zWsA0DcsmVLk23+7//+T+zbt6/RvqioKDEyMtLw63v9PWsrzTm/hvTp00d8//33Db+eP3++GBwc3HqFtaLmnOO+fftEAOLNmzcbbWNN13DLli2iIAjipUuXDPvM+Rrm5eWJAMQDBw402sYcvgutpmdEq9Xi5MmTiIiIMOyTyWSIiIjA0aNHGzzm6NGjRu0BIDIy0tA+IyMDOTk5Rm1cXFwQHh7e6Hu2lZac3+3KyspQVVUFd3d3o/379++Hl5cXevfujZkzZ+L69eutWntztfQcS0pK0KVLF6jVaowbNw6///674TVru4arV6/GxIkT4ejoaLTfXK6hqe72d7A1fs/MiV6vR3Fx8R1/B8+fPw8/Pz9069YN0dHRyMrKkqjClgsJCYGvry8eeeQRHD582LDf2q7h6tWrERERgS5duhjtN9drWFRUBAB3/Jm7lTl8F1pNGCkoKIBOp4O3t7fRfm9v7zvuXdbJyclpsn3dj6a8Z1tpyfnd7u2334afn5/RH6jHHnsM//znP5GQkICPP/4YBw4cwOjRo6HT6Vq1/uZoyTn27t0ba9aswbZt27B27Vro9XoMGzYMly9fBmBd1/DEiRM4ffo0pk+fbrTfnK6hqRr7O6jRaFBeXt4qf+7NyeLFi1FSUoJnn33WsC88PBzffvstdu3ahRUrViAjIwMjRoxAcXGxhJU2n6+vL1auXIl///vf+Pe//w21Wo2RI0ciMTERQOv822Uurl69iv/85z93/B0012uo1+vx+uuvY/jw4ejXr1+j7czhu9AiVu2le/fRRx9h48aN2L9/v9EAz4kTJxp+3r9/fwwYMADdu3fH/v378fDDD0tRqkmGDh2KoUOHGn49bNgwBAUF4auvvsLChQslrKz1rV69Gv3798eQIUOM9lv6Newo1q9fj/fffx/btm0zGk8xevRow88HDBiA8PBwdOnSBZs3b8aLL74oRakm6d27N3r37m349bBhw5Ceno7PP/8c33//vYSVtb7vvvsOrq6uGD9+vNF+c72Gs2bNwunTpyUdg9RcVtMz4uHhAblcjtzcXKP9ubm58PHxafAYHx+fJtvX/WjKe7aVlpxfncWLF+Ojjz7Cnj17MGDAgCbbduvWDR4eHrhw4cI912yqeznHOra2thg4cKChfmu5hqWlpdi4cWOz/mGT8hqaqrG/gyqVCvb29q3yZ8IcbNy4EdOnT8fmzZvv6A6/naurK3r16mUR168xQ4YMMdRvLddQFEWsWbMGkydPhkKhaLKtOVzD2bNnY/v27di3bx86d+7cZFtz+C60mjCiUCgQFhaGhIQEwz69Xo+EhASj/znfaujQoUbtAWDv3r2G9l27doWPj49RG41Gg+PHjzf6nm2lJecH1IyAXrhwIXbt2oVBgwbd9XMuX76M69evw9fXt1XqNkVLz/FWOp0OKSkphvqt4RoCNY/dVVZW4vnnn7/r50h5DU11t7+DrfFnQmobNmzAtGnTsGHDBqNHshtTUlKC9PR0i7h+jUlOTjbUbw3XEKh5SuXChQvN+g+BlNdQFEXMnj0bW7ZswS+//IKuXbve9Riz+C5slWGwZmLjxo2iUqkUv/32W/HMmTPiSy+9JLq6uoo5OTmiKIri5MmTxblz5xraHz58WLSxsREXL14spqamivPnzxdtbW3FlJQUQ5uPPvpIdHV1Fbdt2yb+9ttv4rhx48SuXbuK5eXlZn9+H330kahQKMQffvhBvHbtmmErLi4WRVEUi4uLxbfeeks8evSomJGRIf78889iaGio2LNnT7GioqLdz68l5/j++++Lu3fvFtPT08WTJ0+KEydOFO3s7MTff//d0MaSr2Gd+++/X4yKirpjv7ldw+LiYjEpKUlMSkoSAYhLliwRk5KSxMzMTFEURXHu3Lni5MmTDe0vXrwoOjg4iH/5y1/E1NRUMS4uTpTL5eKuXbsMbe72e2bO57du3TrRxsZGjIuLM/o7WFhYaGjz5ptvivv37xczMjLEw4cPixEREaKHh4eYl5fX7ucniqaf4+effy5u3bpVPH/+vJiSkiK+9tprokwmE3/++WdDG0u+hnWef/55MTw8vMH3NKdrOHPmTNHFxUXcv3+/0Z+5srIyQxtz/C60qjAiiqL45Zdfivfdd5+oUCjEIUOGiMeOHTO89uCDD4pTp041ar9582axV69eokKhEPv27Svu2LHD6HW9Xi++++67ore3t6hUKsWHH35YTEtLa49TaZAp59elSxcRwB3b/PnzRVEUxbKyMvHRRx8VPT09RVtbW7FLly7ijBkzJPkH4lamnOPrr79uaOvt7S2OGTNGTExMNHo/S76GoiiKZ8+eFQGIe/bsueO9zO0a1j3meftWd05Tp04VH3zwwTuOCQkJERUKhditWzfxm2++ueN9m/o9a0+mnt+DDz7YZHtRrHmU2dfXV1QoFKK/v78YFRUlXrhwoX1P7BamnuPHH38sdu/eXbSzsxPd3d3FkSNHir/88ssd72up11AUax5jtbe3F7/++usG39OcrmFD5wbA6O+VOX4XCrXFExEREUnCasaMEBERkWViGCEiIiJJMYwQERGRpBhGiIiISFIMI0RERCQphhEiIiKSFMMIERERSYphhIiIiCTFMEJERESSYhghIiIiSTGMEBERkaQYRoiIiEhS/x+7rxEAmiGWbgAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "loss, val_loss = train_history.history['loss'], train_history.history['val_loss']\n",
        "plt.plot(loss, label='loss')\n",
        "plt.plot(val_loss, label='val_loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q19__xs36CGl"
      },
      "source": [
        "Validation on the Test Set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Al3uVdt934DQ",
        "outputId": "56e309ef-35ce-4f8f-ad68-46b46185c250"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "63/63 [==============================] - 7s 110ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.89      0.94      0.91       275\n",
            "           1       0.85      0.92      0.88       224\n",
            "           2       0.92      0.95      0.94       695\n",
            "           3       0.87      0.71      0.78       159\n",
            "           4       0.96      0.94      0.95       581\n",
            "           5       0.81      0.70      0.75        66\n",
            "\n",
            "    accuracy                           0.92      2000\n",
            "   macro avg       0.89      0.86      0.87      2000\n",
            "weighted avg       0.92      0.92      0.91      2000\n",
            "\n",
            "[[258   4   3   1   9   0]\n",
            " [ 10 205   0   0   6   3]\n",
            " [  3   5 662  14   4   7]\n",
            " [  1   0  45 113   0   0]\n",
            " [ 17   7   7   2 547   1]\n",
            " [  0  19   0   0   1  46]]\n",
            "0.9155\n"
          ]
        }
      ],
      "source": [
        "predictions = bert_model.predict(X_test)\n",
        "prediction_labels = predictions.argmax(axis=1)\n",
        "print(classification_report(y_test, prediction_labels))\n",
        "print(confusion_matrix(y_test, prediction_labels))\n",
        "print(accuracy_score(y_test, prediction_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbuQC0RMw9xt"
      },
      "source": [
        "# PART 4: Comparison with Real Life Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uvYhN5h1w9xt"
      },
      "source": [
        "## First Model: Fully connected NNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZpc7S0uw9xu"
      },
      "source": [
        "Load the real case data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5kRPTdCSw9xu"
      },
      "outputs": [],
      "source": [
        "df_r_train = pd.read_csv('./train.csv')\n",
        "df_r_test = pd.read_csv('./test.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "qy_v5aDd71cu",
        "outputId": "1a2e73a1-2468-4c7d-a975-3e68999ab0dd"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-05d723aa-de3f-4db1-9d18-5b01596ada6f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>article_title</th>\n",
              "      <th>is_ecology</th>\n",
              "      <th>processed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1079</td>\n",
              "      <td>Stations essence à Rochefort-en-Terre : les me...</td>\n",
              "      <td>0</td>\n",
              "      <td>station essence à rochefort - en - terre : les...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1515</td>\n",
              "      <td>À Brest, le chantier de La Boussole se fait at...</td>\n",
              "      <td>0</td>\n",
              "      <td>à brest , le chantier de la boussole se fait a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>51</td>\n",
              "      <td>Météorite en Normandie : à la recherche de pré...</td>\n",
              "      <td>0</td>\n",
              "      <td>météorite en normandie : à la recherche de pré...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1419</td>\n",
              "      <td>Résultats trimestriels mitigés pour les géants...</td>\n",
              "      <td>0</td>\n",
              "      <td>résultats trimestriels mitigés pour les géants...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2181</td>\n",
              "      <td>Opinion | Sobriété énergétique : les solutions...</td>\n",
              "      <td>1</td>\n",
              "      <td>opinion | sobriété énergétique : les solution ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-05d723aa-de3f-4db1-9d18-5b01596ada6f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-2299ae16-62c0-4d88-93fa-a7a3601846ac\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2299ae16-62c0-4d88-93fa-a7a3601846ac')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-2299ae16-62c0-4d88-93fa-a7a3601846ac button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-05d723aa-de3f-4db1-9d18-5b01596ada6f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-05d723aa-de3f-4db1-9d18-5b01596ada6f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   Unnamed: 0                                      article_title  is_ecology  \\\n",
              "0        1079  Stations essence à Rochefort-en-Terre : les me...           0   \n",
              "1        1515  À Brest, le chantier de La Boussole se fait at...           0   \n",
              "2          51  Météorite en Normandie : à la recherche de pré...           0   \n",
              "3        1419  Résultats trimestriels mitigés pour les géants...           0   \n",
              "4        2181  Opinion | Sobriété énergétique : les solutions...           1   \n",
              "\n",
              "                                           processed  \n",
              "0  station essence à rochefort - en - terre : les...  \n",
              "1  à brest , le chantier de la boussole se fait a...  \n",
              "2  météorite en normandie : à la recherche de pré...  \n",
              "3  résultats trimestriels mitigés pour les géants...  \n",
              "4  opinion | sobriété énergétique : les solution ...  "
            ]
          },
          "execution_count": 94,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_r_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8Ib6XRCw9xu"
      },
      "source": [
        "Tokenize the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YwvjBNh3w9xu"
      },
      "outputs": [],
      "source": [
        "df_r_train['processed'] = df_r_train['article_title'].apply(preprocess_text)\n",
        "df_r_test['processed'] = df_r_test['article_title'].apply(preprocess_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vSrNOHwTw9xu"
      },
      "source": [
        "TFIDF transformation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5-FKkU2Ww9xu",
        "outputId": "2654c6a8-ce4b-49a6-a117-03dd31d32566"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "\n",
        "nltk.download('stopwords')\n",
        "final_stopwords_list = stopwords.words('english') + stopwords.words('french')\n",
        "\n",
        "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, stop_words=final_stopwords_list)\n",
        "\n",
        "dtm_r_train = tfidf.fit_transform(df_r_train['processed'])\n",
        "dtm_r_test = tfidf.transform(df_r_test['processed'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UD36B78Aw9xu"
      },
      "source": [
        "Setup the training and test data set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RWbk3Yb1w9xu"
      },
      "outputs": [],
      "source": [
        "X_r_train = dtm_r_train.toarray()\n",
        "y_r_train = df_r_train['is_ecology']\n",
        "\n",
        "X_r_test = dtm_r_test.toarray()\n",
        "y_r_test = df_r_test['is_ecology']\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8z79V-mw9xu"
      },
      "source": [
        "Split the train and validation data set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yO1gFCdHAl1T",
        "outputId": "3887524a-0806-4bb8-cdb1-a6c329ebbf08"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(1772, 2446)"
            ]
          },
          "execution_count": 117,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_r_train.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4uonAINDw9xu"
      },
      "outputs": [],
      "source": [
        "\n",
        "X_r_train, X_r_val, y_r_train_encoded, y_r_val_encoded = train_test_split(X_r_train, y_r_train, test_size=0.2, random_state=42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h4LLCazhw9xv"
      },
      "source": [
        "Build the model with same architecture, but change the last output layer as we have two categories in this case"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zs_6_NoGw9xv",
        "outputId": "8aa94e1b-7a3e-43b4-bf2a-19e3142f7441"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_5\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense_26 (Dense)            (None, 2048)              5011456   \n",
            "                                                                 \n",
            " dense_27 (Dense)            (None, 1024)              2098176   \n",
            "                                                                 \n",
            " dense_28 (Dense)            (None, 512)               524800    \n",
            "                                                                 \n",
            " dense_29 (Dense)            (None, 256)               131328    \n",
            "                                                                 \n",
            " dense_30 (Dense)            (None, 64)                16448     \n",
            "                                                                 \n",
            " dense_31 (Dense)            (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 7,782,273\n",
            "Trainable params: 7,782,273\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(2048, activation='selu', kernel_initializer='lecun_normal', input_shape=(X_r_train.shape[1],), kernel_regularizer=tf.keras.regularizers.l2(0.01)))\n",
        "model.add(Dense(1024, activation='selu', kernel_initializer='lecun_normal', kernel_regularizer=tf.keras.regularizers.l2(0.01)))\n",
        "model.add(Dense(512, activation='selu', kernel_initializer='lecun_normal', kernel_regularizer=tf.keras.regularizers.l2(0.1)))\n",
        "model.add(Dense(256, activation='selu', kernel_initializer='lecun_normal', ))\n",
        "# model.add(Dense(128, activation='selu', kernel_initializer='lecun_normal', ))\n",
        "model.add(Dense(64, activation='selu', kernel_initializer='lecun_normal',))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer='Adam', loss=tf.losses.binary_crossentropy, metrics=['accuracy'])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WLcofVxAw9xv"
      },
      "source": [
        "Set up early stopping and dynamic learning rate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FLMxWrL2w9xv"
      },
      "outputs": [],
      "source": [
        "def step_decay(epoch):\n",
        "    initial_lrate = 0.001\n",
        "    drop = 0.5\n",
        "    epochs_drop = 10.0\n",
        "    lrate = initial_lrate * math.pow(drop, math.floor((1 + epoch) / epochs_drop))\n",
        "    return lrate\n",
        "\n",
        "lr_scheduler = LearningRateScheduler(step_decay)\n",
        "\n",
        "early_stopping = EarlyStopping(monitor='val_loss', restore_best_weights=True, patience=3) #"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9KZKL35zw9xv"
      },
      "source": [
        "Train the model on new dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e6K639XJw9xv",
        "outputId": "2d95bfe9-4506-4715-a15a-294a7b05c7e5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/15\n",
            "111/111 [==============================] - 1s 12ms/step - loss: 0.5566 - accuracy: 0.9707 - val_loss: 0.2197 - val_accuracy: 0.9887 - lr: 0.0010\n",
            "Epoch 2/15\n",
            "111/111 [==============================] - 1s 10ms/step - loss: 0.2163 - accuracy: 0.9797 - val_loss: 0.2129 - val_accuracy: 0.9887 - lr: 0.0010\n",
            "Epoch 3/15\n",
            "111/111 [==============================] - 1s 10ms/step - loss: 0.2057 - accuracy: 0.9853 - val_loss: 0.1972 - val_accuracy: 0.9842 - lr: 0.0010\n",
            "Epoch 4/15\n",
            "111/111 [==============================] - 1s 9ms/step - loss: 0.1654 - accuracy: 0.9876 - val_loss: 0.1961 - val_accuracy: 0.9752 - lr: 0.0010\n",
            "Epoch 5/15\n",
            "111/111 [==============================] - 1s 9ms/step - loss: 0.1319 - accuracy: 0.9915 - val_loss: 0.1370 - val_accuracy: 0.9887 - lr: 0.0010\n",
            "Epoch 6/15\n",
            "111/111 [==============================] - 1s 10ms/step - loss: 0.0925 - accuracy: 0.9949 - val_loss: 0.1338 - val_accuracy: 0.9819 - lr: 0.0010\n",
            "Epoch 7/15\n",
            "111/111 [==============================] - 1s 11ms/step - loss: 0.0708 - accuracy: 0.9972 - val_loss: 0.2138 - val_accuracy: 0.9887 - lr: 0.0010\n",
            "Epoch 8/15\n",
            "111/111 [==============================] - 1s 13ms/step - loss: 0.6251 - accuracy: 0.9769 - val_loss: 0.3292 - val_accuracy: 0.9887 - lr: 0.0010\n",
            "Epoch 9/15\n",
            "111/111 [==============================] - 1s 9ms/step - loss: 0.2431 - accuracy: 0.9865 - val_loss: 0.1905 - val_accuracy: 0.9797 - lr: 0.0010\n"
          ]
        }
      ],
      "source": [
        "history_NN = model.fit(X_r_train, y_r_train_encoded, validation_data=(X_r_val, y_r_val_encoded), epochs=15, batch_size=16, callbacks=[early_stopping, lr_scheduler])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eBXrutCLw9xv"
      },
      "source": [
        "Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_0WqBGEgw9xv",
        "outputId": "b107b229-1ddf-47c6-b4b2-40b6a88e961c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10/10 [==============================] - 0s 9ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "     class 0       0.84      0.98      0.90       241\n",
            "     class 1       0.62      0.15      0.24        53\n",
            "\n",
            "    accuracy                           0.83       294\n",
            "   macro avg       0.73      0.57      0.57       294\n",
            "weighted avg       0.80      0.83      0.78       294\n",
            "\n",
            "[[236   5]\n",
            " [ 45   8]]\n",
            "0.8299319727891157\n"
          ]
        }
      ],
      "source": [
        "predictions_r = model.predict(X_r_test)\n",
        "target_names = ['class 0', 'class 1']\n",
        "\n",
        "predictions_r = (predictions_r > 0.5)*1\n",
        "print(classification_report(y_r_test, predictions_r,target_names=target_names))\n",
        "print(confusion_matrix(y_r_test, predictions_r))\n",
        "print(accuracy_score(y_r_test, predictions_r))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQDoZyzL7hzf"
      },
      "source": [
        "As you can see, we are having some trouble in recalling class 1, probably due to a very unbalanced dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKi08vKl-ITe"
      },
      "source": [
        "## LSTM Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8VJNBx0k-Jz-"
      },
      "outputs": [],
      "source": [
        "#Adapting tokenization to French Dataset\n",
        "\n",
        "tokenizer_lstm = Tokenizer() #(num_words = vocab_size, oov_token=oov_tok)\n",
        "tokenizer_lstm.fit_on_texts(X_r_train)\n",
        "word_index = tokenizer_lstm.word_index\n",
        "sequences = tokenizer_lstm.texts_to_sequences(df_r_train['article_title'])\n",
        "padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)\n",
        "\n",
        "testing_sequences = tokenizer_lstm.texts_to_sequences(df_r_test['article_title'])\n",
        "testing_padded = pad_sequences(testing_sequences,maxlen=max_length)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJfAXjlz_K7J"
      },
      "source": [
        "Building a new model for french dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Exft9CzR-vCD",
        "outputId": "e4cc191b-71dd-4f90-d1f2-f8c57b8ec6f4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_6\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_1 (Embedding)     (None, 100, 100)          765500    \n",
            "                                                                 \n",
            " bidirectional_1 (Bidirectio  (None, 128)              84480     \n",
            " nal)                                                            \n",
            "                                                                 \n",
            " dropout_40 (Dropout)        (None, 128)               0         \n",
            "                                                                 \n",
            " dense_32 (Dense)            (None, 32)                4128      \n",
            "                                                                 \n",
            " dropout_41 (Dropout)        (None, 32)                0         \n",
            "                                                                 \n",
            " dense_33 (Dense)            (None, 1)                 33        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 854,141\n",
            "Trainable params: 854,141\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "vocab_size = len(tokenizer_lstm.word_index) + 1\n",
        "embedding_dim = max_length\n",
        "\n",
        "\n",
        "model = tf.keras.Sequential([\n",
        "    Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    Bidirectional(LSTM(64)), #32\n",
        "    Dropout(0.4),\n",
        "    Dense(32, activation='leaky_relu', kernel_regularizer='l1_l2'),\n",
        "    Dropout(0.4),\n",
        "    Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "\n",
        "loss_function = 'binary_crossentropy'\n",
        "optimizer = 'adam'\n",
        "\n",
        "model.compile(loss=loss_function, optimizer=optimizer, metrics=['accuracy', Precision(), Recall(), AUC()])\n",
        "\n",
        "print(model.summary())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LFdUHOc8_9wx",
        "outputId": "33a9dbac-35c6-4a8c-a516-32ce85d9d5b8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "((1772,), (2215, 100))"
            ]
          },
          "execution_count": 115,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "y_r_train_encoded.shape, predictors.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vq4J8ml5_WGU",
        "outputId": "adbd2eec-f17f-4511-b990-f74e6da5c874"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "56/56 [==============================] - 8s 145ms/step - loss: 1.6726 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4532 - val_loss: 1.0680 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 2/50\n",
            "56/56 [==============================] - 4s 70ms/step - loss: 0.6632 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.6279 - val_loss: 0.3977 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 3/50\n",
            "56/56 [==============================] - 3s 49ms/step - loss: 0.2638 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5238 - val_loss: 0.2365 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 4/50\n",
            "56/56 [==============================] - 1s 19ms/step - loss: 0.2045 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4852 - val_loss: 0.2125 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 5/50\n",
            "56/56 [==============================] - 2s 34ms/step - loss: 0.1959 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4538 - val_loss: 0.2089 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 6/50\n",
            "56/56 [==============================] - 1s 21ms/step - loss: 0.1800 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5070 - val_loss: 0.1968 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 7/50\n",
            "56/56 [==============================] - 2s 42ms/step - loss: 0.1760 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4516 - val_loss: 0.1911 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 8/50\n",
            "56/56 [==============================] - 2s 32ms/step - loss: 0.1657 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4515 - val_loss: 0.1849 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 9/50\n",
            "56/56 [==============================] - 1s 20ms/step - loss: 0.1566 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5322 - val_loss: 0.1964 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 10/50\n",
            "56/56 [==============================] - 1s 15ms/step - loss: 0.1542 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4323 - val_loss: 0.1725 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 11/50\n",
            "56/56 [==============================] - 1s 20ms/step - loss: 0.1430 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5368 - val_loss: 0.1690 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 12/50\n",
            "56/56 [==============================] - 1s 20ms/step - loss: 0.1434 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5189 - val_loss: 0.1856 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 13/50\n",
            "56/56 [==============================] - 1s 20ms/step - loss: 0.1437 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4446 - val_loss: 0.1694 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5551\n",
            "Epoch 14/50\n",
            "56/56 [==============================] - 1s 19ms/step - loss: 0.1345 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5591 - val_loss: 0.1680 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 15/50\n",
            "56/56 [==============================] - 1s 19ms/step - loss: 0.1419 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4270 - val_loss: 0.1616 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 16/50\n",
            "56/56 [==============================] - 1s 27ms/step - loss: 0.1324 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.4930 - val_loss: 0.1698 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 17/50\n",
            "56/56 [==============================] - 2s 32ms/step - loss: 0.1311 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5362 - val_loss: 0.1627 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n",
            "Epoch 18/50\n",
            "56/56 [==============================] - 2s 43ms/step - loss: 0.1262 - accuracy: 0.9786 - precision_3: 0.0000e+00 - recall_3: 0.0000e+00 - auc_3: 0.5527 - val_loss: 0.1637 - val_accuracy: 0.9684 - val_precision_3: 0.0000e+00 - val_recall_3: 0.0000e+00 - val_auc_3: 0.5000\n"
          ]
        }
      ],
      "source": [
        "predictors = np.array(padded)\n",
        "epochs_value = 50\n",
        "validation_split_value = 0.2\n",
        "early_stopping = tf.keras.callbacks.EarlyStopping(patience=3)\n",
        "history = model.fit(predictors, df_r_train['is_ecology'], epochs=epochs_value, verbose=1, validation_split=validation_split_value, callbacks=[early_stopping])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RMHK3DxMBv-4"
      },
      "source": [
        "Plotting Losses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "dLqs01rCBIMe",
        "outputId": "e9482f39-5bf5-449b-8b98-cfa01a990713"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAikAAAGdCAYAAADXIOPgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABK/UlEQVR4nO3deXxU5aE//s+ZPZNkJglJJguBALIKxoiSC1wrQhSjjXD1qlVegli0WrQqtRfTVtD2W6lduNwqilIBve76E2oLVwQEUYoLSyyyCRKSAFmAJDPJJJn1/P44M5NtksyZzJbk8369jnPmzHNmnplMyMdnO4IoiiKIiIiIYowi2hUgIiIi8ochhYiIiGISQwoRERHFJIYUIiIiikkMKURERBSTGFKIiIgoJjGkEBERUUxiSCEiIqKYpIp2BQLhdrtx7tw5JCYmQhCEaFeHiIiIAiCKIhobG5GVlQWFQn67SL8IKefOnUNOTk60q0FERERBqKysxNChQ2Wf1y9CSmJiIgDpTRoMhijXhoiIiAJhsViQk5Pj+zsuV78IKd4uHoPBwJBCRETUzwQ7VIMDZ4mIiCgmMaQQERFRTGJIISIiopjUL8akEBERuVwuOByOaFeD2lEqlVCpVGFbHoQhhYiIYl5TUxPOnDkDURSjXRXqRK/XIzMzExqNJuTPzZBCREQxzeVy4cyZM9Dr9UhLS+OinjFCFEXY7XacP38eZWVlGD16dFALtvWEIYWIiGKaw+GAKIpIS0tDXFxctKtD7cTFxUGtVqO8vBx2ux06nS6kz8+Bs0RE1C+wBSU2hbr1pMNzh+2ZiYiIiPqAIYWIiIhiEkMKERFRGMyYMQOPPvpotKvRrzGkEBERUUwa1CFl08Gz+PWmQ9h3ui7aVSEiIqJOBnVI2X60Bq9/UYHSyoZoV4WIiAIkiiKa7c6obMEuJldfX4/58+cjOTkZer0eRUVFOHHihO/x8vJyFBcXIzk5GfHx8bj00kuxZcsW37nz5s3zTcEePXo01q9fH5LPMtYN6nVSspOl+fZnG1qiXBMiIgpUi8OFCcu2RuW1j/xmNvQa+X8677nnHpw4cQIffvghDAYDli5dihtvvBFHjhyBWq3G4sWLYbfbsXv3bsTHx+PIkSNISEgAADz55JM4cuQI/u///g+pqak4efIkWloGx9+tQR1ShiZ5Qkr94PhhExFR5HnDyZ49ezBt2jQAwBtvvIGcnBxs2rQJt912GyoqKnDrrbdi0qRJAICRI0f6zq+oqEB+fj6uvPJKAEBubm7E30O0DOqQwpYUIqL+J06txJHfzI7aa8t19OhRqFQqFBQU+I4NGTIEY8eOxdGjRwEAP/vZz/Dggw/i448/RmFhIW699VZcdtllAIAHH3wQt956Kw4cOIDrr78ec+fO9YWdgW5Qj0nJTtIDYEghIupPBEGAXqOKyhauVW8XLVqEU6dO4e6778ahQ4dw5ZVX4rnnngMAFBUVoby8HI899hjOnTuHWbNm4fHHHw9LPWLN4A4pnpaUhmYHrDZnlGtDREQD0fjx4+F0OvHll1/6jl28eBHHjx/HhAkTfMdycnLwwAMP4IMPPsDPf/5zrF271vdYWloaFixYgNdffx2rVq3Cyy+/HNH3EC2DursnQauCMU4Nc4sDZxtaMMaUGO0qERHRADN69GjMmTMH9913H1566SUkJibiiSeeQHZ2NubMmQMAePTRR1FUVIQxY8agvr4eO3fuxPjx4wEAy5Ytw+TJk3HppZfCZrPhH//4h++xgW5Qt6QAQDYHzxIRUZitX78ekydPxg9/+ENMnToVoihiy5YtUKvVAACXy4XFixdj/PjxuOGGGzBmzBi88MILAACNRoOSkhJcdtll+MEPfgClUom33347mm8nYgQx2EnfEWSxWGA0GmE2m2EwGEL63Ite3YftR2vw27kTcfe/DQ/pcxMRUd+1trairKwMI0aMgE6ni3Z1qJOefj59/fs96FtShiazJYWIiCgWDfqQ4uvu4QwfIiKimMKQ4mtJaY5yTYiIiKg92SFl9+7dKC4uRlZWFgRBwKZNm3o9x2az4Ve/+hWGDx8OrVaL3NxcrFu3Lpj6hhxbUoiIiGKT7CnIVqsVeXl5uPfee3HLLbcEdM7tt9+OmpoavPLKK7jkkktQVVUFt9stu7Lh4G1JqW20we50Q6Ma9I1LREREMUF2SCkqKkJRUVHA5T/66CN8+umnOHXqFFJSUgDE1nUHhsRroFMr0Opwo8rcguFD4qNdJSIiIkIExqR8+OGHuPLKK/GHP/wB2dnZGDNmDB5//PEer+Bos9lgsVg6bOEiCAKyuFYKERFRzAn7irOnTp3C559/Dp1Oh40bN+LChQv46U9/iosXL2L9+vV+z1mxYgWefvrpcFfNJzspDqfOW3GG41KIiIhiRthbUtxuNwRBwBtvvIEpU6bgxhtvxMqVK/Hqq69225pSUlICs9ns2yorK8NaR66VQkREFHvCHlIyMzORnZ0No9HoOzZ+/HiIoogzZ874PUer1cJgMHTYwokzfIiIKNbk5uZi1apVAZUNdLZtfxP2kDJ9+nScO3cOTU1NvmPfffcdFAoFhg4dGu6XD0g2W1KIiIhijuyQ0tTUhNLSUpSWlgIAysrKUFpaioqKCgBSV838+fN95e+66y4MGTIECxcuxJEjR7B792784he/wL333ou4uLjQvIs+yk7SA2BLChERUSyRHVL27duH/Px85OfnAwCWLFmC/Px8LFu2DABQVVXlCywAkJCQgG3btqGhoQFXXnkl5s2bh+LiYvzlL38J0VvoO29LSpW5BW53zF9vkYhocBNFwG6NzhbgNXlffvllZGVldVkTbM6cObj33nvx/fffY86cOTCZTEhISMBVV12F7du3h+wjOnToEGbOnIm4uDgMGTIE999/f4cejV27dmHKlCmIj49HUlISpk+fjvLycgDAN998g2uvvRaJiYkwGAyYPHky9u3bF7K6ySF7ds+MGTPQ04WTN2zY0OXYuHHjsG3bNrkvFTGmRC2UCgEOl4jaRhsyjLzKJhFRzHI0A89kRee1f3kO0PS+ntZtt92Ghx9+GDt37sSsWbMAAHV1dfjoo4+wZcsWNDU14cYbb8Tvfvc7aLVavPbaayguLsbx48cxbNiwPlXRarVi9uzZmDp1Kr7++mvU1tZi0aJFeOihh7BhwwY4nU7MnTsX9913H9566y3Y7XZ89dVXEAQBADBv3jzk5+fjxRdfhFKpRGlpKdRqdZ/qFKywT0HuD1RKBTIMOpxtaMHZhmaGFCIi6pPk5GQUFRXhzTff9IWU999/H6mpqbj22muhUCiQl5fnK//b3/4WGzduxIcffoiHHnqoT6/95ptvorW1Fa+99hri46VA9fzzz6O4uBjPPvss1Go1zGYzfvjDH2LUqFEApAktXhUVFfjFL36BcePGAQBGjx7dp/r0BUOKR3ZyHM42tOBMfQsmD492bYiIqFtqvdSiEa3XDtC8efNw33334YUXXoBWq8Ubb7yBH/3oR1AoFGhqasJTTz2FzZs3o6qqCk6nEy0tLR2GSwTr6NGjyMvL8wUUQJrE4na7cfz4cfzgBz/APffcg9mzZ+O6665DYWEhbr/9dmRmZgKQhnEsWrQI//u//4vCwkLcdtttvjATabxQjcdQTkMmIuofBEHqconG5ukSCURxcTFEUcTmzZtRWVmJzz77DPPmzQMAPP7449i4cSOeeeYZfPbZZygtLcWkSZNgt9vD9al1sH79euzduxfTpk3DO++8gzFjxuCLL74AADz11FM4fPgwbrrpJnzyySeYMGECNm7cGJF6dcaQ4sFpyEREFEo6nQ633HIL3njjDbz11lsYO3YsrrjiCgDAnj17cM899+A//uM/MGnSJGRkZOD06dMhed3x48fjm2++gdVq9R3bs2cPFAoFxo4d6zuWn5+PkpIS/POf/8TEiRPx5ptv+h4bM2YMHnvsMXz88ce45ZZbul0hPtwYUjy4oBsREYXavHnzsHnzZqxbt87XigJI4zw++OADlJaW4ptvvsFdd93VZSZQX15Tp9NhwYIF+Pbbb7Fz5048/PDDuPvuu2EymVBWVoaSkhLs3bsX5eXl+Pjjj3HixAmMHz8eLS0teOihh7Br1y6Ul5djz549+PrrrzuMWYkkjknx8LaknGNIISKiEJk5cyZSUlJw/Phx3HXXXb7jK1euxL333otp06YhNTUVS5cuDdnFdPV6PbZu3YpHHnkEV111FfR6PW699VasXLnS9/ixY8fw6quv4uLFi8jMzMTixYvxk5/8BE6nExcvXsT8+fNRU1OD1NRU3HLLLRG9nl57gtjTfOIYYbFYYDQaYTabw7ZE/qnzTZj5508Rr1Hi26dn+6ZiERFRdLW2tqKsrAwjRoyATsfZl7Gmp59PX/9+s7vHI8vT3WO1u2BucUS5NkRERMSQ4qFTK5GaoAUAnOHgWSIiihFvvPEGEhIS/G6XXnpptKsXVhyT0k52chwuNNlwtqEFE7ONvZ9AREQUZjfffDMKCgr8PhatlWAjhSGlnewkHb6p5DRkIiKKHYmJiUhMTIx2NaKC3T3tcBoyEVHs6gfzPAalcP5cGFLa8YUUtqQQEcUMpVIJABFbjZXkaW5uBhCerid297STnSxdk4EtKUREsUOlUkGv1+P8+fNQq9VQKPj/17FAFEU0NzejtrYWSUlJvjAZSgwp7bC7h4go9giCgMzMTJSVlaG8vDza1aFOkpKSkJGREZbnZkhpx7vqbJ3Vjma7E3oNPx4iolig0WgwevRodvnEGLVaHZYWFC/+FW7HGKdGolaFRpsT5xpacEn64BxNTUQUixQKBVecHWTYsdeJtzWFC7oRERFFF0NKJxyXQkREFBsYUjrxtqRwGjIREVF0MaR0wpYUIiKi2MCQ0glbUoiIiGIDQ0onbEkhIiKKDQwpnXhbUmosrXC43FGuDRER0eDFkNJJarwWGpUCbhGoNrdGuzpERESDFkNKJwqF4Ovy4VopRERE0cOQ4gfHpRAREUUfQ4ofvpDClhQiIqKoYUjxwzcNuaE5yjUhIiIavBhS/GB3DxERUfQxpPjBBd2IiIiijyHFD29LyrmGVrjdYpRrQ0RENDgxpPiRYdRBIQB2lxsXmmzRrg4REdGgJDuk7N69G8XFxcjKyoIgCNi0aVPA5+7ZswcqlQqXX3653JeNKLVSgQyDDgDHpRAREUWL7JBitVqRl5eH1atXyzqvoaEB8+fPx6xZs+S+ZFS0zfBhSCEiIooGldwTioqKUFRUJPuFHnjgAdx1111QKpWyWl+iJSspDkA9B88SERFFSUTGpKxfvx6nTp3C8uXLAypvs9lgsVg6bJHGachERETRFfaQcuLECTzxxBN4/fXXoVIF1nCzYsUKGI1G35aTkxPmWnbFachERETRFdaQ4nK5cNddd+Hpp5/GmDFjAj6vpKQEZrPZt1VWVoaxlv6xJYWIiCi6ZI9JkaOxsRH79u3DwYMH8dBDDwEA3G43RFGESqXCxx9/jJkzZ3Y5T6vVQqvVhrNqvRrKlhQiIqKoCmtIMRgMOHToUIdjL7zwAj755BO8//77GDFiRDhfvk+yPC0pjTYnzC0OGOPUUa4RERHR4CI7pDQ1NeHkyZO++2VlZSgtLUVKSgqGDRuGkpISnD17Fq+99hoUCgUmTpzY4fz09HTodLoux2ONXqNCSrwGdVY7zta3MKQQERFFmOwxKfv27UN+fj7y8/MBAEuWLEF+fj6WLVsGAKiqqkJFRUVoaxklHJdCREQUPYIoijF/cRqLxQKj0Qiz2QyDwRCx133gf/fjo8PVeKp4Au6ZHrtdU0RERLGor3+/ee2eHnDVWSIiouhhSOkBu3uIiIiihyGlB1zQjYiIKHoYUnrAlhQiIqLoYUjpgXdBtwtNdrQ6XFGuDRER0eDCkNIDY5wa8RolALamEBERRRpDSg8EQeC4FCIioihhSOkFx6UQERFFB0NKL9iSQkREFB0MKb3ITtIDYEsKERFRpDGk9IItKURERNHBkNILjkkhIiKKDoaUXnjXSqm2tMLpcke5NkRERIMHQ0ov0hK00CgVcLlFVFtao10dIiKiQYMhpRcKhYDMJB0AjkshIiKKJIaUAGQZOS6FiIgo0hhSAuCd4XOOIYWIiChiGFICwBk+REREkceQEgBvS8oZjkkhIiKKGIaUAAxlSwoREVHEMaQEoP2YFFEUo1wbIiKiwYEhJQCZxjgIAtDqcOOi1R7t6hAREQ0KDCkB0KgUSE/UAuBaKURERJHCkBIgzvAhIiKKLIaUAGUn6wGwJYWIiChSGFICxJYUIiKiyGJICRDXSiEiIooshpQAca0UIiKiyGJICZC3JeVsfXOUa0JERDQ4MKQEyDsmxdLqRGOrI8q1ISIiGvgYUgIUr1UhSa8GwC4fIiKiSGBIkcE3w4eDZ4mIiMJOdkjZvXs3iouLkZWVBUEQsGnTph7Lf/DBB7juuuuQlpYGg8GAqVOnYuvWrcHWN6o4DZmIiChyZIcUq9WKvLw8rF69OqDyu3fvxnXXXYctW7Zg//79uPbaa1FcXIyDBw/Krmy0tQ2eZUghIiIKN5XcE4qKilBUVBRw+VWrVnW4/8wzz+Bvf/sb/v73vyM/P1/uy0eVtyXlDFtSiIiIwk52SOkrt9uNxsZGpKSkdFvGZrPBZrP57lsslkhUrVdD2ZJCREQUMREfOPunP/0JTU1NuP3227sts2LFChiNRt+Wk5MTwRp2LzvJc/0etqQQERGFXURDyptvvomnn34a7777LtLT07stV1JSArPZ7NsqKysjWMvueceknG+0odXhinJtiIiIBraIdfe8/fbbWLRoEd577z0UFhb2WFar1UKr1UaoZoFL1qsRp1aixeFClbkVI1Ljo10lIiKiASsiLSlvvfUWFi5ciLfeegs33XRTJF4yLARB4AwfIiKiCJHdktLU1ISTJ0/67peVlaG0tBQpKSkYNmwYSkpKcPbsWbz22msApC6eBQsW4H/+539QUFCA6upqAEBcXByMRmOI3kbkZCXF4WRtE8428Bo+RERE4SS7JWXfvn3Iz8/3TR9esmQJ8vPzsWzZMgBAVVUVKioqfOVffvllOJ1OLF68GJmZmb7tkUceCdFbiCyuOktERBQZsltSZsyYAVEUu318w4YNHe7v2rVL7kvENO80ZK6VQkREFF68do9MbEkhIiKKDIYUmbwDZ8+ZGVKIiIjCiSFFJm9LSlVDK1zu7ru9iIiIqG8YUmQyGXRQKQQ43SJqG1ujXR0iIqIBiyFFJqVCQIZRB4DjUoiIiMKJISUIvsGznOFDREQUNgwpQfAOnj3DlhQiIqKwYUgJwlC2pBAREYUdQ0oQeP0eIiKi8GNICUJ2kh4AW1KIiIjCiSElCO1bUnq6RAAREREFjyElCJmeKcgtDhfqmx1Rrg0REdHAxJASBJ1aibRELQCOSyEiIgoXhpQgta2V0hzlmhAREQ1MDClB4lopRERE4cWQEiSulUJERBReDClB4lopRERE4cWQEiRev4eIiCi8GFKC5GtJYUghIiIKC4aUIHlbUhqaHbDanFGuDRER0cAzuENKSwNweg/QWCP71ESdGgadCgBbU4iIiMJhcIeUd+cDG24ETmwN6vTsZM81fDh4loiIKOQGd0jJmCTdVn8b1OnZSdLy+GfYkkJERBRygzukmCZKtzXBhhROQyYiIgqXQR5SLpVua74FgriaMWf4EBERhc/gDilpYwGFCmg1A+Yzsk/PTvKOSeH1e4iIiEJtcIcUlRZIHSPtB9Hlw5YUIiKi8BncIQXo07gU75iU2kYb7E53KGtFREQ06DGkZHhCShAzfFITNNCqFBBFoMrM1hQiIqJQYkjxDZ49LPtUQRB4DR8iIqIwYUgxedZKqfsesMsfAMurIRMREYUHQ0qiCYhPA0Q3UHtU9ulsSSEiIgoP2SFl9+7dKC4uRlZWFgRBwKZNm3o9Z9euXbjiiiug1WpxySWXYMOGDUFUNYzar5ciExd0IyIiCg/ZIcVqtSIvLw+rV68OqHxZWRluuukmXHvttSgtLcWjjz6KRYsWYevW4K6XExZ9meHDachERERhoZJ7QlFREYqKigIuv2bNGowYMQJ//vOfAQDjx4/H559/jv/+7//G7Nmz5b58eJiCn+HD7h4iIqLwCPuYlL1796KwsLDDsdmzZ2Pv3r3hfunAeach1xyWvTy+tyWlqqEVbrf8pfWJiIjIv7CHlOrqaphMpg7HTCYTLBYLWlr8tz7YbDZYLJYOW1ilepbHt5kBc6WsUzMMOigVAuwuN8432cJUQSIiosEnJmf3rFixAkaj0bfl5OSE9wVVGimoALLXS1EpFcgw6AAAZzh4loiIKGTCHlIyMjJQU1PT4VhNTQ0MBgPi4uL8nlNSUgKz2ezbKivltW4EV1GOSyEiIoolsgfOyjV16lRs2bKlw7Ft27Zh6tSp3Z6j1Wqh1WrDXbWOfNOQD8k+NTs5DjjNachEREShJLslpampCaWlpSgtLQUgTTEuLS1FRUUFAKkVZP78+b7yDzzwAE6dOoX/+q//wrFjx/DCCy/g3XffxWOPPRaadxAqpnaDZ2Vqa0mRv2ItERER+Sc7pOzbtw/5+fnIz88HACxZsgT5+flYtmwZAKCqqsoXWABgxIgR2Lx5M7Zt24a8vDz8+c9/xl//+tfYmX7sleFZHv/i94DdKutULo1PREQUerK7e2bMmAGxh2m6/laTnTFjBg4ePCj3pSIrIV1aHt96Hqg9BgydHPCpHJNCREQUejE5uydqfF0+8saltG9J6SnAERERUeAYUtoLcoaPtyXFanfB3OIIda2IiIgGJYaU9oIcPKtTK5GaoAHAtVKIiIhChSGlPVMflsfnuBQiIqKQYkhpL3UMoFBLy+M3VPRevp2sJM7wISIiCiWGlPZUGiAtuOXx2ZJCREQUWgwpnfm6fGQOnuVaKURERCHFkNKZb3n84Gb4sCWFiIgoNBhSOgt2GnIyQwoREVEoMaR05u3uqTsla3n8oUl66TSrHc12ZzhqRkRENKgwpHSWkA7EpwMQgdqjAZ9miFMhQStdZeAcW1OIiIj6jCHFH1+XT+DL4wuC4BuXwgXdiIiI+o4hxZ9gB89yXAoREVHIMKT4Y5ok3Qa5Vgq7e4iIiPqOIcWfjOCWx+daKURERKHDkOLPkNGe5fEtspbH51opREREocOQ4o9KA6SNk/ZljEthSwoREVHoMKR0xzt4VsaibkM9LSnVllY4XO5w1IqIiGjQYEjpTob8a/ikJmihUSrgFoFqc2uYKkZERDQ4MKR0J4gLDSoUArKSdAA4LoWIiKivGFK641sevwywNQV8GselEBERhQZDSncS0oAEE+Quj88ZPkRERKHBkNITX5dP4MvjZ3suNMiWFCIior5hSOmJb3n8wFee5dL4REREocGQ0pMMz/L4MqYhs7uHiIgoNBhSetK+JcUd2LonQ9u1pLjdgS+pT0RERB0xpPQkdQyg1AD2RsAc2PL4GUYdFAJgd7pxwWoLcwWJiIgGLoaUnijVQNpYaT/ALh+1UgGTwbNWCgfPEhERBY0hpTdBLOqWxXEpREREfcaQ0psgQopv8CxbUoiIiILGkNKbIC40yGnIREREfceQ0hvvNOT6wJfHZ0sKERFR3zGk9CY+FUjIkPZrjwR0CltSiIiI+i6okLJ69Wrk5uZCp9OhoKAAX331VY/lV61ahbFjxyIuLg45OTl47LHH0NraGlSFo8LX5RPY8vhD2ZJCRETUZ7JDyjvvvIMlS5Zg+fLlOHDgAPLy8jB79mzU1tb6Lf/mm2/iiSeewPLly3H06FG88soreOedd/DLX/6yz5WPmAzv4NnAlsf3tqQ02pwwtzjCVSsiIqIBTXZIWblyJe677z4sXLgQEyZMwJo1a6DX67Fu3Tq/5f/5z39i+vTpuOuuu5Cbm4vrr78ed955Z6+tLzHF5BmXEuAMH71GhWS9GgBbU4iIiIIlK6TY7Xbs378fhYWFbU+gUKCwsBB79+71e860adOwf/9+Xyg5deoUtmzZghtvvLHb17HZbLBYLB22qPItj38k4OXxOS6FiIiob2SFlAsXLsDlcsFkMnU4bjKZUF1d7fecu+66C7/5zW/w7//+71Cr1Rg1ahRmzJjRY3fPihUrYDQafVtOTo6caoZe6ui25fEbygM6pW2GT3M4a0ZERDRghX12z65du/DMM8/ghRdewIEDB/DBBx9g8+bN+O1vf9vtOSUlJTCbzb6tsrIy3NXsWfvl8QPs8slO0gNgSwoREVGwVHIKp6amQqlUoqampsPxmpoaZGRk+D3nySefxN13341FixYBACZNmgSr1Yr7778fv/rVr6BQdM1JWq0WWq1WTtXCzzRJmt1TcxgYX9xrcXb3EBER9Y2slhSNRoPJkydjx44dvmNutxs7duzA1KlT/Z7T3NzcJYgolUoAgCiKcusbPd4ZPgFOQ+aCbkRERH0jqyUFAJYsWYIFCxbgyiuvxJQpU7Bq1SpYrVYsXLgQADB//nxkZ2djxYoVAIDi4mKsXLkS+fn5KCgowMmTJ/Hkk0+iuLjYF1b6Bd/g2cC6e4b6WlL60XowREREMUR2SLnjjjtw/vx5LFu2DNXV1bj88svx0Ucf+QbTVlRUdGg5+fWvfw1BEPDrX/8aZ8+eRVpaGoqLi/G73/0udO8iErzTkOtPA7ZGQJvYY3FvS8qFJhtaHS7o1P0okBEREcUAQewHfS4WiwVGoxFmsxkGgyF6FfnzOKCxCrj3Y2BYQY9FRVHEpcu3otnuwic/vwYj0xIiVEkiIqLY0Ne/37x2jxwyunwEQWgbl8LBs0RERLIxpMhh8i6PH+A05GQOniUiIgoWQ4oc3pBSHehaKWxJISIiChZDihzeaci1gS2Pz5YUIiKi4DGkyDHEuzx+E9Bwutfi3paUM2xJISIiko0hRQ6lCkgbJ+0H0OUzlC0pREREQWNIkSvDs15KzeFei3qv31NtaYXTFdjVk4mIiEjCkCKXjGnI6YlaqJUCXG4RNY22MFeMiIhoYGFIkUvGNGSFQkCmkV0+REREwWBIkcsbUupPA62WXotnJekAAGcbmsNYKSIiooGHIUWu+CFAYqa0X3uk1+LecSlsSSEiIpKHISUYMrp8fGulcBoyERGRLAwpwcgIfOXZod61UtiSQkREJAtDSjDYkkJERBR2DCnB8IWU3pfH9646e66hBaIohrtmREREAwZDSjCGXAIotYDDCtSX9Vg00zO7p9XhxkWrPRK1IyIiGhAYUoKhVAHpnuXxe1l5VqtSIj1RC4AzfIiIiORgSAmWybs8PselEBERhQNDSrC8y+MHMMPHOy6FLSlERESBY0gJVgZn+BAREYUTQ0qwvDN8Gsp7XR6fa6UQERHJx5ASLH0KkJgl7fcyeJYtKURERPIxpPRFgF0+3uv3nKlrhsvNtVKIiIgCwZDSF97Bs72ElJFp8UjUqdBoc6K0siH89SIiIhoAGFL6wrfybM/dPWqlAjPGpgMAth+tCXetiIiIBgSGlL7I8K6V0vvy+IXjPSHlCEMKERFRIBhS+iJlVMDL488Ymw6VQsCJ2iacvmCNUAWJiIj6L4aUvlCqgPTx0n4v41KMcWoUjEwBwC4fIiKiQDCk9JV3XEoAK88WjjcBALaxy4eIiKhXDCl9lRHY4FmgLaTsK69HPa+ITERE1COGlL7yzfA51GvRnBQ9xmUkwuUWseu72jBXjIiIqH9jSOkr71opDRVAq7nX4tdNYJcPERFRIIIKKatXr0Zubi50Oh0KCgrw1Vdf9Vi+oaEBixcvRmZmJrRaLcaMGYMtW7YEVeGYo08BDNnSfs2RXot7u3w+PX4eNqcrnDUjIiLq12SHlHfeeQdLlizB8uXLceDAAeTl5WH27NmorfXffWG323Hdddfh9OnTeP/993H8+HGsXbsW2dnZfa58zDAFfkXkSdlGpCdqYbW78MWpujBXjIiIqP+SHVJWrlyJ++67DwsXLsSECROwZs0a6PV6rFu3zm/5devWoa6uDps2bcL06dORm5uLa665Bnl5eX2ufMzwdvlU9z4uRaEQMMvTmsKF3YiIiLonK6TY7Xbs378fhYWFbU+gUKCwsBB79+71e86HH36IqVOnYvHixTCZTJg4cSKeeeYZuFwDqKtDxgwfALjeMy5l+9EaiCIvOEhEROSPSk7hCxcuwOVywWQydThuMplw7Ngxv+ecOnUKn3zyCebNm4ctW7bg5MmT+OlPfwqHw4Hly5f7Pcdms8Fms/nuWywWOdWMPG93T+0RwO0CFMoei08dNQRxaiWqzK04fM6CidnGCFSSiIiofwn77B6324309HS8/PLLmDx5Mu644w786le/wpo1a7o9Z8WKFTAajb4tJycn3NXsm5RRgEoHOJqB+tO9FteplfjBmFQAnOVDRETUHVkhJTU1FUqlEjU1Hf+w1tTUICMjw+85mZmZGDNmDJTKttaF8ePHo7q6Gna7/wXNSkpKYDabfVtlZaWcakZe++XxAxiXArTN8uES+URERP7JCikajQaTJ0/Gjh07fMfcbjd27NiBqVOn+j1n+vTpOHnyJNztrhL83XffITMzExqNxu85Wq0WBoOhwxbzvINnA5jhAwAzx6VDIQCHz1lwtqEljBUjIiLqn2R39yxZsgRr167Fq6++iqNHj+LBBx+E1WrFwoULAQDz589HSUmJr/yDDz6Iuro6PPLII/juu++wefNmPPPMM1i8eHHo3kUsME2SbgMcPDskQYvJw5MBADvYmkJERNSFrIGzAHDHHXfg/PnzWLZsGaqrq3H55Zfjo48+8g2mraiogELRln1ycnKwdetWPPbYY7jsssuQnZ2NRx55BEuXLg3du4gFGYFfaNCrcLwJX5+ux7YjNZg/NTc89SIiIuqnBLEfzIG1WCwwGo0wm82x2/XTUg88myvtLy0H4pJ6PeX7802Y9edPoVYKOPDkdUjUqcNaRSIiokjq699vXrsnVOKSAcNQab+29+XxAWBUWgJGpsbD4RLx6Xfnw1g5IiKi/ochJZR8K88G3uXjveAgV58lIiLqiCEllDICv4aPV6EnpHxyrBYOl7uX0kRERIMHQ0ooybjQoNcVw5KRrFfD0urEvtP1YaoYERFR/8OQEkq+kOJZHj8ASoWAmeOk1hSuPktERNSGISWUhniWx3e2AHVlAZ/mHZey7Wg1LzhIRETkwZASSgpl2/L4NYEtjw8AV49OhUalQGVdC07UNoWpckRERP0LQ0qomeQv6havVWH6qCEA2OVDRETkxZASahnylsf38s7yYUghIiKSMKSEmswLDXp5r4pcWtmA2sbWUNeKiIio32FICTVvSDFXAi0NgZ9m0CFvqBEA8MnR2jBUjIiIqH9hSAm1uGTAmCPty+3y8bSmbOdVkYmIiBhSwiLYLh/PuJTPTlxAs90Z6loRERH1Kwwp4RDEyrMAMC4jEUOT42BzuvH5iQthqBgREVH/wZASDkFcaBAABEFglw8REZEHQ0o4eKch1x4NeHl8L+/qszuO1sLl5uqzREQ0eDGkhEPKSEAV51ke/5SsU6eMSEGiToWLVjtKKxvCUz8iIqJ+gCElHNovj18d+PL4AKBWKnDt2HQAXNiNiIgGN4aUcMnwDp6VNw0ZaJvlw3EpREQ0mDGkhEuQM3wA4JoxaVApBJysbULZBWuIK0ZERNQ/MKSESxAXGvQyxqlRMDIFALCDrSlERDRIMaSEi3casuUM0FIv+/TrPFORP+a4FCIiGqQYUsIlLino5fEBYJYnpOw7XYd6qz2EFSMiIuofGFLCqQ9dPjkpeozLSIRbBHYe5wUHiYho8GFICaeM4AfPAm0Lu3GWDxERDUYMKeEU5IUGvbwh5dPj52Fzylu5loiIqL9jSAknU/DL4wPAxCwjTAYtrHYX9n5/McSVIyIiim0MKeGUMsKzPH4rcPF72acrFIJvAC27fIiIaLBhSAknhRIwTZD2a+Qtj+/lnYq8/UgtRJEXHCQiosGDISXcTMEvjw8AU0cNgV6jRLWlFd+etYSwYkRERLGNISXc+jANGQB0aiV+MDoNALCNXT5ERDSIMKSEWx8uNOjlu+AgV58lIqJBhCEl3Novj99cF9RTzByXDoUAHKmy4GxDSwgrR0REFLuCCimrV69Gbm4udDodCgoK8NVXXwV03ttvvw1BEDB37txgXrZ/0hkB4zBpP8jWlJR4Da4cLl1wkK0pREQ0WMgOKe+88w6WLFmC5cuX48CBA8jLy8Ps2bNRW9vz0u2nT5/G448/jquvvjroyvZbIenySQfAqchERDR4yA4pK1euxH333YeFCxdiwoQJWLNmDfR6PdatW9ftOS6XC/PmzcPTTz+NkSNH9qnC/ZJv5dngpiEDQKFnKvIXpy7C0uoIRa2IiIhimqyQYrfbsX//fhQWFrY9gUKBwsJC7N27t9vzfvOb3yA9PR0//vGPA3odm80Gi8XSYevX+jjDBwBGpiVgZFo8HC4Ru787H6KKERERxS5ZIeXChQtwuVwwmUwdjptMJlRXV/s95/PPP8crr7yCtWvXBvw6K1asgNFo9G05OTlyqhl7MvOk2+p/Aac/D/ppvNfy2cZxKURENAiEdXZPY2Mj7r77bqxduxapqakBn1dSUgKz2ezbKisrw1jLCEgZAeTdCYhu4P0fA009j9/pjnf12Z3HauFwuUNZQyIiopijklM4NTUVSqUSNTUd/0++pqYGGRkZXcp///33OH36NIqLi33H3G7pj6tKpcLx48cxatSoLudptVpotVo5VYt9N/0ZOHcQOH8M+P8WAXdvlJbNlyF/WDJS4jWos9rx9ek6TBsVePAjIiLqb2S1pGg0GkyePBk7duzwHXO73dixYwemTp3apfy4ceNw6NAhlJaW+rabb74Z1157LUpLS/t/N44cmnjgtlcBtR4o+xT49A+yn0KpEDBznGeWz5HgWmOIiIj6C9ndPUuWLMHatWvx6quv4ujRo3jwwQdhtVqxcOFCAMD8+fNRUlICANDpdJg4cWKHLSkpCYmJiZg4cSI0Gk1o302sSx8H/HCVtP/ps8D3n8h+Ct+4lKPVvOAgERENaLK6ewDgjjvuwPnz57Fs2TJUV1fj8ssvx0cffeQbTFtRUQGFggvZdivvDqB8D3DgVeD/uw944DPAkBXw6VePToVGpUBlXQu+q2nC2IzEMFaWiIgoegSxH/zvuMVigdFohNlshsFgiHZ1+s7RAvz1OmndlGHTgAV/B5SB58V7N3yNT47V4hezx2LxtZeEsaJERETB6+vfbzZ5RIM6Drj9VUCTCFT8E9j5/2Sd7l3YjVORiYhoIGNIiZYho4A5z0n7n/838N3WgE8tHC8Nni2tbEBtY2s4akdERBR1DCnRdOl/AFPul/Y/uB9oqAjotHSDDnk5SQCAHUc5y4eIiAYmhpRou/7/AVn5QGsD8N5CwGkP6LTrxnunIrPLh4iIBiaGlGhTaYHbNgA6I3B2H7B9eUCnFXqmIn9+8gKa7c4wVpCIiCg6GFJiQXIuMHeNtP/FC8CRD3s9ZawpETkpcbA53fj8xIXw1o+IiCgKGFJixbgbgWkPS/t/WwzUneqxuCAInOVDREQDGkNKLJm1HMgpAGwW4N0FgKPnmTveCw5+cqwWLnfML3dDREQkC0NKLFGqgf9cD8SlANX/Arb+ssfiV41IgUGnwkWrHaWV9RGqJBERUWQwpMQaYzZwy1oAArDvFeDQ+90WVSsVuNZzwcFtvOAgERENMAwpsWh0IXD1z6X9D38GnP+u26Jt41KqI1EzIiKiiGFIiVUzSoDcqwGHFXhvAWBv9lvsmrFpUCkEfH/eilPnmyJcSSIiovBhSIlVShVw61+B+HSg9giw5Rd+ixl0avzbyCEAuPosERENLAwpsSwxA/jPVwBBAZS+Dhx83W+x6zwLu207yqnIREQ0cDCkxLoRPwBmeGb5bH4cqDncpcgszxL5+07Xoc4a2LL6REREsY4hpT+4+ufAqFmAs0VaP8XW2OHhocl6jM80wC0CO4+xy4eIiAYGhpT+QKEAbnkZSMwCLp4A/vEYIHZcvM13wUF2+RAR0QDBkNJfxKcCt60HBCVw6D1g//oOD3svOPjpd+fR6nBFo4ZEREQhxZDSnwz7N6DwKWn//5YC50p9D03KNsJk0KLZ7sLeUxejUj0iIqJQYkjpb6Y9DIwpAlx2af2UVjOAjhcc3M4LDhIR0QDAkNLfCAIw9wXAOAyoPy1dMdkzPsXb5bP9aA1EkRccJCKi/o0hpT/SpwC3bQAUauDo34Ev1wAApo4cAr1GiRqLDQcqeMFBIiLq3xhS+quhk4HZv5P2P/41cGYfdGolZoxNAwDc+fKX+PWmQzhT7385fSIioljHkNKfTbkfmDAHcDuB9+4BmuvwyxvH46rcZNhdbrz+RQVm/HEX/uv9b3D6gjXatSUiIpJFEPvB4AWLxQKj0Qiz2QyDwRDt6sSWVjPw8gyg7hQw5gbgR29BFAR8caoOz+88gT0npZk+CgG4OS8Li6+9BKNNidGtMxERDQp9/fvNkDIQVP0L+Gsh4LIBhU8D//6o76H95fV4/pMT2Hn8PABp3G3RxAwsvvYSXJpljFKFiYhoMGBIIcn+DcDfH5EWe7vnH8DwaR0e/vasGc99cgJbD7dNTy4cn46HZo7G5TlJka0rERENCgwpJBFFYONPgH+9AyRmSoNqMy8HkkdIy+p7HK9uxOqdJ/GPf52D2/OTv3p0Kh6eORpTRqREp+5ERDQgMaRQG1sTsPZa4MJ3bce0BiDjMiAzD8i6XLodcglOXWzBC7u+x8aDZ+HypJUpI1Lws5mjMf2SIRAEITrvgYiIBgyGFOrIfAbY8z/A2QNAzbeAs7VrGXU8kDEJyMzDRcN4vF6RjBcPK9HqklpcLs9Jws9mXYJrx6YzrBARUdAYUqh7Lidw4ThQ9Y10nZ+qb4DqQ4Cj63RkUanDWe1I7G7KxjeuXHzrzoUqYwIenDUe10/IgELBsEJERPIwpJA8bhdw8aQUWLzhpfpfgM3SpahdVOK4mINK7RhkT/g3TLzyGigzLgXUcZGvNxER9TtRCSmrV6/GH//4R1RXVyMvLw/PPfccpkyZ4rfs2rVr8dprr+Hbb78FAEyePBnPPPNMt+X9YUgJM7cbqC/zBJdSoOobuM+VQtHa0LWooISQNg6Cd3xL0jBApQVUOkCp9ex7NmWnfQXXDiQiGkwiHlLeeecdzJ8/H2vWrEFBQQFWrVqF9957D8ePH0d6enqX8vPmzcP06dMxbdo06HQ6PPvss9i4cSMOHz6M7OzsgF6TISUKRBFoqIC1/ACO7N8NW+UBjBNPIVXo2uISMKWm5yDjCzsa6bZ9GXUcEJ8GJJqk2UsJJiAxA9DEh+49ExFRSEU8pBQUFOCqq67C888/DwBwu93IycnBww8/jCeeeKLX810uF5KTk/H8889j/vz5Ab0mQ0r0WW1OvPHFaWzcvR/ZLccxUVGGK9QVGKVvhhYOqOGAWrRD5XZA6bZB4bIDLhsEhLk3UWtoCyze28QMICGj3b4J0CZKK9kREVHE9PXvt0pOYbvdjv3796OkpMR3TKFQoLCwEHv37g3oOZqbm+FwOJCS0v2aHDabDTabzXffYunD/71TSMRrVbj/mkswf9oIvP1VBdZ8egqrLK1AS09niYhTuJEWB6TqRAyJA1K1IpK1biRpRCRp3DCq3TCoXUhUuZCgciNB4YRe6YJOcEDhsgEuuzRDyd4MWGuBxhqgsQpoqgEczdJYGpsFuHii5zeg1rcLL51aYxI89xNNgC6JYYaIKEbICikXLlyAy+WCyWTqcNxkMuHYsWMBPcfSpUuRlZWFwsLCbsusWLECTz/9tJyqUYTo1ErcM30E7iwYhr9/U4XvahrR0GxHQ7MDDS0OmJsdaGixo77ZAbvTjRa3EhVWoELm9Q0FATDo1EjSq5EUp0aSXoPs5DjkjNRjWIoeOck65MS7keS6AKGpRgovTdVAo2drqmnbtzdKgabulLT1RKUD4lKkbiRNPKBJaLff+b53X9/N8XhpuvdAGYsjitK1oppqpc+3qabTfg3QXAck57atyZN5OaDnIoFEFBxZIaWvfv/73+Ptt9/Grl27oNPpui1XUlKCJUuW+O5bLBbk5OREoooUIK1Kif+cPLTHMq0Olye8eEJMswPmlrZA0+F+swPmFgcamu2w2l0QRcDcIh0r7+E1ErUqDE3RY1jKCOQkX4phQ/TIGa5HTooeQ5PjoFMrAbu1XXCp6j7QtDZIrTaN50L6WUGt7xpgvMe0idIxbYLnNrHTscR2j3nuK0P8a+u0ecKGN3BUt9vvFEj8rbvTWVUpcGRT231jTltgycyTtkRTNycTEbWR9a9damoqlEolampqOhyvqalBRkZGj+f+6U9/wu9//3ts374dl112WY9ltVottFqtnKpRDNKplcgwKpFh7D6Q+mN3uj0BpS3A1FntOFPfjIq6ZlTWt6CirhnnG21otDlxtMqCo1X+uwRNBi1ykqXWl6EpqRiWMgw5pjgMm6CHKVHXcf0XR4v0x7ilQQo2ditgb+q672judLxzOc990e153mZps54P8pPsRBXXNbi0v681dA0+bmfX0NHoufUzi6tHWqMUMhJMQEJ6x1udEbhwwjdLDHWnAHOltB37R9tzJGS0a23xbIbswdfV5nYBlnPS7Lq6MikEJg2XWqOShkmtdESDmKyQotFoMHnyZOzYsQNz584FIA2c3bFjBx566KFuz/vDH/6A3/3ud9i6dSuuvPLKPlWYBj6NSoG0RC3SEnsOqq0OV1twqWvx3Db7bq12F2osNtRYbNhXXt/1dZQKqQspRY+c5DipGylFj9SEYdCpFdDqlNKtSgmtSgGdWgmNSgFlIAvbiWLbWBp/Qcdulbqh7Fbpcga2Rum+rUkq47u1tO277NJzO1ukLVShBwAUas/YHH/hI6Pdfrq8dXJazdICgu3X5bnwndRa891H0ualT+0YWjLzpD/W/T24OFqA+vK2IOK7PQ00lLf9XP1JMEmfgb8tIWPgdCUSdSOoKcgLFizASy+9hClTpmDVqlV49913cezYMZhMJsyfPx/Z2dlYsWIFAODZZ5/FsmXL8Oabb2L69Om+50lISEBCQkJAr8nZPSSXKIqob3a0hZZ6Kbh4w8y5hhY43cHNPFIrBWhVHQOMVu259YSZ9sc6Bx1vuTiNEkl6DVITNEiJ12JIggaJWlX3lyJw2j3BpdETarxhplO4sVk6BZ1GQFB4Bgl7w0f7LR2IS45cGLBbgepv24JLVSlQexQQXV3L6oztQsvl0pYyMrb+OIsi0FLfKYC0CyK9dR8q1FKrScoIKQA2VAB1pwGbuefzlFogebj/AJM0XGpJI4qyqCzm9vzzz/sWc7v88svxl7/8BQUFBQCAGTNmIDc3Fxs2bAAA5Obmory866iC5cuX46mnngro9RhSKNScLjeqzK1dwktlfTMamh2wOVxodbphc7hgc7qDDjRyaZQKDEnQICVegyEJWqTGazz3pRCTmqDBkHgtUuI1SE3QIk6jjEi9ws7RCtQe7tjiUnvEfyuDJkG6aOaQkVLXl0rbbm0dTddFBLusy6Npt/igpt3aPD0sOuh2AZazXYNI/enAAoXWIIWHlBHSlcm9t8m5gHEooPDzc2ypl57f39ZQ6T/UtRef1n0rTGKm/9ek3rndUvepShPtmvQLXBafKAKcLjdsTu/mQqtDurU53Gj1BBmbs/2+5zHPrb/HWhwu1FntuGi1oa5JGjAsl16j9BtoUhOkfW+gGZKgQbJeIw0k7i+cduD8sY4tLtXfSl1d4aRQdww8CpXUPdVTtwwg/eFvH0DaBxF9SmhbqlxOwHKm+xDT0rV7swOFWmq1ERRSWBGU7W4Vne73dNzf+Z2OKzVdB4a3HxyuTWwbV6VNjMyMOFGUWhlbGqQxWS317fa7OdZSL+23mqXxZkotoDNIAbTDrbGb434eV8sbr9cfMaQQDRAtdpcUWKx2XGyy40KTZ98q7V9ssnses+GC1Q670y37NXRqBZL1GiTpNUjWqz37HW9T4tv2k/UaJOpUsXOBSZdTWhPnXKnUsuFdR8cpLR4Ip2frsN+ujLPVc79TuUAWHVSoPd0r/oLI8Ni6plVLgzTexW8rTIXUEhDLvIHGF2y8A8QTuwYeX9l46ecbSOBobYiNz0Cp8R9m2u8rVFIoEl2e23abu9P9LmVEqRWw83miS3rM9zyeMlcvAXL/PaRvkSGFaBASRRFWu0sKLO3CizfQtA86F6121FvtQXdZKQQgSd8+uKh9ISfJE2R8+/FqaJQKqJUKqJQCVAoF1EoBKqUCKoUAtTLAgceRJIqAy+EJLN4g027f7ZTG7RiyB0YXicsptQw5bZ4/Tq5Ot+7QHnfZuh8z5Rtb5Rln1VsXVqgpNdICjnHJQFySZz/J/7G45LZ9lVaqb6tnMUnfrbnjfb9lPLfhXo07GLe+Akz6z5A+ZURXnCWi2CAIAhK0KiRoVRg+pPfrF4miiCabEw3NDtQ3S4vtNTRL4aXOu++7taPe2rZmjVsE6qxSEAJkrsrnt+6AWuENMQI0KgVUnvtqT5hRKT3hpsN+261KKUCvUSItUYvUBG2XW4OuhwHI/iqk0kjbYFj5QKmSxsHEGu+MuA4Dwxv9z3TrroxK1zVQ9BQ41HHBd8PFJQf/Xt3utvfUW9AR3VL3meDpbhMUUp19XWqKTpuy7XHv1qWc0O652pUZGnuzbxlSiAYBQRCQqFMjUadGTkrga2/YnC6Ymx2o94Ybq9233zHYSMfMnpWGHW43nC7Rb+uNKAJ2lxtBDMEJmEalQFqCFqmJWqQlaLoNM2mJWsRrlIEHGgofQZBCgzpOarkayBQKqTtHZwCM0a5MbGNIIaJuaVVKpBuUSDcEN8BPFKWg4nSJbcHF5YbD7bl1iXC2CzSdjzlc0swqh8tbxvO4y40mmxMXmuw432TD+UYbLnhuG1udsDvdONvQgrMNvQ+y1akVbaHFF2za3SZofF1UIqSQBYhSl77vfUrv1fu46H1ERIdjYtthX3m0Ky89T/vnFTu8Bro8h3e/c33awmGX1/bcJuhUSIlvG5tkjFNDpYyhqd1EYEghojASBAFqpQC1EohDZMZztDpcvsByocneIcB0vrXapZlalXUtqKwL86yhfsDgCS5JnQZQe/dTvIOu49v2NarQBBtRFNFsd8HS6kBjqxOWFs9tqwOWTve9j7ffb7I5odcopTFSnvAlhTCN71hKvLrDfVndghQVDClENKDo1EoMTdZjaHLv3VrNdicuNHZtjWl/e6HJDpdb9A1dEARAgOC5he+PnOD5j/eY909f+/LoVF4Q/DzuOanj+X7OQ9sJ3mPSXcH3vO3vt39t7xglb/educUBAFIYaHUCF5sD+agBAAlaVdug6nYtM8meMKNTKdvChJ8A0v7W1cf1iJrtLlxo6mWqeDsqhdA20y1egxQ/AccX0OKlx3pccJFCjiGFiAYtvUaFYUNUGDZkcF8jx+mSrpflHVQthZee9xua7XCLQJPNiSabE2fqQ9MSpVIISNSpYIhTS7e69rdqGOJU0q1O5btv0KmRoFWhxeHyBa8678BwqzR+qs5T/zrPe2i2u+B0i54gapNdP6VCgEIQfLcKBaAUBCi8xz37SgWkx31l4dtXKgQIggClgHb7nnKesjqVEnEaJXRqJfQaJeLUSug0SujV0vE4z7E4z+M6z3G9RgmdShk7ywcEiSGFiGiQUykVGJKgxZCEwKc3ud0iGludUhhoN6i6odkbBKRQYHO6OoULdZcQYmh3P04dmYHM3qu01/mClzfUOHz3fY95jnmDTX2zI+z1CxWtSuELNx1CjUaFOLWi3b4S/5GfjUlDY2skL0MKERHJplAIMOrVMOrVGIHep8HHmmCu0t7qcKG+2Y7GVifcogiXW4TbDbhEEW5RhNvtOSai7XHPYy434HKLEEURLs9joigdc3nOdYtoty+VsTndaLG70OJwodnuQqvDhRa7C80OF1rtLjQ7nGjxjK1qtjvR4pD2vbyrYdej92CVPyyJIYWIiKg/0qmVyDTGITO2/o534XaLaHV6wow32HhCTos33Hj2vQGoxeHCaFPsXZSSIYWIiGgAUSgE6DUq6DUqDIl2ZfqIk+KJiIgoJjGkEBERUUxiSCEiIqKYxJBCREREMYkhhYiIiGISQwoRERHFJIYUIiIiikkMKURERBSTGFKIiIgoJjGkEBERUUxiSCEiIqKYxJBCREREMYkhhYiIiGJSv7gKsiiKAACLxRLlmhAREVGgvH+3vX/H5eoXIaWxsREAkJOTE+WaEBERkVyNjY0wGo2yzxPEYONNBLndbpw7dw6JiYkQBCFkz2uxWJCTk4PKykoYDIaQPW9/xM9Cws9Bws+hDT8LCT8HCT+HNoF8FqIoorGxEVlZWVAo5I8w6RctKQqFAkOHDg3b8xsMhkH/ZfPiZyHh5yDh59CGn4WEn4OEn0Ob3j6LYFpQvDhwloiIiGISQwoRERHFpEEdUrRaLZYvXw6tVhvtqkQdPwsJPwcJP4c2/Cwk/Bwk/BzaROKz6BcDZ4mIiGjwGdQtKURERBS7GFKIiIgoJjGkEBERUUxiSCEiIqKYNOBDyurVq5GbmwudToeCggJ89dVXPZZ/7733MG7cOOh0OkyaNAlbtmyJUE3DZ8WKFbjqqquQmJiI9PR0zJ07F8ePH+/xnA0bNkAQhA6bTqeLUI3D46mnnurynsaNG9fjOQPx+wAAubm5XT4LQRCwePFiv+UHyvdh9+7dKC4uRlZWFgRBwKZNmzo8Looili1bhszMTMTFxaGwsBAnTpzo9Xnl/jsTbT19Dg6HA0uXLsWkSZMQHx+PrKwszJ8/H+fOnevxOYP5/Yq23r4P99xzT5f3dMMNN/T6vP3t+wD0/ln4+/dCEAT88Y9/7PY5Q/GdGNAh5Z133sGSJUuwfPlyHDhwAHl5eZg9ezZqa2v9lv/nP/+JO++8Ez/+8Y9x8OBBzJ07F3PnzsW3334b4ZqH1qefforFixfjiy++wLZt2+BwOHD99dfDarX2eJ7BYEBVVZVvKy8vj1CNw+fSSy/t8J4+//zzbssO1O8DAHz99dcdPodt27YBAG677bZuzxkI3wer1Yq8vDysXr3a7+N/+MMf8Je//AVr1qzBl19+ifj4eMyePRutra3dPqfcf2diQU+fQ3NzMw4cOIAnn3wSBw4cwAcffIDjx4/j5ptv7vV55fx+xYLevg8AcMMNN3R4T2+99VaPz9kfvw9A759F+8+gqqoK69atgyAIuPXWW3t83j5/J8QBbMqUKeLixYt9910ul5iVlSWuWLHCb/nbb79dvOmmmzocKygoEH/yk5+EtZ6RVltbKwIQP/30027LrF+/XjQajZGrVAQsX75czMvLC7j8YPk+iKIoPvLII+KoUaNEt9vt9/GB+H0AIG7cuNF33+12ixkZGeIf//hH37GGhgZRq9WKb731VrfPI/ffmVjT+XPw56uvvhIBiOXl5d2Wkfv7FWv8fQ4LFiwQ58yZI+t5+vv3QRQD+07MmTNHnDlzZo9lQvGdGLAtKXa7Hfv370dhYaHvmEKhQGFhIfbu3ev3nL1793YoDwCzZ8/utnx/ZTabAQApKSk9lmtqasLw4cORk5ODOXPm4PDhw5GoXlidOHECWVlZGDlyJObNm4eKiopuyw6W74Pdbsfrr7+Oe++9t8cLeA7E70N7ZWVlqK6u7vAzNxqNKCgo6PZnHsy/M/2R2WyGIAhISkrqsZyc36/+YteuXUhPT8fYsWPx4IMP4uLFi92WHSzfh5qaGmzevBk//vGPey3b1+/EgA0pFy5cgMvlgslk6nDcZDKhurra7znV1dWyyvdHbrcbjz76KKZPn46JEyd2W27s2LFYt24d/va3v+H111+H2+3GtGnTcObMmQjWNrQKCgqwYcMGfPTRR3jxxRdRVlaGq6++Go2NjX7LD4bvAwBs2rQJDQ0NuOeee7otMxC/D515f65yfubB/DvT37S2tmLp0qW48847e7yInNzfr/7ghhtuwGuvvYYdO3bg2WefxaeffoqioiK4XC6/5QfD9wEAXn31VSQmJuKWW27psVwovhP94irIFDqLFy/Gt99+22u/4NSpUzF16lTf/WnTpmH8+PF46aWX8Nvf/jbc1QyLoqIi3/5ll12GgoICDB8+HO+++25A/0cwUL3yyisoKipCVlZWt2UG4veBeudwOHD77bdDFEW8+OKLPZYdiL9fP/rRj3z7kyZNwmWXXYZRo0Zh165dmDVrVhRrFl3r1q3DvHnzeh08H4rvxIBtSUlNTYVSqURNTU2H4zU1NcjIyPB7TkZGhqzy/c1DDz2Ef/zjH9i5cyeGDh0q61y1Wo38/HycPHkyTLWLvKSkJIwZM6bb9zTQvw8AUF5eju3bt2PRokWyzhuI3wfvz1XOzzyYf2f6C29AKS8vx7Zt23psRfGnt9+v/mjkyJFITU3t9j0N5O+D12effYbjx4/L/jcDCO47MWBDikajweTJk7Fjxw7fMbfbjR07dnT4P8L2pk6d2qE8AGzbtq3b8v2FKIp46KGHsHHjRnzyyScYMWKE7OdwuVw4dOgQMjMzw1DD6GhqasL333/f7XsaqN+H9tavX4/09HTcdNNNss4biN+HESNGICMjo8PP3GKx4Msvv+z2Zx7MvzP9gTegnDhxAtu3b8eQIUNkP0dvv1/90ZkzZ3Dx4sVu39NA/T6098orr2Dy5MnIy8uTfW5Q34k+DbuNcW+//bao1WrFDRs2iEeOHBHvv/9+MSkpSayurhZFURTvvvtu8YknnvCV37Nnj6hSqcQ//elP4tGjR8Xly5eLarVaPHToULTeQkg8+OCDotFoFHft2iVWVVX5tubmZl+Zzp/F008/LW7dulX8/vvvxf3794s/+tGPRJ1OJx4+fDgabyEkfv7zn4u7du0Sy8rKxD179oiFhYViamqqWFtbK4ri4Pk+eLlcLnHYsGHi0qVLuzw2UL8PjY2N4sGDB8WDBw+KAMSVK1eKBw8e9M1a+f3vfy8mJSWJf/vb38R//etf4pw5c8QRI0aILS0tvueYOXOm+Nxzz/nu9/bvTCzq6XOw2+3izTffLA4dOlQsLS3t8G+GzWbzPUfnz6G3369Y1NPn0NjYKD7++OPi3r17xbKyMnH79u3iFVdcIY4ePVpsbW31PcdA+D6IYu+/G6IoimazWdTr9eKLL77o9znC8Z0Y0CFFFEXxueeeE4cNGyZqNBpxypQp4hdffOF77JprrhEXLFjQofy7774rjhkzRtRoNOKll14qbt68OcI1Dj0Afrf169f7ynT+LB599FHf52YymcQbb7xRPHDgQOQrH0J33HGHmJmZKWo0GjE7O1u84447xJMnT/oeHyzfB6+tW7eKAMTjx493eWygfh927tzp93fB+17dbrf45JNPiiaTSdRqteKsWbO6fD7Dhw8Xly9f3uFYT//OxKKePoeysrJu/83YuXOn7zk6fw69/X7Fop4+h+bmZvH6668X09LSRLVaLQ4fPly87777uoSNgfB9EMXefzdEURRfeuklMS4uTmxoaPD7HOH4TgiiKIqy22yIiIiIwmzAjkkhIiKi/o0hhYiIiGISQwoRERHFJIYUIiIiikkMKURERBSTGFKIiIgoJjGkEBERUUxiSCEiIqKYxJBCREREMYkhhYiIiGISQwoRERHFJIYUIiIiikn/PwRC95P3qPDSAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "loss, val_loss = history.history['loss'], history.history['val_loss']\n",
        "plt.plot(loss, label='loss')\n",
        "plt.plot(val_loss, label='val_loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GjKYJOc_Byoq"
      },
      "source": [
        "Printing Classification Report and Confusion Matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xu81uyjK_uuk",
        "outputId": "834ec835-483e-4856-955f-511f20b18076"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10/10 [==============================] - 1s 8ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.82      1.00      0.90       241\n",
            "           1       0.00      0.00      0.00        53\n",
            "\n",
            "    accuracy                           0.82       294\n",
            "   macro avg       0.41      0.50      0.45       294\n",
            "weighted avg       0.67      0.82      0.74       294\n",
            "\n",
            "[[241   0]\n",
            " [ 53   0]]\n",
            "0.8197278911564626\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        }
      ],
      "source": [
        "predictions = model.predict(testing_padded)\n",
        "predictions = (predictions > 0.5)*1\n",
        "print(classification_report(df_r_test['is_ecology'].to_numpy(), predictions))\n",
        "print(confusion_matrix(df_r_test['is_ecology'].to_numpy(), predictions))\n",
        "print(accuracy_score(df_r_test['is_ecology'].to_numpy(), predictions))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-pVvpPAUB3dU"
      },
      "source": [
        "As you can see the result is not the best, the unbalanced dataset has really affected our results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "up-2P2mcCQLR"
      },
      "source": [
        "# Bert Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0JroAxgtBnq_"
      },
      "outputs": [],
      "source": [
        "bert_X_train, bert_X_valid, bert_Y_train, bert_Y_valid = train_test_split(df_r_train['article_title'], df_r_train['is_ecology'],test_size=0.2,random_state=42,stratify=df_r_train['is_ecology'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9_9tifFFCfjl"
      },
      "source": [
        "Encoding raw test for training validation and test."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RWxwDsAfCfjl",
        "outputId": "cd22a883-1a4f-45b0-b76d-72d825c0d2e0"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 7/7 [00:00<00:00, 34.57it/s]\n",
            "100%|██████████| 2/2 [00:00<00:00, 37.82it/s]\n",
            "100%|██████████| 2/2 [00:00<00:00, 56.04it/s]\n"
          ]
        }
      ],
      "source": [
        "X_train = fast_encode(bert_X_train, fast_tokenizer, maxlen=max_length)\n",
        "X_valid = fast_encode(bert_X_valid, fast_tokenizer, maxlen=max_length)\n",
        "X_test = fast_encode(df_r_test['article_title'], fast_tokenizer, maxlen=max_length)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CWPwh8jvCfjm",
        "outputId": "c37fd4b7-3cb5-4a39-f93c-aafbaf634c36"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(1772, 100)"
            ]
          },
          "execution_count": 130,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t1L2xN-WCfjm"
      },
      "source": [
        "Encoding the labels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7s3A_2TESd2"
      },
      "source": [
        "Creating TensorFlow Datasets with our data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ib1oGkhcCfjm"
      },
      "outputs": [],
      "source": [
        "\n",
        "y_train_encoded = bert_Y_train\n",
        "y_valid_encoded = bert_Y_valid\n",
        "y_test_encoded = df_r_test['is_ecology']\n",
        "\n",
        "\n",
        "\n",
        "BATCH_SIZE = 16\n",
        "\n",
        "train_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_train, y_train_encoded))\n",
        "    .repeat()\n",
        "    .shuffle(2048)\n",
        "    .batch(BATCH_SIZE)\n",
        "\n",
        ")\n",
        "\n",
        "valid_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_valid, y_valid_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        "    .cache()\n",
        "\n",
        ")\n",
        "\n",
        "test_dataset = (\n",
        "    tf.data.Dataset\n",
        "    .from_tensor_slices((X_test,y_test_encoded))\n",
        "    .batch(BATCH_SIZE)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OeJ9tulPCfjo"
      },
      "source": [
        "Building the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q_vz-_cnCfjo"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.optimizers.experimental import Adam\n",
        "\n",
        "def build_model(transformer, max_len=512):\n",
        "    \"\"\"\n",
        "    function for training the BERT model\n",
        "    \"\"\"\n",
        "    input_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n",
        "    sequence_output = transformer(input_word_ids)[0]\n",
        "    cls_token = sequence_output[:, 0, :]\n",
        "    out = Dense(1, activation='sigmoid')(cls_token)\n",
        "\n",
        "    model = tf.keras.Model(inputs=input_word_ids, outputs=out)\n",
        "    model.compile(Adam(learning_rate=1e-5), loss='binary_crossentropy', metrics=['accuracy', Precision(), Recall(), AUC()])\n",
        "\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ChwN0_htCfjo",
        "outputId": "1618ffa4-102a-42c7-8c78-0b1aaceb9b3d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFDistilBertModel: ['vocab_layer_norm.bias', 'vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_projector.bias']\n",
            "- This IS expected if you are initializing TFDistilBertModel from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing TFDistilBertModel from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "All the weights of TFDistilBertModel were initialized from the PyTorch model.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
          ]
        }
      ],
      "source": [
        "transformer_layer = (\n",
        "    transformers.TFDistilBertModel.from_pretrained('distilbert-base-multilingual-cased')\n",
        ")\n",
        "\n",
        "\n",
        "bert_model = build_model(transformer_layer, max_len=max_length)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "srN7zJzgCfjp",
        "outputId": "9013df7a-3420-41d6-f0b4-f4f185f53b87"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_3\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_word_ids (InputLayer)  [(None, 100)]            0         \n",
            "                                                                 \n",
            " tf_distil_bert_model_3 (TFD  TFBaseModelOutput(last_h  134734080\n",
            " istilBertModel)             idden_state=(None, 100,             \n",
            "                             768),                               \n",
            "                              hidden_states=None, att            \n",
            "                             entions=None)                       \n",
            "                                                                 \n",
            " tf.__operators__.getitem_3   (None, 768)              0         \n",
            " (SlicingOpLambda)                                               \n",
            "                                                                 \n",
            " dense_35 (Dense)            (None, 1)                 769       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 134,734,849\n",
            "Trainable params: 134,734,849\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "bert_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TRql7QMKCfjp"
      },
      "source": [
        "Freezing all other layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bueu3OpICfjp"
      },
      "outputs": [],
      "source": [
        "bert_model.layers[0].trainable = False\n",
        "bert_model.layers[1].trainable = False\n",
        "bert_model.layers[2].trainable = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8a-anHWCfjp"
      },
      "source": [
        "Now that we have frozen the layers, we can see that the trainable params (thankfully) have reduced significantly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w4OUq3fqCfjp",
        "outputId": "44054c60-32b1-4535-f751-bb774dd2c31d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_3\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_word_ids (InputLayer)  [(None, 100)]            0         \n",
            "                                                                 \n",
            " tf_distil_bert_model_3 (TFD  TFBaseModelOutput(last_h  134734080\n",
            " istilBertModel)             idden_state=(None, 100,             \n",
            "                             768),                               \n",
            "                              hidden_states=None, att            \n",
            "                             entions=None)                       \n",
            "                                                                 \n",
            " tf.__operators__.getitem_3   (None, 768)              0         \n",
            " (SlicingOpLambda)                                               \n",
            "                                                                 \n",
            " dense_35 (Dense)            (None, 1)                 769       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 134,734,849\n",
            "Trainable params: 769\n",
            "Non-trainable params: 134,734,080\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "bert_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EW4v1FRGCfjp"
      },
      "source": [
        "Now we can train the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6oe6g3C7Cfjq",
        "outputId": "f26e8e25-59b6-4cba-bd08-6177e6653c65"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "110/110 [==============================] - 59s 276ms/step - loss: 0.1164 - accuracy: 0.9710 - precision_5: 0.0625 - recall_5: 0.0270 - auc_5: 0.5255 - val_loss: 0.1139 - val_accuracy: 0.9774 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00 - val_auc_5: 0.3711\n",
            "Epoch 2/3\n",
            "110/110 [==============================] - 24s 215ms/step - loss: 0.1231 - accuracy: 0.9744 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - auc_5: 0.5033 - val_loss: 0.1082 - val_accuracy: 0.9774 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00 - val_auc_5: 0.4251\n",
            "Epoch 3/3\n",
            "110/110 [==============================] - 22s 202ms/step - loss: 0.1198 - accuracy: 0.9750 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - auc_5: 0.5199 - val_loss: 0.1093 - val_accuracy: 0.9774 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00 - val_auc_5: 0.4326\n"
          ]
        }
      ],
      "source": [
        "n_steps = X_train.shape[0] // BATCH_SIZE\n",
        "train_history = bert_model.fit(\n",
        "    train_dataset,\n",
        "    steps_per_epoch=n_steps,\n",
        "    validation_data=valid_dataset,\n",
        "    epochs=3\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7azBMKQ6FD17"
      },
      "source": [
        "Plotting Losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "eUjCHc2XEk3k",
        "outputId": "6da766f9-3c47-45a5-96bf-2083da575191"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQxUlEQVR4nO3deViU5f4/8Pew7yggm6ID4r6gKUxYCRaKS6YnsuWYkufYqhRRpnRS63vqYOkpjsnJ6ldSnRazxEzJTBItQy2QRFBSQ0VWcWFVUOb+/TEwMjAsAzPM9n5d11zBM/c8c9880Lz9zOd5RiKEECAiIiIychb6ngARERGRNjDUEBERkUlgqCEiIiKTwFBDREREJoGhhoiIiEwCQw0RERGZBIYaIiIiMgkMNURERGQSrPQ9gd4il8tRXFwMZ2dnSCQSfU+HiIiIukAIgerqavj6+sLCouNajNmEmuLiYvj5+el7GkRERNQNhYWFGDBgQIdjzCbUODs7A1D8UFxcXPQ8GyIiIuqKqqoq+Pn5KV/HO2I2oab5LScXFxeGGiIiIiPTldYRNgoTERGRSWCoISIiIpPAUENEREQmwWx6aoiIiIQQuHHjBhobG/U9FWrB0tISVlZWPb7kCkMNERGZhYaGBpSUlKCurk7fUyE1HBwc4OPjAxsbm27vg6GGiIhMnlwuR0FBASwtLeHr6wsbGxteiNVACCHQ0NCACxcuoKCgAEOGDOn0InvtYaghIiKT19DQALlcDj8/Pzg4OOh7OtSKvb09rK2tcfbsWTQ0NMDOzq5b+2GjMBERmY3uVgBI97RxbHh0iYiIyCQw1BAREZFJYKghIiIyYOHh4YiNjdX3NIwCQw0RERGZBJ79RERG71R5NX7IK8cIH2fcFugBa0v+e43IHPEvn4iM1o1GOd5JP42Z63/G67tO4JFNv2Liq3uwbMvv2JtfjoYbcn1PkQyYEAJ1DTd6/SaE6PacL1++jIULF6Jv375wcHDAjBkzcPLkSeX9Z8+exezZs9G3b184Ojpi1KhRSE1NVT52/vz56NevH+zt7TFkyBBs2rSpxz9HQ8JKDREZpVPl1Xhuy1H8XngFADDOrw/OX65DRU0DtmSex5bM83C1t8a0kV6YOdYHtw32gI0V/x1HN1293oiRq77v9efN+79IONh07+X3kUcewcmTJ7F9+3a4uLhg+fLlmDlzJvLy8mBtbY0lS5agoaEB+/fvh6OjI/Ly8uDk5AQAWLlyJfLy8vDdd9/Bw8MDp06dwtWrV7W5NL1jqCEio3KjUY73fyrAW3v+QMMNOZztrLDy7pGYN2EA5AI4VHARqTkl2HWsVCXguNhZYdoob8xiwCEj1RxmDhw4gEmTJgEAPv30U/j5+WHbtm2YN28ezp07h6ioKIwZMwYAEBAQoHz8uXPnMH78eEycOBEAIJVKe30NusZQQ0RG41R5NZ7fchTZTdWZ8GH9kHDvGPi42gMALCXApMEemDTYA6/cMxqHCy5hZ04xdh0rQ0VNPb7KPI+vWgacMT64LZABx1zZW1si7/8i9fK83XH8+HFYWVlBJpMpt7m7u2PYsGE4fvw4AODpp5/Gk08+id27dyMiIgJRUVEYO3YsAODJJ59EVFQUsrKyMG3aNMydO1cZjkwFQw0RGbxGucD7P/2JN39oqs7YWmHlbEV1pr3P77G0kCB0sDtCB7srA05qTgm+O1baJuBMHemNu8cy4JgbiUTS7beBDNXixYsRGRmJnTt3Yvfu3UhISMC///1vxMTEYMaMGTh79ixSU1Pxww8/4K677sKSJUuwbt06fU9baySiJx1LRqSqqgqurq6orKyEi4uLvqdDRF10qrwGz2/5XVmdCRvaD2uiblZnNNUoF/j1zCXsPHoz4DRrDjizxnrj9sB+DDgm5Nq1aygoKIC/v3+3P1dIX8LDwzFu3DgsWbIEQ4cOVXn76eLFi/Dz88PHH3+M++67r81j4+PjsXPnThw9erTNfe+++y6WLVuGqqoqna+hK9o7Rpq8fptWRCUik9EoF/h/P/2Jf7esztw9EvMmtl+d6QpLCwluDXDHrQHuePmeUfj1zM0KzoXqenyddR5fZ52Hs50Vpo70wt1jfRhwyCAMGTIEc+bMwaOPPop3330Xzs7OWLFiBfr37485c+YAAGJjYzFjxgwMHToUly9fxt69ezFixAgAwKpVqzBhwgSMGjUK9fX12LFjh/I+U8FQQ0QGR111JuHeMfDt073qTHtaBpzVs0fhtzOXsLNFwNmaVYStWUXKgDNrjA9uH+IBW6vu9UQQ9dSmTZvwzDPP4O6770ZDQwMmT56M1NRUWFtbAwAaGxuxZMkSnD9/Hi4uLpg+fTreeustAICNjQ3i4+Nx5swZ2Nvb44477sAXX3yhz+VoHd9+IiKDoavqTHfm8VuLCk559c23qJztrDB1hBdmjWXAMSbG/PaTueDbT0RkMk6V12DZV7/jyLkrAIDJQ/thjQ6qM11haSGBLMAdsgB3rJo9CplnL2Pn0WJlwNl6pAhbjxTB2VZRwZk5xgd3DGXAIdI3hhoi0it11ZmX7h6B+yf69Wp1pj2WFhKE+LshxN9N8RbV2ctIzSlBak5Jm4AT0fQWFQMOkX4w1BCR3hhSdaYrLFoEnFV3j0TmucvYefRmwEk5UoSUFgFn5hgf3DHEA3bdvC4JEWmGoYaIel2jXOCDn//Eut2GWZ3pCgsLCYKlbgiWqgac746VoKzqZsBxavkWFQMOkU4x1BBRrzK26kxXtA44WecuY0c7ASdihCdmjvHB5KH9GHCItIyhhoh6hSlUZ7rCwkKCiVI3TGwRcHbmlOC7nFKUVl3DtuxibMsuZsAh0gGGGiLSudMXarBsy+/IaqrO3DHEA69HjTXq6kxXtAw4K2eNxJHCpgqOmoBzV1PACWPAIeo2hhoi0plGucCHPxdg3e581N+Qw8nWCi/NGoEHgk2rOtMVFhYSTBjkhgmDbgacnUdL8d2xEpRUXsM32cX4JrsYjjaWyiZjBhwizTDUEJFOqKvOrIkai/4mXp3pipYB56VZI3Ck8Iqyybh1wLmr6UJ/DDhEnWOoISKtYnVGM4qA0xcTBvVVBpzm6+CUVF7D9t+Lsf33mwFn5hgfhA9jwKGuk0qliI2NRWxsbKdjJRIJUlJSMHfuXJ3PSxcYaohIa05fqMELXx1F5tnLAFid0VTLgPOPmTcDznc5JShuFXDuHKG40B8DDtFNDDVE1GPqqjP/mDUCD7I6022tA072+StIbbrQX3HlNXz7ezG+VQk43ggf5smAQ2bNojsPSkpKglQqhZ2dHWQyGQ4fPtzu2NzcXERFRUEqlUIikSAxMbHNmISEBAQHB8PZ2Rmenp6YO3cu8vPzlfdfunQJMTExGDZsGOzt7TFw4EA8/fTTqKys7M70iUiL/rxQg/vfzcBrqcdRf0OOO4Z44PtnJ+OhkIEMNFpiYSHBLQP74qW7R+Ln5Xdi61OTsPh2f/TvY4/ahkZ8+3sxnvhfFm755w9Y+lkWdh0rwbXrjfqetuETAmio7f2bBp8j/d5778HX1xdyuVxl+5w5c/C3v/0Np0+fxpw5c+Dl5QUnJycEBwdjz549WvsR5eTk4M4774S9vT3c3d3x2GOPoaamRnl/eno6QkJC4OjoiD59+uC2227D2bNnAQC///47pkyZAmdnZ7i4uGDChAn47bfftDY3dTSu1GzevBlxcXHYuHEjZDIZEhMTERkZifz8fHh6erYZX1dXh4CAAMybNw/PPvus2n3u27cPS5YsQXBwMG7cuIEXX3wR06ZNQ15eHhwdHVFcXIzi4mKsW7cOI0eOxNmzZ/HEE0+guLgYX331learJqIea5QLbDpQgLXfszrTm5oDzi0D++Ifs0YgW9mDU4qiK1ex42gJdhwtgYONJe4c7tn0FpUn7G1YwWnjeh3wL9/ef94XiwEbxy4NnTdvHmJiYrB3717cddddABT/0N+1axdSU1NRU1ODmTNn4rXXXoOtrS0+/vhjzJ49G/n5+Rg4cGCPpllbW4vIyEiEhobi119/RXl5ORYvXoylS5ciOTkZN27cwNy5c/Hoo4/i888/R0NDAw4fPqz8+58/fz7Gjx+Pd955B5aWlsjOzoa1tXWP5tQZiRAaREYAMpkMwcHB2LBhAwBALpfDz88PMTExWLFiRYeP7Wqz0oULF+Dp6Yl9+/Zh8uTJasds2bIFDz/8MGpra2Fl1Xk20+Sjy4moY39eqMEy9s4YFCFEm4DTzMHGElOGe+JuMw44165dQ0FBAfz9/WFnZ6fY2FBr8KEGAObOnQt3d3d88MEHABTVm1deeQWFhYWwsGj7hsvo0aPxxBNPYOnSpQC63yj8/vvvY/ny5SgsLISjo2K+qampmD17NoqLi2FtbQ13d3ekp6cjLCyszb5cXFzw9ttvIzo6ukvrVHuMoNnrt0aVmoaGBmRmZiI+Pl65zcLCAhEREcjIyNBkVx1qflvJzc2twzEuLi7tBpr6+nrU19crv6+qqtLa/IjMVevqjKONJf4xayQeCmF1Rt8kEgnGD+yL8QP74sWZI/D7+Uqk5pRg59ESFF25ip1HFV/bW1vizhGKCs4UMw04StYOioChj+fVwPz58/Hoo4/iv//9L2xtbfHpp5/iwQcfhIWFBWpqavDyyy9j586dKCkpwY0bN3D16lWcO3eux9M8fvw4goKClIEGAG677TbI5XLk5+dj8uTJeOSRRxAZGYmpU6ciIiIC999/P3x8fAAAcXFxWLx4MT755BNERERg3rx5GDx4cI/n1RGNemoqKirQ2NgILy8vle1eXl4oLS3VyoTkcjliY2Nx2223YfTo0e3O45///Ccee+yxdveTkJAAV1dX5c3Pz08r8yMyV39eqMED72bg1Z2K3pnbAxW9M3+VsXfG0EgkEozz64MXZ47Az8unYNuS2/DY5AD072OPq9cbsfNoCZ76VNGDs+SzLKTmlOBqgxn24EgkiopJb980/HuZPXs2hBDYuXMnCgsL8dNPP2H+/PkAgOeffx4pKSn417/+hZ9++gnZ2dkYM2YMGhoadPETa2PTpk3IyMjApEmTsHnzZgwdOhQHDx4EALz88svIzc3FrFmz8OOPP2LkyJFISUnR6XwM7uynJUuW4NixY/j555/V3l9VVYVZs2Zh5MiRePnll9vdT3x8POLi4lQex2BDpDlWZ4xbc8AZ59cH8TOG42hTBWeHugrOcMVHNUwZ3g8ONgb38mC27OzscO+99+LTTz/FqVOnMGzYMNxyyy0AgAMHDuCRRx7BX/7yFwBATU0Nzpw5o5XnHTFiBJKTk1FbW6us1hw4cAAWFhYYNmyYctz48eMxfvx4xMfHIzQ0FJ999hluvfVWAMDQoUMxdOhQPPvss3jooYewadMm5Vx1QaPfWg8PD1haWqKsrExle1lZGby9vXs8maVLl2LHjh3Yv38/BgwY0Ob+6upqTJ8+Hc7OzkhJSemw4cjW1ha2trY9nhOROSuoqMWyLb/jt6bemdsDPbAmagwG9NWsfE6GQSKRIMivD4L8+mBFi4CzM6cE5y9fxc6mrxlwDM/8+fNx9913Izc3Fw8//LBy+5AhQ7B161bMnj0bEokEK1eubHOmVE+ec/Xq1YiOjsbLL7+MCxcuICYmBgsWLICXlxcKCgrw3nvv4Z577oGvry/y8/Nx8uRJLFy4EFevXsWyZctw3333wd/fH+fPn8evv/6KqKgorcytPRr9ptrY2GDChAlIS0tTXm1QLpcjLS1N2ZDUHUIIxMTEICUlBenp6fD3928zpqqqCpGRkbC1tcX27dtVmoiISLtYnTF9rQNOTlGlomqjJuBMGd4PM8f44M7hngw4enLnnXfCzc0N+fn5+Otf/6rc/uabb+Jvf/sbJk2aBA8PDyxfvlxrPaQODg74/vvv8cwzzyA4OBgODg6IiorCm2++qbz/xIkT+Oijj3Dx4kX4+PhgyZIlePzxx3Hjxg1cvHgRCxcuRFlZGTw8PHDvvffilVde0crc2qPx2U+bN29GdHQ03n33XYSEhCAxMRFffvklTpw4AS8vLyxcuBD9+/dHQkICAEVzcV5eHgBg5syZmD9/PubPnw8nJycEBgYCAJ566il89tln+Oabb1RKWq6urrC3t0dVVRWmTZuGuro6pKSkqDQt9evXD5aWnTe68ewnoq5hdca8CSEUAafpoxoKL908i8rO2kJZwTG2gNPemTVkOLRx9pPGoQYANmzYgLVr16K0tBTjxo3D+vXrIZPJAADh4eGQSqVITk4GAJw5c0Zt5SUsLAzp6emKSbTzL79NmzbhkUceQXp6OqZMmaJ2TEFBAaRSaadzZqgh6phcLrDplzNY+/0JXLuuqM68OGsE/sqL6JktIQSOFVVhR06x2oAzZZgnZo01joDDUGP49BZqjBFDDVH7Cipq8cJXv+PXM4rqzG2B7ng9aiyrM6TUHHCaKzjnLtUp72sOOM0VHEdbwws4DDXAp59+iscff1ztfYMGDUJubm4vz0gVQ40GGGqI2mJ1hrpDCIHc4irsOKo+4IQPvVnBMZSAw1CjONmm9Yk+zaytrTFo0KBenpGqXr/4HhGZjjMVtVjWqjqz5t6x8HNjdYY6JpFIMLq/K0b3d8Xy6cOQW3yzgnP2Yh125ZZiV24pbK2aKjhjfXCXAQUcc+Xs7AxnZ2d9T0On+BtGZGbkcoHkX87gjRbVmfiZIzCfF9GjbmgZcF6I7DjghA/rh1ljffUacMzkzQmjpI1jw1BDZEZaV2cmDVb0zrA6Q9qgLuCkNgWcMxfr8H1uGb7PLVMGnJljfHDXCC849ULAab6uWV1dHezt+RllhqiuTvE2Zk8+9JI9NURmgNUZ0ichBPJKqrDz6M2A06w3A05JSQmuXLkCT09PODg48HffQAghUFdXh/LycvTp00f52VHN2CisBkMNmaszFbV44aujOHzmEgBWZ0i/mgNO86eJF1TUKu+zsbJA+NB+mDVWNwFHCIHS0lJcuXJFq/sl7ejTpw+8vb3bhE2GGjUYasjctK7OODRXZ0IGwsKC/0Il/RNC4HhJNXbmFPdqwGlsbMT169e1tj/qOWtr63YvpMtQowZDDZkTVmfI2DQHnOYenD9bBZywof0wa4wP7hrhCWe77vdckPFhqFGDoYbMgVwu8FHGGby+i9UZMl5CCJworVb24DDgmDeGGjUYasjUnb1Yi2VfHcXhAkV1JjTAHW/cx+oMGbfmgJOaU4KdR9sGnMlD+mHWWG9EjPBiwDFRDDVqMNSQqZLLBT7OOIPXd+Xj6vVGVmfIZKkEnJwS/HmhRcCxtMDkoQw4poihRg2GGjJFrM6QuRJCIL+sGqlHS7BDbcDxwMwxPogY6QUXBhyjxlCjBkMNmRK11ZkZwzFfNojVGTI7LQPOzpwSnGbAMSkMNWow1JCpOHtRcWbToabqzK0Bblh7XxCrM0RQBJw/ymqwM6cEO48Wtwk4dwzxwKyxDDjGhKFGDYYaMnaszhBppmXASc0pwanyGuV9zQGnuYLjas+AY6gYatRgqCFjdu5iHZZ99btKdeaNqCAMdGd1hqir/ihTnCa+s1XAsbaU4I4hitPEGXAMD0ONGgw1ZIzkcoFPDp7Fmu9OKKszK2YMx8OszhD1SHPASc0pwUk1AWfmGB9MZcAxCAw1ajDUkLFhdYaod5wsq27qwWkbcG4P9MCssb4MOHrEUKMGQw0Zi9bVGXtrS8TPZHWGqDc0B5zUnBL8UdY24Mwc44NpI73h6sCA01sYatRgqCFjcO5iHV74+ncc/FNRnZH5K85sYnWGqPcx4BgGhho1GGrIkMnlAv87pKjO1DUoqjMrZgzHgltZnSEyBKfKq7HzaClSc0qQX1at3G5tKcFtTQEnkgFHJxhq1GCoIUOlrjrzxn1jMcjdUc8zIyJ12gs4VhYS3D6EAUfbGGrUYKghQ8PqDJHxO1Veg9Smt6hOlKoGnNsCPTBrjA+mjfJCHwcbPc7SuDHUqMFQQ4ak8JLizCZWZ4hMBwOObjDUqMFQQ4aA1Rki83D6Qo3ys6haB5xJgR6YNcYb00Z6o68jA05nGGrUYKghfSu8VIcXvjqKjD8vAgBC/N2wltUZIpPHgNMzDDVqMNSQvsjlAp8eOouEFtWZ5dOHYWGolNUZIjPz5wXFW1Q7c0pxvKRKud3KQoLQwe64e6wPA04rDDVqMNSQPrA6Q0TtaS/gWFpIMGmwO2aN8UHkKAYchho1GGqoN8nlAp8ePoeE1OOszhBRp/68UIPvjpVi59ES5LUTcKaN8oabGQYchho1GGqot7A6Q0Q9UVBRq6jgtBNwZjZVcMwl4DDUqMFQQ7rWujpjZ22B5dOHI5rVGSLqpuaAk5pTgtxi1YATGuCOWWNNP+Aw1KjBUEO6VHipDsu/PopfTjdVZ6SK685IPVidISLtOFNRq/wsKnUBR1HB8YK7k60eZ6l9DDVqMNSQLrA6Q0T6cKaiFqnHFAHnWJFpBxyGGjUYakjbWJ0hIkNw9uLNCk7rgHNrgBtmjvHB9FHeRhtwNHn9tujOEyQlJUEqlcLOzg4ymQyHDx9ud2xubi6ioqIglUohkUiQmJjYZkxCQgKCg4Ph7OwMT09PzJ07F/n5+Spjrl27hiVLlsDd3R1OTk6IiopCWVlZd6ZP1CNCCPzv4FlMT9yPX05fhJ21BVbdPRJfPHYrAw0R9bpB7o54KjwQO2LuwL5l4Vg+fThG93dBo1zgwKmL+EfKMQS/tgfz/99BfHroLCpq6vU9ZZ3RONRs3rwZcXFxWL16NbKyshAUFITIyEiUl5erHV9XV4eAgACsWbMG3t7easfs27cPS5YswcGDB/HDDz/g+vXrmDZtGmpra5Vjnn32WXz77bfYsmUL9u3bh+LiYtx7772aTp+oRwov1WH+/zuEl7YdQ21DI0Kkbtj1zGT87XZ/vt1ERHo3yN0RT4YPVgk4Y/q7Qi6gDDghr+3BX98/iP8dNL2Ao/HbTzKZDMHBwdiwYQMAQC6Xw8/PDzExMVixYkWHj5VKpYiNjUVsbGyH4y5cuABPT0/s27cPkydPRmVlJfr164fPPvsM9913HwDgxIkTGDFiBDIyMnDrrbd2Om++/UQ9IYTAp4cUvTO1Tb0zL0QOxyOT2DtDRIbv3MU6pB5TnCaeU1Sp3G4hAW5t6sGZPtobHgb4FpUmr99Wmuy4oaEBmZmZiI+PV26zsLBAREQEMjIyujdbNSorFT9wNzc3AEBmZiauX7+OiIgI5Zjhw4dj4MCB7Yaa+vp61NffTKBVVVVtxhB1xfnLit6ZA6cUvTPB0r54474g+POtJiIyEgPdHfBE2GA8ETZYGXBSc0pw9Hwlfjl9Eb+cvohV3xyDzN8dM8cqenD6ORtewOmMRqGmoqICjY2N8PLyUtnu5eWFEydOaGVCcrkcsbGxuO222zB69GgAQGlpKWxsbNCnT582z1taWqp2PwkJCXjllVe0MicyT0IIfHb4HP61k9UZIjIdLQNO4aW6po9qUAScjD8vIuPPi1htpAFHo1DTG5YsWYJjx47h559/7tF+4uPjERcXp/y+qqoKfn5+PZ0emQlWZ4jIHPi5OeDxsMF4vEXASc0pwe+tAk6Iv5vis6hGe8PT2U7f026XRqHGw8MDlpaWbc46Kisra7cJWBNLly7Fjh07sH//fgwYMEC53dvbGw0NDbhy5YpKtaaj57W1tYWtrXEkSzIc6qozy5qqM5aszhCRCWsdcL5r6sH5/XwlDv55CQf/vIRV23MhM+CAo1GosbGxwYQJE5CWloa5c+cCULxdlJaWhqVLl3Z7EkIIxMTEICUlBenp6fD391e5f8KECbC2tkZaWhqioqIAAPn5+Th37hxCQ0O7/bxELZ2/XIcVX+fg51MVAICJg/pi7TxWZ4jI/Pi5OeCxyYPx2OQWASenFL8XXlEJOCFSN9w91nACjsZvP8XFxSE6OhoTJ05ESEgIEhMTUVtbi0WLFgEAFi5ciP79+yMhIQGAork4Ly9P+XVRURGys7Ph5OSEwMBAAIq3nD777DN88803cHZ2VvbJuLq6wt7eHq6urvj73/+OuLg4uLm5wcXFBTExMQgNDe3SmU9EHWF1hoiofR0FnEMFl3Co4GbAmTXWB1G3DICjrX66W7p1ReENGzZg7dq1KC0txbhx47B+/XrIZDIAQHh4OKRSKZKTkwEAZ86caVN5AYCwsDCkp6crJiFR/8KxadMmPPLIIwAUF9977rnn8Pnnn6O+vh6RkZH473//2+W3vXhKN6nD6gwRUfecv1yH73JKsTOnBNmFVwAANlYWyFo5FU5aDDX8mAQ1GGqoJSEEPj9ciH+lHkdN/Q3YWllgWeQwLLrNn9UZIiINNQecS3UNWD59uFb3rbPr1BCZgqIrV7Hi66P46SSrM0RE2jCgrwMenRyg72kw1JD5EELgi18L8dpOVmeIiEwRQw2ZhdbVmQmD+mLtfWMR0M9JzzMjIiJtYaghk8bqDBGR+WCoIZOlrjrzxn1jMZjVGSIik8RQQyaH1RkiIvPEUEMmpXV15paBfbB2XhCrM0REZoChhkyCEAKbfy3Eqy2qM89PG4a/3c7qDBGRuWCoIaPH6gwREQEMNWTEWJ0hIqKWGGrIKBVfuYoVW3Ow/48LAFidISIihhoyMkIIfPlbIV7dcRzVrM4QEVELDDVkNFidISKijjDUkMFrXZ2xsbLA89OG4u+3B7A6Q0RESgw1ZNCKr1xF/NYc7Guqzowf2Adr7wtCoCerM0REpIqhhgwSqzNERKQphhoyOCWVV7Hia1ZniIhIMww1ZDCEENjy23n8c0eesjrz3NShWHwHqzNERNQ5hhoyCKzOEBFRTzHUkF4JIbAls6k6c43VGSIi6j6GGtKbkkrFmU3p+YrqzDi/Plg3bywCPZ31PDMiIjJGDDXU61idISIiXWCooV5VWnkN8VuPYi+rM0REpGUMNdQr1FVn4qYOxeLb/WFlaaHv6RERkQlgqCGda12dCfLrg3+zOkNERFrGUEM6I4TAV5nn8X+szhARUS9gqCGdUFedWXffWAzxYnWGiIh0g6GGtKpNdcbSAs9OHYpH72B1hoiIdIuhhrSmtPIaXkzJwY8nygGwOkNERL2LoYZ6TAiBr7OK8Mq3uazOEBGR3jDUUI+0qc4McMW6eUGszhARUa9jqKFuaa7O/N+3uahqqs7ETh2Cx+4IYHWGiIj0gqGGNFZWdQ3xW1WrM2vnBWEoqzNERKRHDDXUZazOEBGRIevWK1FSUhKkUins7Owgk8lw+PDhdsfm5uYiKioKUqkUEokEiYmJbcbs378fs2fPhq+vLyQSCbZt29ZmTE1NDZYuXYoBAwbA3t4eI0eOxMaNG7szfeqGsqprWPzRb3h+y++ounYDYwe4YsfTt+Op8EAGGiIiMggavxpt3rwZcXFxWL16NbKyshAUFITIyEiUl5erHV9XV4eAgACsWbMG3t7easfU1tYiKCgISUlJ7T5vXFwcdu3ahf/97384fvw4YmNjsXTpUmzfvl3TJZAGhBD4OvM8pr65D2knymFjaYFlkcOw9clJfLuJiIgMikQIITR5gEwmQ3BwMDZs2AAAkMvl8PPzQ0xMDFasWNHhY6VSKWJjYxEbG9v+hCQSpKSkYO7cuSrbR48ejQceeAArV65UbpswYQJmzJiBV199tdN5V1VVwdXVFZWVlXBxcel0PCmqMy9uzUFaU+/M2KYzmxhmiIiot2jy+q1RpaahoQGZmZmIiIi4uQMLC0RERCAjI6N7s+2iSZMmYfv27SgqKoIQAnv37sUff/yBadOm6fR5zZEQAluzWJ0hIiLjolGjcEVFBRobG+Hl5aWy3cvLCydOnNDqxFp7++238dhjj2HAgAGwsrKChYUF3n//fUyePFnt+Pr6etTX1yu/r6qq0un8TAWrM0REZKyM5uynt99+GwcPHsT27dsxaNAg7N+/H0uWLIGvr69K5ahZQkICXnnlFT3M1DgJIZBypAgvb795ZtMzEUPw+GSe2URERMZBo1Dj4eEBS0tLlJWVqWwvKytrtwlYG65evYoXX3wRKSkpmDVrFgBg7NixyM7Oxrp169SGmvj4eMTFxSm/r6qqgp+fn87maMzKqxRXBd5z/GZ1Zu19QRjmzeoMEREZD43+CW5jY4MJEyYgLS1NuU0ulyMtLQ2hoaFan1yz69ev4/r167CwUJ2upaUl5HK52sfY2trCxcVF5UaqmntnIt7chz3Hy2FtKVH2zjDQEBGRsdH47ae4uDhER0dj4sSJCAkJQWJiImpra7Fo0SIAwMKFC9G/f38kJCQAUDQX5+XlKb8uKipCdnY2nJycEBgYCEBxDZpTp04pn6OgoADZ2dlwc3PDwIED4eLigrCwMCxbtgz29vYYNGgQ9u3bh48//hhvvvlmj38I5qh1dWZMf0XvDMMMEREZK41P6QaADRs2YO3atSgtLcW4ceOwfv16yGQyAEB4eDikUimSk5MBAGfOnIG/v3+bfYSFhSE9PR0AkJ6ejilTprQZEx0drdxPaWkp4uPjsXv3bly6dAmDBg3CY489hmeffRYSiaTTOfOUbgUhBLZlF+Hl7XmovHod1pYSxEYMZe8MEREZJE1ev7sVaowRQ01zdeYY9hxX9ESxOkNERIZOk9dvozn7ibpPXXXmmbuG4PGwwbBmdYaIiEwEQ42JY3WGiIjMBUONiRJC4JvsYqzensvqDBERmQWGGhNUXn0NL269WZ0Z3d8F6+YFYbi3efYSERGReWCoMSGszhARkTljqDER5dXX8I+UY/ghj9UZIiIyTww1Rk5ddebpO4fgiXBWZ4iIyLww1Bix8upreCnlGHa3qM6svS8II3xYnSEiIvPDUGOEhBDY/ruiOnOljtUZIiIigKHG6LSuzozyVfTOsDpDRETmjqHGSKirzsTcOQRPsjpDREQEgKHGKLA6Q0RE1DmGGgPG6gwREVHXMdQYqAvV9XhpWw6+z2V1hoiIqCsYagxM6+qMlYWiOvPUFFZniIiIOsJQY0BaV2dG+iiqMyN9WZ0hIiLqDEONARBC4NujJVj9zTFcZnWGiIioWxhq9OxCdT1WbjuGXbmlAFidISIi6i6GGj1RV51ZemcglkwJZHWGiIioGxhq9IDVGSIiIu1jqOlFQgjsOFqCVazOEBERaR1DTS+pqKnHSymszhAREekKQ42OtVedeSo8EDZWrM4QERFpC0ONDlXUKHpnvjumqM6M8HHBunljMcrXVc8zIyIiMj0MNTqy42gxVm67WZ1ZMkXRO8PqDBERkW4w1GhZRU09Vn1zDKk5rM4QERH1JoYaLdpxtBirvsnFpdoGVmeIiIh6GUONFrSuzgz3dsa/7w9idYaIiKgXMdT00InSKvz1/UPK6sxTUwKxlNUZIiKiXsdQ00MBHk7wdLaFp7Mt1s0Lwuj+rM4QERHpA0NND9lYWeDDR4Lh4WTL6gwREZEeMdRogW8fe31PgYiIyOyxtEBEREQmgaGGiIiITAJDDREREZmEboWapKQkSKVS2NnZQSaT4fDhw+2Ozc3NRVRUFKRSKSQSCRITE9uM2b9/P2bPng1fX19IJBJs27ZN7b6OHz+Oe+65B66urnB0dERwcDDOnTvXnSUQERGRidE41GzevBlxcXFYvXo1srKyEBQUhMjISJSXl6sdX1dXh4CAAKxZswbe3t5qx9TW1iIoKAhJSUntPu/p06dx++23Y/jw4UhPT8fRo0excuVK2NnZaboEIiIiMkESIYTQ5AEymQzBwcHYsGEDAEAul8PPzw8xMTFYsWJFh4+VSqWIjY1FbGxs+xOSSJCSkoK5c+eqbH/wwQdhbW2NTz75RJPpKlVVVcHV1RWVlZVwcXHp1j6IiIiod2ny+q1RpaahoQGZmZmIiIi4uQMLC0RERCAjI6N7s+0CuVyOnTt3YujQoYiMjISnpydkMlm7b1MBQH19PaqqqlRuREREZLo0CjUVFRVobGyEl5eXynYvLy+UlpZqdWItlZeXo6amBmvWrMH06dOxe/du/OUvf8G9996Lffv2qX1MQkICXF1dlTc/Pz+dzY+IiIj0zyjOfpLL5QCAOXPm4Nlnn8W4ceOwYsUK3H333di4caPax8THx6OyslJ5Kyws7M0pExERUS/T6IrCHh4esLS0RFlZmcr2srKydpuAtcHDwwNWVlYYOXKkyvYRI0bg559/VvsYW1tb2Nra6mxOREREZFg0qtTY2NhgwoQJSEtLU26Ty+VIS0tDaGio1ifX8nmDg4ORn5+vsv2PP/7AoEGDdPa8REREZDw0/uynuLg4REdHY+LEiQgJCUFiYiJqa2uxaNEiAMDChQvRv39/JCQkAFA0F+fl5Sm/LioqQnZ2NpycnBAYGAgAqKmpwalTp5TPUVBQgOzsbLi5uWHgwIEAgGXLluGBBx7A5MmTMWXKFOzatQvffvst0tPTe/QDICIiIhMhuuHtt98WAwcOFDY2NiIkJEQcPHhQeV9YWJiIjo5Wfl9QUCAAtLmFhYUpx+zdu1ftmJb7EUKIDz74QAQGBgo7OzsRFBQktm3b1uU5V1ZWCgCisrKyO0smIiIiPdDk9Vvj69QYK16nhoiIyPjo7Do1RERERIaKoYaIiIhMAkMNERERmQSGGiIiIjIJDDVERERkEhhqiIiIyCQw1BAREZFJYKghIiIik8BQQ0RERCaBoYaIiIhMAkMNERERmQSGGiIiIjIJDDVERERkEhhqiIiIyCQw1BAREZFJYKghIiIik8BQQ0RERCaBoYaIiIhMAkMNERERmQSGGiIiIjIJDDVERERkEhhqiIiIyCQw1BAREZFJYKghIiIik8BQQ0RERCaBoYaIiIhMAkMNERERmQSGGiIiIjIJDDVERERkEhhqiIiIyCQw1BAREZFJYKghIiIik8BQQ0RERCaBoYaIiIhMQrdCTVJSEqRSKezs7CCTyXD48OF2x+bm5iIqKgpSqRQSiQSJiYltxuzfvx+zZ8+Gr68vJBIJtm3b1uHzP/HEE+3ui4iIiMyTxqFm8+bNiIuLw+rVq5GVlYWgoCBERkaivLxc7fi6ujoEBARgzZo18Pb2VjumtrYWQUFBSEpK6vT5U1JScPDgQfj6+mo6dSIiIjJhGoeaN998E48++igWLVqEkSNHYuPGjXBwcMCHH36odnxwcDDWrl2LBx98ELa2tmrHzJgxA6+++ir+8pe/dPjcRUVFiImJwaeffgpra2tNp05EREQmTKNQ09DQgMzMTERERNzcgYUFIiIikJGRofXJtSSXy7FgwQIsW7YMo0aN6nR8fX09qqqqVG5ERERkujQKNRUVFWhsbISXl5fKdi8vL5SWlmp1Yq29/vrrsLKywtNPP92l8QkJCXB1dVXe/Pz8dDe5zI+AMwd0t38iIiLqlFGc/ZSZmYn//Oc/SE5OhkQi6dJj4uPjUVlZqbwVFhbqZnJXCoHvXgCSZwJfLgQun9HN8xAREVGHNAo1Hh4esLS0RFlZmcr2srKydpuAteGnn35CeXk5Bg4cCCsrK1hZWeHs2bN47rnnIJVK1T7G1tYWLi4uKjedsHYAgh4CJBZA3jfAhhBgzytAfbVuno+IiIjU0ijU2NjYYMKECUhLS1Nuk8vlSEtLQ2hoqNYn12zBggU4evQosrOzlTdfX18sW7YM33//vc6et0sc3YHZicDjPwH+k4HGeuDnN4H1twBZnwByuX7nR0REZCasNH1AXFwcoqOjMXHiRISEhCAxMRG1tbVYtGgRAGDhwoXo378/EhISACiai/Py8pRfFxUVITs7G05OTggMDAQA1NTU4NSpU8rnKCgoQHZ2Ntzc3DBw4EC4u7vD3d1dZR7W1tbw9vbGsGHDurdybfMeDSzcDuSnAt//A7hcAGxfChx+D5i+BpDepu8ZEhERmTSNQ80DDzyACxcuYNWqVSgtLcW4ceOwa9cuZfPwuXPnYGFxswBUXFyM8ePHK79ft24d1q1bh7CwMKSnpwMAfvvtN0yZMkU5Ji4uDgAQHR2N5OTk7qxLPyQSYPgsIDBCEWb2vQGUHlX024ycA0z9P6CvVN+zJCIiMkkSIYTQ9yR6Q1VVFVxdXVFZWam7/prWai4Ae18Dsj4ChBywtAVClwB3xAG2zr0zByIiIiOmyeu3UZz9ZLSc+nXSb9Oo7xkSERGZDIaa3tDcb/Pg54BbAFBbrui3eS+c17chIiLSEoaa3iKRAMNnAk8dAqa9Cti63Oy34fVtiIiIeoyhprdZ2QCTYoCnjwAT/9bi+jbBwJ6XeX0bIiKibmKo0RdHD+Dut4Anfgb8w4DGBuDnt9hvQ0RE1E0MNfrmNQpY+A37bYiIiHqIocYQsN+GiIioxxhqDEln/TbXqvQ9QyIiIoPFUGOI2uu3eXsCkPUx+22IiIjUYKgxZM39Ng99AbgNbuq3iWnqt/lZ37MjIiIyKAw1hk4iAYbNAJ46CEx7DbB1beq3mQVsXgBcKtD3DImIiAwCQ42xsLIBJi0Fns662W9zfDuQFMJ+GyIiIjDUGB/22xAREanFUGOs2G9DRESkgqHGmLHfhoiISImhxhSo9Nv8XbXf5ofV7LchIiKzwFBjShw9gLvfBJ44AASEK/ptDiSy34aIiMwCQ40p8hoJLNimpt8mjP02RERkshhqTFXLfpvIfzX12+Q09ds8zH4bIiIyOQw1ps7KBghd0qrf5lv22xARkclhqDEX7LchIiITx1BjbpT9Npvb9tsU/KTv2REREXUbQ405kkiAYdPb9tt8dDf7bYiIyGgx1JgzZb/NESB4cat+m1XstyEiIqPCUEOAozsw69+t+m3+A7x9C5D5EfttiIjIKDDU0E1t+m0uAN8+zX4bIiIyCgw1pIr9NkREZKQYakg99tsQEZGRYaihjqn020xhvw0RERkshhrqGq+RwIIURb+Ne+DNfpt32W9DRESGgaGGuq653+bJDCAyAbBzBcqa+m2+mA9c+lPfMyQiIjPGUEOas7IBQp8CYlr025zYASTJ2G9DRER6w1BD3ddhv00y+22IiKhXMdRQz6ntt3mG/TZERNSruhVqkpKSIJVKYWdnB5lMhsOHD7c7Njc3F1FRUZBKpZBIJEhMTGwzZv/+/Zg9ezZ8fX0hkUiwbds2lfuvX7+O5cuXY8yYMXB0dISvry8WLlyI4uLi7kyfdIH9NkREpGcah5rNmzcjLi4Oq1evRlZWFoKCghAZGYny8nK14+vq6hAQEIA1a9bA29tb7Zja2loEBQUhKSmp3X1kZWVh5cqVyMrKwtatW5Gfn4977rlH0+mTrqn02zwKSCzZb0NERL1CIoQQmjxAJpMhODgYGzZsAADI5XL4+fkhJiYGK1as6PCxUqkUsbGxiI2NbX9CEglSUlIwd+7cDvf166+/IiQkBGfPnsXAgQM7nXdVVRVcXV1RWVkJFxeXTseTlpQfB3bFA3/uVXzv2A+48yVg/ALAwlK/cyMiIoOnyeu3RpWahoYGZGZmIiIi4uYOLCwQERGBjIyM7s22myorKyGRSNCnTx+199fX16OqqkrlRnrgOULRb/PXL1v120wGCvbre3ZERGRCNAo1FRUVaGxshJeXl8p2Ly8vlJaWanViHbl27RqWL1+Ohx56qN3UlpCQAFdXV+XNz8+v1+ZHrUgkwNDIVv02x4CPZrPfhoiItMbozn66fv067r//fggh8M4777Q7Lj4+HpWVlcpbYWFhL86S1Oqo32b3SvbbEBFRj2gUajw8PGBpaYmysjKV7WVlZe02AWtTc6A5e/Ysfvjhhw7fW7O1tYWLi4vKjQyEozswax3wZIvr2/yynte3ISKiHtEo1NjY2GDChAlIS0tTbpPL5UhLS0NoaKjWJ9dSc6A5efIk9uzZA3d3d50+H/UC9tsQEZEWWWn6gLi4OERHR2PixIkICQlBYmIiamtrsWjRIgDAwoUL0b9/fyQkJABQNBfn5eUpvy4qKkJ2djacnJwQGBgIAKipqcGpU6eUz1FQUIDs7Gy4ublh4MCBuH79Ou677z5kZWVhx44daGxsVPbwuLm5wcbGpmc/BdKf5n6bwXcCv/4/ID3hZr/N8LuBaf8E3AL0PUsiIjICGp/SDQAbNmzA2rVrUVpainHjxmH9+vWQyWQAgPDwcEilUiQnJwMAzpw5A39//zb7CAsLQ3p6OgAgPT0dU6ZMaTMmOjoaycnJ7e4DAPbu3Yvw8PBO58xTuo1E3SVg77+A3z4ERCNgaQPIngAmP69oMCYiIrOiyet3t0KNMWKoMTLlx4HvXwRO/6j43sFDcX2bWxby+jZERGZEZ9epIeo1niOAh7fe7LepqwB2xLLfhoiI2sVQQ4arud/mqYPA9DVtr29z8bS+Z0hERAaEoYYMn6U1cOuTwNPZ7VzfplLfMyQiIgPAUEPGw8Ht5vVtBt8JyK8rrm+z/hbgt028vg0RkZljqCHjo+y32QK4D2G/DRERAWCoIWMlkQBDpwFPZbDfhoiIADDUkLFr2W8T8lirfpuX2G9DRGRGGGrINDi4ATPXAk/+Agy+q6nf5m322xARmRGGGjItnsOBh79W32/z5z59z46IiHSIoYZMj0q/zeuAXR9Fv83H97DfhojIhDHUkOmytAZufQJ4+gj7bYiIzABDDZk+9tsQEZkFhhoyH57DgQVbgflfAR5D2W9DRGRiGGrI/AyZqqjatO63+fyv7LchIjJiDDVkntT12+TvZL8NEZERY6gh89Zhv82H7LchIjIiDDVEQDv9Ns8CG+9gvw0RkZFgqCFqqbnfZsYbin6b8lz22xARGQmGGqLWLK0B2eNN/TaPs9+GiMhIMNQQtcfBDZj5huLKxIER7LchIjJwDDVEnek3TPF5Umr7bdL1PTsiImrCUEPUVWr7beYAnz/EfhsiIgPAUEOkCbX9NqmKfpvv/wFcvaLvGRIRmS2GGqLuUOm3marot8nYALzd1G/TeEPfMyQiMjsMNUQ90W8Y8PBXLfptLir6bd6dzH4bIqJexlBDpA3styEi0juGGiJtadlvI3uC/TZERL2MoYZI2xzcgBmvq++3+fUD9tsQEekIQw2Rrqjrt9kZB7zL69sQEekCQw2Rrin7bdYC9n2B8jz22xAR6QBDDVFvsLQGZI8BMVnstyEi0hGGGqLe1LLfZsg09tsQEWkRQw2RPvQbBszfAsz/GvAYxn4bIiItYKgh0qchEcCTB9hvQ0SkBd0KNUlJSZBKpbCzs4NMJsPhw4fbHZubm4uoqChIpVJIJBIkJia2GbN//37Mnj0bvr6+kEgk2LZtW5sxQgisWrUKPj4+sLe3R0REBE6ePNmd6RMZFvbbEBFphcahZvPmzYiLi8Pq1auRlZWFoKAgREZGory8XO34uro6BAQEYM2aNfD29lY7pra2FkFBQUhKSmr3ed944w2sX78eGzduxKFDh+Do6IjIyEhcu3ZN0yUQGaYO+23+H/ttiIg6IRFCCE0eIJPJEBwcjA0bNgAA5HI5/Pz8EBMTgxUrVnT4WKlUitjYWMTGxrY/IYkEKSkpmDt3rnKbEAK+vr547rnn8PzzzwMAKisr4eXlheTkZDz44IOdzruqqgqurq6orKyEi4tL5wsl0reTe4DvXwQq8hXfe44EIv8FDJ6i33kREfUiTV6/NarUNDQ0IDMzExERETd3YGGBiIgIZGRkdG+2XVBQUIDS0lKV53V1dYVMJmv3eevr61FVVaVyIzIq6vptPpkLfPYg+22IiNTQKNRUVFSgsbERXl5eKtu9vLxQWlqq1Ym11LxvTZ43ISEBrq6uypufn5/O5kekMyr9Nk8CFlbAH9+x34aISA2TPfspPj4elZWVylthYaG+p0TUfQ5uwIw1wJPstyEiao9GocbDwwOWlpYoKytT2V5WVtZuE7A2NO9bk+e1tbWFi4uLyo3I6PUbqub6Ns8prm9zeq++Z0dEpFcahRobGxtMmDABaWlpym1yuRxpaWkIDQ3V+uSa+fv7w9vbW+V5q6qqcOjQIZ0+L5HB6qjfpuKUvmdHRKQXVpo+IC4uDtHR0Zg4cSJCQkKQmJiI2tpaLFq0CACwcOFC9O/fHwkJCQAUzcV5eXnKr4uKipCdnQ0nJycEBgYCAGpqanDq1M3/ERcUFCA7Oxtubm4YOHAgJBIJYmNj8eqrr2LIkCHw9/fHypUr4evrq3KWFJFZae63GXMfsO8N4Nf3Ff02p34AQh4Hwl4A7Pvoe5ZERL1G41O6AWDDhg1Yu3YtSktLMW7cOKxfvx4ymQwAEB4eDqlUiuTkZADAmTNn4O/v32YfYWFhSE9PBwCkp6djypS2p6lGR0cr9yOEwOrVq/Hee+/hypUruP322/Hf//4XQ4cO7dKceUo3mbwLfwC7XwJOfq/43t4NuPMfwC2PAJYa//uFiMggaPL63a1QY4wYashsnNoD7GpxfZt+I4Dp/wIG36nfeRERdYPOrlNDREYgMAJ48hdg5jpFv82F48Anf2G/DRGZPIYaIlNkaQWEPAo8fUT1+jb/lSmqOLy+DRGZIIYaIlNm37fF9W0iAfkN4GASsH48r29DRCaHoYbIHPQbCsz/Eni46fo2Vy8prm+z8Xbg9I/6nh0RkVYw1BCZk3b7bR4AKk7qe3ZERD3CUENkblr229z6VFO/zS7gv7c29dtc1vcMiYi6haGGyFzZ9wWmJwBPHWzVb3MLcPh99tsQkdHhdWqISOHUHsUnf184ofie17choo4IoajsVhU33c4DDXXApKVafRpefE8NhhqiLmi8AWRuAva+dvNtqKHTgWmvAh5D9Ds3Iuo9QgB1l4CqoqbAUtTq66Ygc71O9XFWdsA/SgGJRGtT0eT1m9dOJ6Kbmvttmj9P6vB7in6bU3uaPk9qmeJtKyIyXkIAdReByvOtQkqr/9641rX9ObgDLv2bbr5AYwNgZavbNbSDlRoial/FScVbUi0/T2rKi8CERfw8KSJDJJcDdRWKYFKpLqw0fd3Y0LX9OfZTBBWXAU3/9VWEF9emAOPsC1jb6XRJfPtJDYYaoh5o028zHIj8FxB4l37nRWRO5HKgtrydwNLU01JVAsivd2FnEsDJ82ZIaa6yNP/XtT/g7KO3iktLDDVqMNQQ9ZCy3+Zfiov3Aey3IdIWeSNQU3YzqFQWtQosRUB1ieIsxU5JAGdv1cpK69Di7ANY2eh8WdrAUKMGQw2Rlly9fLPfRn5DcZ2bkMeAsBfYb0OkTuMNoKZUtbLSOrRUlwCisfN9SSwUgaTDwOINWFrrfl29hKFGDYYaIi2rOAnsfknRSAyw34bMU+N1oLq01dlBxS2acIsVgUbIO9+XxFIRWFxbhZSWwcXJy+z+vhhq1GCoIdKRU2lN/TbHFd+z34ZMxY0GRQVF3SnNzT0tNWUAuvAyamGlaKpt7ldRCS1NTbhOnoCFpc6XZWwYatRgqCHSIfbbkLG5Ud+iX6W5ybZYNbTUlndtXxbWbc8Kal1lcewHWPAi/t3BUKMGQw1RL7h6Gdi3Fjj8LvttSH+uX2vbZNv6Giy1F7q2L0vbtmcFKQNLU5XFwZ2BRYcYatRgqCHqRRWngN3/YL8NaV9DneItIbUXjmv6uu5i1/ZlZaf+VOaW2xzctXp1XNIcQ40aDDVEesB+G9JEQ23bnpXWoaWrnyJv7aDm7CBfwHXAze32fRlYjABDjRoMNUR60ngDyEoGfnztZr/NkEgg8jX225iT+uoOAktTT8u1yq7ty9qxRe9KO1e6tevDwGIiGGrUYKgh0jP225iua5UdX4Olqgior+ravmxdOr4Gi2t/xRgGFrPBUKMGQw2RgWjTb9MXmPIP9tsYIiGaAktRx6Globpr+7NzbdtkqxJgfAE7/v+ZVDHUqMFQQ2Rg2G+jX0Ioqmdtzg5qdeG467Vd259dnxb9KuqqLD6ArbNOl0SmiaFGDYYaIgPEfhvdEAKou9QiqKi5BktVMXDjatf2Z+/W6hosLUNLU2CxcdTtmshsMdSowVBDZMCuXgH2rwUObbzZbxP8qKLfxsFN37MzLEIAtRXqT2Vu+RbRjWtd25+DRycXjvMFrO11uyaiDjDUqMFQQ2QEKk41fZ7Ud4rvza3fRi4H6ipU3/5Rd+G4xoau7c/Rs+PA4uwDWNvpdk1EPcRQowZDDZEROf0jsOvFVv02rwGBEfqdV0/IGxVXse3oGixVJYD8ehd2JlF8sGGbU5lbvD3k7ANY2ep8WUS6xlCjBkMNkZFR228zDZj2GtBvqF6n1oa8UfHBhio9K63eEqouUby11ikJ4Ozd8ZVunbwBKxudL4vIEDDUqMFQQ2Sk9N1v03gDqCnt+Eq31aWAaOx8XxILRQWldWBpeaVbJy/A0lr36yIyEgw1ajDUEBk5XfTbNF5XVFA6ugZLTSkg5J3vS2LZ9poryipLU2Bx9DSP3iAiLWKoUYOhhshEnP5RcX2b8jzF9+3129xoaAosrS8c16IJt6YMQBf+F2hhrThtWSWs9FftZ3HsB1hYan25ROaOoUYNhhoiE9J4A8j6CNj72s1PZPafrLh8fnNoqS3v2r4sbdq+FdT6SreO/QALC92th4japcnrN+ugRGR8LK2A4L8Do6Nu9tsU7FczzrbVJzOrqbI4uDOwEJmIbv0lJyUlQSqVws7ODjKZDIcPH253bG5uLqKioiCVSiGRSJCYmNitfZaWlmLBggXw9vaGo6MjbrnlFnz99dfdmT4RmQr7Poq3np46BNy1Gpj1b+ChL4DHfwKW/Qm8VAY8kw08sgO49z0g4mUg5FFg+EzAdxzgxAoMkSnR+K958+bNiIuLw+rVq5GVlYWgoCBERkaivFx9qbeurg4BAQFYs2YNvL29u73PhQsXIj8/H9u3b0dOTg7uvfde3H///Thy5IimSyAiU+MRCNwRBwQvBobNAHzGAo7u/CRnIjOjcU+NTCZDcHAwNmzYAACQy+Xw8/NDTEwMVqxY0eFjpVIpYmNjERsbq/E+nZyc8M4772DBggXKx7m7u+P111/H4sWLO503e2qIiIiMjyav3xpVahoaGpCZmYmIiJtnGVhYWCAiIgIZGRndmmxX9zlp0iRs3rwZly5dglwuxxdffIFr164hPDxc7X7r6+tRVVWlciMiIiLTpVGoqaioQGNjI7y8vFS2e3l5obS0tFsT6Oo+v/zyS1y/fh3u7u6wtbXF448/jpSUFAQGBqrdb0JCAlxdXZU3Pz+/bs2PiIiIjIPRdMitXLkSV65cwZ49e/Dbb78hLi4O999/P3JyctSOj4+PR2VlpfJWWFjYyzMmIiKi3qTRKd0eHh6wtLREWVmZyvaysrJ2m4C1sc/Tp09jw4YNOHbsGEaNGgUACAoKwk8//YSkpCRs3LixzX5tbW1ha8sPcyMiIjIXGlVqbGxsMGHCBKSlpSm3yeVypKWlITQ0tFsT6Mo+6+rqFJNtdeqlpaUl5PIuXL6ciIiITJ7GF9+Li4tDdHQ0Jk6ciJCQECQmJqK2thaLFi0CoDj1un///khISACgaATOy8tTfl1UVITs7Gw4OTkp+2E62+fw4cMRGBiIxx9/HOvWrYO7uzu2bduGH374ATt27NDKD4KIiIiMm8ah5oEHHsCFCxewatUqlJaWYty4cdi1a5ey0ffcuXMqFZXi4mKMHz9e+f26deuwbt06hIWFIT09vUv7tLa2RmpqKlasWIHZs2ejpqYGgYGB+OijjzBz5syerJ+IiIhMBD/7iYiIiAyWzq5TQ0RERGSoGGqIiIjIJDDUEBERkUlgqCEiIiKToPHZT8aquR+anwFFRERkPJpft7tyXpPZhJrq6moA4GdAERERGaHq6mq4urp2OMZsTumWy+UoLi6Gs7MzJBKJVvddVVUFPz8/FBYWmuTp4qa+PsD018j1GT9TX6Oprw8w/TXqan1CCFRXV8PX17fNJwu0ZjaVGgsLCwwYMECnz+Hi4mKSv6jNTH19gOmvkeszfqa+RlNfH2D6a9TF+jqr0DRjozARERGZBIYaIiIiMgkMNVpga2uL1atXw9bWVt9T0QlTXx9g+mvk+oyfqa/R1NcHmP4aDWF9ZtMoTERERKaNlRoiIiIyCQw1REREZBIYaoiIiMgkMNQQERGRSWCoaUdSUhKkUins7Owgk8lw+PDhDsdv2bIFw4cPh52dHcaMGYPU1FSV+4UQWLVqFXx8fGBvb4+IiAicPHlSl0vokCbre//993HHHXegb9++6Nu3LyIiItqMf+SRRyCRSFRu06dP1/Uy2qXJ+pKTk9vM3c7OTmWMoR0/QLM1hoeHt1mjRCLBrFmzlGMM6Rju378fs2fPhq+vLyQSCbZt29bpY9LT03HLLbfA1tYWgYGBSE5ObjNG079rXdF0fVu3bsXUqVPRr18/uLi4IDQ0FN9//73KmJdffrnN8Rs+fLgOV9E+TdeXnp6u9veztLRUZZyhHD9A8zWq+/uSSCQYNWqUcowhHcOEhAQEBwfD2dkZnp6emDt3LvLz8zt9nL5fCxlq1Ni8eTPi4uKwevVqZGVlISgoCJGRkSgvL1c7/pdffsFDDz2Ev//97zhy5Ajmzp2LuXPn4tixY8oxb7zxBtavX4+NGzfi0KFDcHR0RGRkJK5du9Zby1LSdH3p6el46KGHsHfvXmRkZMDPzw/Tpk1DUVGRyrjp06ejpKREefv88897YzltaLo+QHEFzJZzP3v2rMr9hnT8AM3XuHXrVpX1HTt2DJaWlpg3b57KOEM5hrW1tQgKCkJSUlKXxhcUFGDWrFmYMmUKsrOzERsbi8WLF6u88Hfn90JXNF3f/v37MXXqVKSmpiIzMxNTpkzB7NmzceTIEZVxo0aNUjl+P//8sy6m3ylN19csPz9fZf6enp7K+wzp+AGar/E///mPytoKCwvh5ubW5m/QUI7hvn37sGTJEhw8eBA//PADrl+/jmnTpqG2trbdxxjEa6GgNkJCQsSSJUuU3zc2NgpfX1+RkJCgdvz9998vZs2apbJNJpOJxx9/XAghhFwuF97e3mLt2rXK+69cuSJsbW3F559/roMVdEzT9bV248YN4ezsLD766CPltujoaDFnzhxtT7VbNF3fpk2bhKura7v7M7TjJ0TPj+Fbb70lnJ2dRU1NjXKbIR3DlgCIlJSUDse88MILYtSoUSrbHnjgAREZGan8vqc/M13pyvrUGTlypHjllVeU369evVoEBQVpb2Ja0pX17d27VwAQly9fbneMoR4/Ibp3DFNSUoREIhFnzpxRbjPUYyiEEOXl5QKA2LdvX7tjDOG1kJWaVhoaGpCZmYmIiAjlNgsLC0RERCAjI0PtYzIyMlTGA0BkZKRyfEFBAUpLS1XGuLq6QiaTtbtPXenO+lqrq6vD9evX4ebmprI9PT0dnp6eGDZsGJ588klcvHhRq3Pviu6ur6amBoMGDYKfnx/mzJmD3Nxc5X2GdPwA7RzDDz74AA8++CAcHR1VthvCMeyOzv4GtfEzMyRyuRzV1dVt/gZPnjwJX19fBAQEYP78+Th37pyeZtg948aNg4+PD6ZOnYoDBw4ot5va8QMUf4MREREYNGiQynZDPYaVlZUA0OZ3riVDeC1kqGmloqICjY2N8PLyUtnu5eXV5v3dZqWlpR2Ob/6vJvvUle6sr7Xly5fD19dX5Rdz+vTp+Pjjj5GWlobXX38d+/btw4wZM9DY2KjV+XemO+sbNmwYPvzwQ3zzzTf43//+B7lcjkmTJuH8+fMADOv4AT0/hocPH8axY8ewePFile2Gcgy7o72/waqqKly9elUrv/eGZN26daipqcH999+v3CaTyZCcnIxdu3bhnXfeQUFBAe644w5UV1frcaZd4+Pjg40bN+Lrr7/G119/DT8/P4SHhyMrKwuAdv6/ZUiKi4vx3XfftfkbNNRjKJfLERsbi9tuuw2jR49ud5whvBaazad0k3asWbMGX3zxBdLT01WaaR988EHl12PGjMHYsWMxePBgpKen46677tLHVLssNDQUoaGhyu8nTZqEESNG4N1338U///lPPc5MNz744AOMGTMGISEhKtuN+Riak88++wyvvPIKvvnmG5WekxkzZii/Hjt2LGQyGQYNGoQvv/wSf//73/Ux1S4bNmwYhg0bpvx+0qRJOH36NN566y188sknepyZbnz00Ufo06cP5s6dq7LdUI/hkiVLcOzYMb3192iClZpWPDw8YGlpibKyMpXtZWVl8Pb2VvsYb2/vDsc3/1eTfepKd9bXbN26dVizZg12796NsWPHdjg2ICAAHh4eOHXqVI/nrImerK+ZtbU1xo8fr5y7IR0/oGdrrK2txRdffNGl/0Hq6xh2R3t/gy4uLrC3t9fK74Uh+OKLL7B48WJ8+eWXbcr8rfXp0wdDhw41iuOnTkhIiHLupnL8AMXZPx9++CEWLFgAGxubDscawjFcunQpduzYgb1792LAgAEdjjWE10KGmlZsbGwwYcIEpKWlKbfJ5XKkpaWp/Gu+pdDQUJXxAPDDDz8ox/v7+8Pb21tlTFVVFQ4dOtTuPnWlO+sDFB3r//znP7Fr1y5MnDix0+c5f/48Ll68CB8fH63Mu6u6u76WGhsbkZOTo5y7IR0/oGdr3LJlC+rr6/Hwww93+jz6Oobd0dnfoDZ+L/Tt888/x6JFi/D555+rnIrfnpqaGpw+fdoojp862dnZyrmbwvFrtm/fPpw6dapL/7DQ5zEUQmDp0qVISUnBjz/+CH9//04fYxCvhVppNzYxX3zxhbC1tRXJyckiLy9PPPbYY6JPnz6itLRUCCHEggULxIoVK5TjDxw4IKysrMS6devE8ePHxerVq4W1tbXIyclRjlmzZo3o06eP+Oabb8TRo0fFnDlzhL+/v7h69arBr2/NmjXCxsZGfPXVV6KkpER5q66uFkIIUV1dLZ5//nmRkZEhCgoKxJ49e8Qtt9wihgwZIq5du2bw63vllVfE999/L06fPi0yMzPFgw8+KOzs7ERubq5yjCEdPyE0X2Oz22+/XTzwwANtthvaMayurhZHjhwRR44cEQDEm2++KY4cOSLOnj0rhBBixYoVYsGCBcrxf/75p3BwcBDLli0Tx48fF0lJScLS0lLs2rVLOaazn5khr+/TTz8VVlZWIikpSeVv8MqVK8oxzz33nEhPTxcFBQXiwIEDIiIiQnh4eIjy8nKDX99bb70ltm3bJk6ePClycnLEM888IywsLMSePXuUYwzp+Amh+RqbPfzww0Imk6ndpyEdwyeffFK4urqK9PR0ld+5uro65RhDfC1kqGnH22+/LQYOHChsbGxESEiIOHjwoPK+sLAwER0drTL+yy+/FEOHDhU2NjZi1KhRYufOnSr3y+VysXLlSuHl5SVsbW3FXXfdJfLz83tjKWppsr5BgwYJAG1uq1evFkIIUVdXJ6ZNmyb69esnrK2txaBBg8Sjjz6qt//ZCKHZ+mJjY5Vjvby8xMyZM0VWVpbK/gzt+Amh+e/oiRMnBACxe/fuNvsytGPYfIpv61vzmqKjo0VYWFibx4wbN07Y2NiIgIAAsWnTpjb77ehn1ps0XV9YWFiH44VQnMLu4+MjbGxsRP/+/cUDDzwgTp061bsLa6Lp+l5//XUxePBgYWdnJ9zc3ER4eLj48ccf2+zXUI6fEN37Hb1y5Yqwt7cX7733ntp9GtIxVLc2ACp/V4b4WihpmjwRERGRUWNPDREREZkEhhoiIiIyCQw1REREZBIYaoiIiMgkMNQQERGRSWCoISIiIpPAUENEREQmgaGGiIiITAJDDREREZkEhhoiIiIyCQw1REREZBIYaoiIiMgk/H/J7du3HHE8hQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "loss, val_loss = train_history.history['loss'], train_history.history['val_loss']\n",
        "plt.plot(loss, label='loss')\n",
        "plt.plot(val_loss, label='val_loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kB-JuBg1FH5G"
      },
      "source": [
        "As you can see the results are terrible due to the unbalanced dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9VyR_D3-FFgS",
        "outputId": "de7c5674-5d29-4d8d-8874-ced87dc1c152"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10/10 [==============================] - 1s 109ms/step\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.82      1.00      0.90       241\n",
            "           1       0.00      0.00      0.00        53\n",
            "\n",
            "    accuracy                           0.82       294\n",
            "   macro avg       0.41      0.50      0.45       294\n",
            "weighted avg       0.67      0.82      0.74       294\n",
            "\n",
            "[[241   0]\n",
            " [ 53   0]]\n",
            "0.8197278911564626\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        }
      ],
      "source": [
        "predictions = bert_model.predict(X_test)\n",
        "prediction_labels = (predictions > 0.5)*1\n",
        "print(classification_report(df_r_test['is_ecology'], prediction_labels))\n",
        "print(confusion_matrix(df_r_test['is_ecology'], prediction_labels))\n",
        "print(accuracy_score(df_r_test['is_ecology'], prediction_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j0ORsziDFx3j"
      },
      "source": [
        "# Conclusion in Real Life Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZXqVlZn4F2Ze"
      },
      "source": [
        "Surprisingly the most successful model was the Fully Connected Neural Network. This is because it did not have 0 recall in the target class.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U3wVrOzeFcTF"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    },
    "orig_nbformat": 4,
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "00f8fee3a784465980e6874ce6c184a6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0869185102034822b086f01a7f82c698": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_36fc5ac7e3ee4acaac95266078d0201e",
            "placeholder": "​",
            "style": "IPY_MODEL_c625e94451a84cdcaa7f17e3346dc0a2",
            "value": " 996k/996k [00:00&lt;00:00, 4.12MB/s]"
          }
        },
        "1cb25ddeb15142d7b15476897f30783f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20c1cdce5b4e497a879e1d19880afe6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "36fc5ac7e3ee4acaac95266078d0201e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3cb2ac723c354105a1326ed751c16dc3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_488651f1c2c94891958f936842db07fa",
              "IPY_MODEL_8d33b4b1932f4f64a82c7390e3aeb696",
              "IPY_MODEL_43c9ecffb2264a62a5c56a5670b345b0"
            ],
            "layout": "IPY_MODEL_ab5c57f5bde2430399ab49a7d7eb4a15"
          }
        },
        "42343b32671b4f1193656b01b7efa816": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "43c9ecffb2264a62a5c56a5670b345b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d2d83f468bd24cd68cc38e095f75e04f",
            "placeholder": "​",
            "style": "IPY_MODEL_59b46d93eeec4d32a70d3432b9a10165",
            "value": " 466/466 [00:00&lt;00:00, 12.6kB/s]"
          }
        },
        "4868bab11dd14a8abac93b647e723435": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_92b6625bc97a4d6aa57a8bcb4e85bf70",
            "placeholder": "​",
            "style": "IPY_MODEL_c443289e3af94bc191d0abf32a2b1147",
            "value": "Downloading (…)okenizer_config.json: 100%"
          }
        },
        "488651f1c2c94891958f936842db07fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cca7527080ff4144ad92f3b7a9396f83",
            "placeholder": "​",
            "style": "IPY_MODEL_753ee821bd0844c5abc11575df9ef275",
            "value": "Downloading (…)lve/main/config.json: 100%"
          }
        },
        "5337a253cb86488ebe32eb598556413f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_80a1d649f235493092c4222b0483d38b",
            "placeholder": "​",
            "style": "IPY_MODEL_00f8fee3a784465980e6874ce6c184a6",
            "value": "Downloading (…)solve/main/vocab.txt: 100%"
          }
        },
        "59b46d93eeec4d32a70d3432b9a10165": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "694330cda08b453c8ed46d22c463b93c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5337a253cb86488ebe32eb598556413f",
              "IPY_MODEL_dd19cd086a9647a9b5e11c737dae9778",
              "IPY_MODEL_0869185102034822b086f01a7f82c698"
            ],
            "layout": "IPY_MODEL_9e5b62996fef4c9f858d3137f1f7ddcd"
          }
        },
        "753ee821bd0844c5abc11575df9ef275": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "80a1d649f235493092c4222b0483d38b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8393277a5a384980a72222ea03700758": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8958269a6e984badad837614e2bc9f12": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4868bab11dd14a8abac93b647e723435",
              "IPY_MODEL_af72e62cebb7458e897fd2e76595012b",
              "IPY_MODEL_f13bc3c2724b4b8d8d1ff5c82308f002"
            ],
            "layout": "IPY_MODEL_b764ed364d9444c89ac118389c8c8e48"
          }
        },
        "8d33b4b1932f4f64a82c7390e3aeb696": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ed4d36090a9a4d6ea01d912edad8dcc8",
            "max": 466,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8393277a5a384980a72222ea03700758",
            "value": 466
          }
        },
        "92b6625bc97a4d6aa57a8bcb4e85bf70": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9e5b62996fef4c9f858d3137f1f7ddcd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a5e829830d014b32b6fe08f3b1da0477": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ab5c57f5bde2430399ab49a7d7eb4a15": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "af72e62cebb7458e897fd2e76595012b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a5e829830d014b32b6fe08f3b1da0477",
            "max": 29,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_42343b32671b4f1193656b01b7efa816",
            "value": 29
          }
        },
        "b764ed364d9444c89ac118389c8c8e48": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd98daaf6f934b23abff82c2fecb3fc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c443289e3af94bc191d0abf32a2b1147": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c625e94451a84cdcaa7f17e3346dc0a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cca7527080ff4144ad92f3b7a9396f83": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d2d83f468bd24cd68cc38e095f75e04f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dd19cd086a9647a9b5e11c737dae9778": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1cb25ddeb15142d7b15476897f30783f",
            "max": 995526,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bd98daaf6f934b23abff82c2fecb3fc1",
            "value": 995526
          }
        },
        "ed24da5ac71c4999a3609e32b809314e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ed4d36090a9a4d6ea01d912edad8dcc8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f13bc3c2724b4b8d8d1ff5c82308f002": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ed24da5ac71c4999a3609e32b809314e",
            "placeholder": "​",
            "style": "IPY_MODEL_20c1cdce5b4e497a879e1d19880afe6e",
            "value": " 29.0/29.0 [00:00&lt;00:00, 678B/s]"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}