Prediction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Prediction.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cdbm/F1DataAnalysis/blob/master/Prediction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3Y0pk8fnH_B"
      },
      "source": [
        "# Classificação \n",
        "o seguinte notebook foi feito para fazer predições usando modelos de classificação em cima do dataset da fórmula 1. Escolhemos o dataset de resultados, que contém os resultados de cada piloto em todas as corridas, para fazer nossa classificação. Para a classificação escolhemos a columa positions, que define em que posição o piloto terminou a corrida."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DoVoCe3xAaIe"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1Kt_gC7AiCN"
      },
      "source": [
        "results=pd.read_csv(\"https://raw.githubusercontent.com/cdbm/F1DataAnalysis/master/f1_data/results.csv\")\n",
        "races=pd.read_csv(\"https://raw.githubusercontent.com/cdbm/F1DataAnalysis/master/f1_data/races.csv\")"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UBEzsW39npp2"
      },
      "source": [
        "Limpando o dataset para ter apenas pilotos que de fato terminaram as corridas."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fugQtyVrGG30"
      },
      "source": [
        "results = results[results.position != '\\\\N']\n",
        "results = results[results.milliseconds != '\\\\N']"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9T5p3jcIn1Om"
      },
      "source": [
        "O código a seguir foi feito para tornar mais possível a classificação. Como depois da limpeza restam poucas linhas (aproximadamente 6 mil) e o dataset não tem tantas features que podem ser usadas para o modelo e existem muitas classes possíveis para se predizer (19), resolvemos então transformar as 19 classes que seriam cada posição de chegada em classes mais abrangentes. O nosso modelo agora busca classificar entre as quatro classes de posições que um piloto pode chegar, entre os 5 primeiros, entre o 6º e o 10º, entre o 11º e o 15º e entre o 16º e 19º."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7K31gY-9y8e9"
      },
      "source": [
        "results.position = results.position.astype('int64')"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1T-UFEQq_twD"
      },
      "source": [
        "results.loc[results['position'] <= 5, 'beans'] = 'TOP5'\n",
        "results.loc[(results['position'] > 5) & (results['position'] <= 10) , 'beans'] = 'SIX_TO_TEN'\n",
        "results.loc[(results['position'] > 10) & (results['position'] <= 15) , 'beans'] = 'EL_TO_FIFTEEN'\n",
        "results.loc[(results['position'] > 15) & (results['position'] <= 20) , 'beans'] = 'SIXTEEN_TO_TWENTY'"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zj0Nd-2ipNDn"
      },
      "source": [
        "Adicionando a informação de qual circuito foi o dado resultado\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TYcRLhbpdnHx"
      },
      "source": [
        "results[\"circuitId\"] = results.raceId.map(races.set_index('raceId')['circuitId'])"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uz5x95lqydwG",
        "outputId": "525b1fbf-945e-40b6-adab-9660de662437",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "results.head()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>resultId</th>\n",
              "      <th>raceId</th>\n",
              "      <th>driverId</th>\n",
              "      <th>constructorId</th>\n",
              "      <th>number</th>\n",
              "      <th>grid</th>\n",
              "      <th>position</th>\n",
              "      <th>positionText</th>\n",
              "      <th>positionOrder</th>\n",
              "      <th>points</th>\n",
              "      <th>laps</th>\n",
              "      <th>time</th>\n",
              "      <th>milliseconds</th>\n",
              "      <th>fastestLap</th>\n",
              "      <th>rank</th>\n",
              "      <th>fastestLapTime</th>\n",
              "      <th>fastestLapSpeed</th>\n",
              "      <th>statusId</th>\n",
              "      <th>beans</th>\n",
              "      <th>circuitId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>18</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>22</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>10.0</td>\n",
              "      <td>58</td>\n",
              "      <td>1:34:50.616</td>\n",
              "      <td>5690616</td>\n",
              "      <td>39</td>\n",
              "      <td>2</td>\n",
              "      <td>1:27.452</td>\n",
              "      <td>218.300</td>\n",
              "      <td>1</td>\n",
              "      <td>TOP5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>18</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>8.0</td>\n",
              "      <td>58</td>\n",
              "      <td>+5.478</td>\n",
              "      <td>5696094</td>\n",
              "      <td>41</td>\n",
              "      <td>3</td>\n",
              "      <td>1:27.739</td>\n",
              "      <td>217.586</td>\n",
              "      <td>1</td>\n",
              "      <td>TOP5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>18</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>6.0</td>\n",
              "      <td>58</td>\n",
              "      <td>+8.163</td>\n",
              "      <td>5698779</td>\n",
              "      <td>41</td>\n",
              "      <td>5</td>\n",
              "      <td>1:28.090</td>\n",
              "      <td>216.719</td>\n",
              "      <td>1</td>\n",
              "      <td>TOP5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>11</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>5.0</td>\n",
              "      <td>58</td>\n",
              "      <td>+17.181</td>\n",
              "      <td>5707797</td>\n",
              "      <td>58</td>\n",
              "      <td>7</td>\n",
              "      <td>1:28.603</td>\n",
              "      <td>215.464</td>\n",
              "      <td>1</td>\n",
              "      <td>TOP5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>18</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>23</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>4.0</td>\n",
              "      <td>58</td>\n",
              "      <td>+18.014</td>\n",
              "      <td>5708630</td>\n",
              "      <td>43</td>\n",
              "      <td>1</td>\n",
              "      <td>1:27.418</td>\n",
              "      <td>218.385</td>\n",
              "      <td>1</td>\n",
              "      <td>TOP5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   resultId  raceId  driverId  ...  statusId beans  circuitId\n",
              "0         1      18         1  ...         1  TOP5          1\n",
              "1         2      18         2  ...         1  TOP5          1\n",
              "2         3      18         3  ...         1  TOP5          1\n",
              "3         4      18         4  ...         1  TOP5          1\n",
              "4         5      18         5  ...         1  TOP5          1\n",
              "\n",
              "[5 rows x 20 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TA3fX_afpZ8Z"
      },
      "source": [
        "Dropando features que ou não podem ser usadas, ou não são úteis ou são úteis até demais para o modelo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9sryAt2dTq9-"
      },
      "source": [
        "results = results.drop('number', axis=1)\n",
        "results = results.drop('statusId', axis=1)\n",
        "results = results.drop('time', axis=1)\n",
        "results = results.drop('fastestLapTime', axis=1)\n",
        "results = results.drop('positionOrder', axis=1)\n",
        "results = results.drop('positionText', axis=1)\n",
        "results = results.drop('resultId', axis=1)\n",
        "results = results.drop('fastestLapSpeed', axis=1)\n",
        "results = results.drop('rank', axis=1)\n",
        "results = results.drop('fastestLap',axis=1)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6NGRe6mLBd_5"
      },
      "source": [
        "results = results.drop('position', axis=1)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbtikj1spja3"
      },
      "source": [
        "Definição de tipos"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nJAFRV2fU6g6"
      },
      "source": [
        "results.milliseconds = results.milliseconds.astype('int64')\n",
        "results.beans = results.beans.astype('category')"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NXZUDadRC1In",
        "outputId": "dbd4f1eb-ed93-46a7-e6aa-2d1bd25189ba",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "results.dtypes"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "raceId              int64\n",
              "driverId            int64\n",
              "constructorId       int64\n",
              "grid                int64\n",
              "points            float64\n",
              "laps                int64\n",
              "milliseconds        int64\n",
              "beans            category\n",
              "circuitId           int64\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GLuVqsXEqdTT"
      },
      "source": [
        "# Divisão do dataset em treinamento validação e teste"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UwsFFiaSWvGN"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "features_results = results.drop('beans', axis=1)\n",
        "labels_results = results['beans']"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1YCQASmOW8Lm",
        "outputId": "88b26a86-90f0-4fa3-f46c-a3217623d304",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "features_results"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>raceId</th>\n",
              "      <th>driverId</th>\n",
              "      <th>constructorId</th>\n",
              "      <th>grid</th>\n",
              "      <th>points</th>\n",
              "      <th>laps</th>\n",
              "      <th>milliseconds</th>\n",
              "      <th>circuitId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>18</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>10.0</td>\n",
              "      <td>58</td>\n",
              "      <td>5690616</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>18</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>8.0</td>\n",
              "      <td>58</td>\n",
              "      <td>5696094</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>18</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>6.0</td>\n",
              "      <td>58</td>\n",
              "      <td>5698779</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>18</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "      <td>5.0</td>\n",
              "      <td>58</td>\n",
              "      <td>5707797</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>18</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4.0</td>\n",
              "      <td>58</td>\n",
              "      <td>5708630</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24711</th>\n",
              "      <td>1035</td>\n",
              "      <td>20</td>\n",
              "      <td>6</td>\n",
              "      <td>11</td>\n",
              "      <td>0.0</td>\n",
              "      <td>52</td>\n",
              "      <td>4855363</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24712</th>\n",
              "      <td>1035</td>\n",
              "      <td>832</td>\n",
              "      <td>1</td>\n",
              "      <td>12</td>\n",
              "      <td>0.0</td>\n",
              "      <td>52</td>\n",
              "      <td>4856063</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24720</th>\n",
              "      <td>1036</td>\n",
              "      <td>1</td>\n",
              "      <td>131</td>\n",
              "      <td>1</td>\n",
              "      <td>25.0</td>\n",
              "      <td>66</td>\n",
              "      <td>5505279</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24721</th>\n",
              "      <td>1036</td>\n",
              "      <td>830</td>\n",
              "      <td>9</td>\n",
              "      <td>3</td>\n",
              "      <td>18.0</td>\n",
              "      <td>66</td>\n",
              "      <td>5529456</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24722</th>\n",
              "      <td>1036</td>\n",
              "      <td>822</td>\n",
              "      <td>131</td>\n",
              "      <td>2</td>\n",
              "      <td>16.0</td>\n",
              "      <td>66</td>\n",
              "      <td>5550031</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>6452 rows × 8 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "       raceId  driverId  constructorId  ...  laps  milliseconds  circuitId\n",
              "0          18         1              1  ...    58       5690616          1\n",
              "1          18         2              2  ...    58       5696094          1\n",
              "2          18         3              3  ...    58       5698779          1\n",
              "3          18         4              4  ...    58       5707797          1\n",
              "4          18         5              1  ...    58       5708630          1\n",
              "...       ...       ...            ...  ...   ...           ...        ...\n",
              "24711    1035        20              6  ...    52       4855363          9\n",
              "24712    1035       832              1  ...    52       4856063          9\n",
              "24720    1036         1            131  ...    66       5505279          4\n",
              "24721    1036       830              9  ...    66       5529456          4\n",
              "24722    1036       822            131  ...    66       5550031          4\n",
              "\n",
              "[6452 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RbR392i8W959",
        "outputId": "ff1d95d0-fad9-4510-fe5d-78473ec0d420",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "labels_results"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0                 TOP5\n",
              "1                 TOP5\n",
              "2                 TOP5\n",
              "3                 TOP5\n",
              "4                 TOP5\n",
              "             ...      \n",
              "24711    EL_TO_FIFTEEN\n",
              "24712    EL_TO_FIFTEEN\n",
              "24720             TOP5\n",
              "24721             TOP5\n",
              "24722             TOP5\n",
              "Name: beans, Length: 6452, dtype: category\n",
              "Categories (4, object): ['EL_TO_FIFTEEN', 'SIXTEEN_TO_TWENTY', 'SIX_TO_TEN', 'TOP5']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gCZo_TIiYcug"
      },
      "source": [
        "feature_train, feature_test, label_train, label_test = train_test_split(features_results, labels_results, test_size=0.2,random_state=1)\n",
        "feature_train, feature_validation, label_train, label_validation =  train_test_split(feature_train, label_train, test_size=0.25,random_state=1)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-YEeG_A4peS",
        "outputId": "d447c29f-1278-4c3a-bea9-0590366b3f1a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(feature_train.shape)\n",
        "print(feature_test.shape)\n",
        "print(feature_validation.shape)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(3870, 8)\n",
            "(1291, 8)\n",
            "(1291, 8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nkdr8yL5x2rT"
      },
      "source": [
        "Dicionário para guardar as métricas\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Km1nfVs_xzxQ"
      },
      "source": [
        "stats = {}"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL9wUFQ6xNFq"
      },
      "source": [
        "# MLFLOW\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDJ9oWXiGHmD",
        "outputId": "8dc06a44-74ba-4ebe-ac60-ec6eafa8b913",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install mlflow --quiet\n",
        "!pip install pyngrok --quiet\n",
        "\n",
        "import mlflow\n",
        "\n",
        "with mlflow.start_run(run_name=\"MLflow on Colab\"):\n",
        "  mlflow.log_metric(\"m1\", 2.0)\n",
        "  mlflow.log_param(\"p1\", \"mlflow-colab\")\n",
        "\n",
        "# run tracking UI in the background\n",
        "get_ipython().system_raw(\"mlflow ui --port 5000 &\") # run tracking UI in the background\n",
        "\n",
        "\n",
        "# create remote tunnel using ngrok.com to allow local port access\n",
        "# borrowed from https://colab.research.google.com/github/alfozan/MLflow-GBRT-demo/blob/master/MLflow-GBRT-demo.ipynb#scrollTo=4h3bKHMYUIG6\n",
        "\n",
        "from pyngrok import ngrok\n",
        "\n",
        "# Terminate open tunnels if exist\n",
        "ngrok.kill()\n",
        "\n",
        "# Setting the authtoken (optional)\n",
        "# Get your authtoken from https://dashboard.ngrok.com/auth\n",
        "NGROK_AUTH_TOKEN = \"\"\n",
        "ngrok.set_auth_token(NGROK_AUTH_TOKEN)\n",
        "\n",
        "# Open an HTTPs tunnel on port 5000 for http://localhost:5000\n",
        "ngrok_tunnel = ngrok.connect(addr=\"5000\", proto=\"http\", bind_tls=True)\n",
        "print(\"MLflow Tracking UI:\", ngrok_tunnel.public_url)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 13.9MB 307kB/s \n",
            "\u001b[K     |████████████████████████████████| 153kB 46.6MB/s \n",
            "\u001b[K     |████████████████████████████████| 163kB 40.8MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.1MB 46.5MB/s \n",
            "\u001b[K     |████████████████████████████████| 81kB 9.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 61kB 7.9MB/s \n",
            "\u001b[K     |████████████████████████████████| 337kB 37.2MB/s \n",
            "\u001b[K     |████████████████████████████████| 204kB 31.9MB/s \n",
            "\u001b[K     |████████████████████████████████| 71kB 8.5MB/s \n",
            "\u001b[K     |████████████████████████████████| 81kB 9.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 92kB 10.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 133kB 38.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 2.6MB 51.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 51kB 5.8MB/s \n",
            "\u001b[?25h  Building wheel for prometheus-flask-exporter (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for alembic (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for databricks-cli (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for pyngrok (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "MLflow Tracking UI: https://0a8a4a4276bc.ngrok.io\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgQx_OHkusWL"
      },
      "source": [
        "# Criação, treinamento e avaliação dos modelos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s393ewI3yAas"
      },
      "source": [
        "Modelos escolhidos:\n",
        "\n",
        "\n",
        "*   Naive bayes\n",
        "*   KNN\n",
        "*   MLP\n",
        "*   Random forest\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yKFFOROQzRZb"
      },
      "source": [
        "from sklearn.metrics import accuracy_score"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DulPFloK0VjO"
      },
      "source": [
        "Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vjJRwqH1x6-x",
        "outputId": "9282993a-8a9a-414f-b3e2-ed339ea33166",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.naive_bayes import GaussianNB \n",
        "model_naive = GaussianNB()                       \n",
        "model_naive.fit(feature_train, label_train)                  \n",
        "\n",
        "prediction_test = model_naive.predict(feature_test)\n",
        "prediction_train = model_naive.predict(feature_train)   \n",
        "\n",
        "stats['naive_bayes_test'] = accuracy_score(label_test, prediction_test)\n",
        "stats['naive_bayes_train'] = accuracy_score(label_train, prediction_train)\n",
        "\n",
        "print('precisão nos casos de teste: ' + str(stats['naive_bayes_test']))\n",
        "print('precisão nos casos de treino: ' + str(stats['naive_bayes_train']))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "precisão nos casos de teste: 0.6893880712625872\n",
            "precisão nos casos de treino: 0.6545219638242894\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7qPax-F0Xvj"
      },
      "source": [
        "KNN"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZnGQAZDlYsn6",
        "outputId": "5d0aef01-5be5-477d-f3fb-d2b1b3c00b11",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "model_knn = KNeighborsClassifier(n_neighbors=4)             \n",
        "model_knn.fit(feature_train, label_train) \n",
        "\n",
        "prediction_test = model_knn.predict(feature_test)\n",
        "prediction_train = model_knn.predict(feature_train)   \n",
        "\n",
        "stats['knn_test'] = accuracy_score(label_test, prediction_test)\n",
        "stats['knn_train'] = accuracy_score(label_train, prediction_train)\n",
        "\n",
        "print('precisão nos casos de teste: ' + str(stats['knn_test']))\n",
        "print('precisão nos casos de treino: ' + str(stats['knn_train']))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "precisão nos casos de teste: 0.5894655305964369\n",
            "precisão nos casos de treino: 0.7423772609819121\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0L6gST1m07iz"
      },
      "source": [
        "MLP\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HOAJpRSS-7wk",
        "outputId": "6f9b4896-f8fb-4f91-aa51-903c8e99a0fd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.neural_network import MLPClassifier\n",
        "model_mlp = MLPClassifier(hidden_layer_sizes=(150,100,50), max_iter=300,activation = 'relu',solver='sgd',random_state=1)\n",
        "model_mlp.fit(feature_train, label_train) \n",
        "\n",
        "prediction_test = model_mlp.predict(feature_test)\n",
        "prediction_train = model_mlp.predict(feature_train)   \n",
        "\n",
        "stats['mlp_test'] = accuracy_score(label_test, prediction_test)\n",
        "stats['mlp_train'] = accuracy_score(label_train, prediction_train)\n",
        "\n",
        "print('precisão nos casos de teste: ' + str(stats['mlp_test']))\n",
        "print('precisão nos casos de treino: ' + str(stats['mlp_train']))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "precisão nos casos de teste: 0.6901626646010844\n",
            "precisão nos casos de treino: 0.6591731266149871\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3qIrVCgS3eaH"
      },
      "source": [
        "Random Forrest"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gy0TLpvW3hDq",
        "outputId": "0c619305-6f5f-4043-c0a1-9decc8042f87",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "model_forrest = RandomForestClassifier(n_estimators=1000)\n",
        "model_forrest.fit(feature_train, label_train) \n",
        "\n",
        "prediction_test = model_forrest.predict(feature_test)\n",
        "prediction_train = model_forrest.predict(feature_train)   \n",
        "\n",
        "stats['forest_test'] = accuracy_score(label_test, prediction_test)\n",
        "stats['forest_train'] = accuracy_score(label_train, prediction_train)\n",
        "\n",
        "print('precisão nos casos de teste: ' + str(stats['forest_test']))\n",
        "print('precisão nos casos de treino: ' + str(stats['forest_train']))\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "precisão nos casos de teste: 0.975987606506584\n",
            "precisão nos casos de treino: 1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qSJxqQuF7LV7",
        "outputId": "621f2b43-5fa9-48ff-bfe7-e2bdf793dc4a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "stats"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'forrest_test': 0.975987606506584,\n",
              " 'forrest_train': 1.0,\n",
              " 'knn_test': 0.5894655305964369,\n",
              " 'knn_train': 0.7423772609819121,\n",
              " 'mlp_test': 0.6901626646010844,\n",
              " 'mlp_train': 0.6591731266149871,\n",
              " 'naive_bayes_test': 0.6893880712625872,\n",
              " 'naive_bayes_train': 0.6545219638242894}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nJwPjIue7f9O"
      },
      "source": [
        "#Adição das métricas de cada algoritmo ao MLFLow"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ucfGGOmt7yW1"
      },
      "source": [
        "mlflow.log_metric(\"NaiveBayes\",stats['naive_bayes_test'])\n",
        "mlflow.log_metric(\"KNN\",stats['knn_test'])\n",
        "mlflow.log_metric(\"MLP\",stats['mlp_test'])\n",
        "mlflow.log_metric(\"RandomForest\",stats['forest_test'])"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDP2U9grx401"
      },
      "source": [
        "# Seleção de Hiper-parâmetros"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzC7-I5fyx-c",
        "outputId": "d4c9b2a8-51d4-40b3-f01c-07efb9e37ef9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install optuna"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting optuna\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/87/10/06b58f4120f26b603d905a594650440ea1fd74476b8b360dbf01e111469b/optuna-2.3.0.tar.gz (258kB)\n",
            "\r\u001b[K     |█▎                              | 10kB 18.0MB/s eta 0:00:01\r\u001b[K     |██▌                             | 20kB 11.0MB/s eta 0:00:01\r\u001b[K     |███▉                            | 30kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████                           | 40kB 6.7MB/s eta 0:00:01\r\u001b[K     |██████▍                         | 51kB 4.3MB/s eta 0:00:01\r\u001b[K     |███████▋                        | 61kB 4.9MB/s eta 0:00:01\r\u001b[K     |████████▉                       | 71kB 5.0MB/s eta 0:00:01\r\u001b[K     |██████████▏                     | 81kB 5.6MB/s eta 0:00:01\r\u001b[K     |███████████▍                    | 92kB 5.7MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 102kB 5.4MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 112kB 5.4MB/s eta 0:00:01\r\u001b[K     |███████████████▏                | 122kB 5.4MB/s eta 0:00:01\r\u001b[K     |████████████████▌               | 133kB 5.4MB/s eta 0:00:01\r\u001b[K     |█████████████████▊              | 143kB 5.4MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 153kB 5.4MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 163kB 5.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████▌          | 174kB 5.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████▉         | 184kB 5.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 194kB 5.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 204kB 5.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▋     | 215kB 5.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▉    | 225kB 5.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▏  | 235kB 5.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▍ | 245kB 5.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▊| 256kB 5.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 266kB 5.4MB/s \n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.6/dist-packages (from optuna) (20.4)\n",
            "Requirement already satisfied: alembic in /usr/local/lib/python3.6/dist-packages (from optuna) (1.4.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from optuna) (4.41.1)\n",
            "Collecting cmaes>=0.6.0\n",
            "  Downloading https://files.pythonhosted.org/packages/8d/3c/06c76ec8b54b9b1fad7f35e903fd25010fe3e0d41bd94cea5e6f12e0d651/cmaes-0.7.0-py3-none-any.whl\n",
            "Collecting cliff\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/8a/61/5b64d73b01c1218f55c894b5ec0fb89b32c6960b7f7b3ad9f5ac0c373b9d/cliff-3.5.0-py3-none-any.whl (81kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 5.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from optuna) (1.18.5)\n",
            "Collecting colorlog\n",
            "  Downloading https://files.pythonhosted.org/packages/4e/c8/c16d30bbed11a1722060014c246d124582d1f781b26f5859d8dacc3e08e1/colorlog-4.6.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: scipy!=1.4.0 in /usr/local/lib/python3.6/dist-packages (from optuna) (1.4.1)\n",
            "Requirement already satisfied: sqlalchemy>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from optuna) (1.3.20)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from optuna) (0.17.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging>=20.0->optuna) (2.4.7)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging>=20.0->optuna) (1.15.0)\n",
            "Requirement already satisfied: Mako in /usr/local/lib/python3.6/dist-packages (from alembic->optuna) (1.1.3)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from alembic->optuna) (2.8.1)\n",
            "Requirement already satisfied: python-editor>=0.3 in /usr/local/lib/python3.6/dist-packages (from alembic->optuna) (1.0.4)\n",
            "Collecting pbr!=2.1.0,>=2.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fb/48/69046506f6ac61c1eaa9a0d42d22d54673b69e176d30ca98e3f61513e980/pbr-5.5.1-py2.py3-none-any.whl (106kB)\n",
            "\u001b[K     |████████████████████████████████| 112kB 8.2MB/s \n",
            "\u001b[?25hCollecting cmd2!=0.8.3,>=0.8.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/27/54/af6e2703f064485d717cb311d3f9440cd302a823ba6d80a020b59eae166d/cmd2-1.4.0-py3-none-any.whl (133kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 7.8MB/s \n",
            "\u001b[?25hCollecting PrettyTable<0.8,>=0.7.2\n",
            "  Downloading https://files.pythonhosted.org/packages/ef/30/4b0746848746ed5941f052479e7c23d2b56d174b82f4fd34a25e389831f5/prettytable-0.7.2.tar.bz2\n",
            "Collecting stevedore>=2.0.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b8/a1/004f04ba411a8002b02aadb089fd6868116c12ddc9f6d576175e89d07587/stevedore-3.2.2-py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 6.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: PyYAML>=3.12 in /usr/local/lib/python3.6/dist-packages (from cliff->optuna) (3.13)\n",
            "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.6/dist-packages (from Mako->alembic->optuna) (1.1.1)\n",
            "Collecting colorama>=0.3.7\n",
            "  Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl\n",
            "Collecting pyperclip>=1.6\n",
            "  Downloading https://files.pythonhosted.org/packages/6f/4c/0b1d507ad7e8bc31d690d04b4f475e74c2002d060f7994ce8c09612df707/pyperclip-1.8.1.tar.gz\n",
            "Requirement already satisfied: attrs>=16.3.0 in /usr/local/lib/python3.6/dist-packages (from cmd2!=0.8.3,>=0.8.0->cliff->optuna) (20.2.0)\n",
            "Requirement already satisfied: importlib-metadata>=1.6.0; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from cmd2!=0.8.3,>=0.8.0->cliff->optuna) (2.0.0)\n",
            "Requirement already satisfied: wcwidth>=0.1.7 in /usr/local/lib/python3.6/dist-packages (from cmd2!=0.8.3,>=0.8.0->cliff->optuna) (0.2.5)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=1.6.0; python_version < \"3.8\"->cmd2!=0.8.3,>=0.8.0->cliff->optuna) (3.4.0)\n",
            "Building wheels for collected packages: optuna\n",
            "  Building wheel for optuna (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for optuna: filename=optuna-2.3.0-cp36-none-any.whl size=359761 sha256=95b5a431805ebdb3dc19f3adf2e778d5d88e35a8fd102b96a1ebc557cd2f6d04\n",
            "  Stored in directory: /root/.cache/pip/wheels/fa/91/19/64b0ec6b964f89c0695a9dc6db6f851d0b54c5381a5c9cadfb\n",
            "Successfully built optuna\n",
            "Building wheels for collected packages: PrettyTable, pyperclip\n",
            "  Building wheel for PrettyTable (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for PrettyTable: filename=prettytable-0.7.2-cp36-none-any.whl size=13700 sha256=9210d4ce10b05692605e5fa7c1a5c2adebbfb6b2c7038d77c687b40a61cb58a2\n",
            "  Stored in directory: /root/.cache/pip/wheels/80/34/1c/3967380d9676d162cb59513bd9dc862d0584e045a162095606\n",
            "  Building wheel for pyperclip (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyperclip: filename=pyperclip-1.8.1-cp36-none-any.whl size=11119 sha256=2e2e513efc4a8659e74f8253bb8ce35f5ddc353cf82de697ec627da23c1386b9\n",
            "  Stored in directory: /root/.cache/pip/wheels/44/10/3a/c830e9bb3db2c93274ea1f213a41fabde0d8cf3794251fad0c\n",
            "Successfully built PrettyTable pyperclip\n",
            "Installing collected packages: cmaes, pbr, colorama, pyperclip, cmd2, PrettyTable, stevedore, cliff, colorlog, optuna\n",
            "  Found existing installation: prettytable 1.0.1\n",
            "    Uninstalling prettytable-1.0.1:\n",
            "      Successfully uninstalled prettytable-1.0.1\n",
            "Successfully installed PrettyTable-0.7.2 cliff-3.5.0 cmaes-0.7.0 cmd2-1.4.0 colorama-0.4.4 colorlog-4.6.2 optuna-2.3.0 pbr-5.5.1 pyperclip-1.8.1 stevedore-3.2.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ixpxmwNrz3Og"
      },
      "source": [
        "Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CKpdWTlnzhjk",
        "outputId": "63f13687-4737-4dc4-90bf-7bbce55980d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import optuna\n",
        "\n",
        "def objective(trial):\n",
        "    nb_var_smoothing = trial.suggest_float('var_smoothing', 1e-9, 1e9, log=True)\n",
        "    naive_obj = GaussianNB(var_smoothing=nb_var_smoothing)\n",
        "    naive_obj.fit(feature_train, label_train)     \n",
        "\n",
        "    prediction_test = naive_obj.predict(feature_test)\n",
        "    prediction_train = naive_obj.predict(feature_train)   \n",
        "\n",
        "    accuracy_test = accuracy_score(label_test, prediction_test)\n",
        "    accuracy_train = accuracy_score(label_train, prediction_train)\n",
        "    return accuracy_test\n",
        "\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=100)\n",
        "print(study.best_trial)\n",
        "stats['NaiveOptMetric'] = study.best_trial.value\n",
        "stats['NaiveOptParams'] = study.best_trial.params"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2020-11-15 22:11:09,021]\u001b[0m A new study created in memory with name: no-name-eba3c953-e4df-4dbc-8e0b-a3241b220706\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,046]\u001b[0m Trial 0 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 1.548056178019961e-06}. Best is trial 0 with value: 0.6816421378776143.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,066]\u001b[0m Trial 1 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 178685.0283443523}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,085]\u001b[0m Trial 2 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1.88197156094464e-09}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,103]\u001b[0m Trial 3 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 3.075673166707541}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,120]\u001b[0m Trial 4 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1883.276179373984}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,137]\u001b[0m Trial 5 finished with value: 0.6800929512006196 and parameters: {'var_smoothing': 3.059298303343548e-08}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,155]\u001b[0m Trial 6 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 16328.8866714138}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,172]\u001b[0m Trial 7 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 388250978.7551538}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,189]\u001b[0m Trial 8 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 3033644.1639608205}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,207]\u001b[0m Trial 9 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 3.7557558762707868}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,227]\u001b[0m Trial 10 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 0.0006096972649707702}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,248]\u001b[0m Trial 11 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 552814192.4958407}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,268]\u001b[0m Trial 12 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 0.0006322129219687833}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,287]\u001b[0m Trial 13 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 6981.077303397047}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,307]\u001b[0m Trial 14 finished with value: 0.691711851278079 and parameters: {'var_smoothing': 4.973141888801734e-09}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,326]\u001b[0m Trial 15 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 0.0056587717006694556}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,345]\u001b[0m Trial 16 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2302199.566154424}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,363]\u001b[0m Trial 17 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 10041705.363157991}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,383]\u001b[0m Trial 18 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 102.61275203450785}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,402]\u001b[0m Trial 19 finished with value: 0.6831913245546088 and parameters: {'var_smoothing': 0.10622082175903814}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,421]\u001b[0m Trial 20 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 1.5265863583898173e-06}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,442]\u001b[0m Trial 21 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1.6193683110601464e-09}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,462]\u001b[0m Trial 22 finished with value: 0.6886134779240899 and parameters: {'var_smoothing': 1.0039952097449505e-08}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,484]\u001b[0m Trial 23 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 424983010.7756639}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,502]\u001b[0m Trial 24 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 9.930905341499482e-07}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,522]\u001b[0m Trial 25 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 28.539079922478894}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,550]\u001b[0m Trial 26 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 8676757.418114448}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,570]\u001b[0m Trial 27 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2091.441860382837}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,589]\u001b[0m Trial 28 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 51.54149428808278}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,608]\u001b[0m Trial 29 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 0.0443432919524885}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,628]\u001b[0m Trial 30 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 2.1200793483961452e-05}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,649]\u001b[0m Trial 31 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 1.2125042729736488e-07}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,669]\u001b[0m Trial 32 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1.864235401548747e-09}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,687]\u001b[0m Trial 33 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 262038.8739271781}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,706]\u001b[0m Trial 34 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 825.624005021684}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,725]\u001b[0m Trial 35 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 4.303734267958728e-05}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,745]\u001b[0m Trial 36 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 46.855319253064955}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,764]\u001b[0m Trial 37 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 94737.88504324626}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,782]\u001b[0m Trial 38 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 673.451309151287}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,802]\u001b[0m Trial 39 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 55911531.6965686}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,821]\u001b[0m Trial 40 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1.2928108322107277}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,840]\u001b[0m Trial 41 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1.2123456489856972}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,860]\u001b[0m Trial 42 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 533.5079530108193}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,879]\u001b[0m Trial 43 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 114644853.97526763}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,899]\u001b[0m Trial 44 finished with value: 0.6893880712625872 and parameters: {'var_smoothing': 1.4051735788850811e-09}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,917]\u001b[0m Trial 45 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 5704.775473328095}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,936]\u001b[0m Trial 46 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 18.57547675307822}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,955]\u001b[0m Trial 47 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 107471.89957540305}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,973]\u001b[0m Trial 48 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1804335.1266210587}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:09,993]\u001b[0m Trial 49 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 209810.51254981948}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,015]\u001b[0m Trial 50 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 568759742.7727687}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,037]\u001b[0m Trial 51 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 10.727046623733967}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,057]\u001b[0m Trial 52 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 105.95463340425412}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,081]\u001b[0m Trial 53 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 33048.05441085445}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,101]\u001b[0m Trial 54 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 738283.796527958}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,121]\u001b[0m Trial 55 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 27226.39492225645}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,142]\u001b[0m Trial 56 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1494.3540658495006}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,161]\u001b[0m Trial 57 finished with value: 0.6723470178156468 and parameters: {'var_smoothing': 1.3581417764625947e-08}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,179]\u001b[0m Trial 58 finished with value: 0.6831913245546088 and parameters: {'var_smoothing': 0.1503028808379422}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,199]\u001b[0m Trial 59 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 53235353.497246064}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,219]\u001b[0m Trial 60 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 16462093.588031761}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,241]\u001b[0m Trial 61 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 15471949.3317292}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,260]\u001b[0m Trial 62 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 6.439642451545347}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,279]\u001b[0m Trial 63 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 121.42777126943946}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,298]\u001b[0m Trial 64 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 167.63733006802065}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,318]\u001b[0m Trial 65 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 521.8394374664075}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,336]\u001b[0m Trial 66 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2.59996994415789}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,356]\u001b[0m Trial 67 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 116621.98059970065}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,375]\u001b[0m Trial 68 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 6480.840800625895}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,395]\u001b[0m Trial 69 finished with value: 0.6847405112316034 and parameters: {'var_smoothing': 0.35004882814436644}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,415]\u001b[0m Trial 70 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 6970138.12679325}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,436]\u001b[0m Trial 71 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 566132.7220276644}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,455]\u001b[0m Trial 72 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 1.0994954198872155e-07}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,474]\u001b[0m Trial 73 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 249378.7842392157}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,494]\u001b[0m Trial 74 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 10.449714727534007}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,516]\u001b[0m Trial 75 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 51258.02658346259}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,535]\u001b[0m Trial 76 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2071018.8443161545}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,554]\u001b[0m Trial 77 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 12121.987687389757}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,579]\u001b[0m Trial 78 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2606.5845781172884}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,599]\u001b[0m Trial 79 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 148532552.98852986}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,618]\u001b[0m Trial 80 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 667238316.7149206}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,638]\u001b[0m Trial 81 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 32094.833170217003}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,657]\u001b[0m Trial 82 finished with value: 0.6816421378776143 and parameters: {'var_smoothing': 0.006511764675294732}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,678]\u001b[0m Trial 83 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2950213.4781941995}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,698]\u001b[0m Trial 84 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 27480997.37511437}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,716]\u001b[0m Trial 85 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 571356.3315229603}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,735]\u001b[0m Trial 86 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 26689.378483871336}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,754]\u001b[0m Trial 87 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 30391192.992422517}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,773]\u001b[0m Trial 88 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 146684196.73303625}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,792]\u001b[0m Trial 89 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 1312783.3538000444}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,811]\u001b[0m Trial 90 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 14.92734762892649}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,830]\u001b[0m Trial 91 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 194.70496035747618}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,852]\u001b[0m Trial 92 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 443.11309765647}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,872]\u001b[0m Trial 93 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2.950004283370823}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,893]\u001b[0m Trial 94 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 2.9094420719513443}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,915]\u001b[0m Trial 95 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 153.46976433662851}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,936]\u001b[0m Trial 96 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 38.07302140105767}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,956]\u001b[0m Trial 97 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 4263.212662112233}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,975]\u001b[0m Trial 98 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 333.20537528812116}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:10,995]\u001b[0m Trial 99 finished with value: 0.6901626646010844 and parameters: {'var_smoothing': 8241.800337656021}. Best is trial 14 with value: 0.691711851278079.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "FrozenTrial(number=14, value=0.691711851278079, datetime_start=datetime.datetime(2020, 11, 15, 22, 11, 9, 288766), datetime_complete=datetime.datetime(2020, 11, 15, 22, 11, 9, 307317), params={'var_smoothing': 4.973141888801734e-09}, distributions={'var_smoothing': LogUniformDistribution(high=1000000000.0, low=1e-09)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=14, state=TrialState.COMPLETE)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQkg-ZYN4liK"
      },
      "source": [
        "KNN\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z1K-kXtz4nl9",
        "outputId": "eab57853-4690-4e3a-8efe-124276f1e949",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def objective(trial):\n",
        "    knn_n_neighbors = trial.suggest_int('n_neighbors', 2, 20)\n",
        "    knn_weights = trial.suggest_categorical('weights', ['uniform', 'distance'])\n",
        "    knn_algorithm = trial.suggest_categorical('algorithm', ['auto', 'ball_tree', 'kd_tree', 'brute'])\n",
        "\n",
        "    knn_obj = KNeighborsClassifier(n_neighbors=knn_n_neighbors, weights=knn_weights, algorithm=knn_algorithm)\n",
        "    knn_obj.fit(feature_train, label_train)     \n",
        "\n",
        "    prediction_test = knn_obj.predict(feature_test)\n",
        "    prediction_train = knn_obj.predict(feature_train)   \n",
        "\n",
        "    accuracy_test = accuracy_score(label_test, prediction_test)\n",
        "    accuracy_train = accuracy_score(label_train, prediction_train)\n",
        "    return accuracy_test\n",
        "\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=100)\n",
        "print(study.best_trial)\n",
        "stats['KNNOptMetric'] = study.best_trial.value\n",
        "stats['KNNOptParams'] = study.best_trial.params"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2020-11-15 22:11:19,234]\u001b[0m A new study created in memory with name: no-name-bba37ac4-4682-4a37-85cb-6b51b0f36df9\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:19,805]\u001b[0m Trial 0 finished with value: 0.6467854376452362 and parameters: {'n_neighbors': 15, 'weights': 'distance', 'algorithm': 'brute'}. Best is trial 0 with value: 0.6467854376452362.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:19,973]\u001b[0m Trial 1 finished with value: 0.6390395042602633 and parameters: {'n_neighbors': 7, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 0 with value: 0.6467854376452362.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:20,158]\u001b[0m Trial 2 finished with value: 0.65143299767622 and parameters: {'n_neighbors': 9, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 2 with value: 0.65143299767622.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:20,771]\u001b[0m Trial 3 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:20,814]\u001b[0m Trial 4 finished with value: 0.6359411309062742 and parameters: {'n_neighbors': 9, 'weights': 'distance', 'algorithm': 'kd_tree'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:21,295]\u001b[0m Trial 5 finished with value: 0.6537567776917118 and parameters: {'n_neighbors': 18, 'weights': 'distance', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:21,479]\u001b[0m Trial 6 finished with value: 0.5894655305964369 and parameters: {'n_neighbors': 4, 'weights': 'uniform', 'algorithm': 'ball_tree'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:21,977]\u001b[0m Trial 7 finished with value: 0.6498838109992254 and parameters: {'n_neighbors': 16, 'weights': 'distance', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:22,058]\u001b[0m Trial 8 finished with value: 0.6553059643687065 and parameters: {'n_neighbors': 19, 'weights': 'distance', 'algorithm': 'ball_tree'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:22,228]\u001b[0m Trial 9 finished with value: 0.65143299767622 and parameters: {'n_neighbors': 9, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:22,841]\u001b[0m Trial 10 finished with value: 0.6560805577072037 and parameters: {'n_neighbors': 14, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:23,450]\u001b[0m Trial 11 finished with value: 0.6560805577072037 and parameters: {'n_neighbors': 14, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:24,059]\u001b[0m Trial 12 finished with value: 0.6522075910147173 and parameters: {'n_neighbors': 13, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:24,676]\u001b[0m Trial 13 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:24,872]\u001b[0m Trial 14 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:25,490]\u001b[0m Trial 15 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 3 with value: 0.6692486444616577.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:26,109]\u001b[0m Trial 16 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:26,292]\u001b[0m Trial 17 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:26,468]\u001b[0m Trial 18 finished with value: 0.6545313710302091 and parameters: {'n_neighbors': 12, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:26,647]\u001b[0m Trial 19 finished with value: 0.6638264910921766 and parameters: {'n_neighbors': 16, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:26,834]\u001b[0m Trial 20 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,010]\u001b[0m Trial 21 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,192]\u001b[0m Trial 22 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,369]\u001b[0m Trial 23 finished with value: 0.6537567776917118 and parameters: {'n_neighbors': 11, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,553]\u001b[0m Trial 24 finished with value: 0.6607281177381874 and parameters: {'n_neighbors': 15, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,735]\u001b[0m Trial 25 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:27,944]\u001b[0m Trial 26 finished with value: 0.6638264910921766 and parameters: {'n_neighbors': 16, 'weights': 'uniform', 'algorithm': 'ball_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,116]\u001b[0m Trial 27 finished with value: 0.6522075910147173 and parameters: {'n_neighbors': 13, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 16 with value: 0.6715724244771495.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,291]\u001b[0m Trial 28 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,350]\u001b[0m Trial 29 finished with value: 0.6467854376452362 and parameters: {'n_neighbors': 15, 'weights': 'distance', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,518]\u001b[0m Trial 30 finished with value: 0.512006196746708 and parameters: {'n_neighbors': 2, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,695]\u001b[0m Trial 31 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:28,877]\u001b[0m Trial 32 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,085]\u001b[0m Trial 33 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,263]\u001b[0m Trial 34 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,442]\u001b[0m Trial 35 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,618]\u001b[0m Trial 36 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,799]\u001b[0m Trial 37 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:29,857]\u001b[0m Trial 38 finished with value: 0.6537567776917118 and parameters: {'n_neighbors': 18, 'weights': 'distance', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,039]\u001b[0m Trial 39 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,205]\u001b[0m Trial 40 finished with value: 0.6219984508133231 and parameters: {'n_neighbors': 5, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,387]\u001b[0m Trial 41 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,565]\u001b[0m Trial 42 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,754]\u001b[0m Trial 43 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,934]\u001b[0m Trial 44 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:30,987]\u001b[0m Trial 45 finished with value: 0.6553059643687065 and parameters: {'n_neighbors': 19, 'weights': 'distance', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:31,179]\u001b[0m Trial 46 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:31,398]\u001b[0m Trial 47 finished with value: 0.6638264910921766 and parameters: {'n_neighbors': 16, 'weights': 'uniform', 'algorithm': 'ball_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:31,578]\u001b[0m Trial 48 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:31,750]\u001b[0m Trial 49 finished with value: 0.6607281177381874 and parameters: {'n_neighbors': 15, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:31,922]\u001b[0m Trial 50 finished with value: 0.6390395042602633 and parameters: {'n_neighbors': 7, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:32,104]\u001b[0m Trial 51 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:32,283]\u001b[0m Trial 52 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:32,466]\u001b[0m Trial 53 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:32,642]\u001b[0m Trial 54 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:32,815]\u001b[0m Trial 55 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,000]\u001b[0m Trial 56 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,174]\u001b[0m Trial 57 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,226]\u001b[0m Trial 58 finished with value: 0.6498838109992254 and parameters: {'n_neighbors': 16, 'weights': 'distance', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,428]\u001b[0m Trial 59 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'ball_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,605]\u001b[0m Trial 60 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,778]\u001b[0m Trial 61 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:33,968]\u001b[0m Trial 62 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:34,147]\u001b[0m Trial 63 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:34,332]\u001b[0m Trial 64 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:34,950]\u001b[0m Trial 65 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:35,132]\u001b[0m Trial 66 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:35,313]\u001b[0m Trial 67 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:35,491]\u001b[0m Trial 68 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:35,677]\u001b[0m Trial 69 finished with value: 0.6560805577072037 and parameters: {'n_neighbors': 14, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:35,856]\u001b[0m Trial 70 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,038]\u001b[0m Trial 71 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,224]\u001b[0m Trial 72 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,403]\u001b[0m Trial 73 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,583]\u001b[0m Trial 74 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,777]\u001b[0m Trial 75 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:36,958]\u001b[0m Trial 76 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,157]\u001b[0m Trial 77 finished with value: 0.6638264910921766 and parameters: {'n_neighbors': 16, 'weights': 'uniform', 'algorithm': 'ball_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,333]\u001b[0m Trial 78 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,388]\u001b[0m Trial 79 finished with value: 0.6553059643687065 and parameters: {'n_neighbors': 19, 'weights': 'distance', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,571]\u001b[0m Trial 80 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,748]\u001b[0m Trial 81 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:37,931]\u001b[0m Trial 82 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:38,115]\u001b[0m Trial 83 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:38,292]\u001b[0m Trial 84 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:38,472]\u001b[0m Trial 85 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:38,660]\u001b[0m Trial 86 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:38,838]\u001b[0m Trial 87 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:39,031]\u001b[0m Trial 88 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'auto'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:39,649]\u001b[0m Trial 89 finished with value: 0.6638264910921766 and parameters: {'n_neighbors': 16, 'weights': 'uniform', 'algorithm': 'brute'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:39,823]\u001b[0m Trial 90 finished with value: 0.6444616576297444 and parameters: {'n_neighbors': 10, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,011]\u001b[0m Trial 91 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,202]\u001b[0m Trial 92 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,384]\u001b[0m Trial 93 finished with value: 0.6715724244771495 and parameters: {'n_neighbors': 17, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,563]\u001b[0m Trial 94 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,743]\u001b[0m Trial 95 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:40,920]\u001b[0m Trial 96 finished with value: 0.6661502711076684 and parameters: {'n_neighbors': 20, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:41,107]\u001b[0m Trial 97 finished with value: 0.6723470178156468 and parameters: {'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:41,286]\u001b[0m Trial 98 finished with value: 0.6692486444616577 and parameters: {'n_neighbors': 18, 'weights': 'uniform', 'algorithm': 'kd_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:11:41,367]\u001b[0m Trial 99 finished with value: 0.656855151045701 and parameters: {'n_neighbors': 20, 'weights': 'distance', 'algorithm': 'ball_tree'}. Best is trial 28 with value: 0.6723470178156468.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "FrozenTrial(number=28, value=0.6723470178156468, datetime_start=datetime.datetime(2020, 11, 15, 22, 11, 28, 117838), datetime_complete=datetime.datetime(2020, 11, 15, 22, 11, 28, 291067), params={'n_neighbors': 19, 'weights': 'uniform', 'algorithm': 'kd_tree'}, distributions={'n_neighbors': IntUniformDistribution(high=20, low=2, step=1), 'weights': CategoricalDistribution(choices=('uniform', 'distance')), 'algorithm': CategoricalDistribution(choices=('auto', 'ball_tree', 'kd_tree', 'brute'))}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=28, state=TrialState.COMPLETE)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GF5o4HMc6w88"
      },
      "source": [
        "MLP"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ilBzcoP6y9l",
        "outputId": "8892f1a5-0790-4eb5-8744-e7733731d789",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def objective(trial):\n",
        "    mlp_max_iter = trial.suggest_int('max_iter', 50, 300)\n",
        "    mlp_activation = trial.suggest_categorical('activation', ['identity', 'logistic', 'tanh', 'relu'])\n",
        "    mlp_solver = trial.suggest_categorical('solver', ['lbfgs', 'sgd', 'adam'])\n",
        "\n",
        "    mlp_obj = MLPClassifier(hidden_layer_sizes=(150,100,50), max_iter=mlp_max_iter,activation = mlp_activation,solver=mlp_solver,random_state=1)\n",
        "    mlp_obj.fit(feature_train, label_train) \n",
        "\n",
        "    prediction_test = mlp_obj.predict(feature_test)\n",
        "    prediction_train = mlp_obj.predict(feature_train)   \n",
        "\n",
        "    accuracy_test = accuracy_score(label_test, prediction_test)\n",
        "    accuracy_train = accuracy_score(label_train, prediction_train)\n",
        "    return accuracy_test\n",
        "\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=100)\n",
        "print(study.best_trial)\n",
        "stats['MLPOptMetric'] = study.best_trial.value\n",
        "stats['MLPOptParams'] = study.best_trial.params"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2020-11-15 22:12:03,569]\u001b[0m A new study created in memory with name: no-name-261e53af-a6ff-4c7e-bcf2-a8e233d5e05f\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning:\n",
            "\n",
            "Stochastic Optimizer: Maximum iterations (51) reached and the optimization hasn't converged yet.\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:12:06,434]\u001b[0m Trial 0 finished with value: 0.238574748257165 and parameters: {'max_iter': 51, 'activation': 'identity', 'solver': 'adam'}. Best is trial 0 with value: 0.238574748257165.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:08,672]\u001b[0m Trial 1 finished with value: 0.6901626646010844 and parameters: {'max_iter': 170, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:09,713]\u001b[0m Trial 2 finished with value: 0.6901626646010844 and parameters: {'max_iter': 108, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:10,746]\u001b[0m Trial 3 finished with value: 0.6901626646010844 and parameters: {'max_iter': 244, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:12:19,513]\u001b[0m Trial 4 finished with value: 0.004647560030983733 and parameters: {'max_iter': 70, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:20,546]\u001b[0m Trial 5 finished with value: 0.6901626646010844 and parameters: {'max_iter': 258, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning:\n",
            "\n",
            "Stochastic Optimizer: Maximum iterations (115) reached and the optimization hasn't converged yet.\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:12:29,240]\u001b[0m Trial 6 finished with value: 0.6901626646010844 and parameters: {'max_iter': 115, 'activation': 'relu', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:39,619]\u001b[0m Trial 7 finished with value: 0.6901626646010844 and parameters: {'max_iter': 106, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:44,614]\u001b[0m Trial 8 finished with value: 0.6901626646010844 and parameters: {'max_iter': 277, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:45,500]\u001b[0m Trial 9 finished with value: 0.6901626646010844 and parameters: {'max_iter': 130, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:47,745]\u001b[0m Trial 10 finished with value: 0.6901626646010844 and parameters: {'max_iter': 193, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:49,971]\u001b[0m Trial 11 finished with value: 0.6901626646010844 and parameters: {'max_iter': 181, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:50,989]\u001b[0m Trial 12 finished with value: 0.6901626646010844 and parameters: {'max_iter': 154, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:52,591]\u001b[0m Trial 13 finished with value: 0.6901626646010844 and parameters: {'max_iter': 207, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:53,819]\u001b[0m Trial 14 finished with value: 0.6901626646010844 and parameters: {'max_iter': 91, 'activation': 'logistic', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:56,310]\u001b[0m Trial 15 finished with value: 0.6901626646010844 and parameters: {'max_iter': 157, 'activation': 'tanh', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:12:58,557]\u001b[0m Trial 16 finished with value: 0.6901626646010844 and parameters: {'max_iter': 220, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:03,560]\u001b[0m Trial 17 finished with value: 0.6901626646010844 and parameters: {'max_iter': 294, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/extmath.py:151: RuntimeWarning:\n",
            "\n",
            "overflow encountered in matmul\n",
            "\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/extmath.py:151: RuntimeWarning:\n",
            "\n",
            "invalid value encountered in matmul\n",
            "\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning:\n",
            "\n",
            "Stochastic Optimizer: Maximum iterations (285) reached and the optimization hasn't converged yet.\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:13:17,371]\u001b[0m Trial 18 finished with value: 0.06661502711076685 and parameters: {'max_iter': 285, 'activation': 'identity', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:18,261]\u001b[0m Trial 19 finished with value: 0.6901626646010844 and parameters: {'max_iter': 146, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:19,138]\u001b[0m Trial 20 finished with value: 0.6901626646010844 and parameters: {'max_iter': 132, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:21,369]\u001b[0m Trial 21 finished with value: 0.6901626646010844 and parameters: {'max_iter': 220, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:26,332]\u001b[0m Trial 22 finished with value: 0.6901626646010844 and parameters: {'max_iter': 194, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:27,237]\u001b[0m Trial 23 finished with value: 0.6901626646010844 and parameters: {'max_iter': 137, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:29,479]\u001b[0m Trial 24 finished with value: 0.6901626646010844 and parameters: {'max_iter': 222, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:30,669]\u001b[0m Trial 25 finished with value: 0.6901626646010844 and parameters: {'max_iter': 238, 'activation': 'logistic', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:31,518]\u001b[0m Trial 26 finished with value: 0.6901626646010844 and parameters: {'max_iter': 162, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:33,120]\u001b[0m Trial 27 finished with value: 0.6901626646010844 and parameters: {'max_iter': 206, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:35,375]\u001b[0m Trial 28 finished with value: 0.6901626646010844 and parameters: {'max_iter': 242, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:13:41,517]\u001b[0m Trial 29 finished with value: 0.6901626646010844 and parameters: {'max_iter': 52, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:51,879]\u001b[0m Trial 30 finished with value: 0.6901626646010844 and parameters: {'max_iter': 82, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:13:54,337]\u001b[0m Trial 31 finished with value: 0.6901626646010844 and parameters: {'max_iter': 179, 'activation': 'tanh', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning:\n",
            "\n",
            "Stochastic Optimizer: Maximum iterations (264) reached and the optimization hasn't converged yet.\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:14:12,848]\u001b[0m Trial 32 finished with value: 0.238574748257165 and parameters: {'max_iter': 264, 'activation': 'identity', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:14:15,088]\u001b[0m Trial 33 finished with value: 0.6901626646010844 and parameters: {'max_iter': 232, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:14:22,070]\u001b[0m Trial 34 finished with value: 0.6692486444616577 and parameters: {'max_iter': 54, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:14:24,500]\u001b[0m Trial 35 finished with value: 0.6901626646010844 and parameters: {'max_iter': 291, 'activation': 'tanh', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:14:29,493]\u001b[0m Trial 36 finished with value: 0.6901626646010844 and parameters: {'max_iter': 182, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:14:38,512]\u001b[0m Trial 37 finished with value: 0.004647560030983733 and parameters: {'max_iter': 71, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning:\n",
            "\n",
            "Stochastic Optimizer: Maximum iterations (259) reached and the optimization hasn't converged yet.\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:14:56,374]\u001b[0m Trial 38 finished with value: 0.238574748257165 and parameters: {'max_iter': 259, 'activation': 'identity', 'solver': 'adam'}. Best is trial 1 with value: 0.6901626646010844.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:15:02,922]\u001b[0m Trial 39 finished with value: 0.691711851278079 and parameters: {'max_iter': 55, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:03,811]\u001b[0m Trial 40 finished with value: 0.6901626646010844 and parameters: {'max_iter': 202, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:04,694]\u001b[0m Trial 41 finished with value: 0.6901626646010844 and parameters: {'max_iter': 136, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:05,567]\u001b[0m Trial 42 finished with value: 0.6901626646010844 and parameters: {'max_iter': 114, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:06,457]\u001b[0m Trial 43 finished with value: 0.6901626646010844 and parameters: {'max_iter': 133, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:08,666]\u001b[0m Trial 44 finished with value: 0.6901626646010844 and parameters: {'max_iter': 219, 'activation': 'logistic', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:09,538]\u001b[0m Trial 45 finished with value: 0.6901626646010844 and parameters: {'max_iter': 169, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:10,725]\u001b[0m Trial 46 finished with value: 0.6901626646010844 and parameters: {'max_iter': 241, 'activation': 'logistic', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:21,134]\u001b[0m Trial 47 finished with value: 0.6901626646010844 and parameters: {'max_iter': 101, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:31,576]\u001b[0m Trial 48 finished with value: 0.6901626646010844 and parameters: {'max_iter': 170, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:41,951]\u001b[0m Trial 49 finished with value: 0.6901626646010844 and parameters: {'max_iter': 93, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:43,555]\u001b[0m Trial 50 finished with value: 0.6901626646010844 and parameters: {'max_iter': 230, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:15:53,929]\u001b[0m Trial 51 finished with value: 0.6901626646010844 and parameters: {'max_iter': 247, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:16:01,362]\u001b[0m Trial 52 finished with value: 0.238574748257165 and parameters: {'max_iter': 56, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:02,388]\u001b[0m Trial 53 finished with value: 0.6901626646010844 and parameters: {'max_iter': 267, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:16:11,093]\u001b[0m Trial 54 finished with value: 0.6893880712625872 and parameters: {'max_iter': 71, 'activation': 'relu', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:12,817]\u001b[0m Trial 55 finished with value: 0.6901626646010844 and parameters: {'max_iter': 274, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:13,889]\u001b[0m Trial 56 finished with value: 0.6901626646010844 and parameters: {'max_iter': 85, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:15,592]\u001b[0m Trial 57 finished with value: 0.6901626646010844 and parameters: {'max_iter': 275, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:16,659]\u001b[0m Trial 58 finished with value: 0.6901626646010844 and parameters: {'max_iter': 83, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:17,719]\u001b[0m Trial 59 finished with value: 0.6901626646010844 and parameters: {'max_iter': 67, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning:\n",
            "\n",
            "lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "\n",
            "\u001b[32m[I 2020-11-15 22:16:25,808]\u001b[0m Trial 60 finished with value: 0.06661502711076685 and parameters: {'max_iter': 62, 'activation': 'identity', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:26,830]\u001b[0m Trial 61 finished with value: 0.6901626646010844 and parameters: {'max_iter': 50, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:31,763]\u001b[0m Trial 62 finished with value: 0.6901626646010844 and parameters: {'max_iter': 181, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:34,229]\u001b[0m Trial 63 finished with value: 0.6901626646010844 and parameters: {'max_iter': 296, 'activation': 'tanh', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:39,163]\u001b[0m Trial 64 finished with value: 0.6901626646010844 and parameters: {'max_iter': 182, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:44,077]\u001b[0m Trial 65 finished with value: 0.6901626646010844 and parameters: {'max_iter': 189, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:48,986]\u001b[0m Trial 66 finished with value: 0.6901626646010844 and parameters: {'max_iter': 151, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:53,916]\u001b[0m Trial 67 finished with value: 0.6901626646010844 and parameters: {'max_iter': 197, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:16:58,881]\u001b[0m Trial 68 finished with value: 0.6901626646010844 and parameters: {'max_iter': 210, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:03,920]\u001b[0m Trial 69 finished with value: 0.6901626646010844 and parameters: {'max_iter': 211, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:08,865]\u001b[0m Trial 70 finished with value: 0.6901626646010844 and parameters: {'max_iter': 214, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:09,743]\u001b[0m Trial 71 finished with value: 0.6901626646010844 and parameters: {'max_iter': 112, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:10,624]\u001b[0m Trial 72 finished with value: 0.6901626646010844 and parameters: {'max_iter': 122, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:11,511]\u001b[0m Trial 73 finished with value: 0.6901626646010844 and parameters: {'max_iter': 113, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:12,388]\u001b[0m Trial 74 finished with value: 0.6901626646010844 and parameters: {'max_iter': 163, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:13,272]\u001b[0m Trial 75 finished with value: 0.6901626646010844 and parameters: {'max_iter': 142, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:14,163]\u001b[0m Trial 76 finished with value: 0.6901626646010844 and parameters: {'max_iter': 138, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:15,077]\u001b[0m Trial 77 finished with value: 0.6901626646010844 and parameters: {'max_iter': 140, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:15,968]\u001b[0m Trial 78 finished with value: 0.6901626646010844 and parameters: {'max_iter': 122, 'activation': 'relu', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:17,574]\u001b[0m Trial 79 finished with value: 0.6901626646010844 and parameters: {'max_iter': 273, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:19,167]\u001b[0m Trial 80 finished with value: 0.6901626646010844 and parameters: {'max_iter': 275, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:20,213]\u001b[0m Trial 81 finished with value: 0.6901626646010844 and parameters: {'max_iter': 252, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:21,254]\u001b[0m Trial 82 finished with value: 0.6901626646010844 and parameters: {'max_iter': 78, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:22,277]\u001b[0m Trial 83 finished with value: 0.6901626646010844 and parameters: {'max_iter': 249, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:23,308]\u001b[0m Trial 84 finished with value: 0.6901626646010844 and parameters: {'max_iter': 279, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:24,333]\u001b[0m Trial 85 finished with value: 0.6901626646010844 and parameters: {'max_iter': 266, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:25,404]\u001b[0m Trial 86 finished with value: 0.6901626646010844 and parameters: {'max_iter': 63, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:26,434]\u001b[0m Trial 87 finished with value: 0.6901626646010844 and parameters: {'max_iter': 95, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:27,486]\u001b[0m Trial 88 finished with value: 0.6901626646010844 and parameters: {'max_iter': 77, 'activation': 'tanh', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:29,115]\u001b[0m Trial 89 finished with value: 0.6901626646010844 and parameters: {'max_iter': 300, 'activation': 'tanh', 'solver': 'adam'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:30,311]\u001b[0m Trial 90 finished with value: 0.6901626646010844 and parameters: {'max_iter': 190, 'activation': 'logistic', 'solver': 'lbfgs'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:32,751]\u001b[0m Trial 91 finished with value: 0.6901626646010844 and parameters: {'max_iter': 184, 'activation': 'tanh', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:37,712]\u001b[0m Trial 92 finished with value: 0.6901626646010844 and parameters: {'max_iter': 152, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:42,641]\u001b[0m Trial 93 finished with value: 0.6901626646010844 and parameters: {'max_iter': 193, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:47,598]\u001b[0m Trial 94 finished with value: 0.6901626646010844 and parameters: {'max_iter': 203, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:52,494]\u001b[0m Trial 95 finished with value: 0.6901626646010844 and parameters: {'max_iter': 198, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:17:57,488]\u001b[0m Trial 96 finished with value: 0.6901626646010844 and parameters: {'max_iter': 169, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:18:02,448]\u001b[0m Trial 97 finished with value: 0.6901626646010844 and parameters: {'max_iter': 217, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:18:07,449]\u001b[0m Trial 98 finished with value: 0.6901626646010844 and parameters: {'max_iter': 212, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:18:12,363]\u001b[0m Trial 99 finished with value: 0.6901626646010844 and parameters: {'max_iter': 226, 'activation': 'logistic', 'solver': 'sgd'}. Best is trial 39 with value: 0.691711851278079.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "FrozenTrial(number=39, value=0.691711851278079, datetime_start=datetime.datetime(2020, 11, 15, 22, 14, 56, 376373), datetime_complete=datetime.datetime(2020, 11, 15, 22, 15, 2, 921913), params={'max_iter': 55, 'activation': 'relu', 'solver': 'lbfgs'}, distributions={'max_iter': IntUniformDistribution(high=300, low=50, step=1), 'activation': CategoricalDistribution(choices=('identity', 'logistic', 'tanh', 'relu')), 'solver': CategoricalDistribution(choices=('lbfgs', 'sgd', 'adam'))}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=39, state=TrialState.COMPLETE)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0eds0oD-pAh"
      },
      "source": [
        "Random Forrest\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qeg3hFmd-qiv",
        "outputId": "a132e96d-7dc0-4c87-e1f9-74b1d1661107",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def objective(trial):\n",
        "    forrest_n_estimators = trial.suggest_int('n_estimators', 100, 1000)\n",
        "    forrest_criterion = trial.suggest_categorical('criterion', ['gini', 'entropy'])\n",
        "    forrest_max_features = trial.suggest_categorical('max_features', ['auto', 'sqrt', 'log2'])\n",
        "\n",
        "    forrest_obj = RandomForestClassifier(n_estimators=forrest_n_estimators, criterion=forrest_criterion, max_features=forrest_max_features)\n",
        "    forrest_obj.fit(feature_train, label_train) \n",
        "\n",
        "    prediction_test = forrest_obj.predict(feature_test)\n",
        "    prediction_train = forrest_obj.predict(feature_train)   \n",
        "\n",
        "    accuracy_test = accuracy_score(label_test, prediction_test)\n",
        "    accuracy_train = accuracy_score(label_train, prediction_train)\n",
        "    return accuracy_test\n",
        "\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=100)\n",
        "print(study.best_trial)\n",
        "stats['ForestOptMetric'] = study.best_trial.value\n",
        "stats['ForestOptParams'] = study.best_trial.params"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[32m[I 2020-11-15 22:19:08,213]\u001b[0m A new study created in memory with name: no-name-6b99268e-f6d5-4a9f-b1dd-d35b2fc4a520\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:09,963]\u001b[0m Trial 0 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 245, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 0 with value: 0.9744384198295895.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:15,934]\u001b[0m Trial 1 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 754, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 1 with value: 0.9752130131680867.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:19,899]\u001b[0m Trial 2 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 750, 'criterion': 'gini', 'max_features': 'auto'}. Best is trial 1 with value: 0.9752130131680867.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:22,687]\u001b[0m Trial 3 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 395, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 1 with value: 0.9752130131680867.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:24,253]\u001b[0m Trial 4 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 296, 'criterion': 'gini', 'max_features': 'sqrt'}. Best is trial 1 with value: 0.9752130131680867.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:25,130]\u001b[0m Trial 5 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 141, 'criterion': 'gini', 'max_features': 'log2'}. Best is trial 1 with value: 0.9752130131680867.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:27,387]\u001b[0m Trial 6 finished with value: 0.975987606506584 and parameters: {'n_estimators': 321, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 6 with value: 0.975987606506584.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:32,290]\u001b[0m Trial 7 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 707, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 6 with value: 0.975987606506584.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:39,001]\u001b[0m Trial 8 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 845, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:40,062]\u001b[0m Trial 9 finished with value: 0.9713400464756003 and parameters: {'n_estimators': 194, 'criterion': 'gini', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:47,815]\u001b[0m Trial 10 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 967, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:51,344]\u001b[0m Trial 11 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 509, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:19:58,091]\u001b[0m Trial 12 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 977, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:02,658]\u001b[0m Trial 13 finished with value: 0.975987606506584 and parameters: {'n_estimators': 571, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:05,491]\u001b[0m Trial 14 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 405, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:12,599]\u001b[0m Trial 15 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 899, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:17,005]\u001b[0m Trial 16 finished with value: 0.9713400464756003 and parameters: {'n_estimators': 628, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:21,237]\u001b[0m Trial 17 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 532, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:28,095]\u001b[0m Trial 18 finished with value: 0.975987606506584 and parameters: {'n_estimators': 855, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:34,856]\u001b[0m Trial 19 finished with value: 0.975987606506584 and parameters: {'n_estimators': 853, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:41,523]\u001b[0m Trial 20 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 851, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:46,557]\u001b[0m Trial 21 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 629, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:53,203]\u001b[0m Trial 22 finished with value: 0.975987606506584 and parameters: {'n_estimators': 836, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:20:59,569]\u001b[0m Trial 23 finished with value: 0.975987606506584 and parameters: {'n_estimators': 796, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:07,428]\u001b[0m Trial 24 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 998, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:14,621]\u001b[0m Trial 25 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 909, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:20,806]\u001b[0m Trial 26 finished with value: 0.975987606506584 and parameters: {'n_estimators': 784, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:24,931]\u001b[0m Trial 27 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 646, 'criterion': 'gini', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:28,874]\u001b[0m Trial 28 finished with value: 0.975987606506584 and parameters: {'n_estimators': 453, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:34,582]\u001b[0m Trial 29 finished with value: 0.975987606506584 and parameters: {'n_estimators': 685, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:37,953]\u001b[0m Trial 30 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 427, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:43,352]\u001b[0m Trial 31 finished with value: 0.975987606506584 and parameters: {'n_estimators': 681, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:48,816]\u001b[0m Trial 32 finished with value: 0.975987606506584 and parameters: {'n_estimators': 690, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:21:54,510]\u001b[0m Trial 33 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 727, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:00,668]\u001b[0m Trial 34 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 773, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:07,989]\u001b[0m Trial 35 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 927, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:12,973]\u001b[0m Trial 36 finished with value: 0.9728892331525949 and parameters: {'n_estimators': 824, 'criterion': 'gini', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:18,937]\u001b[0m Trial 37 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 859, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:25,319]\u001b[0m Trial 38 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 794, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:30,033]\u001b[0m Trial 39 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 888, 'criterion': 'gini', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:37,526]\u001b[0m Trial 40 finished with value: 0.975987606506584 and parameters: {'n_estimators': 942, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:45,016]\u001b[0m Trial 41 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 932, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:50,277]\u001b[0m Trial 42 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 668, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:54,011]\u001b[0m Trial 43 finished with value: 0.975987606506584 and parameters: {'n_estimators': 476, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:22:56,811]\u001b[0m Trial 44 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 356, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:02,783]\u001b[0m Trial 45 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 756, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:09,426]\u001b[0m Trial 46 finished with value: 0.975987606506584 and parameters: {'n_estimators': 840, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:14,129]\u001b[0m Trial 47 finished with value: 0.975987606506584 and parameters: {'n_estimators': 597, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:17,345]\u001b[0m Trial 48 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 461, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:18,739]\u001b[0m Trial 49 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 264, 'criterion': 'gini', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:22,560]\u001b[0m Trial 50 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 554, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:26,600]\u001b[0m Trial 51 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 583, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:33,257]\u001b[0m Trial 52 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 967, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:36,571]\u001b[0m Trial 53 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 469, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:41,585]\u001b[0m Trial 54 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 725, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:45,544]\u001b[0m Trial 55 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 498, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:48,092]\u001b[0m Trial 56 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 369, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:51,301]\u001b[0m Trial 57 finished with value: 0.9728892331525949 and parameters: {'n_estimators': 459, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:54,755]\u001b[0m Trial 58 finished with value: 0.975987606506584 and parameters: {'n_estimators': 436, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:23:59,395]\u001b[0m Trial 59 finished with value: 0.975987606506584 and parameters: {'n_estimators': 586, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:03,553]\u001b[0m Trial 60 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 516, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:07,655]\u001b[0m Trial 61 finished with value: 0.975987606506584 and parameters: {'n_estimators': 520, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:10,815]\u001b[0m Trial 62 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 397, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:14,978]\u001b[0m Trial 63 finished with value: 0.975987606506584 and parameters: {'n_estimators': 521, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:21,095]\u001b[0m Trial 64 finished with value: 0.975987606506584 and parameters: {'n_estimators': 769, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:28,633]\u001b[0m Trial 65 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 783, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:34,214]\u001b[0m Trial 66 finished with value: 0.975987606506584 and parameters: {'n_estimators': 705, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:38,627]\u001b[0m Trial 67 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 557, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:43,494]\u001b[0m Trial 68 finished with value: 0.975987606506584 and parameters: {'n_estimators': 614, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:48,462]\u001b[0m Trial 69 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 630, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:52,795]\u001b[0m Trial 70 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 825, 'criterion': 'gini', 'max_features': 'auto'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:24:59,242]\u001b[0m Trial 71 finished with value: 0.975987606506584 and parameters: {'n_estimators': 816, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:03,125]\u001b[0m Trial 72 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 493, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:06,879]\u001b[0m Trial 73 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 471, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:10,819]\u001b[0m Trial 74 finished with value: 0.975987606506584 and parameters: {'n_estimators': 499, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:15,045]\u001b[0m Trial 75 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 537, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:18,351]\u001b[0m Trial 76 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 422, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:21,773]\u001b[0m Trial 77 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 435, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:28,729]\u001b[0m Trial 78 finished with value: 0.975987606506584 and parameters: {'n_estimators': 888, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:30,936]\u001b[0m Trial 79 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 313, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:33,849]\u001b[0m Trial 80 finished with value: 0.975987606506584 and parameters: {'n_estimators': 363, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:39,152]\u001b[0m Trial 81 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 667, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:43,113]\u001b[0m Trial 82 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 499, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:47,165]\u001b[0m Trial 83 finished with value: 0.975987606506584 and parameters: {'n_estimators': 514, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:51,311]\u001b[0m Trial 84 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 517, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:54,709]\u001b[0m Trial 85 finished with value: 0.975987606506584 and parameters: {'n_estimators': 434, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:25:58,100]\u001b[0m Trial 86 finished with value: 0.975987606506584 and parameters: {'n_estimators': 420, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:01,864]\u001b[0m Trial 87 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 479, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:05,710]\u001b[0m Trial 88 finished with value: 0.975987606506584 and parameters: {'n_estimators': 481, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 8 with value: 0.9767621998450813.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:09,621]\u001b[0m Trial 89 finished with value: 0.9775367931835787 and parameters: {'n_estimators': 490, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:10,788]\u001b[0m Trial 90 finished with value: 0.9721146398140976 and parameters: {'n_estimators': 165, 'criterion': 'entropy', 'max_features': 'auto'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:15,061]\u001b[0m Trial 91 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 542, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:18,267]\u001b[0m Trial 92 finished with value: 0.9728892331525949 and parameters: {'n_estimators': 401, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:25,211]\u001b[0m Trial 93 finished with value: 0.9752130131680867 and parameters: {'n_estimators': 875, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:27,834]\u001b[0m Trial 94 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 382, 'criterion': 'entropy', 'max_features': 'sqrt'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:30,579]\u001b[0m Trial 95 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 342, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:33,349]\u001b[0m Trial 96 finished with value: 0.9736638264910922 and parameters: {'n_estimators': 453, 'criterion': 'gini', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:37,014]\u001b[0m Trial 97 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 416, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:41,856]\u001b[0m Trial 98 finished with value: 0.9767621998450813 and parameters: {'n_estimators': 563, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n",
            "\u001b[32m[I 2020-11-15 22:26:44,001]\u001b[0m Trial 99 finished with value: 0.9744384198295895 and parameters: {'n_estimators': 259, 'criterion': 'entropy', 'max_features': 'log2'}. Best is trial 89 with value: 0.9775367931835787.\u001b[0m\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "FrozenTrial(number=89, value=0.9775367931835787, datetime_start=datetime.datetime(2020, 11, 15, 22, 26, 5, 711962), datetime_complete=datetime.datetime(2020, 11, 15, 22, 26, 9, 621482), params={'n_estimators': 490, 'criterion': 'entropy', 'max_features': 'log2'}, distributions={'n_estimators': IntUniformDistribution(high=1000, low=100, step=1), 'criterion': CategoricalDistribution(choices=('gini', 'entropy')), 'max_features': CategoricalDistribution(choices=('auto', 'sqrt', 'log2'))}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=89, state=TrialState.COMPLETE)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TPyo-XeZKzXM",
        "outputId": "76d16448-1ce3-44b2-c747-011f580093de",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "forest_params = study.best_trial.params\n",
        "print(study.best_trial)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "FrozenTrial(number=89, value=0.9775367931835787, datetime_start=datetime.datetime(2020, 11, 15, 22, 26, 5, 711962), datetime_complete=datetime.datetime(2020, 11, 15, 22, 26, 9, 621482), params={'n_estimators': 490, 'criterion': 'entropy', 'max_features': 'log2'}, distributions={'n_estimators': IntUniformDistribution(high=1000, low=100, step=1), 'criterion': CategoricalDistribution(choices=('gini', 'entropy')), 'max_features': CategoricalDistribution(choices=('auto', 'sqrt', 'log2'))}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=89, state=TrialState.COMPLETE)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lKODxLIVC6cV"
      },
      "source": [
        "#Adicionando metricas otimizadas ao MLflow"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w8DSceCJ94uT"
      },
      "source": [
        "mlflow.log_metric(\"NaiveBayes\",stats['NaiveOptMetric'])\n",
        "mlflow.log_param(\"NaiveBayes\",stats['NaiveOptParams'])\n",
        "\n",
        "mlflow.log_metric(\"KNN\",stats['KNNOptMetric'])\n",
        "mlflow.log_param(\"KNN\",stats['KNNOptParams'])\n",
        "\n",
        "mlflow.log_metric(\"MLP\",stats['MLPOptMetric'])\n",
        "mlflow.log_param(\"MLP\",stats['MLPOptParams'])\n",
        "\n",
        "mlflow.log_metric(\"RandomForest\",stats['ForestOptMetric'])\n",
        "mlflow.log_param(\"RandomForest\",stats['ForestOptParams'])"
      ],
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gn6QWPGnHNIO"
      },
      "source": [
        "# Diagnóstico do Modelo\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Ydn_j4eHMia"
      },
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "forest_diag = RandomForestClassifier(n_estimators=forest_params['n_estimators'], criterion=forest_params['criterion'], max_features=forest_params['max_features'])\n",
        "forest_diag.fit(feature_train, label_train) \n",
        "\n",
        "prediction_test = forest_diag.predict(feature_test)\n",
        "prediction_train = forest_diag.predict(feature_train)   \n",
        "\n",
        "accuracy_test = accuracy_score(label_test, prediction_test)\n",
        "accuracy_train = accuracy_score(label_train, prediction_train)\n",
        "\n",
        "mat = confusion_matrix(label_test, prediction_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iUBew3uMMIiR",
        "outputId": "bcce21e7-a55c-411d-b9be-cb4b4ec43d0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "source": [
        "import seaborn as sns\n",
        "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False)\n",
        "plt.xlabel('true label')\n",
        "plt.ylabel('predicted label');"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAEGCAYAAACHNTs8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZsUlEQVR4nO3dd3gVBfr28e8TEkCqQpQuIKDwE0Sk6IoNFVCKsLqWXZHVFbGLFfVde1m7K+qugg0QFLACoiIgiliAKAiKKFVMQrVRhZTn/eMcNOuEcCRnMiG5P9eVK2dmcs7cc0HuTB9zd0RECkqJOoCIlD4qBhEJUDGISICKQUQCVAwiEpAadYCd2b9WmzJ5uGTVph+ijiB/UJn8jxiXuz3LChuvNQYRCVAxiEiAikFEAlQMIhKgYhCRABWDiASoGEQkQMUgIgEqBhEJUDGISICKQUQCVAwiEqBiEJEAFYOIBKgYRCRAxSAiASoGEQlQMYhIgIpBRAJUDCISoGIQkQAVg4gElMtiqFGjOk8Of4h3P5nAtE/Gc1jHtrQ6+EBemzyKd2a+yrMvPEa16lWjjllsKSkpzJk9mddfGxF1lKSqWbMGY8YMY8GC95k//z2OOLx91JGSonu34/jyixksWjiTwdddGmmWclkMt91zPe9N+5DjjziFk44+jSVfL+P+Ibdz7+2P0O2oU3l70jQuvPy8qGMW2xWXD+CrRYujjpF0/374Dt6ZPJ02bY6lffuuZWIZU1JSeHTI3fTq3Y82bbtw5pl9adWqRXR5IptzRKpXr0anI9sz5vlXAcjJyWXDho00bd6YWR9lAPDBex/To/eJUcYstgYN6nHyySfw7LMvRh0lqWrUqM5RRx3Os8/FlisnJ4eff94Qcari69SxHUuXrmD58pXk5OQwbtx4TundPbI8oRWDmbU0s+vN7NH41/Vm1iqs+SWqUeMG/LD+Rx56/C7efG8c9w25jb2q7MU3i5bSrcfxAPTs05169etGnLR4Hnrodm688S7y8/OjjpJUTZvuz/r13/PM0/9mzuzJDH3yAapU2SvqWMVWv0FdvsvM/nU4M2sV9SP8PxhKMZjZ9cAYwIDZ8S8DXjSzG8KYZ6JSUyvQum0rnn9uLD2OO4OtW7ZyyZXnc93lt9D//DOZ9O5YqlWrQk5OTpQxi6VHjxNZt3Y9n81dEHWUpEutUIF27dowdOhIOnbqzubNWxg8+LKoY5U5Ya0xnA90dPd73X1U/OteoFN8WqHMbKCZZZhZxqZt4TzjcVX2GlZlr2Hep7FfmjfHT6H1Ia1Yung5/U67kJ7Hn8n4V97i2+XfhTL/knDkkR3o1asbi7/5hNGj/kuXLp0ZMfzRqGMlRWbWKjIzVzF7zlwAXnl1Eu0ObRNxquLLzlpNo4b1fx1u2KAe2dmrI8sTVjHkA/ULGV8vPq1Q7j7M3Tu4e4dqlWqFEmzd2u9ZlbWaA5o3AaDzsYez+Oul1E6Pzc/MuOKagYwaPi6U+ZeEm266l6YHdKDFgUdwdr9LmD79Q/5+7hVRx0qKNWvWkZmZzYEHNgPg+OOP4quvvok4VfHNyZhH8+ZNadKkEWlpaZxxRh8mvvFOZHnCetr1lcA0M1sM7PjTuz/QHIh8ve+W6+/h0aH3klYxjZUrMrn2sps57aze9D//LADefmMa40a/HnFK2Zkrr7qZkSMeo2LFNJYtX8mAAVdHHanY8vLyGHTlTbw56QUqpKQwfMRYFi6MrvDMPZyHfJtZCrFNhwbxUVnAHHfPS+T9+9dqUyafPr5qUzibSBKeMvkfMS53e5YVNj6sNQbcPR/4JKzPF5HwlLvzGERk11QMIhKgYhCRABWDiASoGEQkQMUgIgEqBhEJUDGISICKQUQCVAwiEqBiEJEAFYOIBKgYRCRAxSAiASoGEQlQMYhIgIpBRAJUDCISoGIQkQAVg4gEqBhEJCC0u0QXV7Zusy4SGa0xiEiAikFEAlQMIhKgYhCRABWDiASoGEQkQMUgIgEqBhEJUDGISICKQUQCVAwiEqBiEJEAFYOIBKgYRCRAxSAiASoGEQlQMYhIgIpBRAJUDCISsNN7PprZAsALmwS4ux8SWioRiVRRN4PtVWIpRKRU2WkxuPu3O16bWWOghbtPNbO9inqfiOz5drmPwcwuAF4GhsZHNQReDzOUiEQrkZ2PlwKdgQ0A7r4Y2C/MUCXlqWEPkZ35OfPmTos6StKV5WXr3u04vvxiBosWzmTwdZdGHSdpStNyJVIM29x9+44BM0ul8J2Se5yRI8fRs9fZUccIRVldtpSUFB4dcje9evejTdsunHlmX1q1ahF1rGIrbcuVSDG8b2b/D9jLzLoCLwETw41VMj6YOYsffvwp6hihKKvL1qljO5YuXcHy5SvJyclh3LjxnNK7e9Sxiq20LVcixXADsA5YAFwIvAnctLszNLPzdve9IvUb1OW7zOxfhzOzVlG/ft0IEyVHaVuuXR5dcPd8MxsBzCK2CfG1uxdnU+J24LnCJpjZQGAggFWoSUpK1WLMRkR21y6Lwcx6Ak8CS4md3NTUzC5097eKeM/8nU0C6uzsfe4+DBgGkFqxQZnYjyHJlZ21mkYN6/863LBBPbKzV0eYKDlK23Ilcj7CQ0AXd18CYGbNgEnATouB2C9/d+DH34034KPdyCkCwJyMeTRv3pQmTRqRlbWaM87owzn99/wjE6VtuRLZx7BxRynELQM27uI9bwDV3P3b332tAN7bvajJN+r5/zBzxgQOOrAZK5ZlcN65Z0UdKWnK6rLl5eUx6MqbeHPSC3wx/z1efnkiCxd+E3WsYitty2U7211gZqfGX3YFGgPjiO1jOB1Y6e6XhBlMmxIi4cvdnmWFjS9qU6J3gddrgGPjr9cBeyUpl4iUQjtdY4ia1hhEwrc7awwAmFll4HzgYKDyjvHu/o+kpRORUiWRnY/PA3WJHWV4n9hFVLva+Sgie7BEiqG5u98MbHb3EUBP4PBwY4lIlBIphpz495/MrDVQkzJydaWIFC6RE5yGmdk+wM3ABKAacEuoqUQkUjoqIVKO/eGjEmZ2dVEf6O4PFzeUiJRORW1KVC+xFCJSqmhTQqQc29mmhB44IyIBKgYRCVAxiEiAjkqISEAiRyUOAjoSO7kJYpdjzw4zlIhEa5dHJcxsBtDT3TfGh6sDk9z9mDCD6aiESPiKc1SiDrC9wPB2irihq4js+RK5VmIkMNvMXosP9wVGhBdJRKKW0AlOZnYYcHR8cIa7zw01FdqUECkJxT3BqQqwwd2HAJlm1jRpyUSk1NllMZjZrcD1wI3xUWnAqDBDiUi0EtnH8GegHfAZgLtnx49MhCo1pULYs4hEbn5e1BFCcWe9LlFHCM3Nq6ZHHaHEJbIpsT3+rEoHMDM9UFKkjEukGMaZ2VBgbzO7AJgKPB1uLBGJUiJPu37QzLoCG4idBXmLu08JPZmIRCaR50rc5+7XA1MKGSciZVAimxJdCxl3crKDiEjpUdTVlRcDlwDNzGx+gUnV0aPsRcq0ojYlXgDeAu4BbigwfqO7/xBqKhGJ1E43Jdz9Z3dfAQwBfnD3b939WyDXzPQkKpEyLJF9DE8AmwoMb4qPE5EyKpFiMC9wpZW755PYGZMisodKpBiWmdkVZpYW/xoELAs7mIhEJ5FiuAg4EsgCMok96XpgmKFEJFqJnPm4FjirBLKISClR1HkMg939fjN7jPgFVAW5+xWhJhORyBS1xvBV/HtGSQQRkdJjp8Xg7hPj33V/R5FypqhNiYkUsgmxg7ufEkoiEYlcUZsSD8a/nwrU5bfbuf0VWBNmKBGJVlGbEu8DmNlD7t6hwKSJZqb9DiJlWCLnMVQ1swN2DMTvEK3bu4mUYYmc2nwV8J6ZLQMMaAxcGGoqEYlUIic4vW1mLYCW8VGL3H1buLFEJEqJ3NqtCnA10NjdLzCzFmZ2kLu/EX685Bs69AFOPvkE1q37nvbtf7s51cUXn8tFF/UnLy+ft956l3/+818Rpiy+p4Y9RM8eJ7J23XoObXdC1HH+kOr1atH73xdRNb0m7s68F6aT8dxk9mu1Pyf96zzSqlTm58x1TBj0BNs3bSUlrQIn/+t86h7SFM/PZ+rto1j5yVe7nlEp0rBhfYY/O4T96qTj7jz99Ggee/yZyPIkso/hOWIPsv1TfDgLuCu0RCF7/vmXOOWU/v8z7thj/0Tv3t3o2PEkDjvsRB55ZGhE6ZJn5Mhx9Ox1dtQxdkt+Xj7T7nqBp068npF9b6N9/xOp3aI+Pe4bwPR7x/JM9xv5ZnIGR1zYE4BD/xp7psUz3W9kTL/7OP6mv4EV+uS1Uis3N5frBt/OIW270Pmo3lx88bm0atUisjyJFEMzd78fyAFw9y3E9jUUycxamtkJZlbtd+NP2q2kSTJz5mx+/PGn/xl3wQXn8OCD/2X79thDvdet+z6KaEn1wcxZ/PC75dxTbF77E2u+WAHA9s2/sH5JNtXr1GKfpnX5btYiAJZ/8AUHndwRgPQWDfj2oy8B2PL9BrZt2EK9Q/aspyiuXr2WufO+AGDTps0sWrSYBvXrRpYnoQfOmNle/PbAmWZAkfsYzOwKYDxwOfCFmfUpMLnUraO3aNGUzp07MWPGeKZMGUf79odEHUniajZMp87Bjcmet5T1izNp0a09AC17Hk71erUAWLtwJc27HoZVSKFmo32p27oJNerXjjJ2sTRu3JBD27Zm1uzQnx29U4kclbgVeBtoZGajgc7Aubt4zwVAe3ffZGZNgJfNrEn8obg7Xdsws4HEL+lOTd2HChWq7exHkyo1NZV99qnJMcf0oUOHtowe/V9atjyqROYtO5dWpRJ/fnIQU+8YxfZNW5l03VN0u60/na/oy5Ipn5GfkwvA5+Pep3bz+pw38U5+zlpP1meLyc/Ljzj97qlatQrjxj7F1dfeysaNm3b9hpAUWQxmlgLsQ+zsxyOI/VIPcvf1u/jcFHffBODuK8zsOGLl0JgiisHdhwHDACpX3n+np2MnW1bWKsaPfxuAjIzPyc930tNrsX697nkblZTUCpz65CC+fP0jvnk7dj7dD0tXMeac+wCo1bQuzY4/FADPy2fanaN/fe85r97CD8tXlXzoYkpNTeWlsU/x4ouv8frrb0WapchNifht3Aa7+/fuPsnd30igFADWmNmhBT5nE9ALSAfaFCtxCCZMeIdjj43tW23evCkVK6apFCLW4/4BfL8kmzlP//YLUqV2jdgLM468vA9zR08DILVyRdL2qgRAk6Nak5+bz/eLs0s8c3E9Newhvlq0hEeGDIs6SkKbElPN7FpgLLB5x8hd3EK+P5BbcIS75wL948/BjMzIkY9x9NF/Ij19H5YsmcVddz3MiBFjGTbsAT79dArbt29nwICro4yYFKOe/w/HHvMn0tNrsWJZBrff8SDPDR8TdayENOxwIG1OO5q1X63kH2/eDcD7D4xjnyZ1ad//RAC+fjuD+eNmAFA1vQZnjrwe93w2rv6RiVftefcq7nxkR87p9xfmL1hIxpx3ALj55nt56+13I8ljBe7zWvgPmC0vZLS7+wGFjE+aktyUKEm5+XlRRwjFnfW6RB0hNDevmh51hNDkbs8qdNM+kTMf96zjPiJSbImc+ViZ2KPqjiJ2yPID4El3/yXkbCISkUT2MYwENgKPxYf/BjwPnB5WKBGJViLF0Nrd/6/A8HQzWxhWIBGJXiJnPn5mZkfsGIg/t1I3ahEpwxJZY2gPfGRmK+PD+wNfm9kCYkcndP6wSBmTSDFEetGTiJS8RA5XflsSQUSk9EhkH4OIlDMqBhEJUDGISICKQUQCVAwiEqBiEJEAFYOIBKgYRCRAxSAiASoGEQlQMYhIgIpBRAJUDCISoGIQkYBd3j4+KqkVG5TOYFLubM3+IOoIoUlLP6DQ28drjUFEAlQMIhKgYhCRABWDiASoGEQkQMUgIgEqBhEJUDGISICKQUQCVAwiEqBiEJEAFYOIBKgYRCRAxSAiASoGEQlQMYhIgIpBRAJUDCISoGIQkQAVg4gEqBhEJEDFICIB5b4Yunc7ji+/mMGihTMZfN2lUcdJmrK6XJUqVeLjD9/g04wpfD7vXW695ZqoI/1hI8e8Rp+zL6Rvv4u47tZ72bZtO59kzOX08y7jtL9fyjkXX8PKzGwAslev4fwrbuDP/S/m3MsGs3rtuhLJWK6LISUlhUeH3E2v3v1o07YLZ57Zl1atWkQdq9jK6nIBbNu2jRO7nUH7Dl1p36Eb3bsdx+GdDos6VsLWrFvP6JfHM/bZR3l91JPk5+fz1tT3ufPB/3DvrYN5ZcR/6Nm1C0OHvwjAg48/zSknncBrI5/g4vP+xiNPDi+RnOW6GDp1bMfSpStYvnwlOTk5jBs3nlN6d486VrGV1eXaYfPmLQCkpaWSmpZGaX1o0s7k5uWxbdt2cnPz2PrLNvZNr4Xx23Jt3LSZfdNrA7B0+Uo6tT8UgE6HtWX6Bx+XSMbQisHMOplZx/jr/zOzq82sR1jz2x31G9Tlu/gqG0Bm1irq168bYaLkKKvLtUNKSgoZc95hVdZ8pk2bwew5c6OOlLA6+6Zz7l9P48RT+9Olz9+oXrUKnQ9vz+03XMnF197CCX37MXHyNAacczoAB7U4gKnvfwjA1Pc/YvOWrfz084bQc4ZSDGZ2K/Ao8ISZ3QM8DlQFbjCzf4YxTyk/8vPz6dCxG42bdqBjh3YcfPBBUUdK2M8bNjL9g0+Y/NJzvDt+NFt/2cbEye8ycuxrPPHgHUx7fRR9e3Tj/kefAuDaSweQMXcBfzn3UjLmLaDOvrVJSQl/RT81pM/9C3AoUAlYDTR09w1m9iAwC7i7sDeZ2UBgIIBVqElKStWQ4sVkZ62mUcP6vw43bFCP7OzVoc6zJJTV5fq9n3/ewHvvfxjb0frl11HHScgnGfNoUL8OtfbZG4ATjj2SufO/5Oslyzjk4JYAnHzCMVx4zU0A7LdvbYbcczMAW7ZsZep7M6lRvVroOcOqnlx3z3P3LcBSd98A4O5bgfydvcndh7l7B3fvEHYpAMzJmEfz5k1p0qQRaWlpnHFGHya+8U7o8w1bWV0ugPT0WtSsWQOAypUrc+IJx/D110sjTpW4enX2Zf4Xi9j6yy+4O7My5tGsyf5s2ryFFSszAfhozlwOaLw/AD/+9DP5+bFfmaeeH8ufe3YrkZxhrTFsN7Mq8WJov2OkmdWkiGIoaXl5eQy68ibenPQCFVJSGD5iLAsXfhN1rGIrq8sFUK9eHZ595hEqVEghJSWFl1+eyKQ3p0YdK2GHHNySrl2O4ozzLqdChQq0PLAZp/c5mTr7pXPVP+/GUowa1atx541XATBn7nweeXI4Zkb7tq256ZpLSiSnhbFH18wqufu2QsanA/XcfcGuPiO1YoM9a1ezlFlbsz+IOkJo0tIPsMLGh7LGUFgpxMevB9aHMU8RSZ5yfR6DiBROxSAiASoGEQlQMYhIgIpBRAJUDCISoGIQkQAVg4gEqBhEJEDFICIBKgYRCVAxiEiAikFEAlQMIhKgYhCRABWDiASoGEQkQMUgIgEqBhEJUDGISICKQUQCVAwiEhDKcyX2NGY20N2HRZ0jDGV12bRc4dIaQ8zAqAOEqKwum5YrRCoGEQlQMYhIgIohJvJtuhCV1WXTcoVIOx9FJEBrDCISoGIQkYByXwxmdpKZfW1mS8zshqjzJIuZPWtma83si6izJJOZNTKz6Wa20My+NLNBUWdKBjOrbGazzezz+HLdHmme8ryPwcwqAN8AXYFMYA7wV3dfGGmwJDCzY4BNwEh3bx11nmQxs3pAPXf/zMyqA58Cfff0fzMzM6Cqu28yszRgJjDI3T+JIk95X2PoBCxx92Xuvh0YA/SJOFNSuPsM4IeocySbu69y98/irzcCXwENok1VfB6zKT6YFv+K7K92eS+GBsB3BYYzKQP/ycoLM2sCtANmRZskOcysgpnNA9YCU9w9suUq78Ugeygzqwa8Alzp7huizpMM7p7n7ocCDYFOZhbZJmB5L4YsoFGB4YbxcVKKxbfBXwFGu/urUedJNnf/CZgOnBRVhvJeDHOAFmbW1MwqAmcBEyLOJEWI76R7BvjK3R+OOk+ymNm+ZrZ3/PVexHaIL4oqT7kuBnfPBS4DJhPbiTXO3b+MNlVymNmLwMfAQWaWaWbnR50pSToD5wDHm9m8+FePqEMlQT1gupnNJ/YHa4q7vxFVmHJ9uFJECleu1xhEpHAqBhEJUDGISICKQUQCVAwiEqBiKEfMbG8zuyTEzz/XzB7fxc/cZmbX/sHP3bTrn5JkUjGUL3sDhRaDmaWWcBYpxVQM5cu9QLP4SUEPmNlxZvaBmU0AFppZk4L3bzCza83stvjrZmb2tpl9Gn9Py6JmZGa9zWyWmc01s6lmVqfA5LZm9rGZLTazCwq85zozm2Nm86O+H0F5p78S5csNQOv4hTqY2XHAYfFxy+NXK+7MMOAid19sZocD/wWOL+LnZwJHuLub2QBgMHBNfNohwBFAVWCumU0CWgMtiF0Kb8AEMzsmfvm4lDAVg8x29+VF/UD8SsYjgZdilyoAUGkXn9sQGBu/sUpFoOA8xrv7VmCrmU0nVgZHAd2AufGfqUasKFQMEVAxyOYCr3P5383LyvHvKcBPO9Y0EvQY8LC7T4ivmdxWYNrvz8N3YmsJ97j70D8wDwmJ9jGULxuB6kVMXwPsZ2a1zawS0Asgfr+D5WZ2OsSucDSztruYV01+u4T977+b1id+j8PawHHELhqaDPwjvnaCmTUws/0SXzRJJq0xlCPu/r2ZfRjfwfgWMOl303PM7A5gNrFf6oKX/Z4NPGFmNxG77dgY4PMiZncbsU2PH4F3gaYFps0ndr+BdOBOd88Gss2sFfBxfHNlE9CP2N2MpITp6koRCdCmhIgEqBhEJEDFICIBKgYRCVAxiEiAikFEAlQMIhLw/wF95xb8C7bVXwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0nkms6cS8Hc",
        "outputId": "e4c4a089-da7a-4452-f9e5-e4ccc81ba9cc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "feature_importance = pd.DataFrame({'feature': feature_train.columns, 'feature_importance': forest_diag.feature_importances_})\n",
        "feature_importance = feature_importance.sort_values('feature_importance', ascending=False).reset_index(drop=True)\n",
        "feature_importance"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>feature</th>\n",
              "      <th>feature_importance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>points</td>\n",
              "      <td>0.497011</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>raceId</td>\n",
              "      <td>0.131661</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>grid</td>\n",
              "      <td>0.123168</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>driverId</td>\n",
              "      <td>0.095789</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>constructorId</td>\n",
              "      <td>0.048717</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>milliseconds</td>\n",
              "      <td>0.042124</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>circuitId</td>\n",
              "      <td>0.030818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>laps</td>\n",
              "      <td>0.030712</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         feature  feature_importance\n",
              "0         points            0.497011\n",
              "1         raceId            0.131661\n",
              "2           grid            0.123168\n",
              "3       driverId            0.095789\n",
              "4  constructorId            0.048717\n",
              "5   milliseconds            0.042124\n",
              "6      circuitId            0.030818\n",
              "7           laps            0.030712"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tXhXYnefWZHv",
        "outputId": "ee601f2c-dc65-4a8a-abeb-44fdbf474771",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 318
        }
      },
      "source": [
        "sns.barplot(x = 'feature_importance', y='feature', data=feature_importance, orient='h', color='royalblue').set_title('Feature Importance', fontsize = 20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Feature Importance')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAEcCAYAAACve0HFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7xVdZ3/8ddbVOAIiYQKI+oRtRRRUY+WZg6WYzWTtzQpNcUupnZx6odll5mh8afTpM0FNY1MMbVELSZ0TJ3BEENN7jfBTKSkzPICqOAF+Mwf63tisdv7nL0PZ58Na7+fj8d67LW/67u+6/Nd+7A/rLW+ey1FBGZmZlu7bRodgJmZWXdwQjMzs0JwQjMzs0JwQjMzs0JwQjMzs0JwQjMzs0JwQjMzs0JwQrMtjqToZBrTw7FM66ntdSdJE3t6fzWKpGmS/KPaJrdtowMw68A3KpTP69EozGyr4IRmW6yIGNfoGMxs6+FTjrZVkzRQ0r9IWiJpraRVkqZKOr5M3R0lXSzpAUkrJL0h6U+Spkg6sqTumNwprL8uOeU5LtUZlX9fZnvLJS0v1256fX86VbYqf7pM0raSLpT0qKTVktZImivps5I2+99s++k5SdtJ+kdJT0l6TdITkj6Vq3e+pIVpv66Q9I3S7UtqTW1NlLSfpP+S9KKkVyX9otznkNbrLemS1P6a1M+HJJ1epm5+G2+TNEnSHyVtyH1Of53q5j+nabk2jpU0QdLjaVtrJS2S9E+S+pTZ5rjUxihJp0l6LMX5oqTbJO1WoV8DJV2W2l6TPtv5kr4paYcydav627Xq+AjNtlqS9gSmAa3AQ8C9wA7AB4F7JX06Ir6XW2V/4DJgOvDfwEvAHsCJwAcknRAR96a688hOef4T8BtgYq6dad0Q/mnA+4GfAdcBe6Y+bQfcBbwPeAL4IfAacCxwFfAO4GPdsH2A21J79wBvppgmSHoTOAg4B7gbmEq2j/4RWAP8a5m29gIeARYC3wWGAKOBn0k6IyImtVeUtD1wH1kSWgpcA7Sk7U+SNDIivlpmG3sDvwR+BdwK9AUWkH1OY8j2Yf409fLc/JeB/YCHyT77PsC7gHHAKEnHRcT6Mtu8MPV9CvBg2l+jgYNTnK/n+rUX8PMUx2zgWrKDhrcBXyD7nF9NdWv927VqRIQnT1vUBESaxpWZxuTqTQM2AB8pWX8AWUJaC+yaK98RGFRme0OB3wNLKsQyrUKco9rjrLB8ObC8pGxMWmcD8P4y64xLy68CeuXKewHfT8tOqnI/Tkz1x5SUT0vlM4EBufJhwBtkif5pYLeSffo88Cdg21x5a+7zuqJkO21kifIl4C258q+k+veUtLVL2mcBHFVhG5dX6Ou07Ous4r4YBqhM+aWp3dEVPofVwIEly36Ylp1eUv5wKv9Kme0MAvp09W/XU3VTwwPw5Kl0yn15lZumpToHp/d3VGjjpLT8wiq3OT7V36NMLNMqrDOKrie0yWXqbwO8ADyb/6LPLR+QvgRvr7JPE+k4ob23zDoPpGUfL7PsxrRsz1xZe7JZCfTvIIZzcmVPpn7sV6b+J1L9G8ps4w9A7wp97TChdbCPBpZuL5W3J7T/X2adY9OyK3Nlh6WyucA2nWyzW/92PW2cfMrRtlgRoQ4Wt1/z2rHCNayd0+v++UJJ7wIuSuvvAmxfst5uwG9rDrZ2j5UpexvZF+yTwNelst1fS0mfNsOsMmW/T6+zyyz7XXodSnYaNm9ORLxcZp1pZKcuDwFuktQf2Af4XUQsLVP/gfR6SJll8yN3iq8W6frVRcApZPu5P5DfwWWviVF+Hz2TXnfKlb0zvd4XERs6CadLf7vWOSc021q9Nb3+TZoq6dc+I+kU4E6ya1L/AzxFdk1jA9nR1l8DvesQazl/KFPW3qd9ya7dVdKvg2VVi4hVZYrXpdeOlm1XZtlzFTbT3s8dS16frVC/vXxAB23VJF2XfAA4AlgETCI7dfpmqvJPVP7cV5Ypa98PvXJl7fH+js7V/Ldr1XFCs61V+xfuRRExvsp1LiW7RtQWEUvyCyR9lzRSrgbt/xOv9O9oAOW/ECE7pVSqvU+TI+JDNcbSaLtWKB+cXleVvA4uUxeywST5enld/eH0SWTJbGJEnJtfIGkIHf/noVrtn3OlI728rvztWhU8bN+2Vo+m13fXsM4+wONlktk2wNEV1tnApv8Tz3spve5eukDSPmw8GqnWUrIvxnemo4qtyaHpdGKpUel1LkA6LfkUsJukfcvUPza9zqlx++sBJJX7rPZJrz8ps6zW/8RU0v73+L4qflrRlb9dq4ITmm2VImIW2XDnD0n6eLk6kg6UtEuuaDmwr6S/ytUR2QCA4RU29QJlElaylGwU3En57UjqSzbIpCYRsY5sdOMQYHxqZxOShkiqFGsj7Ug2rP/PJLUBZ5IdkUzOLbqB7PrVFfkEJGkQ8A+5OrV4Ib3uUWbZ8vQ6qiS+YZT/CULNImI22SjHkWQ/EdiEpLe2/96ti3+7VgWfcrSt2Rlk10a+L+nzZL9RWkk2aOEgYATZBfg/pvr/TvZboLmSfkx2DeVdZMnsLuCEMtuYCnxE0l1kRw1vAtMjYnpEvCnpP8m+hOdKmkz2b+pvyAZX/L5Me525lGwU3PnACZIeILsuswvZtbV3AV8DHu9C2/U0HfikpHcAM9j4O7RtgE9HxOpc3SuBD5CdCpwv6R6y36F9mKyf34qIX9S4/alp/Z+k9tYCv4mIm8k+218DX5R0INnR4h5kv/n6b8onwa44i2wQzOWSTk3zIvvcjif7HdzyVLfWv12rRqOHWXryVDqRhuhXWbc/8FWyUXmvkH2RPU32RXUesENJ/TFkv/N5lex3VZOBA9k4THtUSf1dyH539BzZaa1NhumTfWFdQnYa7Q2yEZLfIvuCXk7lYftjOuiTyH48PRV4MbX7O+AXqa+7V7lvJpbbFh0Mcc+t01pm2V/sIzYOqZ9INirvp2SnYteQJbb3VdhOn9SXRekzezn176Nl6v55Gx30tRdwObCM7D8dm/zcguwo+9a0H9cCi4Evkf0H5C9+mlHp76GzeMgGfPwr2Y/iXyNLUvPIftDfsjl/u546n5R2rJlZzSS1kn0J3xQRYxoajDU9X0MzM7NCcEIzM7NCcEIzM7NC8DU0MzMrBA/bb5BBgwZFa2tro8MwM9uqzJ49+/mI2LncMie0BmltbWXWrHL3PTUzs0okld4Y+898Dc3MzArBR2gNsmzFGk4fW+vt6szMtm63X3lo3dr2EZqZmRWCE5qZmRWCE5qZmRWCE5qZmRWCE5qZmRWCE5qZmRWCE5qZmRWCE1oFkq7v7FH3kk7urI6ZmfUMJ7QKIuKTEdHZY+5PBpzQzMy2AE2T0CS1Sloq6VZJSyTdKalF0nslzZW0UNINknqn+tMktaX5VyRdJmm+pEcl7SrpKOBE4ApJ8yTtLenzkh6XtEDSbY3sr5lZs2mahJa8HfhOROwPrAa+CEwERkfEgWS3ArugzHo7AI9GxMHAdOBTEfEwMAW4OCJGRsRTwCXAIRFxEHB+aSOSzpM0S9Ks19e8VIfumZk1r2ZLaM9ExIw0fwvwXuDpiPhVKrsJOKbMem8Ad6f52UBrhfYXALdKOgtYV7owIiZERFtEtPVu2amLXTAzs3KaLaGVPs10ZZXrvRkbn4S6nso3df474BrgUGCmJN/82cyshzRbQttD0pFp/gxgFtAqaZ9U9jHgwRraexnoDyBpG2D3iPg58GVgR6Bft0RtZmadaraE9gTwGUlLgJ2AfwfOBe6QtBDYAFxXQ3u3ARdLmgvsC9yS2pkLjI+Iao8AzcxsMzXbKbF1EXFWSdlU4JDSihExKjffLzd/J3Bnmp/BpsP2j+7OYM3MrHrNdoRmZmYF1TRHaBGxHBjR6DjMzKw+fIRmZmaF4IRmZmaF4IRmZmaF0DTX0LY0w4a2cPuVhzY6DDOzwvARmpmZFYITmpmZFYITmpmZFYITmpmZFYIHhTTIshVrOH3snEaHUSgeZGPW3HyEZmZmheCEZmZmheCEZmZmheCEZmZmheCEZmZmheCEZmZmheCEZmZmheCE1kWSWiUtqrBsmqS2no7JzKyZNV1CU6bp+m1mVnRN8cWejqaekPQDYBHwfUmzJC2W9I1cvcMlPSxpvqTHJPWX1EvSFZJmSlog6dNl2u8r6TZJSyRNBvr2YPfMzIzmuvXVvsA5EfGopIER8aKkXsBUSQcBS4FJwOiImCnpLcBa4BPAqog4XFJvYIak+4HItX0BsCYi9k9tlb2nlaTzgPMAWvoPrlc/zcyaUjMltN9ExKNp/vSUXLYFhgDDyRLUsxExEyAiVgNIOh44SNJpad0dyZLjr3JtHwOMT+stkLSgXAARMQGYADBw8PAoV8fMzLqmmRLaqwCS9gLGAodHxEuSJgJ9OlhPwOci4r5NCqXW+oRpZmZd0RTX0Eq8hSy5rZK0K/CBVP4EMETS4QDp+tm2wH3ABZK2S+Vvk7RDSZvTgTPS8hHAQfXvhpmZ5TXTERoAETFf0lyya2bPADNS+RuSRgNXSepLdv3sOOB6oBWYI0nAn4CTS5q9FrhR0hJgCTC7J/piZmYbNUVCi4jlwIjc+zEV6s0E3llm0VfTlLeqvc2IWAt8pBtCNTOzLmrGU45mZlZATmhmZlYITmhmZlYITmhmZlYITmhmZlYITTHKcUs0bGgLt195aKPDMDMrDB+hmZlZITihmZlZITihmZlZITihmZlZITihmZlZIXiUY4MsW7GG08eWfQ6odcKjQ82sHB+hmZlZITihmZlZITihmZlZITihmZlZITihmZlZITihmZlZITihdZGk8yWdXaa8VdKiRsRkZtbM/Du0LpC0bURc1+g4zMxsIye0MiT9A3AW8CfgGWA28EFgHnA08CNJ/YFXIuJKSYcBN6TV729AyGZmTc+nHEtIOhw4FTgY+ADQllu8fUS0RcS3S1a7EfhcRBzcSdvnSZoladbra17q1rjNzJqdE9pfehfw04h4LSJeBu7KLZtUWlnSAGBARExPRTdXajgiJqSE2Na7ZaduDdrMrNk5odXm1UYHYGZm5Tmh/aUZwAmS+kjqR3btrKKIWAmslHR0Kjqz3gGamdlf8qCQEhExU9IUYAHwHLAQWNXJaucCN0gKPCjEzKwhnNDKuzIixklqAaYDsyPie/kKETEuNz+bbBBJuy/1SJRmZvZnTmjlTZA0HOgD3BQRfnCZmdkWzgmtjIg4o9ExmJlZbTwoxMzMCsEJzczMCsEJzczMCsHX0Bpk2NAWbr/y0EaHYWZWGD5CMzOzQnBCMzOzQnBCMzOzQnBCMzOzQnBCMzOzQvAoxwZZtmINp49tvjtqeWSnmdWLj9DMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQCpnQJI2TNLZM+fmSzq7zNlolLeqObZiZWfWa5ndokraNiOu6q63uaMfMzLpPp1/MkgScCQyLiH+WtAcwOCIeq3t0NZD0NeAc4I/AM8BsSdOAecDRwI8k9QdeAe4GfhARR6R1W4G7IuJASYcB/wb0A54HxkTEs6VtlWz7MOCG9Pb++vXSzMwqqeaU43eAI4GPpvcvA9fULaIuSAnlI8BI4G+Bw3OLt4+Itoj4dntBRCwFtpe0VyoaDUyStB1wFXBaRLQnqcs6aiu5EfhcRBzcrR0zM7OqVZPQ3hERnwFeA4iIl4Dt6xpV7d4NTI6INRGxGpiSWzapwjq3kyUy0usk4O3ACOB/JM0Dvg4M7agtSQOAARExPRXdXClISedJmiVp1utrXqqiW2ZmVq1qrgW9KakXEACSdgY21DWq7vVqhfJJwB2SfgJERDwp6UBgcUQcWWNbVYmICcAEgIGDh8fmtGVmZpuq5ghtPDAZ2EXSZcAvgMvrGlXtpgMnS+qbrpOd0NkKEfEUsB74BzYeeT0B7CzpSABJ20k6oJN2VgIrJR2dis7sYh/MzGwzdHiEJmkb4GngS8B7AQEnR8SSHoitahExR9IkYD7ZoJCZVa46CbgC2Cu184ak04DxknYk2z//ASzupJ1zgRskBR4UYmbWEIro+MyXpLkRcUgPxdM0Bg4eHseddUujw+hxfnyMmW0OSbMjoq3csmpOOU6VdGoavm9mZrZFqiahfRq4A3hd0mpJL0taXee4zMzMatLpKMeI6N8TgZiZmW2Oau4Ucky58tzvrszMzBqumt+hXZyb7wMcAcwG3lOXiMzMzLqgmlOOm/ymS9LuZEPZbTMMG9riEX9mZt2oK4+PWQHs392BmJmZbY5qrqFdRbrtFVkCHAnMqWdQZmZmtarmGtqs3Pw64EcRMaNO8ZiZmXVJNQltQET8Z75A0kWlZWZmZo1UzTW0c8qUjenmOMzMzDZLxSM0SR8FzgD2kpR/vlh/4MV6B1Z0y1as4fSxjbsU6RGWZlY0HZ1yfBh4FhgE5J/Q/DKwoJ5BmZmZ1apiQouI3wC/ASo97NLMzGyL0ek1NEnvlDRT0iuS3pC03jcnNjOzLU01g0KuBj4KPAn0BT4JXFPPoMzMzGpV1Z1CIuLXQK+IWB8RNwLvr29YZmZmtanmd2hrJG0PzJP0LbKBIl25ZZaZmVndVJOYPpbqfRZ4FdgdOLWeQZmZmdWq04SWRjsKGBIR34iIL6ZTkA0j6avd2NYASRd2cd1XKpRPlHTa5kVmZma1qGaU4wnAPODe9H5kyQ+tG6FsQlOm1tOhA4CaEpqkak7VmplZD6rmy38c2UM9VwJExDxgr2oal3S2pAWS5ku6WVKrpAdS2VRJe6R6EyWNl/SwpGXtRzeShkiaLmmepEWS3i3pm0DfVHZravMJST8AFgG754+cJJ0maWKa31XS5BTPfElHAd8E9k7tXZGS4hVpewsljU7rjpL0UErmj5f0U5KuTnH8L7BLNfvHzMy6TzVHGm9GxCpJ+bKoVLmdpAOArwNHRcTzkgYCNwE3RcRNkj4OjAdOTqsMAY4G9gOmAHeS3Xrrvoi4TFIvoCUiHpL02YgYmbbTCuwLnBMRj6aySmGNBx6MiFNSe/2AS4ARufZOJXtEzsFkd0mZKWl6Wv/QVPfpknZPAd4ODAd2JUt4N5TZJ+cB5wG09B/cyR40M7NaVHOEtljSGUAvSfum56M9XMV67wHuiIjnASLiRbK7jvwwLb+ZLIG1+6+I2BARj5MlBYCZwLmSxgEHRsTLFbb1m/ZkVkVM16Z41kfEqjJ1jiZ7RM76iHgOeBA4PC17rEwyAzgmt87vgQfKbTwiJkREW0S09W7ZqYpwzcysWhUTmqSb0+xTwAHA68CPgNXA39chltfzmweIiOlkyeJ3wERJZ1dY99WS9/kjyD7dFuFfbsfMzLYQHR2hHSbpr4DRZDcnfh9wfJpvqaLtB4APS3orQDrl+DDwkbT8TOChjhqQtCfwXER8D7ie7JQfwJuStutg1eck7Z8GiJySK58KXJDa7iVpR7KbLffP1XkIGJ2W70yWUB/rpK/Tc+sMAY7tpL6ZmXWzjq6hXUeWAIax6VOrRXYENKyjhiNisaTLgAclrQfmAp8DbpR0MfAn4NxO4hsFXCzpTeAVoP0IbQKwQNIc4Gtl1rsEuDttYxbZtTKAi4AJkj4BrAcuiIhHJM2QtAj4GfAlslOj81M/vxQRf5C0XwdxTiY7nfk48FvgkU76ZWZm3UwRHY/vkHRtRFzQQ/E0jYGDh8dxZ93SsO37eWhmtjWSNDsi2sotq+aH1U5mZma2xfM9Gc3MrBCc0MzMrBCc0MzMrBCc0MzMrBB8k90GGTa0xSMNzcy6kY/QzMysEJzQzMysEJzQzMysEJzQzMysEDwopEGWrVjD6WPn9Mi2PPjEzJqBj9DMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQtqiEJulESZek+XGSxqb5iZJOS/PXSxreyDjbSWqVtKjRcZiZ2Rb2w+qImAJM6aTOJ3soHDMz24r02BFaOppZmo62fiXpVknHSZoh6UlJR0gaI+nqTtqZJqlNUq/U1iJJCyV9IS3fW9K9kmZLekjSfql8V0mTJc1P01Gp/IupjUWS/j4X6xJJ35O0WNL9kvqmZYe1twF8JhfXAZIekzRP0gJJ+9ZpV5qZWRk9fcpxH+DbwH5pOgM4GhgLfLXGtkYCu0XEiIg4ELgxlU8APhcRh6V2v5PKxwMPRsTBwKHAYkmHAecC7wDeCXxK0iGp/r7ANRFxALASODWV35jaP7gknvOB/4yIkUAbsKI0YEnnSZoladbra16qsbtmZtaRnk5oT0fEwojYACwGpkZEAAuB1hrbWgYMk3SVpPcDqyX1A44C7pA0D/guMCTVfw9wLUBErI+IVWTJdHJEvBoRrwA/Ad6di3Vemp8NtEoaAAyIiOmp/OZcPI8AX5X0ZWDPiFhbGnBETIiItoho692yU43dNTOzjvR0Qns9N78h934DNV7Pi4iXgIOBaWRHR9eT9WdlRIzMTft3Q6zrO4svIn4InAisBe6R9J4ubtfMzLpgixrlWAtJg4BtIuLHwNeBQyNiNfC0pA+nOpLUfmpwKnBBKu8laUfgIeBkSS2SdgBOSWVlRcRKYKWko1PRmbl4hgHLImI88FPgoG7srpmZdWKrTWjAbsC0dGrxFuArqfxM4BNp0MZi4KRUfhFwrKSFZKcQh0fEHGAi8BjwS+D6iJjbyXbPBa5J21Wu/HRgUSofAfxgM/tnZmY1UHYJy3rawMHD47izbumRbfl5aGZWFJJmR0RbuWVb8xGamZnZnzmhmZlZITihmZlZITihmZlZITihmZlZIWxRNyduJsOGtnj0oZlZN/IRmpmZFYITmpmZFYITmpmZFYITmpmZFYIHhTTIshVrOH3snG5rzwNMzKzZ+QjNzMwKwQnNzMwKwQnNzMwKwQnNzMwKwQnNzMwKwQnNzMwKwQnNzMwKoZAJTdL5ks7uprbukTQgTRfmylslLaqwzjRJZR8RbmZm9VHIhBYR10XED0rLJdX8Q/KI+NuIWAkMAC7srL6ZmTVGIRKapLMlLZA0X9LNksZJGpuWTZP0H5JmARdJOlzSw6nuY5L6Sxoj6epce3dLGpXml0saBHwT2FvSPElXlGy/r6TbJC2RNBno22OdNzMzoAC3vpJ0APB14KiIeF7SQODzJdW2j4g2SdsDS4HRETFT0luAtVVu6hJgRESMTNttzS27AFgTEftLOggoe08rSecB5wG09B9c5WbNzKwaRThCew9wR0Q8DxARL5apMym9vh14NiJmprqrI2JdN8RwDHBLanMBsKBcpYiYEBFtEdHWu2WnbtismZm1K0JCq8arnSxfx6b7ok8dYzEzszooQkJ7APiwpLcCpFOOlTwBDJF0eKrbPw0UWQ6MlLSNpN2BI8qs+zLQv0K704EzUpsjgIO60hEzM+u6rf4aWkQslnQZ8KCk9cBcsgRVru4bkkYDV0nqS3b97DhgBvA08DiwhDLXwCLiBUkz0lD9nwHX5BZfC9woaUlaf3Z39c/MzKqjiGh0DE1p4ODhcdxZt3Rbe34empk1A0mzI6Ls73yLcMrRzMzMCc3MzIrBCc3MzArBCc3MzArBCc3MzAphqx+2v7UaNrTFIxPNzLqRj9DMzKwQnNDMzKwQnNDMzKwQnNDMzKwQPCikQZatWMPpY8s+Nq1LPMDEzJqdj9DMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNDMzKwQnNBKSHql0TGYmVntnNDMzKwQnNAqkNRP0lRJcyQtlHRSKm+VtFTSrZKWSLpTUkta9k1Jj0taIOnKxvbAzKy5+NZXlb0GnBIRqyUNAh6VNCUtezvwiYiYIekG4EJJNwKnAPtFREgaUNqgpPOA8wBa+g/umV6YmTUJH6FVJuBySQuA/wV2A3ZNy56JiBlp/hbgaGAVWRL8vqQPAWtKG4yICRHRFhFtvVt2qnsHzMyaiRNaZWcCOwOHRcRI4DmgT1oWJXUjItYBRwB3Ah8E7u2pQM3MzKccO7Ij8MeIeFPSscCeuWV7SDoyIh4BzgB+Iakf0BIR90iaASxrQMxmZk3LCa2yW4G7JC0EZgFLc8ueAD6Trp89DlxLlgB/KqkP2enKL/ZwvGZmTc0JrURE9EuvzwNHli6X1Aqsi4izShatITvlaGZmDeBraGZmVgg+QqtRRCwHRjQ6DjMz25SP0MzMrBCc0MzMrBCc0MzMrBB8Da1Bhg1t4fYrD210GGZmheEjNDMzKwQnNDMzKwRFlN6W0HqCpJfJ7jjSzAYBzzc6iAZy/91/9792e0bEzuUW+Bpa4zwREW2NDqKRJM1q5n3g/rv/7n/39t+nHM3MrBCc0MzMrBCc0BpnQqMD2AI0+z5w/5ub+9/NPCjEzMwKwUdoZmZWCE5oZmZWCE5odSbp/ZKekPRrSZeUWd5b0qS0/JfpAaKFUUX/j5E0R9I6Sac1IsZ6qqL/X5T0uKQFkqZK2rMRcdZTFfvgfEkLJc2T9AtJwxsRZ7101v9cvVMlhaRCDeWv4vMfI+lP6fOfJ+mTXd5YRHiq0wT0Ap4ChgHbA/OB4SV1LgSuS/MfASY1Ou4e7n8rcBDwA+C0RsfcgP4fC7Sk+QuK9PnXsA/ekps/Ebi30XH3ZP9Tvf7AdOBRoK3Rcffw5z8GuLo7tucjtPo6Avh1RCyLiDeA24CTSuqcBNyU5u8E3itJPRhjPXXa/4hYHhELgA2NCLDOqun/zyNiTXr7KDC0h2Ost2r2werc2x2AIo1Uq+Y7AOBS4F+B13oyuB5Qbf+7hRNafe0GPJN7vyKVla0TEeuAVcBbeyS6+qum/0VWa/8/AfysrhH1vKr2gaTPSHoK+Bbw+R6KrSd02n9JhwK7R8R/92RgPaTafwOnptPud0ravasbc0Iz2wJIOgtoA65odCyNEBHXRMTewJeBrzc6np4iaRvg34D/1+hYGuguoDUiDgL+h41nrGrmhFZfvwPy/9sYmsrK1pG0LbAj8EKPRFd/1fS/yKrqv6TjgK8BJ0bE6z0UW0+p9W/gNuDkukbUszrrf39gBDBN0nLgncCUAg0M6fTzj4gXcn/31wOHdXVjTmj1NcPOZjUAAAT2SURBVBPYV9JekrYnG/QxpaTOFOCcNH8a8ECkK6UFUE3/i6zT/ks6BPguWTL7YwNirLdq9sG+ubd/BzzZg/HVW4f9j4hVETEoIlojopXsOuqJETGrMeF2u2o+/yG5tycCS7q6Md9tv44iYp2kzwL3kY32uSEiFkv6Z2BWREwBvg/cLOnXwItkH3ghVNN/SYcDk4GdgBMkfSMiDmhg2N2mys//CqAfcEcaC/TbiDixYUF3syr3wWfTUeqbwEts/A/eVq/K/hdWlf3/vKQTgXVk34Fjuro93/rKzMwKwacczcysEJzQzMysEJzQzMysEJzQzMysEJzQzMysEJzQzMysEJzQzOpA0uclLZF0a43rtUo6o15x5bZzj6QB9d5ObnsDJF3YU9uz5uTfoZnVgaSlwHERsaLG9UYBYyPigzWu1ysi1teyTk9Jt3QbCtwdESMaHY8Vl4/QzLqZpOvInv/0M0lfk3SDpMckzZV0UqrTKumh9HDTOZKOSqt/E3h3etDhF9LDD6/OtX13SnpIekXStyXNB46UdFbazjxJ35XUq4MYl0salOJYKmmipF9JulXScZJmSHpS0hGp/jhJN0t6JJV/KpVL0hWSFil7SOfoVD4q9W8K8Hjq194ptisk9VP2QNM5ab38flki6XuSFku6X1LftGwfSf8raX5ab+9UfrGkmcru1v6NbvsgbevT6AfAefJUxAlYDgwCLgfOSmUDgF+RPfOrBeiTyvcluw0QwCiyI5n2dsaQe/ghcDcwKs0HcHqa35/sruXbpfffAc6uIr5WslsOHUj2H9zZwA2AyJ5b9V+p/jiyhzP2Tes9A/wVcCrZHdJ7AbsCvwWGpH68CuyV1m8FFuW2vy3pwZ6pvV+nbbbHMzItuz23/34JnJLm+6R9eDwwIa27Tdo/xzT68/fUmMn3cjSrr+OBEyWNTe/7AHsAvweuljQSWA+8rQttrwd+nObfS3aX8pnpnpB9gWpvdvx0RCwEkLQYmBoRIWkhWYJp99OIWAuslfRzsoc3Hg38KLLTnc9JehA4HFgNPBYRT1fYpoDLJR1D9nDX3cgSYns889L8bKBVUn9gt4iYDBARr6V4jyfbx3NT/X5k/0GYXmXfrUCc0MzqS8CpEfHEJoXSOOA54GCyI4tKTypex6aXBvrk5l+LjdfNBNwUEV/pQoz5R9ZsyL3fwKbfEaUX3Du7AP9qB8vOBHYGDouIN5U9OqW9b/l41pMl50oE/EtEfLeTWKwJ+BqaWX3dB3xO6bBJ2eNiIHvu3bMRsQH4GNkpO4CXyZ6R1W45MFLSNsqe5HtEhe1MBU6TtEvazkBJe3ZrT+AkSX0kvZXslOJM4CFgtKReknYGjgEeK7Nuab92BP6YktmxQIexRsTLwApJJwNI6i2phWz/flxSv1S+W/s+sObjhGZWX5cC2wEL0um8S1P5d4Bz0oCO/dh4NLMAWJ8GPnwBmAE8TTawYjwwp9xGIuJxsic93y9pAdl1rSHl6m6GBcDPyZ7ZdWlE/J7s0T8LyK6vPQB8KSL+UCa+F4AZafDIFcCtQFs6rXk2sLSK7X+M7FEjC4CHgcERcT/wQ+CR1NadbJo4rYl42L6ZdSqdIn0lIq5sdCxmlfgIzczMCsFHaGYFJumXQO+S4o+1j2o0KxInNDMzKwSfcjQzs0JwQjMzs0JwQjMzs0JwQjMzs0L4P02lQXDSX0TEAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yJ7-gehwfPCk"
      },
      "source": [
        "Variância"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gEh-KwdLbUi8",
        "outputId": "c6c49628-0ead-40c6-b3e3-14edc489db53",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "variance = accuracy_train - accuracy_test\n",
        "variance"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.025561580170410547"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7F8KT8Hf4EX"
      },
      "source": [
        "Viés"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m1xLahDAf3M9",
        "outputId": "dc145a67-89ca-4571-8166-14fab6966f5a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "bias = 1 - 0.02 - accuracy_test\n",
        "bias"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.005561580170410529"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i0uDaGFnhFsL",
        "outputId": "ffb12b27-a353-4302-fe71-4e06002eefb3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(accuracy_test)\n",
        "print(accuracy_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.9744384198295895\n",
            "1.0\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}