6_MSU_Library_Case_Study.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO1vFcY0NZe8SiRlAIIDiSr",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Cintia0528/AB-Testing/blob/main/6_MSU_Library_Case_Study.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Import libraries"
      ],
      "metadata": {
        "id": "GKSNDA4rtpzS"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DPjpRe2TquZf"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from scipy.stats import chi2_contingency"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Import CSV files"
      ],
      "metadata": {
        "id": "gVVz808nttsS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "url = \"https://drive.google.com/file/d/1OY86OEOZ98nj8PLoqYsxH8ciyg_e6eHs/view?usp=drive_link\" # share file, make sure the share settings are on \"everyone\"\n",
        "path = \"https://drive.google.com/uc?export=download&id=\"+url.split(\"/\")[-2]\n",
        "connect = pd.read_csv(path)\n",
        "\n",
        "url = \"https://drive.google.com/file/d/1-S91ngl4HqSsGKW2h9wS7GwjsFTlCzip/view?usp=drive_link\" # share file, make sure the share settings are on \"everyone\"\n",
        "path = \"https://drive.google.com/uc?export=download&id=\"+url.split(\"/\")[-2]\n",
        "help = pd.read_csv(path)\n",
        "\n",
        "url = \"https://drive.google.com/file/d/1z7PwoJvjZzpFymn8LZMS12J6zCtBBb9n/view?usp=drive_link\" # share file, make sure the share settings are on \"everyone\"\n",
        "path = \"https://drive.google.com/uc?export=download&id=\"+url.split(\"/\")[-2]\n",
        "interact = pd.read_csv(path)\n",
        "\n",
        "url = \"https://drive.google.com/file/d/1yTDefbPZFXo-LiF5KfGXMz_ZCYAUXX91/view?usp=drive_link\" # share file, make sure the share settings are on \"everyone\"\n",
        "path = \"https://drive.google.com/uc?export=download&id=\"+url.split(\"/\")[-2]\n",
        "learn = pd.read_csv(path)\n",
        "\n",
        "url = \"https://drive.google.com/file/d/1NPYBH61plPQO3lp-jANv0crPq0AuvwXT/view?usp=drive_link\" # share file, make sure the share settings are on \"everyone\"\n",
        "path = \"https://drive.google.com/uc?export=download&id=\"+url.split(\"/\")[-2]\n",
        "services = pd.read_csv(path)"
      ],
      "metadata": {
        "id": "2OwGfY4Asj21"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Raw Data Exploration\n"
      ],
      "metadata": {
        "id": "dBlLYwqOtv3C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### The experiment was running for 21 days, the approximate daily number of visitors are 1650.\n",
        "\n",
        "Average daily visitors per website version = 1650*21/5 = 6930"
      ],
      "metadata": {
        "id": "3dnWXOlevJCw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "*   x_all_clicks - all clicks that version received\n",
        "*   x_clicks - number of clicks the element of interest received\n",
        "*   x_click_rate - number of clicks the element of interest received out of all **clicks**\n",
        "*   x_ctr - number of clicks the element of interest received out of all **visits**\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "xkALe_UBzI5H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Connect"
      ],
      "metadata": {
        "id": "XB0A75gsuuB6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.set_option('display.max_colwidth', None)"
      ],
      "metadata": {
        "id": "26pC5AwM5lK1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "connect.head(2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "100DqeYi8L_s",
        "outputId": "9c85d740-3e1f-4589-a985-755f46b5eda0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Element ID Tag name  Name  No. clicks  Visible?  \\\n",
              "0          74        a  FIND         502      True   \n",
              "1          66    input   s.q         357      True   \n",
              "\n",
              "                                                             Snapshot information  \n",
              "0        Homepage Version 2 - Connect   •   http://www.lib.montana.edu/index2.php  \n",
              "1  created 5-29-2013   •   20 days 7 hours 34 mins   •   2742 visits, 1587 clicks  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d5caa158-a771-4e3d-91f1-5bcdc9df7635\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Element ID</th>\n",
              "      <th>Tag name</th>\n",
              "      <th>Name</th>\n",
              "      <th>No. clicks</th>\n",
              "      <th>Visible?</th>\n",
              "      <th>Snapshot information</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>74</td>\n",
              "      <td>a</td>\n",
              "      <td>FIND</td>\n",
              "      <td>502</td>\n",
              "      <td>True</td>\n",
              "      <td>Homepage Version 2 - Connect   •   http://www.lib.montana.edu/index2.php</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>66</td>\n",
              "      <td>input</td>\n",
              "      <td>s.q</td>\n",
              "      <td>357</td>\n",
              "      <td>True</td>\n",
              "      <td>created 5-29-2013   •   20 days 7 hours 34 mins   •   2742 visits, 1587 clicks</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d5caa158-a771-4e3d-91f1-5bcdc9df7635')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d5caa158-a771-4e3d-91f1-5bcdc9df7635 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d5caa158-a771-4e3d-91f1-5bcdc9df7635');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 177
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "connect_visits = 2742"
      ],
      "metadata": {
        "id": "Dsvqeh7c59HG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "connect_all_clicks = connect['No. clicks'].sum()\n",
        "connect_all_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mfyKhlNIuz_K",
        "outputId": "15ec8854-ab07-42a6-ab10-0da7cb55b1e3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1587"
            ]
          },
          "metadata": {},
          "execution_count": 178
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "connect_row = connect.loc[connect['Name'] == \"CONNECT\"]\n",
        "connect_clicks = connect_row[\"No. clicks\"].sum()\n",
        "connect_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1H4I3TM-v0rm",
        "outputId": "e7304db8-c0fa-407f-ff9b-216a56024b7b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "53"
            ]
          },
          "metadata": {},
          "execution_count": 179
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "connect_click_rate = connect_clicks / connect_all_clicks\n",
        "connect_click_rate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_7OXtPKVy9JB",
        "outputId": "3b6f55bb-1837-4826-faee-397236c2e430"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.03339634530560807"
            ]
          },
          "metadata": {},
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "connect_ctr = connect_clicks / connect_visits\n",
        "connect_ctr"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8ZI5Sshrwelq",
        "outputId": "1a797dd8-49ba-4b5e-d1d4-7cd6c656e0ae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.019328956965718454"
            ]
          },
          "metadata": {},
          "execution_count": 147
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "connect_no_click = connect_visits - connect_clicks\n",
        "connect_no_click"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kfTbs9Y--pEE",
        "outputId": "cbbcc0b5-53fc-4520-9c3f-133dee343c63"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2689"
            ]
          },
          "metadata": {},
          "execution_count": 181
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Help"
      ],
      "metadata": {
        "id": "o-sxFxdawwvF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "help_visits = 3180"
      ],
      "metadata": {
        "id": "fw2Wpc-u6UZF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "help_all_clicks = help['No. clicks'].sum()\n",
        "help_all_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7aTiKeUdy035",
        "outputId": "36e84e27-0f00-4221-9527-37221c671aa2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1717"
            ]
          },
          "metadata": {},
          "execution_count": 183
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "help_row = help.loc[help[\"Name\"] == 'HELP']\n",
        "help_clicks = help_row[\"No. clicks\"].sum()\n",
        "help_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CCbE_REtxO4m",
        "outputId": "c1c5c66d-402f-4b07-9bca-afd5d5c1ae4d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "38"
            ]
          },
          "metadata": {},
          "execution_count": 184
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "help_click_rate = help_clicks / help_all_clicks\n",
        "help_click_rate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UIpdbDol0OHq",
        "outputId": "6668ebb4-2130-4937-cbf0-f8e1aae981a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0221316249271986"
            ]
          },
          "metadata": {},
          "execution_count": 151
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "help_ctr = help_clicks / help_visits\n",
        "help_ctr"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9PM4T-e_xaTe",
        "outputId": "49c62847-a893-4167-9979-250f7f026202"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.011949685534591196"
            ]
          },
          "metadata": {},
          "execution_count": 152
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "help_no_click = help_visits - help_clicks\n",
        "help_no_click"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gHFhj_Yi-9iv",
        "outputId": "9b1a3a89-0a9f-4d0d-ad4d-d7a26b5a73de"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3142"
            ]
          },
          "metadata": {},
          "execution_count": 185
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Interact"
      ],
      "metadata": {
        "id": "g-j-RarHxzAn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "interact"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "42qzGKrY6631",
        "outputId": "04bcef2b-082d-4e0b-ecdf-6f4986ba290e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    Element ID Tag name                                               Name  \\\n",
              "0          128     area                    Montana State University - Home   \n",
              "1           69        a                                               FIND   \n",
              "2           61    input                                                s.q   \n",
              "3           67        a                              lib.montana.edu/find/   \n",
              "4           78        a                                            REQUEST   \n",
              "..         ...      ...                                                ...   \n",
              "64         133      img  intext-a.akamaihd.net/items/it/img/assets/g9v.jpg   \n",
              "65         134      div                                          index.php   \n",
              "66         135      div      montana.summon.serialssolutions.com/advanced/   \n",
              "67         136      div                                        montana.edu   \n",
              "68         138      div      montana.summon.serialssolutions.com/advanced/   \n",
              "\n",
              "    No. clicks  Visible?  \\\n",
              "0         1291     False   \n",
              "1          842      True   \n",
              "2          508      True   \n",
              "3          166      True   \n",
              "4          151      True   \n",
              "..         ...       ...   \n",
              "64           1     False   \n",
              "65           1     False   \n",
              "66           1     False   \n",
              "67           1     False   \n",
              "68           1     False   \n",
              "\n",
              "                                                               Snapshot information  \n",
              "0          Homepage Version 1 - Interact   •   http://www.lib.montana.edu/index.php  \n",
              "1   created 5-29-2013   •   20 days 4 hours 21 mins   •   10283 visits, 3714 clicks  \n",
              "2                                                                               NaN  \n",
              "3                                                                               NaN  \n",
              "4                                                                               NaN  \n",
              "..                                                                              ...  \n",
              "64                                                                              NaN  \n",
              "65                                                                              NaN  \n",
              "66                                                                              NaN  \n",
              "67                                                                              NaN  \n",
              "68                                                                              NaN  \n",
              "\n",
              "[69 rows x 6 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8e3f472b-5e7c-44a2-ae87-473775537375\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Element ID</th>\n",
              "      <th>Tag name</th>\n",
              "      <th>Name</th>\n",
              "      <th>No. clicks</th>\n",
              "      <th>Visible?</th>\n",
              "      <th>Snapshot information</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>128</td>\n",
              "      <td>area</td>\n",
              "      <td>Montana State University - Home</td>\n",
              "      <td>1291</td>\n",
              "      <td>False</td>\n",
              "      <td>Homepage Version 1 - Interact   •   http://www.lib.montana.edu/index.php</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>69</td>\n",
              "      <td>a</td>\n",
              "      <td>FIND</td>\n",
              "      <td>842</td>\n",
              "      <td>True</td>\n",
              "      <td>created 5-29-2013   •   20 days 4 hours 21 mins   •   10283 visits, 3714 clicks</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>61</td>\n",
              "      <td>input</td>\n",
              "      <td>s.q</td>\n",
              "      <td>508</td>\n",
              "      <td>True</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>67</td>\n",
              "      <td>a</td>\n",
              "      <td>lib.montana.edu/find/</td>\n",
              "      <td>166</td>\n",
              "      <td>True</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>78</td>\n",
              "      <td>a</td>\n",
              "      <td>REQUEST</td>\n",
              "      <td>151</td>\n",
              "      <td>True</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>133</td>\n",
              "      <td>img</td>\n",
              "      <td>intext-a.akamaihd.net/items/it/img/assets/g9v.jpg</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>134</td>\n",
              "      <td>div</td>\n",
              "      <td>index.php</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>135</td>\n",
              "      <td>div</td>\n",
              "      <td>montana.summon.serialssolutions.com/advanced/</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>136</td>\n",
              "      <td>div</td>\n",
              "      <td>montana.edu</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>138</td>\n",
              "      <td>div</td>\n",
              "      <td>montana.summon.serialssolutions.com/advanced/</td>\n",
              "      <td>1</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>69 rows × 6 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8e3f472b-5e7c-44a2-ae87-473775537375')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8e3f472b-5e7c-44a2-ae87-473775537375 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8e3f472b-5e7c-44a2-ae87-473775537375');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-44806b0b-e15f-48c5-966e-c6630c18f519\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-44806b0b-e15f-48c5-966e-c6630c18f519')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const charts = await google.colab.kernel.invokeFunction(\n",
              "          'suggestCharts', [key], {});\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-44806b0b-e15f-48c5-966e-c6630c18f519 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 172
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "interact_visits = 10283"
      ],
      "metadata": {
        "id": "mMF9SIVK6flH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "interact_all_clicks = interact['No. clicks'].sum()\n",
        "interact_all_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W09FR20y0pZl",
        "outputId": "2e3d20a8-8ff3-4d87-a694-ce76484d8086"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3714"
            ]
          },
          "metadata": {},
          "execution_count": 174
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "interact_row = interact.loc[interact[\"Name\"] == \"INTERACT\"]\n",
        "interact_clicks = interact_row[\"No. clicks\"].sum()\n",
        "interact_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5x6COuuI0eJU",
        "outputId": "54ed683b-9659-4b98-e436-cfedfe3a3e3d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "42"
            ]
          },
          "metadata": {},
          "execution_count": 175
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "interact_click_rate = interact_clicks / interact_all_clicks\n",
        "interact_click_rate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kn3lF6N905pp",
        "outputId": "cebcadfd-f0e0-4d68-f96f-cc01e6e8e02e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.011308562197092083"
            ]
          },
          "metadata": {},
          "execution_count": 155
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "interact_ctr = interact_clicks / interact_visits\n",
        "interact_ctr"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9eopObf71Bzx",
        "outputId": "9354454a-c525-4163-979a-6431c40eb40e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0040844111640571815"
            ]
          },
          "metadata": {},
          "execution_count": 170
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "interact_no_clicks = interact_visits - interact_clicks\n",
        "interact_no_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oQHxD5_1_eJF",
        "outputId": "9adad212-0f50-44eb-835d-f7f356089347"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10241"
            ]
          },
          "metadata": {},
          "execution_count": 186
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Learn"
      ],
      "metadata": {
        "id": "KsTwGqoH1Pcx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "learn_visits = 2747"
      ],
      "metadata": {
        "id": "Anzj-Zyu6tXU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "learn_all_clicks = learn[\"No. clicks\"].sum()\n",
        "learn_all_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e5ulXEGQ1W1E",
        "outputId": "c027831a-a171-4d33-bf12-3e33c0adbd42"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1652"
            ]
          },
          "metadata": {},
          "execution_count": 187
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "learn_row = learn.loc[learn[\"Name\"] ==\"LEARN\"]\n",
        "learn_clicks = learn_row[\"No. clicks\"].sum()\n",
        "learn_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_zjim5yZ1a7Q",
        "outputId": "eaad3bb8-8738-411f-94ff-26161f2e9a63"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "21"
            ]
          },
          "metadata": {},
          "execution_count": 188
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "learn_click_rate = learn_clicks / learn_all_clicks\n",
        "learn_click_rate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yPg7w1Ao1qHo",
        "outputId": "2ec4477c-b36c-48a6-97c9-ad83360b33e9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.012711864406779662"
            ]
          },
          "metadata": {},
          "execution_count": 159
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "learn_ctr = learn_clicks / learn_visits\n",
        "learn_ctr"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TWHIoddE1yJt",
        "outputId": "a2170ed4-14dc-43ad-aec3-ffd75312bc7b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.007644703312704768"
            ]
          },
          "metadata": {},
          "execution_count": 171
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "learn_no_clicks = learn_visits - learn_clicks\n",
        "learn_no_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-UlP7GKf_rXS",
        "outputId": "4618355d-96b4-4221-bc8f-4bdb07c53f45"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2726"
            ]
          },
          "metadata": {},
          "execution_count": 190
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Services"
      ],
      "metadata": {
        "id": "QiLmb2kn2CEx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "services_visits = 2064"
      ],
      "metadata": {
        "id": "0LRyOi2A_7Rv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "services_all_clicks = services['No. clicks'].sum()\n",
        "services_all_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T3sl0jtW2I_F",
        "outputId": "8e55e368-3fe7-4f1f-ceee-66ca9ab193a7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1348"
            ]
          },
          "metadata": {},
          "execution_count": 194
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "services_row = services.loc[services[\"Name\"]==\"SERVICES\"]\n",
        "services_clicks = services_row[\"No. clicks\"].sum()\n",
        "services_clicks"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_IfLlnuD2N-A",
        "outputId": "49f6d16c-757e-4a7c-fa44-e5ee6bd1f2a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "45"
            ]
          },
          "metadata": {},
          "execution_count": 195
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "services_click_rate = services_clicks / services_all_clicks\n",
        "services_click_rate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dkemnB-V2bvN",
        "outputId": "331bdf3a-635b-4087-c887-94d50e05a273"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.03338278931750742"
            ]
          },
          "metadata": {},
          "execution_count": 163
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "services_ctr = services_clicks / services_visits\n",
        "services_ctr"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eQ2zqgWv2o0F",
        "outputId": "4bfaf3dd-fbd3-405e-a965-160ce1afc96b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.02180232558139535"
            ]
          },
          "metadata": {},
          "execution_count": 196
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "services_no_click = services_visits - services_clicks\n",
        "services_no_click"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1qIeA2jGAQZ-",
        "outputId": "26aeb548-5b2c-43e9-ea5f-a69ea6238fd3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2019"
            ]
          },
          "metadata": {},
          "execution_count": 197
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Chi-Square Test - CTR"
      ],
      "metadata": {
        "id": "hQcWkkZxAcsA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "1.   We need Chi-Square test as we have categorical data and multiple versions\n",
        "2.   Metrics of interest: CTR, Drop-off rate and Homepage return rate\n",
        "3.   MDE 30%\n",
        "4.   H0 = significant difference between the 5 versions, H1 = significant difference between the 5 versions\n",
        "5.   alpha = 0.1\n"
      ],
      "metadata": {
        "id": "aAJpEtgZAiVx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Gather Data"
      ],
      "metadata": {
        "id": "OXP9jMERA_5f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Click = [53, 38, 42, 21, 45]\n",
        "Website_visits = [2742, 3180, 10283, 2747, 2064]\n",
        "CTR = [53/2742, 38/3180, 42/10283, 21/2747, 45/2064]\n",
        "\n",
        "\n",
        "ctr_rates = pd.DataFrame([Click, Website_visits, CTR],\n",
        "                                  columns = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"],\n",
        "                                  index = [\"Click\", \"Website_visits\", \"CTR\"])\n",
        "\n",
        "ctr_rates = ctr_rates.round({\"Click\": 2, \"Website_visits\": 2, \"CTR\": 6})\n",
        "\n",
        "ctr_rates"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "P631MT-sBfkP",
        "outputId": "6f485ae3-4df4-4292-de87-e639aac34269"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                    Connect        Help      Interact        Learn  \\\n",
              "Click             53.000000    38.00000     42.000000    21.000000   \n",
              "Website_visits  2742.000000  3180.00000  10283.000000  2747.000000   \n",
              "CTR                0.019329     0.01195      0.004084     0.007645   \n",
              "\n",
              "                   Services  \n",
              "Click             45.000000  \n",
              "Website_visits  2064.000000  \n",
              "CTR                0.021802  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ed61bda0-f303-4629-810d-bc34bdfdb002\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Connect</th>\n",
              "      <th>Help</th>\n",
              "      <th>Interact</th>\n",
              "      <th>Learn</th>\n",
              "      <th>Services</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Click</th>\n",
              "      <td>53.000000</td>\n",
              "      <td>38.00000</td>\n",
              "      <td>42.000000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Website_visits</th>\n",
              "      <td>2742.000000</td>\n",
              "      <td>3180.00000</td>\n",
              "      <td>10283.000000</td>\n",
              "      <td>2747.000000</td>\n",
              "      <td>2064.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CTR</th>\n",
              "      <td>0.019329</td>\n",
              "      <td>0.01195</td>\n",
              "      <td>0.004084</td>\n",
              "      <td>0.007645</td>\n",
              "      <td>0.021802</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ed61bda0-f303-4629-810d-bc34bdfdb002')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ed61bda0-f303-4629-810d-bc34bdfdb002 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ed61bda0-f303-4629-810d-bc34bdfdb002');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 209
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "Click = [53, 38, 42, 21, 45]\n",
        "No_click = [2689, 3142, 10241, 2726, 2019]\n",
        "\n",
        "clicknoclick_results = pd.DataFrame([Click, No_click],\n",
        "                                  columns = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"],\n",
        "                                  index = [\"Click\", \"No-click\"])\n",
        "clicknoclick_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "5Af-K4uIBBWH",
        "outputId": "b27537cf-0aaa-45c3-8ff7-eef60ed29c2e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "          Connect  Help  Interact  Learn  Services\n",
              "Click          53    38        42     21        45\n",
              "No-click     2689  3142     10241   2726      2019"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e48429d8-f460-4292-ae31-f97ac01f025a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Connect</th>\n",
              "      <th>Help</th>\n",
              "      <th>Interact</th>\n",
              "      <th>Learn</th>\n",
              "      <th>Services</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Click</th>\n",
              "      <td>53</td>\n",
              "      <td>38</td>\n",
              "      <td>42</td>\n",
              "      <td>21</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>No-click</th>\n",
              "      <td>2689</td>\n",
              "      <td>3142</td>\n",
              "      <td>10241</td>\n",
              "      <td>2726</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e48429d8-f460-4292-ae31-f97ac01f025a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e48429d8-f460-4292-ae31-f97ac01f025a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e48429d8-f460-4292-ae31-f97ac01f025a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 208
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "Website_visits = [2742, 3180, 10283, 2747, 2064]\n",
        "Click = [53, 38, 42, 21, 45]\n",
        "No_click = [2689, 3142, 10241, 2726, 2019]\n",
        "CTR = [53/2742, 38/3180, 42/10283, 21/2747, 45/2064]\n",
        "\n",
        "all_data = pd.DataFrame([Website_visits, Click, No_click, CTR],\n",
        "                                  columns = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"],\n",
        "                                  index = [\"Website_visits\",\"Click\",\"No_click\",\"CTR\"])\n",
        "\n",
        "all_data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "ps9xUB2bGVEl",
        "outputId": "643821e2-91f0-4cdc-b22d-8e985c01ae67"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                    Connect        Help      Interact        Learn  \\\n",
              "Website_visits  2742.000000  3180.00000  10283.000000  2747.000000   \n",
              "Click             53.000000    38.00000     42.000000    21.000000   \n",
              "No_click        2689.000000  3142.00000  10241.000000  2726.000000   \n",
              "CTR                0.019329     0.01195      0.004084     0.007645   \n",
              "\n",
              "                   Services  \n",
              "Website_visits  2064.000000  \n",
              "Click             45.000000  \n",
              "No_click        2019.000000  \n",
              "CTR                0.021802  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-169478bc-e455-479e-bf4f-75beed725098\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Connect</th>\n",
              "      <th>Help</th>\n",
              "      <th>Interact</th>\n",
              "      <th>Learn</th>\n",
              "      <th>Services</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Website_visits</th>\n",
              "      <td>2742.000000</td>\n",
              "      <td>3180.00000</td>\n",
              "      <td>10283.000000</td>\n",
              "      <td>2747.000000</td>\n",
              "      <td>2064.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Click</th>\n",
              "      <td>53.000000</td>\n",
              "      <td>38.00000</td>\n",
              "      <td>42.000000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>No_click</th>\n",
              "      <td>2689.000000</td>\n",
              "      <td>3142.00000</td>\n",
              "      <td>10241.000000</td>\n",
              "      <td>2726.000000</td>\n",
              "      <td>2019.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CTR</th>\n",
              "      <td>0.019329</td>\n",
              "      <td>0.01195</td>\n",
              "      <td>0.004084</td>\n",
              "      <td>0.007645</td>\n",
              "      <td>0.021802</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-169478bc-e455-479e-bf4f-75beed725098')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-169478bc-e455-479e-bf4f-75beed725098 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-169478bc-e455-479e-bf4f-75beed725098');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 215
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Calculation"
      ],
      "metadata": {
        "id": "I9KYuipJEEUC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy import stats\n",
        "chisq, pvalue, df, expected = stats.chi2_contingency(clicknoclick_results)"
      ],
      "metadata": {
        "id": "XlLYyYh4DzGA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   chi-square value = 96.7432\n",
        "*   alpha = 0.1\n",
        "*   pvalue = 4.8523\n"
      ],
      "metadata": {
        "id": "OzQ-uq3oEtL3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "chisq"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k-FiNE1SEOeL",
        "outputId": "5d6da614-153b-451d-dbe2-5d75487ecdd7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "96.74323537983278"
            ]
          },
          "metadata": {},
          "execution_count": 213
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pvalue"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tfn_VHlTE1lc",
        "outputId": "bc408996-add0-44f2-d68f-782e0d37936b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4.852334301093873e-20"
            ]
          },
          "metadata": {},
          "execution_count": 214
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Based on these calculations: the P-value is larger than alpha, therefore we would fail to reject the H0, meaning the evidence does not suggest that we have a statistical significance between the 5 versions.\n",
        "However we must consider the **cumulated alpha error**."
      ],
      "metadata": {
        "id": "5kaGaytUFPVL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Post hoc test - Bonferroni"
      ],
      "metadata": {
        "id": "MoI2rF_FF1zP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alpha = 0.1\n",
        "\n",
        "# Number of comparisons (number of columns)\n",
        "num_comparisons = all_data.shape[1]\n",
        "\n",
        "# Applying Bonferroni correction to the p-value\n",
        "pvalue_corrected = pvalue * num_comparisons\n",
        "\n",
        "# Creating a DataFrame to hold the p-values and corrected p-values\n",
        "p_values_data = pd.DataFrame({\"p_value\": [pvalue], \"p_value_corrected\": [pvalue_corrected]},\n",
        "                              index=[\"Chi-squared test\"])\n",
        "\n",
        "alpha_corrected = alpha / num_comparisons\n",
        "\n",
        "if pvalue_corrected < alpha_corrected:\n",
        "    print(\"Result is statistically significant after Bonferroni correction.\")\n",
        "else:\n",
        "    print(\"Result is not statistically significant after Bonferroni correction.\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "olZLRk4SF4qZ",
        "outputId": "fe6624a9-0878-4f14-ff55-68fd3bec78e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result is not statistically significant after Bonferroni correction.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "p_values_data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "KfmNRlQqH9R4",
        "outputId": "bf9e53f5-a425-485c-89a9-b8f8e1461606"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                   p_value  p_value_corrected\n",
              "Chi-squared test  0.213309           1.066547"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e9f18f7a-ad9f-4012-a462-6067df9a5ed4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>p_value</th>\n",
              "      <th>p_value_corrected</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Chi-squared test</th>\n",
              "      <td>0.213309</td>\n",
              "      <td>1.066547</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e9f18f7a-ad9f-4012-a462-6067df9a5ed4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e9f18f7a-ad9f-4012-a462-6067df9a5ed4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e9f18f7a-ad9f-4012-a462-6067df9a5ed4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 233
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "alpha_corrected"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7nfVzrCoICb7",
        "outputId": "94d1ade0-9bc1-45dc-b6dd-2584a964b225"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.02"
            ]
          },
          "metadata": {},
          "execution_count": 218
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "After correction, we see that the result is still not statistically significant\n",
        " for the test globally speaking. Therefore we fail to reject the hypothesis. However, we might want to check whether specific pairs show significant difference."
      ],
      "metadata": {
        "id": "UWtpA2rFIzzN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import scipy.stats as stats\n",
        "\n",
        "# Data\n",
        "versions = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"]\n",
        "clicks = [53, 38, 42, 21, 45]\n",
        "No_click = [2689, 3142, 10241, 2726, 2019]\n",
        "visits = [2742, 3180, 10283, 2747, 2064]\n",
        "\n",
        "# Calculate CTR for each version\n",
        "ctr = [click / visit for click, visit in zip(clicks, visits)]\n",
        "for v, rate in zip(versions, ctr):\n",
        "    print(f\"CTR for Version {v}: {rate:.4f}\")\n",
        "\n",
        "# Pairwise chi-squared tests with Bonferroni correction\n",
        "alpha = 0.1\n",
        "num_tests = len(versions) * (len(versions) - 1) // 2 # Bonferroni Correction\n",
        "adjusted_alpha = alpha / num_tests\n",
        "\n",
        "print(\"\\nPairwise Comparisons:\")\n",
        "\n",
        "for i in range(len(versions)):\n",
        "    for j in range(i+1, len(versions)):\n",
        "        contingency = [[clicks[i], visits[i] - clicks[i]],\n",
        "                       [clicks[j], visits[j] - clicks[j]]]\n",
        "        _, p_val, _, _ = stats.chi2_contingency(contingency)\n",
        "        if p_val < adjusted_alpha:\n",
        "            result = \"Significant\"\n",
        "        else:\n",
        "            result = \"Not significant\"\n",
        "\n",
        "        print(f\"Version {versions[i]} vs Version {versions[j]}: p = {p_val:.4f} - {result}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B98tMc9OZuYy",
        "outputId": "9d110583-5bce-4d36-929b-f024ef0d6b9d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CTR for Version Connect: 0.0193\n",
            "CTR for Version Help: 0.0119\n",
            "CTR for Version Interact: 0.0041\n",
            "CTR for Version Learn: 0.0076\n",
            "CTR for Version Services: 0.0218\n",
            "\n",
            "Pairwise Comparisons:\n",
            "Version Connect vs Version Help: p = 0.0281 - Not significant\n",
            "Version Connect vs Version Interact: p = 0.0000 - Significant\n",
            "Version Connect vs Version Learn: p = 0.0003 - Significant\n",
            "Version Connect vs Version Services: p = 0.6189 - Not significant\n",
            "Version Help vs Version Interact: p = 0.0000 - Significant\n",
            "Version Help vs Version Learn: p = 0.1251 - Not significant\n",
            "Version Help vs Version Services: p = 0.0074 - Significant\n",
            "Version Interact vs Version Learn: p = 0.0254 - Not significant\n",
            "Version Interact vs Version Services: p = 0.0000 - Significant\n",
            "Version Learn vs Version Services: p = 0.0001 - Significant\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Taking the pairwise comparisons into consideration, in addition to the drop-off rate and the homepage return rate, SERVICES is a clear winner with 530% relative improvement from Interact.**"
      ],
      "metadata": {
        "id": "zEv3_UKQmVYW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Chi-Square Test - CTR 2"
      ],
      "metadata": {
        "id": "vE9Net-iXTKK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since the data is highly skewed ( one versions had 10283 visits while the others are in the 2000-3000 range) I reran the test defining my CTR as element clicks / all clicks."
      ],
      "metadata": {
        "id": "2U9-57raYJAh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "1.   We need Chi-Square test as we have categorical data and multiple versions\n",
        "2.   Metrics of interest: CTR, Drop-off rate and Homepage return rate\n",
        "3.   MDE 30%\n",
        "4.   H0 = significant difference between the 5 versions, H1 = significant difference between the 5 versions\n",
        "5.   alpha = 0.1\n"
      ],
      "metadata": {
        "id": "uVP8v-xLXYsH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "All_Clicks = [1587, 1717, 3714, 1652, 1348]\n",
        "Click = [53, 38, 42, 21, 45]\n",
        "No_click = [1534, 1679, 3672, 1631, 1303]\n",
        "CTR = [53/1587, 38/1717, 42/3714, 21/1652, 45/1348]\n",
        "\n",
        "second_calculation = pd.DataFrame([Website_visits, Click, No_click, CTR],\n",
        "                                  columns = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"],\n",
        "                                  index = [\"All_Clicks\",\"Click\",\"No_click\",\"CTR\"])\n",
        "\n",
        "second_calculation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "Mce7pUpnXu9t",
        "outputId": "baf3a562-c3c3-42a0-b405-12a1fece3304"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                Connect         Help      Interact        Learn     Services\n",
              "All_Clicks  2742.000000  3180.000000  10283.000000  2747.000000  2064.000000\n",
              "Click         53.000000    38.000000     42.000000    21.000000    45.000000\n",
              "No_click    1534.000000  1679.000000   3672.000000  1631.000000  1303.000000\n",
              "CTR            0.033396     0.022132      0.011309     0.012712     0.033383"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ed092e16-6515-4f2f-9dd3-c2f78728f340\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Connect</th>\n",
              "      <th>Help</th>\n",
              "      <th>Interact</th>\n",
              "      <th>Learn</th>\n",
              "      <th>Services</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>All_Clicks</th>\n",
              "      <td>2742.000000</td>\n",
              "      <td>3180.000000</td>\n",
              "      <td>10283.000000</td>\n",
              "      <td>2747.000000</td>\n",
              "      <td>2064.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Click</th>\n",
              "      <td>53.000000</td>\n",
              "      <td>38.000000</td>\n",
              "      <td>42.000000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>No_click</th>\n",
              "      <td>1534.000000</td>\n",
              "      <td>1679.000000</td>\n",
              "      <td>3672.000000</td>\n",
              "      <td>1631.000000</td>\n",
              "      <td>1303.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CTR</th>\n",
              "      <td>0.033396</td>\n",
              "      <td>0.022132</td>\n",
              "      <td>0.011309</td>\n",
              "      <td>0.012712</td>\n",
              "      <td>0.033383</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ed092e16-6515-4f2f-9dd3-c2f78728f340')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ed092e16-6515-4f2f-9dd3-c2f78728f340 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ed092e16-6515-4f2f-9dd3-c2f78728f340');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 244
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "Click = [53, 38, 42, 21, 45]\n",
        "No_click = [1534, 1679, 3672, 1631, 1303]\n",
        "version_2_contingency_table = pd.DataFrame([Click, No_click],\n",
        "                                  columns = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"],\n",
        "                                  index = [\"Click\", \"No-click\"])\n",
        "version_2_contingency_table"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "P2o80HY4bEYh",
        "outputId": "57ceda70-600a-4377-c6a1-72f7671e229c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "          Connect  Help  Interact  Learn  Services\n",
              "Click          53    38        42     21        45\n",
              "No-click     1534  1679      3672   1631      1303"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c7bbd0be-315e-469b-9c71-ff964d6688e0\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Connect</th>\n",
              "      <th>Help</th>\n",
              "      <th>Interact</th>\n",
              "      <th>Learn</th>\n",
              "      <th>Services</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Click</th>\n",
              "      <td>53</td>\n",
              "      <td>38</td>\n",
              "      <td>42</td>\n",
              "      <td>21</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>No-click</th>\n",
              "      <td>1534</td>\n",
              "      <td>1679</td>\n",
              "      <td>3672</td>\n",
              "      <td>1631</td>\n",
              "      <td>1303</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c7bbd0be-315e-469b-9c71-ff964d6688e0')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c7bbd0be-315e-469b-9c71-ff964d6688e0 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c7bbd0be-315e-469b-9c71-ff964d6688e0');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 238
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy import stats\n",
        "chisq, pvalue, df, expected = stats.chi2_contingency(version_2_contingency_table)"
      ],
      "metadata": {
        "id": "M-eSWElqbG5e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chisq"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kxtv7paRcmRK",
        "outputId": "033cbd65-ca95-4e65-ac71-d610a6efd358"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "46.33660181942125"
            ]
          },
          "metadata": {},
          "execution_count": 240
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pvalue"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bLOSl2M7cnWs",
        "outputId": "c654f5b9-a545-49ca-b4dc-59956e1a624e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2.095949812998472e-09"
            ]
          },
          "metadata": {},
          "execution_count": 241
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "alpha"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1TRQbvA_cpmH",
        "outputId": "b6373ed2-55b2-4a9d-e069-4c1c3817d45f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.1"
            ]
          },
          "metadata": {},
          "execution_count": 242
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Based on these calculations: the P-value is smaller than alpha, therefore we would reject the H0, meaning the evidence suggest that we have a statistical significance between the 5 versions.\n",
        "However we must consider the **cumulated alpha error**."
      ],
      "metadata": {
        "id": "OWGkUB92ct5B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import scipy.stats as stats\n",
        "\n",
        "# Data\n",
        "versions = [\"Connect\" , \"Help\" , \"Interact\" ,\"Learn\", \"Services\"]\n",
        "clicks = [53, 38, 42, 21, 45]\n",
        "no_click = [1534, 1679, 3672, 1631, 1303]\n",
        "all_clicks = [1587, 1717, 3714, 1652, 1348]\n",
        "\n",
        "# Calculate CTR for each version\n",
        "ctr = [click / no_click for click, no_click in zip(clicks, no_click)]\n",
        "for v, rate in zip(versions, ctr):\n",
        "    print(f\"CTR for Version {v}: {rate:.4f}\")\n",
        "\n",
        "# Pairwise chi-squared tests with Bonferroni correction\n",
        "alpha = 0.1\n",
        "num_tests = len(versions) * (len(versions) - 1) // 2 # Bonferroni Correction\n",
        "adjusted_alpha = alpha / num_tests\n",
        "\n",
        "print(\"\\nPairwise Comparisons:\")\n",
        "\n",
        "for i in range(len(versions)):\n",
        "    for j in range(i+1, len(versions)):\n",
        "        contingency = [[clicks[i], all_clicks[i] - clicks[i]],\n",
        "                       [clicks[j], all_clicks[i] - clicks[i]]]\n",
        "        _, p_val, _, _ = stats.chi2_contingency(contingency)\n",
        "        if p_val < adjusted_alpha:\n",
        "            result = \"Significant\"\n",
        "        else:\n",
        "            result = \"Not significant\"\n",
        "\n",
        "        print(f\"Version {versions[i]} vs Version {versions[j]}: p = {p_val:.4f} - {result}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vLGfH9GcdXwc",
        "outputId": "b1230045-24c0-4b05-c4b5-dc879bb37913"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CTR for Version Connect: 0.0346\n",
            "CTR for Version Help: 0.0226\n",
            "CTR for Version Interact: 0.0114\n",
            "CTR for Version Learn: 0.0129\n",
            "CTR for Version Services: 0.0345\n",
            "\n",
            "Pairwise Comparisons:\n",
            "Version Connect vs Version Help: p = 0.1489 - Not significant\n",
            "Version Connect vs Version Interact: p = 0.3138 - Not significant\n",
            "Version Connect vs Version Learn: p = 0.0004 - Significant\n",
            "Version Connect vs Version Services: p = 0.4884 - Not significant\n",
            "Version Help vs Version Interact: p = 0.7423 - Not significant\n",
            "Version Help vs Version Learn: p = 0.0391 - Not significant\n",
            "Version Help vs Version Services: p = 0.5170 - Not significant\n",
            "Version Interact vs Version Learn: p = 0.0121 - Not significant\n",
            "Version Interact vs Version Services: p = 0.8322 - Not significant\n",
            "Version Learn vs Version Services: p = 0.0051 - Significant\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![drop-off.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAKfCAMAAAChRVxOAAAA3lBMVEX///8uY88vZM/+/v78/Pz6+voYUsr4+Pjf39/d3d329vbk4+Pa2tocVcwvZNAnXc0iWMzo6OgqYM/u7u4CAgItYs5KdtR3d3erwe3O2vMPDw9/f38ZGRkkJCTs8ftvk98uLi67u7s3Nzfy8vLz8+uDgHsjOmgfVcEuY9owZNMtZM5fX1+BoOP09vyoqKhchduysrLPz8+enp7JycnCwsJWVlbV1dWVlZVnZ2eNjY0/Pz9PT09KSkqYsuhycnLe5fdFRUVtbW2Ghobs6uSCgoKSpMvi6fp8e3tzktU6acyKpsmuAAA0mElEQVR42uzXsQrDIBQFUOEa0pKp9P//tYLQoQiBSgbJOYRnBn2TF7UAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAN5DB36VdUmAhmVw+TklO10RWWEO+Q6Yikl5/ug7nJ84SVnPFhs0oIS5bLCbtm4xAejmPQTxHWErqVpuU/JWsXrfj+dpL2Y/Huw1N7V1r5IC1fdg3t5ZHYSAMD0RpPdyIA4VsAjlcCkHjqbZFahH6///RJtpvz8vCXnWXPJSOyUymV28nE0lm7IZpy78tIlFu5/NJxkdn1nZTSCHtjukK+IEiLyF0IYF/BKnojhr/ViHZzBDZVJyc4Va7GRgq+oKvGXxLNDznNmgj8K9Qc3zB1/JXx7EvyO+7704hva15LpDd5jb2/uaKH7BVwzckZ3pvv6QJW7DAmyMFik/GrhdELiMgvqMo08h50qL0BrY5oov4izTcZ/fH22Mt8NIR6Bj2zasHGXqkZ2vt407xMvhV+hXdcbw14Dm4/CQoJPDeSI5VAiSWd6TPUtejkdMyFZDb5zJPvh7ozs2Z0/Lo9NdisvsfUsNBVgzFWpvKaWGS6YdAWAsOw5G3QAZ7Oi9rXcKwUBRLfSBafjovU5sGfQTeGi+Q3FkyUrwOWUU5R1RJcqbouJsUjjNlirnBxRw+qkc0LJtfmYOu0MPFZi7JFpH3yDpfiyxH0UDX0y3GpnZ7uJWREVu4yx8IvC9Ecuwb2DdKqskqRKT8U744oziikJCdEZHf3eDeftNJIFMCUdV64t7fV96wc/Fli7WY2qwKsS8Pi0t2vlK8NJ3yUXPaKmTXG0PVRRAIvC+7QAgQqZB35YJ4t+3RchR2kD3iOStOFC8mrxXSSUfaEYFhKMahrSjekmLkqExytAzv9TEGsgsEKeecIb3JSE/XtSnkDWl9MC7YZsUZ2SM5rhSfBQQC7wkBsguEuMGHQKglUFZIP6XbEZfq9BPxEUO0UnrKTHWuzlJPiE8N0F7cGv/dZwANx7MGzy4QxhCRnpoY4mJourHiSGtoFPYJ5AL5o2ufTlN56EICb8zHFot4MbT+n10CFFcUdQRk6JFLJxBqACLD8VNyRcd8HCm6KThUyGtfe645QMfwtcHaFtLVVhTZeHDjxCyCIvrcrfK/1wikXAi+nXEFAu8K2Zt0AhBNFPu8qJC/BOIFMFy/CgQMx+dxoYyxNRspWgDQW7hUeBtcJobVEeCbU6z8jKhMBNlMqbjO900gAvvhQyCX2/V0hEDgTSGwCWSIU+2PeR/6+BJIhfRUbEXj0pZP6vdT+oR4KrrJ0eoJcc4grhXyFnwFGWAXCAHi8+a9F4MvF3hvtnctXZnNfq4TvoIkbuGYZHK0nYZA4G0h24tCO54UomhhqJBJIKnlyB/S3hFPpW/S+dpZgXxM41KX5YEYhmxua+fvC6i/qyCEvCqIAYgtQ1y0dDpLYiNeAhGfhnKmdCn0pO5Whx4k8LYQqBki5e4LxZiS5orUEIDjgsgUR7y1JJvp68y3yl93nbY5ehGIdxmD4agaIC5TnwPZ0ja3PU+5UGS26ZFeK/9DFpqLM0vR3N1U75zhRUjgrZEXzhljXFVGAwwLFxIc7aIYpbyXh62DUMoNKvl1Z5Z82v11BFDfWT8A6RSbM+fzDGempI9se86qpL4wpOou+JSWk6D0lpD6xhDZZQqnvIE3hkBmRo/tisiNSjmajICjkNPpaY4RAd872PZxsvlePnZ05/w28RNHM8oSILNTp8keUcrJHokftONki6hb50ebb3GF/fS0BUTDeJqn9gCBwNtCAKKDdqTpa3zQabQ/Rb7Z8E/HBakBXeof7h46/74qStMD8UZv8eBxWaM9v9ZlDFFZ6j25C3PrYrKt1+FWSOC9IR/mR/uhBQJ5hTj5iZ+v1e5hf0j/k5+EG7eB/4hsFcJAIBD4zN69mwAQAkEAPZgTjivGxP57MzVQEFPf62E/sDA79bdaPz0fFgvYW0pUB0gNhSNiSGAuRghYsgAAAAAAAAC4QZ44gsPIX1nYFAMElqI0oLN3dz1qAmEUgA++M3woUj5iddsbkbuSmHEFWZAYt7XZ7P//Q10wTauN29B0pF3fR0m8nMSczGEIM9yuGOO7dMY0sCQBaM/Q5KOcGDvjLjcSgFiXu0PGu+AydtKnnMdaCUBmeZXWW2WB0MnrG8Px24rsv/77o+zhwyIGYD/MP2bFIl+DCB0QaR888YYQrCdDtZ3P84wgVDUNEB0Wpew8gfzVGYT3IGX/zBMJQlCmRZnGEkm5VwCy7TQ6HcfvV7hI88AJII4Iuz4ChptNonJlITjUMQGbdBrgnJBSCnmJENSGRBMiai4hGSNcGwF0nyvZBEShCcjOxhmKHMexL3Hs0Rf9Rja7bc7LdzTEdVFzyeJ7QAjUBIRwSkSu67jOJfbqq35rh90617WHhCsjoAmIoOBzHR9nkABnSFivEXfvzXcDY6DPZPw8E9Jit04SeiDvcyWQFPslgKx+SrquqN49m6Y/1sX3/bHpz8BYP44Vi+JqN6LoaVEIdEL0MoP4kzAMJ1o+YWiE5mTGq1isJ1ZZLSVh9LiYxuU8XaGrtmJ5A108Lxx7HBDWF9muYpFYp1W+z5X4o4B4hj6D0PS4YrG+iJVyBNCeuVyuZOfn1nS8SdfF44CwfsmhIDSSIGp/EKhzxfpk6NIEhCsW6xP9ekYUdQyI8TOuWOyNIZzoXLGMHzgg39irYxoAAAAGYf5d792FgtYDARwEBAICgScQCAKBIBAIAoEgEAgCgSAQCAKBIBAIAoEgEAgCgSAQCAKBIBAIAhn7dCAAAAAAIMjfeoMJyiEYgsAQBIYgMASBIQgMQWAIAkMQGILAEASGIDAEgSEIDEFgCAJDEBiCwBAEhiAwBIEhCAxBYAgCQxAYgsAQBIYgMASBIQgMQWAIAkMQGILAEASGIDAEgSEIDEFinw4EAAAAAAT5W28wQTkEQxAYgsAQBIYgMASBIQgMQWAIAkMQGILAEASGIDAEgSEIDEFgCAJDEBiCwBAEhiAwBIEhCAxBYAgCQxAYgsAQBIYgMASBIQgMQWAIAkMQGILAEASGIDAEgSEIsWdvrY0CYRjHn/gq8RR3J0LbJV0wmcuBEBO1Y8xKbLLk+3+j9dBdaEJD90J74fsD8YDg1Z+ZcdgdHAhjn0PvzsSBMHbVB+EGcSCM3e+BOBA2bpbwa47nhSa9PXCaB5KnWIwBcqOUrspSb0PUyMzOSmtVJcQjCGOQm0o/nsvo+VWiRu52tV9c9OOOp1iMAWZoO164jYodtT2IRbF2XeF1ufAUizHALvcbk0DNZVUlQrj8m5exFgHWS7QI0clUUZY6t68HEPpUIMGkP0ETCBHYyBEGRrtiv/v73eQYqbKIqgw3yLyDqB1BehxCjC4Qk40ehkSw8tVZ4k18WWziZfozd2/ek57tfUicnh5m80mPjPks+HU6eWzkbDHFoIQ6JvSvllCaoOSoHFwxhZ/5H3K8JhBj0qP5d+O3bfts5DLbxZDM5UELgLobGaIWahXjmuXeMbV+PMz6XIMExrdZ8DSdug3pshEjEIYj82NuoUEEudYJCHGhbIDwH9o1yKQ3QbcG4TX62BG1x4BEeVibAEgmS2Ftn9OYfB3lXaaE9+hL/2IZHAhrEYbjqcOSUHN0mqBuQ73oVeUDhBv0hfsg7RQLXAgbmMhzvyvlohPAf03T9OzzRiFjLTMMLdTIEkICkH4Sh7yTzv6wc4dWAIQwEAXF9t/ziROBiKWBGQMN/BeVMNYuYWwUwsohOcKIQGBiiKMNUBrJ/5ogcMv8YoLAq5QIBA4Ox8EmECgEAoVAoBAIFAKBQiBQCAQKgUAhECgEAoVAoBAIH3t315wmEIZh+G0WYlWSykDqDNm1fJyFhoJ8idI0OJnm//+jhthOzckexdm39bmQSTxRZ5hbQHcRNBAIgAYCAdBAIAAaCARAA4EAaCAQAA0EAqCBQAA0EAiABgIB0EAgABoIBEADgQBoIBAADQQCoIFAADQQCIAGAgHQQCAAGggEQAOBAGggEAANBAKggUAANBAIgAYCAdBAIAAaCAT+M+Jwe3ufDwQCZonX5RWvNBAI8GHblmCZCAIBo8S4ON3QlV5iMdyPIBAwzS5T6cdt3lUWtzMQBALmORu5S9O2DfuSWx4IBMzL4rponrtiE2w8bkdYCARME23c0nR4mjmPcWQRMwgETMviIXGXtwl1YZdwO0tHIGCUIGpq2VbL1vI2YTtj1gcCAaPEuEaxjLfL6En2pUAgAH8dgoi2fqCUDNcTdp/zIhAwyZ7Z5BbTar2s+00+ExiLBXDELtusq1uL7GnjzbyEWR4IBMwSVa9kELdRMZ9Y2VNlYw8CcMTLdtL3A7XddOtYthNihnkg4sXqx88T+7Gyxudi9c51NhKnjcM0jaWUfhwRN8wDIUH337+d2ufvK2YfLp4VN4/mH6v102aTsfuekHkgYgzky+WLq5dlvL27q/FBv62IBKfNci7E743sjWUk3viv4FUI90BI3H+9vFx8WizG9b19+jQ+7OXll5VAIea42fCcV43reTPB7BydeSCj+6+LxYeLl+VELm4WV19WqMMcr5NSKfX4sOwabn38M4FcXFxfX7y763H9MAaCM3RDBIloq/YPSoXSrwt2m4F3IIJIjIHcXH84jbu7u8MehN071xlZx7deXj9n6TgfhJt/IpCrMZATuTkEwu7Y93yIdttOonRtFenDlN024B3I4RDranFzwpd4/RoIgTFZnDplemtR169n+CYdgcAxQdNU7stNPZ88h/uEUxwIBIEwIKw8foye1EMXh5nFaveBQBCIcYLIjlqnfPQD+ewSNwgEgZgkiLzGJRKTKN3t57g2LwKBN6xmvxuihEjMEstmN5gXgSAQs4paBoHqPCJKyqzBZX8QCBxJuuBxvQxUREmeqj5HIAgEjkw3KiNvLx+qfejLobF4nYEgEARiVpH2paB5HcYyTNfNhFkeCASBmFXUfUmU3AZBGiUWv18/QCAIxKhDIN5SDXPBMA8EgkCMEkW9KwS5wzb/0wazmWsIBIGYVOzC1iVv2FVjHoLZfFsEgkAMc1JfpW233WYJszIQCAJhwFtvpZSBL/t9VjqehaEmCASOiMSJ2qHuw0AqFe8/srs6GQJBICYJQTRxnSJvh1752wo/f4BA4A1BB14TdS3GYiEQeEMc/7XY9YFAEIhJgiYzW9ArfnNBEAgCMc3Nc5eI344DgSAQFqpd6thFw2+iFAJBIBxESuXV0CUkCGOxEAj8Yu9Od5sGojAMHzg2XimYGIqMjbz8QoOM93gpkQlFcP9XRJMAYilIBYYZ4HsFFQiJ9kefHo/Hy1exPybLEi+Zp+WlvAACIKpr+ySKkovxsmydwMW1WACCPq9om+0uT9I4TtJZNIF2L/oCEABRWSveBm3fN80TMSfh2knGwQwgAPI3lSVp14knLlHQlud77KQDCPq8oY8WkY9tUBARGwYzyYyJGEAA5O/JquY4ipJl3Fe1V5gkzYdZHDIZQADkr8qtSxFHaRIn6TqWgawzvUG13W6n7UA3i49Abkns7ACE6ReS94apz4FoeQ7+P6m9fFKW+3FOwgtZl7ub2ZKL/kJkTMQ3niASv/tOQOgX4sME+RIIJsi/ldVm/obModxd1kxS2ky5KMttOXy1DDn9VSGQs9v37tx+TJpPkDsAoiomsstdv4ipC8h0LUm7IN752JkmMd90mc5HINK+/R7+6gTh0xpEPhDtHjfzf8TEbR+fErW8o1xnFGVW+SYT3eiN+MxyJ8jZw+MinZnpJ2OWDgQv8VQY+0uYTlXVvErCPJO2EmyXvBfzct2jUwzzBxmGTCCns1h3nz02DPOn4+L53bsygRzfk75hE0mOrqvYh3NmHU8z9eHckqS6ZdmXl+lcMX8tdGM79nfzXj9+dPeBxHOot6+AnL30PPune+28u39fNpAX/mvPRhJzHM+la6qXpKJTfh/uCkkzxC6zgIpt+ir4+j83Nr7vfDdbIpBTZ1dA3nie87PZni8dyL37LwbPdhzfQbLyfftaIN3cD5+w5HlLvzk+fbSO88sT/cBfCzSsH2ReHWLdlbhR+PDW7Xt3bz82rJ/ONJ7KXoPcOqxBDNMyLSQr07Ku38TOZvEJiLtLG0vKACnaKjgCEcNNgcndKDx7+Bt20qUDwT7IH4rpm+qLtPu4GWE+SSeXiH//ZyymfHLJbfJdcO2XwT8GIvcQ62ofhAEEXd/mSTQGdIipEGljMknIrPJ0n23ntTKISKed9LOrRbruG4UAojBu8+jcP6qwyjiv6bfHR3vlsvYXS1n8Y5e7MybIPxF//x/cJg2XcvDsYZuEe0vWjelmO51PrUs3vmNR7hrkBETzQ6zjRiE2CVVVlGkYz0LMcTh6Ep8dtwk2H/WxThPk3p0zzQ+xcLGiypjc7nxO0jxdJ1vS+GBt7yg8u5ogfw8QTBFFbbqyKbPWJQ3Tew2CG6b+8T4dUJmOY2j5AwpAAERl/OGDvxttLUc4gACIDrVz2pKOAQiAKI1Pv+o1BxAAQd9T0uaYIACCvjdCONiJgXQMQABEg6xiqE3SMQABENUx282+yRxXxxcgAAiAKI2JjK5PojhdLx39fAAIgCiNidjvw3wclyTUcRkCIACiuiZZW8sKyjl6FWg3QgAEQFTXzHuXmHhY09LU7SgLQABEXScLrRgdOnSZTtpdsAggAKI0JnbGWGQFkz0umanbiSwAARCVMTE5Yximr6ZJpG8L3XwACICojYmCclzzJI7CSDRV55kAAiDoEw8mw90EfjW9usjTJEr2gU4+AARAtMgwi8Bum92c7HwAARD0eWa3E2LXeERBXbUb0ikAARDVWdkaxUkU95mG1ysCCIAojYnqPhZNuV+jOdPq6ApAAESLmlz4hlE4b+Pe0+49RgACIIrj7by16CpP5KV2B1kAAiCqK3MRHKVM+b7Qa34ACICojZlo6KO3AxMHY95Ymm2kAwiAqK/No6XpsvN4GbS7qRBAAER1rlcezvPGUV5qd60igACI2pgo2zVtI+Z1rPS7VhFAAER5TSIKsus6ID7yYJ2IAAiAKI2J2j7PPv6ZdRshAAIgSmOiYBeulVfg6e4Agq7xcbxhKun3ZR0UhlaHVwACIDoUVE9EnsRJPl86pFsAAiCKYyLLa5u34xxftJqtQAAEQBTHRBvHsQODrLopHb10AAiAKM+s9/3Si6ktyLQM0i0AARClWdUcRXEcRcnOJyLWbasQQABEafUcrlPTTCIOF/1WIAACIGpzp2ipDy7sZg6FfiexAARAlGaPSUVMh7o8mkztRgiAAIjKBpF3fEJhNHEfkG4BCICorO7zjk6xs661dqsQAAEQldV9mn1AwUE/Z6RbAAIgKhuWSJTdsGFiKsa50m4jBEAARGX2eRIl+bLbVm1gjyuAAAj6PLedxjWJ4yiex106ZwACIOjzzE0wdOWlWNM4CjV8tCKAAIgGFXZdbc/781qzG24BBED06IDC9YcNa+YDQABEh/j4W7sH8wIIgGiXZkQABEAQgACI/mk2Ot6zd2/LSUNRAIbX7kogJOGQMtUqHhCubBwqhVCQAUW0+v5PJAEdj1tpNDHQ/y/lohedPUO/WdlJCQABSOlKjZTsNBZAAEIAAQgBpGAghb6DWtOvMh163JUA8ncTRFXT57ydlPISwZ0IIDsgpf+7hcd+AaRUQNJUc2eipbsbzp0JIH8PRCT/7QEbkH0DSJmAqKYrfJJ/yiZkzwBSJiCpED1/evk03y4XN6LMkH0CSJmAaPrtLCqVSn3T5jmHKukvvvdaaL8AUjIg4izqlXorzwByiwBSIiDbdNGsn55sMif/OrN7NCuXANkzgJQOiLM4rZ+d5BlA9g8gJQSSThCTWycNgOwfQMydA8IEuUUAMWUEslngxYW5MLsAcjwB5F/sQTjEOtoA8s8miGGCHGEAYQ9CAAEIAQQgJAAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQAhewAxACF7ADEAIXsAMQChbJ9/qgABCP0pBQhA6JtPr9efVChAAEJV3fQLFMoEAQhFo5ejtn7h4Ea1MJzU2lsfChCA3PGc0TJJkuXI2U0MnXTi6TRZjZVNOkBIZJR0V1fLXjyUrRD/qvfqXWf6YcRZLICQiD97vAwk+NCdudsBEnXmg6rfDvoAAQiJ9AfLdHaMX93fkaitVuMo8GXbMe1BNAXSap01TC5dfAdE9fbLUxXNFcjFkQBRKTKnHTkq/Vlv5kraJJ4vV/HMu/VCNAXSMHmVAqk3zlUyVwAQkwLRzC9iMUBu5PBTKTZn0JuPVNLGb7rxct5dTX7xOju/SXdAzkxuNbZA1MmYOporkMaXCZIuUTXbEguZIOocelJkKuIOet2Bv9Mw7HSeTZ4lj2f+j0y17XmBtWh9fq/ePMnzCN/UG4v1Osjc2vuY5wRJa9Yvw3X2BQYvTlu5Atks8N5kHRx2XlCV4lKR9lWve92XNFW33VfR8Zu49uMAcaJwElqrBRsgpymQ/KqfvPe8MHNB+LHVzAvI2ZZws3U5DMKsebUigIyCWnjQTWq+FFn/uvdq4O+wqNNub4fFNB7Kj7n+b6q6hexBqlU/c26/iD3IjZt5ia6/OM3/EOuBU/UPPJXiSudHMnY+7zGkfzUdi8hwHnu33R3lD6RS9k36X5zmVUljk1663Jfd7nUwGY0C6Y+fRe7Vo2TohNPuzC/XdZBGCuT5uWims6i78VggkBKf5r1ROfRUiitYPuyulvE8GYg3TcaysRHPpr1lyD8rcqGQRLzreLWKN0BeStBZjUVqb5MkeReKaKkmCEAA8l+qtqMo8LxarS9uFPki0g9Hk7aIHtmVdIAAJFNq/ZnyhimA0NeUmzYAhH6Z6ndO0gdAAEK/S3lPOkDIbkOUTTpAaBM3jgMI/RRADEDIHkAMQMgeQAxAyB5ADEDIHkAMQMgeQAxAyB5ADEDIHkAMQMgeQD6xTwdCAAAADITu/aVHMINyKEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEH4BEkQPkEShE+QBOETJEHGPh0IAQAAMBC6+UuP4A3KIZogE4QmyAShCTJBaIJMEJogE4QmyAShCTJBaIJMEJogE4QmyNkv252EYSgMn+2M+ZUNtkY3s0TFycc2FRJERRHFNKne/w3ZToQEIskKx7mEhzX055t2T9/V2AmyozgYe8z6mRcVhG53Tbe+uSCNxrFr0GDKJxcEERCKg2r8iSAI6rdDE2vwPo2ahyuGoJrjWkGcupkaZKS+nYSo7z2pIArXnDWIliIoH9UgsuJSw023vnoLQaqvR1n5UY5ac/iUjce3e4VzqQYh9MM1fTsNQR8kFmTeIHpbiJKZIGSY7rcggBVXpMT4B9kFHzxcDgd5CiwQSgliuCkV6vQzQwR9UAoyf//SbZOf0rkg+luOL74MaFLkM3LyBqm4HKWBILH6T8ID9n45qkEhUAniJ25Ch+8kIWz6iZWQogQBtDSOOMSZIImEbhXrTu+14gVSYnYEYKO3vvy7ew4YLmXB9dnw9cTp0uJ0N7qDIF43iCPWnfMb/SsIQNglX8PeWaXvIFiWI/g9PD5uytl9FniLPLNJzNYRn37aHVrszssV0yf2PjrkEXtHMdPlil13qLFPTmNWbTxmQWl4XDUI3meT/WVTa80gENGv8EmbnkceadKSCcWwTY/gItKER1mbngmPKo2YjPahNI74uJ83yHRVkP50GqyjRY8INkK0lvlnCUWLnqDifLFrbisKw0AYDvwJroeb0gtBFOzWO0Gq1dZWEa0Xvv8brVssbkyzpbvGHpwPDKZBmOSfiWlmDtuky6oCVrJ38B0gB0s5grnWr5xn5jlb/8KeGcdut4F1oG91WGW44XHBGJz9xS15gYD8wbpcX+BVRqLeVzQNfj/PQJWT4E58+mD28DPkaR91WXy0RF0tDb981dMaR0Da9k/Tg5dM50FJy4C2bFAlaO7O3lidUHUmhPvbeHfcOKJNq9oS+4k6wINwGC2z+AC5mIaGBm/jFQSrCLAM17K1ud6S0BG9pvI3eC0bbDpBEARBEARBEARBEARBEARBEESrQdpIfRSUZKIoHQiT1aDQZyMlG4GKqoSg7yN3DtqfmpMamlHVA/RC401Sgbh+OM+ZNfCoKdI2a+QHMJvAF6IjOIPGAfEovxxUGm1NwVEQ5cWe+fx45eLKrS8pC+TukBxaQ1GbWsSXAAbL8bqAHACpU+IPtf7ggj9bXnu9iqJoEYjM91WXUh/ih4lQBeUdM9r2lp7vKjOAFMSyVTDva66/CqNwtbbvhtxiA7knCT7w/J6qBN6zjISvN/FKsAf6nmffY0b+lqIInDbuehGAPZWOM4zHo9Fosgs/JP9H4SEH0Az2/IUvmAmsJN4GOpuQ/4dnmkFynFwXcBxflrkHuseHENF03oe6yoA6n8qr/r7Iu7IV13EgKpCM1xdjQUCRwMujwci2vJuQhED+/4+mSnZuuiezMJAMTE/dpb0pPjpVR6XFTn/cHK341f0dV8EAFP2eIfpKJf74ymS75Pq98IIq5yJd11xycQjJC4AXpb7kOPrahenLvHHJB4x6I5/bV3m8DuPov6UQ6p0EV+d1VpyX9e/I+ePkEJz42XtBSl+bnP/Fa0U0U2Jwfpdli5J3CRD4qg36l2O0ikvzVnRMp1wdTBj3zZmLW/yHIvjep36NzJeBkkll9Rm3hie+1l8RvIri9dRHQ6wYJO+0H/u64/wUvvSq6KsfWXZvkj+Kfvq/e16XokBSCQKh+1iOEbRg5GPyOOY6O1cUXOlsO8x1HbZ7d9uz25VQ5q34+pKrKsEt16xcVtstHddlAOchCHt3Svd/gPcZkq7rPoMAYdpr2lRVjHzAaHwQS/1d4m7wZTTn7Lzt6B1mSUVkjHzGooWXvf3wGrjU30HtOBxmcWwGm05SQAlLZmDh/vI5fejGQS5/vFHySyCJMUGib7fBBISF1VmchzZBfpvjYcqsWLysj/V1aAMSmOZ6nKra2Xv0ze0+tC4p2pNUh8xj7wM4CHENdsdVSg740UU23I9DjZtBncWBmY6TjvFEmyWFnm4D4LXmVdfDtYr38n11vU9ZAIWOSp10FJB3GyXx6SmQHcLtftU7nsIMtxvwFhDKvKwGNpvaM7UbNjfY+oBEsDHI+bDv6TzX1uVRcztcLUk0BBjZ1GhtNoxuXQEegxBxJDoB+K2vzfrmCPQVu8uH23HI4p+eQMhTIP1lmU5KCHluEmBSSCnHngRZqYQU+S0CJnRXHnIuD0k8zVIpqRbbtsfNKsHmIUguUEzMlfu+MW/J5/6xEzZWFf3VolubxH513nGahRD50Qcxn87XW4rD0SlGX5sRwEt1qtGPSdUpKJZPSXGUQsm0Kcjb7btAKHEzCyE92GPelO+8FcStlvIObB6zshxGCaA7XXxiFaleedm6uEUSXXnwM9CWifQOzDrZBmPOl8y6rR/npqjyMYRzZkzxurFFbVedRPpuPtZywACAsOh//BD9l0DqhUP1y0XyVBOzCqnyQ+TonKvysApx8QipFOdKpcdoEjy/HODSPNv6uOl4mrlsvCt4P+30+wRSz/ySPF/BZwyi7IJ3v4BTp4SEoxAqXcqUi6kgfceFTLsu5aoJCGvPXHaHTvKupsSp4Gh5Ogsx9VMK7j1XHxcIYWZ+QIBg9E9CzCfkbTbEaRQHFOqa5Uh8d5Y4uv+AxVfBV0hUcbD1QSnbnNoJXkZAi+ICnHZW/BJuKVtp98Zzn5J22cEvLWGNQi+vgp9CGjRKnE/gcjHG5MfbI4OUUN06CYGIU+BCqJV1giE266TwblJC/OF4eTCm1zmeLEJQxpURk/NzliRtx5UpBpFW8RsDL8uxh0W/rCYUkxSXOklMhx3q+CD4WcdJNvM1ov7IeWfiGLRQxgT20iYu4iZFkdULzwFaP+KYRqdq8gtG0D4pkKjkyEcMpB4C0khx6i1vYqKssmxmfg0aOvSJdxNiCN7dHFM7ZSe4yMt70yYMJdKvPNcA4wowApoBjMmY9sbTFgguRr7EZBJnD6rC5eQV4ST5gUYzernwD0I0xOv46BcFhEzekh9vu0Cibpt+iS1DzoVfAkImnmYErLjz2SeZErcE54W7bao/7IC54iZye00254ZkMq3f6GOmAVvwbV4SPFWGuGFyCLrkLlKNp0CZLfFGMRu4vXvks0cqKTVGiAPnatKIbeKqXvKKRKnCUxvSzwgEPxtvarlxJ5HXRK8Xz16z8jsjVSqOCbFjhEsMl/eLOLy7PbYgWHSdBcfcOuqCkgDoyPAEeG6uCQjkgDCMEg1FNGJiZJIQAxkAjO2wSl2LQcztXr8u9s9APUK+3qL/TRcrWsQRm/7gKtaIFDiLhWI5m6iue2/C6VstVQbXs7COCSt805zFDVVyKrYBaBaTSqSGvFsg33yQSVFZ2MUBgCYHUYZ4OkNsfilOyRaYeWTV7dVt3Q9KanpA3YM5beYRg7NYzw/9VAYJ7jw3Xl0DBCkrJ65DwpLINJAWGauUbXyoUcIqN76I0SNPe++jCKclBZHkTUKSka8m2kCpimRqm9kKSz6GqB5ICs4k1ti2ObZRijIfznZtj2RewcPJiatLZULiJMF/+tcz/DOByKnAuLMCSUoUSN/hCtP5fF5SLjTRChlD65vjpcuVkAPxFn5kv0K4ea9AqMEu1rc1g0pCXKE5jcwNCGQbo2grkA6v3gVCb0Ja8GvOReOcsEL72i/NQCAOedpnBIIxr84LIFiBv2tBSb3zpgbGGpW3iMcohYqn4QcEQp8a6fVQKhxfxiU6dQfVELPr1LHExh2/FIRNYkGB5P1j6jxauVr3Qkozs3Iu5xInNH++WYGwXSCYeJ8CqTsu01SB5XPZ0moTCC30KrhQaa42gdye6/BvFgjpz3tco7mt7qlGP6LXGHh0EwjdBUIjKxBK0Lc+uQuZWvR53mXBhZ+SRzdtF8iuuk8K5AlhqYqiOgsuYVupiVHQdw1wUCAakXgjCuTtxsLWd21VWTwofnLicgeVglMzkqk8swT3OB1cp6CZPYOgQB4ai85YSG2FDGHmkkvB+TzE5Mf/Ft7vGYQUR7H6vzKIKLM2A6vDOCC7QJwKZ0GOTV+X4mgFwghaEhbv7mKR4sLBS7t5nTxQg6mColwgwtpfGaTCw1Yg2OChQOidz3oD78WurZD1dREX5F/KIMnIzw8IYeE2KU/Ha9UDrTeHoUD2DKIJfQjk7cHmDmnp032R5sKXBJjorFMNgAqIRoFYsCd+CiqJA1HnkUEighbERb/y0thCfRgWlLGkbg6r5Kr5+X2sPxPIxY5Bynh/HtQku0AwR+c6TBySdPzEtqSMh4+rRoG07yQMwkhci335vJL8QGolKoIG2lj85LQJhD4FgnvoW3rlc2ixOL1Jgss+BnGnpWG4kk4+Ngbpvz7TtPHn1G1RlHzO4oKRAHhz9wyyC4R8FcgbkVGcsJDZA88BBFKAU4MdlImfAmFazdUob+4jgzRi6zAw3V37kZ8eabwtYq0L4sZRk/Mx/OGjdPoQSP/sYu0ZJACiuKoYHpzSc/0QiDfzC7UBq/jdBUFtgwIt+QD/wSXvtKjkalvRwzWZNKP+mZf+pmt+COLL1wzifxUIHBJDYUWWrwYOYVUoqXNwdZ0q/ZElLoqzPNG+bWexBmxi2ZB3UbHw0UYi8rZnEEJ/L5B3A6Ja8a5luIkMnlyIfutUWgAos3ex8HRYynTbcSa5hgQIvheEYju4xM3u5eSWj147pxVCx2Tkk59uTpaKLwJJ7mKNKLR9eeU77Qw/orCfJF99DMIWvB532BIm/ZByfk+oyfmaeWG2cDvNK251/M7AA/+mVxP59ZBzOSQkmIS4tJ6nV55qGo+/BCIz+sggbOBpTXzQ1tCHfqP4XEP9bFUMHKxIn/LRhO/vZFESnkQ66Qqsadog2iBEAGEJg/LJ2y4Q+iWD+B8RCGZ2wc+T6XszzNjCkBqYQ6cOuHwJ+kkztgUCHICMgFvbOsiBy6kP+6sQV9bPfNZ+WF8FvyR1jtEB1Ur5LfnZCWRLrXxyaH8Wt4JQmhzw4Q4GZMmxx9Us2ZWr4Kth2B4aTNTYl4aFdKUkX3oSDLBblgpYD3Dl+62PmlBSwN142pWz4OoYbrM9Isf7qSGxk5PoIlpxkZF+5UebQa5ctYSYM5fnsZN8BmR2/bgrQWXXgnoLfOYQk/dbeOFcSiGk5AKCLZu5XC0EzdiAi9HAm1S8i+lge6P7vDUlqOfyI81xNEqo7bKkAigLtjGk7MYVdJMxVkml925dC86rXNhy9pX0Toh1XLm4RAQHUKobz0IsLQ1sWIyLxKWzn29mThsH/JM2ge1NqdIjpAZW54y6urMvK404I67z2Q4rk8E+DLVUQ37OCMHJGbBzVeCpVEAEvhNekV2UAIP7JdTG4JTi/lolBFo5dbezXDrNDfFHNdlBeqPWHqTfjlLAny5DrweZrUqOugiqWcjTR9prfNoG/yohLx4+ziQRe2kcSuIBgG+8AbwqXS2bbW67NTQ8qUNI394cU0K9YZUCTJXaZtvAdJaXsWWEZvnZbJfR4i6XfsslqguBv/5knX/3sZBeBBRSl5ohf6t1ydg6P/yBd4ymxGQhwydhPQZ7LMpwdtv1daORpKiapiGznouNKWyRxDTXqepZASVh162bCXZdGyCmqmrnne4FRRh8Y1RHj49NjL1fgEPQPuvt4TgzCT7b6zPc87K2wKKh3gviDuuhKk1r1Zu0VQVb7zc3ynbTukaE3g6BPoE7uKZKAfEGxZjYhi1UJSBvNySw6LNmGJrMd/ZDUYWgfGAFnZo8gjzStUt2/gLCyMaf2aZynR4LZaGNGbffwuJ/MMn7998swJw/eLnHYfT79z585Hsunu8osK9vqlLYf3336PsOfRRkX4Axh/77b/lQhz65dRAOVuCjv67ilQpK2ZM/SwV95e4rw4+/zpdz7Gshxv4HT/L+xt4dowAAglAAHf7971xriFuFw3sQCS4uuol3VtpyvO19genWPjPjdkVqVPMfe6QfX6l/0zozDs8AAAAAAAAAAAAAAAAAACz27qCnQSAIw/AkHw3twmWDiQc1UThycaVio40JxsT//48E22paxGJ6KIf3KQkTdrq97GSmJwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZ6MTVofz1I+HM3jpPiZD/cMrO5l6d+mv2tFhIGoE56dtfYwvDck0cuv9465ffkDbR/0FugimQb34mMhk/6DD6pNpuGGp+zBkYSq0N9lso/ET2TDtXR1ZNNhoet8yUSKYBrnF8std5e04V4fMRohdbC35kMy6u8Vple33hu6R/24aipI626zP5nVBeWAiQpM/r9fr59uyGjHb1I9t2nGLh/via7Ply1VirTZYpQd/RBbNevnTRJKPMlhH6WVZM2VhAmRmT9c35euqKR/fx5zK+aoJx3eNn/IHWefuJq+sla0uXnftYse/74pNXWaep5vF+9s8GDAFcs31m3dZ5n3hvI9k8omLnEtCGlsnC9VcZjZzsyydt1fhijgJiclXdbHN+GTn7FomBcEw/IBFTXUiDgTzKpgeCqGWZRbRRND//0dbu7PfH7An++5BFzRz49w+nmhKzF1+OhKJssJggKwdaZ+d1aGbuD5FzYQBJLumO9ujo0ZxI+nZP28+9bZCNd0tOhVT+dluyOtU1xQJXFz8exBAPormFAfE7gaQKXtcWPs2Ki0RoGIbl2ebANysXf1W6zyuevs2lV0+LM6SLw4EhdX1c9kL6GZ6V+asifCT9QCA3aOP425wk+sxglTbXpW17mKp/TTNbQZRL1zpfJVAtLE1S6qjfSgQAKndMQ6+TlwX7wCCJoiG4AOCso3u8qbokNSMjSMTdYKMY4vjQUegOeWjH0NFykMEPikXGK/AOLo4FqoIac4mJWiJzXSnvoEDJAe2xRDbx4ShPWsJthNUjZQL74KWPWNuZIuOopUeii4tZP6oagVzC/M3yAZxlKczhouLf0+kxcN5pcZxwKCFqHfquni+i7qZqcqJZ1vTbUzdYLjTtbVCtF24u0qP9+W0WDmfjvXlGIzlopEbfX5+ZqWZx5AuzEI2hGdbaM4NWPoodTXyFvul7oyis8z8UbU9FMHjhLsg6q51vI/SwOqiPopeW8jFO5D1nE7jMgmx5UhaIbgzUExMn3tLMC1jVdFZxpv4+VgzqITHmgWDUsVsBFbolvOvjk2CmUILPdcIPtHwpUv6h8IQt1Y3pppCF71RheE2Lnlc18Y0M5tJEUQLoPmYm+Dx8FBF1zwfSuKRbgbavojg4uKfQzyb26Ztq6rIAOHpPhlAWigC0LnJ9JR77x13XTGJDpBlQzSwNQMtxgJSv1SfHOPhuE3CAFRCpbniLby4LcHcAtMAsRmUUiPzRDquAbRwWNbz0ca5zWoxJwjaZSy02Ar1WJ5eLXzPoBYP0WO41sfFPwcBpIuoTnFeCAy/zwiSQawZQBVU98bU7Od901IHlUP2xisojwt6thNoJ9XsHx3lpjMtXA5g2ZC0Yepe9QGrSVs2E4hax8f5baF7nE/j7eMouBdC7TM7T2XBRgC1eKZrqLtFzE9f7kObZVGjKJ1zuLh4B1quMHzGjPThDCQzGxKIVjoQz3V6S3WfwsA3AqkShrhQQFIyG0Mt9vSzIz8cJQZZcg1WHAoQnMh1mT6enrB/PE1eeN7HlXhmKHsK3QVRF7gKI5aKWwTZk1rigzHcmTQtrE5uQwO5Do8hhouLf05i2dgUXdc1RiIzPfadexn5+9TFNeMGSlpK0OcN/8nqGCqxFC0fMUodbwGtdMjevnHYBMzIG1hpieFFrMWdDhLQbbr3kFkuKrmyDUE6hcKwBUPnqY/IdFe57B8hx2HJ08A0kFWsyPDQQKLu5bVALt4Bsj0eo3LOBWWwvzuSB9rI54P7ndMhgzbQU5UYj6IBsEylA31KqLgoQHpaQ/NyEDyxCkCfP/R3ZiWCTxhxXwpAQAYWhjXcRYMVswAtnwh2zA+e3ksoFiqeM2M1qpiTSc/4m6Kihlw9lv6NT9X1DOviHSB2FMsSQuDKGOUMoN7pRo2rE4uVAFHluJgOVZQbRqB9Taw/txK/EcCrN4C0E6cDFWWZA6r8SqBRYsXwApeLTU6Rb0Ko3e8Gr3MDqHkOyVl+eu6+Rt02bAuf6iiuVJ8AGQIXqo0hMnMIQVXXBnLxz0EAEb4Vnek6Y24Z7vIIQOa3le+yqLoMDuK0rYoEQGIcASJ5FuFcngIjSHCevBzx2fFwAMlJBHFubgnA61+6OM0+jSabymCCZYJxAkjmBEF8q9qcYBJJLElTFRFE5Gg/vKZqMZyQQ+XX+rh4B9Cv4huQj7T/NuH0neWl0R9j5OhHhdDHj+/GQ9+UQeg3ScSTKx5y8Z/waWLmaqp++94F9IfwFPqL2fw1GPWrsCG6QrcX/ytI6gr/4eUKPyZkfznJ0beWb0F/Dp6j30QTr+Vy8e6gz98R+u35Cf18g0d/qAS/ES+Nrkn/oT04IAEAAAAQ9P91OwIVAAAAAAAAAAAAAACmAqOMhnGPsYlNAAAAAElFTkSuQmCC)![homepage-return.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAKnCAMAAABJbNmOAAAA2FBMVEX///8uY8/+/v/X19fq6uovZM/w8PAmXM0qYM7y8vHt7e37/P1fX18CAgItY88fVsyXsecaU8wODg64ye23t7cYGBi5ubkiIiI0NDT19fXk5OQYUsorKyve3t5NTU1WVlY4aNL5+fnR0dFpaWnLx7tKTlooXtcbQ5QwZtc8PDywsLBERER7e3tzc3PIyMjb4/c3adJVgNiqqqrDw8ODg4O+vr7Z2dmIiIiRkpLMzMykpKSXl5egoKDq8Plskd6ampqKp+OdnZ2MjIyOjo4ISMgNOZBofadBXp6/WLSXAAA5j0lEQVR42uzBgQAAAACAoP2pF6kCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABmn15SAApBKIDC3f+i32cgFDhKGp1DQolFAwUAAAAAAAAAAAAAAAAArsi33sh/qGQjTSZt/bRsz6ZiKYLxfqtUtklJ7ebmIGc/rvvm4mHfTHYchYEwbJkoEosvVsCADAg4QYADkLBKUdLv/07zF53ZWrNqMjf+A23sctmH+lIuoHf9X/FfpJWf2vIfM8b/y7Z+lS748/4bcnZWdr04Dj8kDPtjB/9w4f/+283/eCrB+j2J/LvpH+Cwv1jtjOx6ldKmPUU2GmHZTubvgvYjJPwn/X+aWQjINDzYfwhw0JTv8rLjRzsrNPf0sevlsnLXv6eIrFa5dfRjIhh/Nv4ye/A/MUhP9Xr8OPCTFcPcfap4hN8dveysGg47H7teK8TTIRFyCNBYhEiMHzwk4j+IdP4BATL7ETA/6+JfndulrwbzxxP5x8XCWHyWugXfbItfHekEfE8hu16tgyPcMUBgzVI4V0YKjunX8dS0cbXN9Bmz2yD/ZpDaZPCln5voJW3xfwzYB9lBYL+vQuZpLdwFU39gRfH/tH4qBM3xMA7VWYh4et9rsJFUKuG8e0mtgO3a9SpZjlArlQ+lFDkASae1r4YyRGhe57m7vFVv3aEZq+FkwTqa+7ovQ87SbG2b01C9NQE5OT2qul8zG0QYt6paJ+/uwf54GupqNb7yYtzmy1qNVxatVU0ug1MhZHI/pN3SHpAi5qVL7Wxuu7ZfOm/usrmqlix9ohc6QrfUAs26xLnq1tfVeDmyyZHCX7KAWSVWnKM9lex6YQZ5hNbBWjdAgjaWQgi3vrJ01a6vcXeuCnTpNWBh5aKlqohZA+oAJTBYpuwwKgFJx2AscyQszlomGTNveutuvjxfWqT0XaG9sJbk8pa+H5rcyzEXuuHspERuBaNyyap33CKGZ3L1rg0QG1llBg8Xu8NeIf8Ev7RQZR4fihq5sT/I2vUiQHIhihrCqSWPWOcjXH2JGLPsBTRIogBRrelMkz4QtrEv5JBavYQhDZ4N1vrCjR0l3HtqVVK4Z5+wyuxWo7+Qwok+VxTjBlLsgQ8/Voj2MJcU4g0OT6pjvHSFYwWPjVFndmDrF5LW+wKIdMpLO4DY/BBhrp9j37VVKgJ3MbFlRRvMQ7Zr16sAERLCnzyku+SULVqoNpgR8jeDskNlgAG/mXxRlKF3FkUTECuLQQek0fTyfI7CHnbHi4/yObokcBNR0I9R5hA4PIBsdgc+ldesSjhZWPoiOVwxnHeBkSCDMHZykUHSEWHen6IJgDidUREBXwARUmntAtwLi3pnaKyLxnaQ0MTZOxix0KcQ81xvzyC7XgaI6xe+rwCICQR0yZlZS/pZ1iI+sK4QeuJRIoqslCIuszKmiF9cUZvMHpAfwiCMpnI4C1kdV7rfAt2xDI26YOpyKapwzXOn7jgNR8xE+smnqQR0BhuEmm2WfQXkSL6dA2MTFvLQh70cvgLiugKX/BIwLJtdbrlLZ6xOi9pmLWDxsiYXckl3Qna9qgbJS+/k0Q910OjtWa/dKhxhSlwsdinoFGUkuLZSuNqn3+8xvUkxMsYpC4T29BZT2MrenKUcjoxFvnCOmRby3byaCgH1Ng2bjCoQlDdaCt3ZvXCX4AkI954ZBKacNcgHGWMlwYqlCBDa7M3B7EfAmB21OdVIgIIhj+S2vSisuCWY3mK7dr0GELXajG8/1OaEpJEhFOcnIPWRe76IrzyLUVQQIL7vn+PkFCCI3zCr1MgYXYH+PJEAZJWytjjL0G1ligDxi7MzT7F0XTXa7/xQ4aHI0TnP0grrp2xKNhg6yiAEyGAzAsRHX/s5g3ACRM8cV+GXKUcNIlWca4HjVqkIkBtWfN/gvAOy6yWychQNBEgrEZtZQQce6pSV1W4ZZAOEbYDgAJRcQmNe2tB8uECIBb0U1aGGdRTcCJDWRUXADg/5PGLNYVQusxG2j/FtzPgdNhY70BErgqPxZBIg94COU6pj9voZkP4dkMLYNkbMMf6FZhijIqLUcr6YGUqijhMgAXDF6mF0GudrynbteuGLQtTPVKTXCLrVq5TQnr1+AcQgQPwJ+OjWbhKdG+abK9y6m32hSjOBi9RIqAaBnUxutRI4eYXo6c2w1nEZBEfzeLTZApsD5yVK6s4sY12baY0Jsxk5GMngDksGBMjXDDJ/l0G2t4p3+K/SVmJnMAYglEH8NbrGoM3Kch03bNeuV3zOG8aI4gDtG2La4M1ZCOVL4fam/SZFcmCtpjBtkFq2c5VKEMTOlkGE9DcQOBKCTqjKiA0b5sKVErGbpSd6zHuWwu/481+zBiHqkGoJzIgVbgIq86nKHjHtjB6U5wElGGBwwZoI9JXIfb4oPG+PfDlyFjJdg7s4dtFc2aTxp45mhRWxwSJju3b9m56nllz5C2WQUit6O1g6Wkq36COWroWqkUFilaAGcdT5wsLed4XUecOOgyuKwpXKuXCW1UpKHRcqnnjaxlo7laZPQYI1hrmK24A9dVf67QBQplxL8FVljHcOvKzMcOCjcHxVWcHNV1uRjjWpSPfV8OUxryruKRpTQp9WlmcXVMW+HnEoVK6bROZY0AadU7C/Sd/1khQSdPfWsNGI7ktjMmaH7VCNHbWu7dqkPCzv3pFZl3sZwvryqPoS4WoNUo7ZUq1X8hPNVe+FzW0Orea0zE2QaZFgxIZFdY++flVo3NcsxQR4HaoBful7+7EfM5tH92roLO/WpbYx3zNsCTclLKJ1mVIGcSx/a6/kLM3WR2mll6G6G9d2aWxurP3DM5nZjVXVRjsduz6xcze9icNAGIBHkyhSsPdSbbuOVRo5OQFJD3x/VEJl//9/WjtgUE3UE6IW+z5CqSdOkx48cg0ebkUYI857CLlr5IW49LA7doHrZNtpiNh9kv6XTG78rsVcENuexeR5NB0MG/eW2PHy7hL+ulHRP8VvRyzcvbsTwvUL/8hCCCI+/47vcYw9625iXNO4E8f7uSdigQ43wO7VV4Qe1qLzVYsptSnQdNHXrqp0O7HUk10iBHek/hL4sGYxrKAVxP376310CXwLu97hVsJKDw7OsQ/D5BnsJmofnHfMrFa/3Hp7r3tSILwp9yZk2OebHvu4t8CEGIXpcEP8TZFsOE4vbbP8PR72XcnD8Xq++pwVZPUO5ABfRxyG3F+sdXUt40tOIH5CCAxQAAAAgJvC91IBAAAAAAAAAAAAAAAAAAAAAAAA/C+4O6CCCKAf+5/YZgvwDRRqA/Rj+0KtNkBfaqSbWZWdMsRoLRgpAuBxNS9LudZdmzbNeoD0ADjbSFUfStXobjJpn+sU/2QBePn8ua0GCzka20B8/HlqMgKAk2xXV8Q0VTsmMZOjt3mKCQR+Dkc2+oweGjePqC1RJevtvNV0hPe04H5iHnRM4kPJhLLVaJytak1XRJ7lAHeQdYesiCpDxFSpF+Lp26HQTauvZjkuhlUCcA/LxNGCIsFExadSU0ObcrQoXqR8N8zhDDNIAe4ojyZBiLO1mowN0VS9tqt6MqpnNuCY103w2DiqEZfvlKyIid/nbdvUbzZBBHHUiyZ4ZJENNbN9VVu9rJJBkaU6XUi5ERz1XwwPL6YRt5RPqpblpHxhsiopNQHAyfIg29ZmSD0ji4e7fRpbDgP8nEInSWUlGTGxDbVBxdQ/9u3YBAAYhoEg/P5Dpwtp3Mdw12gCNQLBlZUKRr2RSyFMcruFPQMb/CUdAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIDD3v31pgmFARh/+6ooQsTOZhEVqXjlsH9SnZ3Vi872+3+mqWk3tfVdTMom9flpYrgxBvLkcOAQAQAAAAAAAAD/hqrneZqN9Vfzz+3Is5v6Yz1LNyJCI8grbZSeg0o5K+2g7gmQX43yczs4/2jBy/u5/UggyLPG5Xm5Vjn70Fetsrb8bJfrHqdYyCtdB1IpZeUsKNeZpSPPGrftSuksG6XKMhD6QH7pagTJLpBaUCEQ5Frj9pwRBNg/SScQgEAAAgEIBNiLQAADgQAGAgEMBAIYCAQwEAhgIBDAQCCAgUAAA4EABgIBDAQCGAgEMBAIYCAQwEAggIFAAAOBAAYCAQwEAhgIBDAQCGAgEMBAIICBQAADgQAGAgEMBAIYCAQwEAhgIBDAQCCAgUAAA4EABgIBDAQCHERFdHND9LBAbgkEp0B/l0EgwCt9Z0sJBNhNQYURBHjDjVyVpeL190IoyhwE2BDOx4moSmHsx/2HqjKCAH+482Z3qiKTq2b/vjuYh0ogwAsvGXa++AUReWj2k+pdGieMIMCr1v0gjq9WUTx1ZiqRn45km/41kDaB4CMd0xFv/ZwlP/rXqjLy+xfJrDtuySE0+0COa4fhPZ+3kJ7b8578axHtzZup30yn8obnOnu5YaZLTWpB+dEJXQcnpOqER1SISjheBSLRcOAP0/SutxuwhlFS2GMSRfXLINtAJlEBp6XlyRFx1oFUF+l9En3td7/KbiG9anGvqlvPegQpOkWcFveoAnGXgagkcZqo6F1zER52Oti4bGcYCJP0k3RUx3wViOgykIKoXAwW4e7P0/98mfeo9hZ2feI0VlScYTwSqQ4740lx5A++cZkX2OBcre99TP2OP+42xy3upAMbwofFRES96TCO/VmL1bzA1uOExVaoslQcXSQhy92RbyqZ0PVbVHlgCtjubSsKlrv/Yu9eehuFoTAMHx1QpMRkY+HaIC4KXnFdhEAKiRSl8///06Qzs2ijWc6op9L3VL2susor22AMfG9MxP9zPMIiHb4xJuL/8w8xgsD3d9htFP1j/PFPxiIdviN+/07izr9WaaoE3XTEwXEgxeHojCnWazewlDwQCMjARDS2tjv525stKiZiGY0gEPh6TA/q1XXh0N8Db05bIiGjCAIBIdSpXmmILlTVxYA1CMBn6ur8rvI3TrxbGVMsgM+3K4LSnq/+lgx1PZEUCARkYKKx1cb5c627nYzhA4GADPz7ayyNsUb7mcRAICBBEoZUHZcgvp36WypmiY5AQIbD2HWl68JfaYTBRsy9dAQCIgyt3uvMry9LmHcel3kBPmDaxl2rtbXG+Ui7FyF5IBAQY7cW0Xoq60zvfSplAEEgIATTbknVIR+Or/dAzFYsBAKChOOykbI6RyAgCROpwbu2v8RVNW9ICgQCQvAcaWu0tq7uZpJyIicCASmuWRl3rij2+w4jCMATda5XaqJr1Wd3OesQBAJCqLf6dmiilWbf5QgE4Enj2mUqz5T07ZGFPFCIQECMsDf+pSzmqnWNmDkWAgEBmN4tUTScTN3qKCcpEAjIkQ+bwBtTjgojCMAnm6EZt0Q03++CnpdCICABU/qjtu40KyJSYZqQFAgEJFBXozNrfEoqbfxpljLDQiAgQurNOWhqG3NcGB0hEICPqrJdiI4u6rN91r8kUvpAICABD2VZMaXeaNe9hGK2KiIQkIBpKIqK3gMppoOY6RUCASmGsgyI8qhuxDxLiEBAjqFoF+I5KisiJjlP3CIQEGFojV+XuCgXeifmWCwEAhJw3ltt69pk5zHYkiAIBCRIqosvnLXGuqK/jHIaQSAgg9qGy/HSFdZq7WfGFAvgCatkNw/HvjjleAUbwF8wEe/mNBFzGQuBgDgs5yIWAgEJ5H5KEAh8PSYSsypHICAS//klbEhBICCG2qYb9ScOKef+IBAQgpPGl9FpHYOEHoRseUcgIMXQ7q21mSu6eEdSIBAQQr3a4rp2RWZNFIj52CAQEEK91ZdEbfKqOd3wCjaAZ6PvNvTAm91BzJUsBAJShH12TomYWNBdEQQCUuR+r6NLXOVyjo1DICAGV11tjXFtdB7lJIJAQARmXi5r8+Zba4yfMcUC+Ezd2rdtklbTrb/gDVMAT9TV+S098C48kBQIBARgeggid9xK2wSPQECKudzbbprDhIiJWcZmLAQCQvAQGa2zsv8R50rKAIJAQAimTTXefeGMyfAaaIC/2lXxrStPuZiPDQIBAZge5vtlyhNiwivYAJ4xxbV2RX8f53SLy7wAz+YfpbPG2NavIdYgAE/UJl9ebj7b6z4lKRAIfD3+UEnYFPZVSTl5FIGAGPz7h5qiN0yxAJ7lx+OQ7xRvfRGTFAgEBGBioqkwddStwdi6BnuxAD5hTm+lMzpzmY5mkgKBgASKiZK8mq5nX2d+UIqEQCAgAB/maVi7iYi2wyTn2DgEAjKoqawzXcZzqIiCKRWyAkEg8JO9+9pxEwjDMPzpBzkyQ4qIMSUwxHAEBhQZXIgtOeX+7ylVSo9SSPJb+R7tencPtkjLKwYYBiXyKgruBrGtTv1gt3sleTAQUmKT7rsosnEQxtHdbQ8tGAjpIJjG87R+PhbZ9rLQcpaXgZAKAmDpbgCYfBhcNc/wZCCkgbx9N+X52KSJnptBGAgpYtbbMLS75uQoGV4xENJBAAEmG1Wdfz1Gvp6FFRkIaWHOcZUP3fOpCI8JD9KJPmXG+qn0/hl9bCctt4MwEFJBAHONu2VZXSWp4h20YCCkgkAmG12b7vmmtPVKyf6DgZAOAkB2WZjV3XMbVLwnnegLSbMNgigMbK/nGWwMhDQQQBKgPLe+P5aGq7sTfcK8buM0GSTuUiBq7pdiIKRDWYRhEDcGgDsMKf6Qnx67PXzCQOifk8UxrEd7Ny7hrfzIL/GGzJuECPDujYHQTREc/GyXHKrw1LfZ3aBNZ+tDPn2BAAyEbo1MhS2BycZ1mFWXg8GM5JNCGAjdoN7aEnCru6G/8uZMA1jmCwEg6bCfFoAwELo97wLJ/bD1MCtJLmMpgJm6ovBPOQBhIHRbBL0tSiAv7DDrzYSCxfPMTgAGG3enImtTHqTTDept3Gzk4PvTrJcITdlmge8AizZq88Xg1yseg9ANKou7cXUcazvgLZknk/RYF0VXAt65ywFc4uf4gnw/kEcMhOYk+AXucxsFYRiE/unp4HgGEMwgvVzLk98DmzQ3gNfWDSA//ufKnw9EzfIU9BfIrxUiMMmhedHZOg6DMLZnF4I5bBJjXgcieEOSc1blgs+Ypbf4Bm+Z/NGpJvfu33m2SBb0n/DevniJ/FJYJvFSZ32sbBT4DgCZo1dBMvo9BACSa1YP8tkPFiR56XxLnj94fP/PBnLIHfq/pObXR2VmmR7Wz1fufPNMlm8CgQCLU1avDETwqY3nfpO3ePCn9yDuwnPpf/H2f52Y35o1JYAxmIfgfSCAO2bF3shn4YmG2bw8CPnPyPsPP/kNf2JTeR+IwHsRV1+/Sij/7jTvvdeBMA/6qazmrmTR2gFIrlG8y8updL/ya3mal/5XgkVnB4hTBHHl27rYcy4W3R7BO0ZE5rxQCCDZnQ7A4WXVtlXldz3nYtEtEgBId6dVKu++EgjmIF6aCJI0zx3HKQ9LTjWhWzVVUVg0+106e3nvo+MdhXTDcj+ot8W1jVpvtjNa8nETvGGKbpeYS1Y4Q3dpsuC5zDfC+uQzYSB0q8yxfoqpOi+byDoy7xiLexC6eeZlfDRldcKyqndqNhsGQloMddYM3Q5pUTfQggvHkQYCQfIyyux27MfQP3DpUaJP+wC8UxYEWRbEKyV1MBBSQd6/Jv2psrW/32jJg4GQHvlqSOA6ZQromQPOQEgDAdD7dXW8rhw30fP0AwZCaqTV3btBFNvqdFmnWgphIKSF10a2bYsoCMOoKqEDAyElzM5W5dLLL3W4jYPCgQoMhP49AQRmjK8GgGnsafLDU8IhFtGHRMw5q1wAcIsWU+0fVGw6DIRUEKC04egsNssmKpKD9UvuQYg+LmQVB9v23EXBOblmXapi02EgpIIA2OyLIIyCYFzuwnAH0bDtMBBSJG/O47HxcK3bVMejoBkIqbE59PnmXSiDCx0YCGlhBj+249N1PzlLaMFASAlx/CAKgzDK4jaHFgyEVBDgEhfrMbY2uNsutUzFYiCkxdtFGxr/Urbxcy15MBBSw7yIz5vGf4q8anMGQvSZJt72++KIpNvuRMuj+xgI6SBwu7BaF9aZtnGjZozFQEgJweT70zGK68DPeZBO9MVjPPMySdsoLAajZrNhIKSBQAABlma5elrqmGXCQEgPAWSxP3bjpV9qGV4xENJCAEmuWZBFQTbmBjqm8jIQ0kKAdRxUu0uVhb6WFRsYCOlhXmRVDpOu/bD1INBxJYSBkBLmaq8GgJTbeg0tGAhpIAB6v+gFgkVVN2o2GwZCOghKe3f7tEzd1bZzeaGQ6LNHbfZFEIR15WfRdZloWeCdgZACIgDc/ulYbOPgblAX7cXRcbWQgZAK8q6TdHr6YqzqKIxeeio2HQZC/568f9mUfW6AdH/piqsHDRgIKfCKvTttShuKwjh+OEAgi1CQNIskNMmrAEEFISxWlPL9P1NZ7LRq7YK9eqjPz8EZ3zgZ4M/NTXIDExOxv+i52cwPIjK9khaRBAgE3t6uDzL6tuOE13lvzkRCzhMiEJCCu647nCd3C7st5zwhAgERmCjK3QXFydLPm5ZPUiAQEID3dzW5o5k1JaOTtpgIh3kBHt20IY6zazYsdyjlancEAkIwGVkzzKylMXTCmZi3DQIBAXj3q5hsrzUJHXsh4xAvAgEp9oUEC6vnOulCyq3dEQjIwPtpuqaX4o/d2JRzGgSBgBjROMvmo1lgeDIOYCEQkGN/ubvddtxOlo8ixnqQ3xCzYgZeiblod65X2+vd7SwQ8+pLDoRN04w2P/9eZO4hQymYyMzdqRkFg26eLAJiIS+M6EBOb9U6l/EawBbT2MoC2mDP8EgKyYGYt2frekWBen3zbxvr6imBJNduNitFUo5fHUEgHzaBKNNYV06xiyWJuWw202w5KpY8Oa+J7EAqZ9VytaZEoXp2gUAEYRqHTtq2nbYbzg0xr4zwQOq1sirVxqdTKa8CbJh5Oxu1VknoNJMJSSE5EN4FUlCxebVCeTuCsJgPKiDzOpyaRHqxuxpiPcifjyCFmoI8CrtAMEkXpZiEA5OYyDRZzAcXAoG3x9sHB52TdD4xPFNKHAiEQAomGoUnJ3aa5POBjFtiIRAEIgVvH35rmoVu23bc3MeZdAQCT5n+bN7vpBmOYiEQeHpZajQZzIIoimcaJukIBB4rLUO311kEki42QSAghZbbdhg6tjuMsKIQgcBDTC3XWcSzoXXSbpEYCAQkYCJz5a4iYvL7zcSXMXwgEJDh/s6K1yZtlKy0hcO8CAQeJmLO3XBCxKRn6Ry3HkUg8EhgNTsjPYrGbm+Ew7wIBB5/jecgPGlb/cxt5lhyi0DgaSKzzHXatpMFJAYCAQHYGLVi3yOO7/J8KOdb0hEIyBBbdpqsxr5hCPluQgSCQCTxc6fZtJ2e1V98DATd9QeBgABMpj+af0nck6ZtO2m4KInZx0IgIIZ21+71k9BpWhMheSAQkGScTqNoMlwNfTHTdAQCcgzdqUlkRpGINhAIApGC94+5m+syBg4EgkAE4fs7/bTclaADWAjkFN9bIgT7d1fDuDRMl0wbLOd5RiBiC5Hw9ng1xc6J3UtCO5v5GkmCQCS+gaUshng92jxsO3az6fSsq+7Aj5iIRTwFCOQgbJ6r9VnGaohXw54fj5f9pOdsTxX2fTGfEAjkEEznN58uFbq4/UzvDVNUCmbdRdZzkomYfUwEcpjzT+t140yRxnp9+f4C+UaLWwNNzG4mAjkAE51frhsVZc7WN58lvDveEGMOcryB0DaQRqVcUKRcadx8lvEB+p/gg4+snF4gkMMCOasU1Dxx5XJhE8j73cVSgB/+hUCOOpACAlGA739jBEEg8KtEmBHIsQZS2z4QiAImEWmD1qC0+wOBHGkgGEEU8qedsJNPmIkYgSAQeKh05VpTy8kCYkzSEQg81k2tgTeweiNM0o87EMxBlCj1wwF7UdANiOkH/PtAzpQHwgceceBNIPuNUxJI4wWB8DaQMkaQo7lEP86Sj93+akZP8FsG0qi+ZATh/QiyI2sE4e2ZdMWByH2zHeGV+ly8CjuWlYYfoyfbZ+rGs3Rd6aUmhXr1VtONv6ftNk673QSidATRdU0ztAO2zwtUBlLeBOJ7mnF8ds+mJ7CQoN/udIOlGw6ejB56EBefEwQfPtUVjiCbQCZB8UDBRFkg2x39Wv3iQzApHmQSDC7rVWWB1Cr1y3izbUcpLvry1rLwxEq7Jukr986kRyKtZDxL/3DRUBuIoRmHKenGJhAFU/TC/n9W65sR5NCN07cjiMpAbgLdOE4lw5MXCBWtTkBEo97D26YwSZikH4aJ1R/FYj5wZolJ+jOE3XPhm1KWjol4ni69v9owVn+Yl180Sa++t8O894G8+/Ug/9owtUZ+N3x6HoT5l6UrD8R86XkQBbOQcm03SX/hehCMIMfEv3I7WegudJwoPPIRpIZAVGD/zuokc4OxHgSBwM9mRV48LkZMhMvdEQg8BwumEAj8BCMQBAK/yoOxJh2BwLOYeN8JRhAEAj/B2MVCIPAdbhyHQOAr+3RAAwAAgDCof2sDuD0BdOAJIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiCCEAQRhCCIIARBBCEIIghBEEEIgghCEEQQgiBj12570obCMI7fXk4zhahkTSNscGLRAkVeDCwzsM1kCfr9v9Eog8VGe3jqGXe76+cTxqi39fxtj8BAyIKBMBCyYCAMhCwYCAMhCwbCQMiCgTAQsmAgDIQsGAgDIQsGwkDIgoEwELJgIAyEHEsH8vl/DORCbSAXRwzEESTPYCDFDuSIgbiDxbLaJZDlKmQgDKSksHoDYSCF3oMwEDfQGHlBfXELJbrE+shAKJfdx/G9aVUnp2XbgzgIJJFPIH8xEOUgzcGX9q/W9fBml0BWhTAQBlJWw147OB32TFCyQC5OThgI7e38vj+av37oRfIasDaQbq3rzmW4TyByFV52nQlrSSDY8ZL2x9da2HUmrP2sSzHh9U09jtumKVIZtiZID5s8Z86KRSChO7XwSvYKxOlw3U+WY2OF+RnE6WjzM4iuBbYpKJ06aJuxCCLzvS6CVMPnceT5WeLHzqzjzmz27Mf+9pYDP7sdrvMY+zuKhy8zh7PNZi9R7BeS50XBjagzD+RYpBK1pvU3Hxq0TDXbnVumugdz51Z1D3dpqmY7KGOe6vrOIkG7OoYgMk/pepdnkGzxrVuxt04xh/NvHfOKKoqCimiD5sCMBZVJsgdJgdjA+V4KuX66i/Gg8Vobyva4G4LG/XkCZw99D9IY9P3UbEietuxDxwL8AxB3sO6b5Hbg1H39t/T8TXQDUc/Ejel1uynqQOGhhuLfpeZ1VtTJIR++XZtBr+9XNI5HdGhoTKtmENfZB9G7zoJRk3UQZe6G1d6NSXRYkBWo3CMRHRaK+79zomylf5AYkQYQ8AKLqBiQvGSc2Er8cAxomgZluH9PEeS+yUfORx3pd6Dr4lLDDOlpsM1YUPlj6IGcdzMQINfli3euKKF1GR98lWHTA58+ngpPyirl0se/WClYLgQN6/c3e1e2I6sNRC2DWqKNXyzcNpYNAp5YH9h6lVrd9///KWXT3CydKIuYZJJJ6WoGaA8cV9VxLebOfFa3+hTK+S/Ij2vyv8Jx1k1R/BdJtcp/+uXbPxNC/t8p+D39xPlUVdF5h/B2HkPIRvCCPPfJr+URn4m1mTmb7PNQBZshqqLJ/FEbxF6+x5+joPt0glGWnxIthK7HnGzmL53JtglL+NyGffwLByd+l/95f8TEdDPZWn8OgjqmT4U+i+yiogSLyrD3/9iKYdq0/0T8/lwSVzVjXGvOaDKRbQJIHsKttlE4qSQbg188Y3eqj8GfpYf1g+RC0EdIntDU+yz+Fdw1ZUIKRvmo/pBiziUb4/830n492Yg0FW11zqOC03p4+/yta4t/v81Ebkx3W6GMNDvGv3iASmgTv6F4K91fR3g96DS/IJDtteilNJzfG6xvF/6OLR1SSSqfQz5ca8qP8buh3ntWpgkj8j4Q1PZZWun/nHgJ1Q8XidWV09BDTt79bHW9n/EF/wZJHkyajfSKJ8lP8S88ap/Qb9m74d8YTX5eiJ4Fj9CHiAlZaN6cbz3Cv3G8/ZaRC68hrQersayradmR317nVpviTKn4fYg7+Lot4GXC2Ymxa7ycq5CWkz3cT5f7Zdi5w7NBfnS7TMrlquc98vpb3wXLLeLucr3BwEXU4Xbv8wyRvOH6lu8Q3o4gUKqTeLjdIkMQUVXJioOtivFc3e+VwXYmnheo6jIohPyhf9yqPF6e79vZnDOUmasQY6fIOvftCbIKxia63/t5ec6KJ4DTAFCavs/3npdl3Q2ubg7H1HRc7kkeQlbEmTZ6XHuP2Gsq9/dTfxgGQ9xzd+cuwMbbWx4Qr3o8IoWc7Ifb9dLFy0pjosejP3+9/5OHvZTW1o7OxabmmwFlnBspBC+PVqFD2FSN5lw2OQaFp6c+FVzUV+VMcSphoBw9q+rsUEghRFplQSsYF8ngFqeNCOK1YfUsOdfFRLKLZELIp0JBFAqAEx7AjMExPDWCJRPyAL8WOnGBMesaC6t+BPFTMDi4LFb+MIK4tuChAFiAK4BT0Gfp8JxAaXMTfis4L4ZneDuVHLBH8aZwXP6ZemSB9bhaAsZDASYE0+zg+BY2o+SyrhvjxldJqOIxjQhwpU+1ABUPGRAph1HiZel4CoVt45w+4a89+GA5SDbu1xhK4owg3CWU16FkzOZbkWAaNKMpKxQiIwOdyVJQfgWzmoKxMqxh4AwKPZRUJKmmulIFBykP7p5bEORbDKAAh6hLRpM8u2kGdhx9coHHpamgdp3chYyD1AfgPKvDRFBhQ2OX2DPJ+FVZgnD5WCv+j4sgWS8pd7qwbBycPlOrtMBW8xZmOhWARZQwn/Kw8Ws58chY2pt9TMAqmf0SlQAnlFTcIRSPnAIArRmP7FODkNZ7v2anDMd30FmacFpOBOWpNW7JWKsQPtRMpHAD1nyeVt3fJJXgt583+PyQitOs8oaxxscdaKXtzBCC42N05VSfPNNLcFMEhuCjp+YjZ6OP5oTKi1EHm/aaket7vtuyBslTStODsahOZG9TrEGRg6TppFRV0+SM4pbS+tZPMMTiN31JE4X8lvLnrDqY1MW/CXF821TZvgZBAyihN2pKwfudPq+zQ5nOaIZ5hH0/GUCbVMZ7ctpsG0KQ0xWv21NlYnd+dqaxOpAVik+cymsV9Qlr9oC2k+yJTcmuBB1AY5EyN0FD47dMHD3fazgsT6Sg9aDUkFI9oS8k1i4XLqr1zAmJxNJW3TdUD+gsaaGsze06jh6M21U5u3J5QLl0A3F2ZTCwZ/xGMCK9lBOquPQQ2pAgLhXsAKNJaBFDkQ40QUFL69wCz0s6kgzw2jzBtOXdUp48aWmQ7YHtXB9WPNCsRbS2HT6KIHDnuKFl5zy1pi32C7k0GR6szNEc0sTOQxUwBiNb94Vbr8qkazWljOukGTKEyZPpyXW5UxrG5MpdZCUnCwfFT/vNK/kNxe0COz7p0AyCOS3uYC6KpLS1J3nb5uiLyUqQ7x6zG1l9dkedpjf4wtznKrVsAKU6N5jA03DPxc2YefYjzno0ssTYu+wOvcIXLs+bEgQcHJ6PkbViukcqYcfMXmshgs0KLJ/GpKWhpULQDTEiu3loWa3QNyo9CysbLh5MBib7JpunWCqhobG45oImKjtPO4una1id4xnmEQMgBT4XW7QjS+fNayIVHcMaSEKh4rIhLLVqMgYYoIAu9exsKNjFYW3jhSCqpk3miphq2J2onvzZM+rKRIcaqscKGjQ7/9/6lxn+8isha4r1/Yoq2GtJmxN6tD412evGEeQbS9xnByBI8OQ8SUHCmtIbaVjro5eQG5fDulJvRRAbuzBEDZb6yNSWIJGk0gFIBeTRWbG4HMJBdzu2qeQs9NFIAfEqByF6tMr2BFnneq6pBFAgmsrO1siPFx4PeQm7vhacI8zHrkcJMHhz0xLV9WPCaDlgk1K9wtFnIEjiLwhosQOa8B4tBPEku5KXG+wbW7VY3ZYUeDQklIqk7fPsy/V7J+lSkGWTLe6Gnd8CQVajj2jQerI68dzK942lyg6NhD4ER860FiBa1lPc2moO4ZUgm20Uku8Eucc2wEGc2C8EwZWkQgsrUra7rGCOIEQdNWNcSM1T3wa2FRb+CUE+IsWa15coh5q9cGlZnBHgoYBHcxiyEATjhSDOE1m6MUH2niFOc9k8ctbEJv0OR4d5dmSpI0j24OXZnnkLQfAs+Z28dKNaxtefSfoYDSHolLL64X+tF1Kw3QJOzTrhrtb9rvlOEIggL4KgF0GePyEIZK/yOlV91U+eiRW4p+/uE/tZBhGk20iP+EeCZC+CrBHkIFkbRVXfR7lRKCiYK3dNy1lZjJV34rWyEWSxabaP0bQSZIH2USnWuWRFdKiqPjrPoBmH51jlR17nVo93vEYQwOEIsime+FIX87qhWkBCuk9Z6OBUZzBUvBLE1pAXBTYmCLsIMpcvgpAg9ltWXyb7M8M8qwxjf7gWJaPiQr7Ym/HxN8qd0ywFGbuBApP8pUB6WwmCHUHwTwgSAQtcTYcx8bpg34KHuLbhKa3iC5c5nGD4tzFBAhtB8FyzZ4a6kp+W9FB1MwpeEaTitMlVgNGNSYOOdKmaslv6wGsNgje2MLYEKdS6N21cMr9usw6atrkCZD2T55+nWPgjIkhwpPyCX8AamvhxSIt4MfA5z1aCADcLVjyEPsDY3NUgJT062KZt8m+0Xgif5ec4ng4xilX+kLTw0dcR5yddSetDhi1XbsI2IivOnNtBhBAHRxD8nmJpXdn6vQlc36suI3DHxfkmTU9Zv10XC72nWC6CnIitPuvctWcK3WQxRBA74kp553w2peWMes4fdjZzSQuUaxF9zOo3A0H89dZB8epiqVaOJOJ8WoIMLfNf1iB4e4KQSNOkI865I0nbADVUOgR+I4u9S7EcUlJpIYHX+FWDBCFNPJcgMz1EfOliZfcyNX4hH8Ty3bbcvlL4QC9alPezMeen3eRzzUD9MGo+CVoom5V8T7ECdFwyelLZXGVXUH703IYDkMgDokXKNds70gt+ygMg1jYp1nMt0pF9aOq7zeJJwWPsN9UVjDYkDmlr1+QLY99M4A8ho7Vn3VL2RuWtXVZzzdqzTyyq7QlSPw4VyKUy5AAuejDq3DKbpHI2mp0PigE647xeCQL6XGuQbcNZ0DCa3Lt57k6SlgeCBwmmAR3A9SLORpaq5YEKIPGeIBdBgAE9WLxTqiqBBvCZvs/K3AVNFFBHVgYsLen49d428UdBmQxDyai+BghDQKAsLRJuNx5QxN2Gq2vpB2iktYET0nN+QahLKUuKUFB9gwgM2hNhIZm4ZmgqKasjgrd7mxfy+lO8LGKJbxMtqkd/d+RUhgU8rDBol7pMAucpwB8LyQW18WKqqUgLmNLo26oKFoMAbSzYvc1LuRMKdM5OnOoFl2ebSDxtAA+3MTYv6enVCnbh1/+BvDPoURUGAnAzGhOo72IKlQ1IgJO6cgARVBIi/v//9GaGbeKm8k74dhMnGhllysdMR6mlbTb05UyJc8qwPR3GMUWUfnaDTiMO+WCZUieN+dsCGvwOOdF2uqarhgVm0DZbhnxHnk+971mMeygphtjGerXJxdsJuAWeOYrOlEO6zJceq7mkDsJtOVwg6D4QV80NePDXawVCpgc23PCApkBtSNm2Dgi3Cf94TTJF4KHch9cAa2B44X6DTy9zBVR4rDgV825LxwzrSIBz0EfaA/FXK28VHosYDc3ZhP2HgKDFqB/mYnqJDtrTnofPldc7wr3smesYCZGUCOB5zNOJKNbtcL+CHhL+qLNocppm73FEYxWQ7hQbcon+nFGjUy/nIFjyja4XnFKbkFrfH80arbzYTwZsUpalRKMdFYBKLt+qhT4IJJHq+r5T0dfZy0g19VV9AG67VekKlCBXJwmRf044vFU15y4lv6nvxSxhu2RW9P0tD0iZV7d2ovGJbukjmJP7fGEgZ4rKDdKi9V18PbdIzohJqmaS8U9FX1+rBaQqDSj6RVN3+QKo1yG/XBBwWgEByKO+ZKeihGDae30rORdlRDy+K9NdKp1ccT4kZ/Qn+X6mzoGYWpxU3e73W0Uogn1zudddNWfvqHNiGks5upbwnRK9y2909f1CYIgW7Zq64UrBBWCV2EVvlxxGQMpvw31QtZdyA2tSGTYcPjFmZg9TFsn0K0LBAypYBgOUQWasly2iZ1zyHc+40GwbHmnQXy58fMNlEEaC8hjVR1dJNmIBVN41OVjsITJGHd/PSpbXzZEAFtA4FFihnxjJRhun4zr6SGal6iugnh0N/hlCYNBfPE3LzwqMKU/TAEYqLFgvABPScYFWDRszsOH48Z/Wz4XnGvzYZEHwRAcb+9dP8fqXvXPHARiEYWj17n/oTl5qxV2aFCSyAF4w+fTDEP9v2HPNkBL3OR8QsqLQKr/CeBClMdC8H188GM8/CipijqMIWLP38EqGexcrJHqFZNEYIBJ3JlWcbWI+HMw5XubuIQfZQhi42TztsChnrYUmnatcCwaVOGNCXFQUKXKzhU+uCC401r473UnDkckfEky7D/36kDdkkTa0+BKdg7HtiTcUEOLPbtLyx2724EAAAAAAAMj/tRFUVVVVVVVVpb0763HVhgIAfOQgJLDzgnBsLNvI+In1gSWQRYqS+///UyGdtFdNl9uqnXsrnW+SsBjseThHxjATI4QQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEELfCYGX/65igvP0of8f8l8lCllfbzPU/r3Z3wjOyYR+COQ/mv78ifxF/Vv527mYHeiH8DaN/7+YTOR93msgf9GF4FT86EdDXu9/fdDyPtMvgTfkDzKWYJKgHwCN1IeIwjcgmQrJN+YHpc9FFGfws0zFb42s+9xXueDCkMITzeIMUwR9RwSAntq6tnVdl7Z33xL38WJ7+CbZ0EWwSdn1uULCJa/eEqSyc0DgF4NfAnhSD18AQt8PAXCPZK+l1lrInsI3UK24w7dwlV2yZyPXxB5hkwpewW9dRKvgV3Pig59/tUGKARD6Xsj2Clni07RaHdTHLqAUVhRe6NcjeXfqzfNkQr4+ijh4+SgJGL/D0yTKDlah3/uYvK68Xo79yW0VOdhEXvSEwKYX9giEbM1/oPgkBX0iAgC7Unavzd2gKFAzHWl8NMO1UrBFs6lu58JtEW+Cop+UCqgzW3karGX3IQQAV/SXSlEAp4wqzuedg6iXfA5gs7O6+jni5Q6IOtzvw1pAYqWOVXpUygEEw/V2UITALi/76TxEAPBFtAHQY3/rC7olqKlu11MGgPe40Gcgz49U568xN11kE0KRywqudZ5LznYAZMq1FHXlgPTMW+EXdo9Ms5Zr2fReCnmJIbqWWmpWAJg5b5kUdoChTpI6ha1u5fllXQQ2mR3tmJZSt0egB8+YZpe2VVC02052otBZYe1aaQZZK+4OUqvX6g4UyMlLKcpLjMmBPscWu+7Gy2pK07Q6BFAJfYQmsSZjezHe5L6NYCq57xtdd5A1+8S2d89neqr3oplrrvPF87ogZ63HnnFmoLB72dxY4gOTJ/VyhE008saBuya1IYPl+flaJ21GbiIp/XWWuTKM1/c+T1gAqUzyeyPkAVSuu3WTt30rbEECr9u+EeKOd7bQ54kePJGllFrYAqJmDXtdn8hJJmMAqZDGeT6GQBs+Zirf2yk8MpFCJfhDrQkjKziV5amoxZmCyUVKBp00BgZRFzCLL9nHSKcXTIGpkwvQKm8UwDmpQ9rsy964UTTRifkdwEFoA0tSHkjU8nVnWas4548MMs8XepR8AjeXjwi7EPRZSMC4zFe2bg1AwXhST0DvIg8AYiZ3hRTXburaxAed1ANAUcsC5sTHoBg/E5hKZi77suqmlCU9nBNrAA6aqbAV6etSrtNlQW5JbgCCQzUdqlb7LLT8QsF5cabZUE1TNWobUM8XB9BrryrNoknovhsGn7Rh7Hl+PqqhyHAEgj5NIXUfxkGgVEgJuHnPK4Cs0bMDUKwsKi0Yy1mtl/CsWQxQSWucFz3AVJZHAncxR2Mit4PKcoBxi+9sFg9yqmX3iuRjKU/HkvcAdPfILcs5X+Ao6x0QY2UH5pLbPBdJkwW2LgDgrFu16DleuH7WLL9E0OU8sWmGI3T0WQhAKnIDL9sIoj49u44eACZpdxfJLstyeSxH1+hH9nxqEZutF4GrZgqyRlRZq8fL43J7nAPF9AREeVFBJfPjqxnFyurGvQI4WWGbqpeiopX22dZIHgZM1E2f1vziBskygC3Polb3QaPZbW1+vhRh6MxSJvLsAKHPQNaf7Aufn6tPO5ZoPWdgSlEB0CZp1ajHMIxO8yXKmKwouEZcXKqtAjKKMQTD9Mkx2WVxNjUHMpTlbssCOZCLmKNXxfFc2lJOANmSsCKAoiyLbBQPAu4mPJ247RRVVk/0ylkEa7noslyeYq+XKAq78RwN7AyuaBMbYAeCPgcBUD5pT90wDIfJgGHc91KksJN7prIrF527CLuDnZejM2tQA5hcDOTBfUQCJs6UTFIeiU+amBysPpCryBVApesdtMkYvNpxvdzzMQKIxqSh0LGERSrXZyBRK2ZScZatjSS1oeNe9JlhiY8KWSs389zAicmFpnzrm5akDgDwcSH6JEe75+VG171qtgF2u2dRJxPpW8G/ONgxbhvL7eQqkccAk5aGMj47OJW6A7hzayCVCWt0wgw0SRsCeSQ+WENd3wL40Mn9NuAAMpWiXRjfe7qTcgJQNT+TneV+8WJv48gLbduc25NbEpZBkfO8qXk+bKks51nqi8PnhOjTmKaWG61tVbA6Bejy1txKP9a6vkQAcPJay7YgrsovDujEmsiN+USh860C6PMlBFdZrctZgfuSV47Quz1nMOW6MfBBteXNwSq61tq2I1uo8Vux8r54nl/7kY1hOI5XLzUbILvbGyVkYFrKZkcAdq0sZX0LAaFPQsCp4654MmFQGAfgVHBk8hydDjsHK6qGtIvXZWxCABKbgFBlMgKRUYTAupcCUHNIdxlsJREADbZPZ7pj9tEM/HzKJivSQYUqoG59r5tGbY0e08lESjmqYmcOkwKyVhJ+NH+KYRMX6doIdh7os5HnB6Gv1ZOWw9sxv7dCfqean1+v9d8uCJD3Kl+exW9Nka+K8eIKfReEfL2R8vz4R/8c/tz89f2bgH5tvucA+dNjCAHy28PeE2Zbwz9VRJ+EvCLwTdf00Vsg/l7Ev0fzXyF/Fe7kT6si2Img7464DB/GIfQOr2AQQgj9A/gVoQj9GbydihBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCH0g/gJX/c7loGQqPcAAAAASUVORK5CYII=)"
      ],
      "metadata": {
        "id": "H1teuBgNDO84"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Compare CTR vs. CTR 2\n",
        "\n",
        "The differences are less significant in this case, and when compared to the other version:\n",
        "* While the difference between connect and interact were significant in the first verion, it is not in the second.\n",
        "* While the difference between help and learn were significant in the first verison, it is not in the second.\n",
        "* While the difference between interact and services were significant in the first verison, it is not in the second.\n",
        "\n",
        "However, when taking both Homepage return and Drop-off rates into consideration, the version with \"Services\" is a clear winner in both cases."
      ],
      "metadata": {
        "id": "UisuNdQiDV-q"
      }
    }
  ]
}