13_Case-Study-AB-Tests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0009cf1b",
   "metadata": {},
   "source": [
    "# A/B Testing \n",
    "\n",
    "A/B tests test changes on a web page by running an experiment where a **control group** sees the old version while the **experiment group** sees the new version. A metric is then chosen to measure the level of engagement from users in each group. These results are then used to judge whether one version is more effective than the other. A/B testing is very much like hypothesis testing with the following hypotheses:\n",
    "\n",
    "- **Null Hypothesis:** The new version is no better, or even worse, than the old version\n",
    "- **Alternative Hypothesis:** The new version is better than the old version\n",
    "\n",
    "If we fail to reject the null hypothesis, the results would suggest keeping the old version. If we reject the null hypothesis, the results would suggest launching the change. These tests can be used for a wide variety of changes, from large feature additions to small adjustments in color, to see what change maximizes your metric the most.\n",
    "\n",
    "A/B testing also has its drawbacks. It can help you compare two options, but it can't tell you about an option you haven’t considered. It can also produce bias results when tested on existing users due to factors like change aversion and novelty effect.\n",
    "\n",
    "- **Change Aversion** Existing users may give an unfair advantage to the old version simply because they are unhappy with the change, even if it’s ultimately for the better.\n",
    "- **Novelty Effect:** Existing users may give an unfair advantage to the new version because they’re excited or drawn to the change, even if it isn’t any better in the long run. You'll learn more about factors like these later.\n",
    "\n",
    "## Case Study:\n",
    "In this case study, you’ll analyze the A/B test results for Audacity. Here's the customer funnel for typical new users on their site:\n",
    "\n",
    "View home page > Explore courses > View course overview page > Enroll in course > Complete course\n",
    "\n",
    "### Experiment I:\n",
    "The first change Audacity wants to try is on their homepage. They hope that this new, more engaging design will increase the number of users who explore their courses and move on to the second stage of the funnel.\n",
    "\n",
    "The metric we will use is the click-through rate for the Explore Courses button on the home page. Click-through rate (CTR) is often defined as the number of clicks divided by the number of views. Since Audacity uses cookies, we can identify unique users and make sure we don't count the same one multiple times. For this experiment, we'll define our click-through rate as:\n",
    "\n",
    "CTR: # clicks by unique users / # views by unique users\n",
    "\n",
    "Now that we have our metric, let's set up our null and alternative hypotheses:\n",
    "\n",
    "$$H_{0}: CTR_{new} \\leqslant CTR _{old} $$\n",
    "\n",
    "$$H_{1}: CTR_{new} > CTR _{old} $$\n",
    "\n",
    "Our alternative hypothesis is what we want to prove to be true. In this case, the new homepage design has a higher click-through rate than the old homepage design. And the null hypothesis is what we assume to be true before analyzing data, which is that the new homepage design has a click-through rate that is less than or equal to that of the old homepage design. As you’ve seen before, we can rearrange our hypotheses to look like this:\n",
    "\n",
    "$$H_{0}: CTR_{new} - CTR _{old} \\leqslant 0 $$\n",
    "\n",
    "$$H_{1}: CTR_{new} - CTR _{old} > 0 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4a933f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b2e4a2a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>action</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-09-24 17:42:27.839496</td>\n",
       "      <td>804196</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-09-24 19:19:03.542569</td>\n",
       "      <td>434745</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-09-24 19:36:00.944135</td>\n",
       "      <td>507599</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-09-24 19:59:02.646620</td>\n",
       "      <td>671993</td>\n",
       "      <td>control</td>\n",
       "      <td>view</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-09-24 20:26:14.466886</td>\n",
       "      <td>536734</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group action\n",
       "0  2016-09-24 17:42:27.839496  804196  experiment   view\n",
       "1  2016-09-24 19:19:03.542569  434745  experiment   view\n",
       "2  2016-09-24 19:36:00.944135  507599  experiment   view\n",
       "3  2016-09-24 19:59:02.646620  671993     control   view\n",
       "4  2016-09-24 20:26:14.466886  536734  experiment   view"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('support-files/13_Case-Study-AB-Tests/homepage_actions.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db965756",
   "metadata": {},
   "source": [
    "#### 1. Match the following characteristics of this dataset: \n",
    "- total number of actions\n",
    "- number of unique users\n",
    "- sizes of the control and experiment groups (i.e., the number of unique users in each group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6026bc36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8188 entries, 0 to 8187\n",
      "Data columns (total 4 columns):\n",
      " #   Column     Non-Null Count  Dtype         \n",
      "---  ------     --------------  -----         \n",
      " 0   timestamp  8188 non-null   datetime64[ns]\n",
      " 1   id         8188 non-null   int64         \n",
      " 2   group      8188 non-null   object        \n",
      " 3   action     8188 non-null   object        \n",
      "dtypes: datetime64[ns](1), int64(1), object(2)\n",
      "memory usage: 256.0+ KB\n"
     ]
    }
   ],
   "source": [
    "# convert the timestamp column to datetime\n",
    "df['timestamp'] = pd.to_datetime(df['timestamp'])\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee2e1aef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8188"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total number of actions\n",
    "df['action'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d77ff04e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6328"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique users\n",
    "df['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "047c9483",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6328, 4)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df containing only the unique users\n",
    "df_unique = df.drop_duplicates(subset=['id'])\n",
    "df_unique.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "212953fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3332"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# size of control group\n",
    "df_unique.query('group == \"control\"').count()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "01556a59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3332"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# another possibility for the size of control group\n",
    "df.query('group == \"control\"')['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "34df9e03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2996"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# size of experiment group\n",
    "df_unique.query('group == \"experiment\"').count()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "37387dac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2996"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# another possibility for the size of control group\n",
    "df.query('group == \"experiment\"')['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "38d13e50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking to see if everyone is accounted for\n",
    "df_unique.query('group == \"control\"').count()[0] + df_unique.query('group == \"experiment\"').count()[0] == df_unique.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b596af73",
   "metadata": {},
   "source": [
    "#### 2. How long was the experiment run for? \n",
    "> Hint: the records in this dataset are ordered by timestamp in increasing order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0610cb6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas._libs.tslibs.timedeltas.Timedelta"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duration = df['timestamp'].iloc[-1] - df['timestamp'].iloc[0]\n",
    "type(duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048cf4f1",
   "metadata": {},
   "source": [
    "[Documentation](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.isoformat.html) on dealing with pandas Timedeltas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "91b2d268",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "115"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# duration of this experiment\n",
    "duration.days # about four months"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b15a182",
   "metadata": {},
   "source": [
    "#### 3. What action types are recorded in this dataset?\n",
    "(i.e., What are the unique values in the action column?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "23d80917",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['view', 'click'], dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# action types in this experiment\n",
    "df['action'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "673f0739",
   "metadata": {},
   "source": [
    "#### 4. Why would we use click through rate instead of number of clicks to compare the performances of control and experiment pages?\n",
    "\n",
    "- The control and experiment pages could have different numbers of total visitors\n",
    "- More total clicks could occur in one version, even if there is a greater percentage of clicks in the other version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35e3da4a",
   "metadata": {},
   "source": [
    "#### 5. Define the click through rate (CTR) for this experiment.\n",
    "\n",
    "CTR = # clicks by unique users / # unique users "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6620c045",
   "metadata": {},
   "source": [
    "#### 6. What are the null and alternative hypotheses?\n",
    "Use $CTR_{old}$ and $CTR_{new}$ in your hypotheses.\n",
    "\n",
    "$$H_0: CTR_{new} - CTR_{old} \\leqslant 0 $$\n",
    "\n",
    "$$H_1: CTR_{new} - CTR_{old} > 0 $$\n",
    "\n",
    "#### 7. Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "756d9178",
   "metadata": {},
   "outputs": [],
   "source": [
    "# retriving the control group from the original dataset\n",
    "control_df = df.query('group == \"control\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ec368c8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['view', 'click'], dtype=object)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking to see the types of actions before calculating the click through rate (CTR)\n",
    "control_df['action'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e2c3d343",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2797118847539016"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting the click through rate for the control group\n",
    "control_ctr = control_df.query('action == \"click\"')['id'].nunique() / control_df.query('action == \"view\"')['id'].nunique()\n",
    "control_ctr # about 28%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f1d7079c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# retriving the experiment group from the original dataset\n",
    "experiment_df = df.query('group == \"experiment\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5878fe56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3097463284379172"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting the click through rate for the experiment group\n",
    "experiment_ctr = experiment_df.query('action == \"click\"')['id'].nunique() / experiment_df.query('action == \"view\"')['id'].nunique()\n",
    "experiment_ctr # about 31%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7d585400",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.030034443684015644"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# difference between CTRs\n",
    "obs_diff = experiment_ctr - control_ctr\n",
    "obs_diff # about 3%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "081cc528",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now, we have to see if this difference is significant, and not just due to chance\n",
    "# bootstrap the sample to simulate the sampling distribution for the difference in proportions\n",
    "\n",
    "diffs = []\n",
    "\n",
    "for _ in range(int(1e4)):\n",
    "    bootsample = df.sample(df.shape[0], replace=True) # bootstrap the WHOLE sample (df.shape[0])\n",
    "    control_df = bootsample.query('group == \"control\"')\n",
    "    experiment_df = bootsample.query('group == \"experiment\"')\n",
    "    control_ctr = control_df.query('action == \"click\"')['id'].nunique() / control_df.query('action == \"view\"')['id'].nunique()\n",
    "    experiment_ctr = experiment_df.query('action == \"click\"')['id'].nunique() / experiment_df.query('action == \"view\"')['id'].nunique()\n",
    "    diffs.append(experiment_ctr - control_ctr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "787e2102",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the p-value for our statistic, which is the observed difference in proportions \n",
    "# by simulating the distribution under the null hypothesis and then finding the probability\n",
    "# that our statistic came from this distribution \n",
    "\n",
    "# simulating from the null creating a normal distribution centered at zero\n",
    "# with the same standard deviation as our sampling distribution above \n",
    "\n",
    "diffs = np.array(diffs)\n",
    "null_vals= np.random.normal(0, np.std(diffs), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "95eaa30c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAQFElEQVR4nO3df+xddX3H8edrrTI2JcJaWG2btTNdMiAZStORuD9Y2KQBs2Iyk5pMSGZSJZho5rIU/UP/aQI6f4xksFQllAwlTdTQCGwiMTEmCH7LkFIqo0qVr+1onX/I/mEpvvfHPV2u39723u/33u/9tt/P85Gc3PN9n88553M+uXlx+rnnXlJVSJLa8FtL3QFJ0vQY+pLUEENfkhpi6EtSQwx9SWrIyqXuwDCrVq2qDRs2LHU3JC0X+/f3Xq++emn7scj279//i6paPbd+zof+hg0bmJmZWepuSFoukt7rMs+VJD8dVHd6R5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGnLOfyNXOldt2Pnwkp37yB03Ltm5dX4z9HXeW8rwlc43Tu9IUkMMfUlqyNDQT7I+yXeSHEpyMMlHuvqnkvw8yTPdckPfPrcnOZzkhSTX99WvTnKg23ZXcurn7iRJ0zDKnP5J4GNV9XSSNwP7kzzWbft8Vf1jf+MklwPbgSuAtwLfTvJHVfU6cA+wA/g+8AiwFXh0MpciSRpm6J1+VR2rqqe79VeBQ8Das+yyDXiwql6rqpeAw8CWJGuAi6rqiaoq4H7gpnEvQJI0unnN6SfZALwdeLIrfTjJs0nuTXJxV1sLvNy322xXW9utz60POs+OJDNJZk6cODGfLkqSzmLk0E/yJuBrwEer6lf0pmreBlwFHAM+e6rpgN3rLPXTi1W7q2pzVW1evfq0/9uXJGmBRgr9JG+gF/gPVNXXAarqlap6vap+DXwR2NI1nwXW9+2+Djja1dcNqEuSpmSUp3cCfBk4VFWf66uv6Wv2HuC5bn0fsD3JBUk2ApuAp6rqGPBqkmu6Y94MPDSh65AkjWCUp3feCbwfOJDkma72ceB9Sa6iN0VzBPggQFUdTLIXeJ7ekz+3dU/uANwK3AdcSO+pHZ/ckaQpGhr6VfU9Bs/HP3KWfXYBuwbUZ4Ar59NBSdLk+I1cSWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaMjT0k6xP8p0kh5IcTPKRrn5JkseSvNi9Xty3z+1JDid5Icn1ffWrkxzott2VJItzWZKkQUa50z8JfKyq/hi4BrgtyeXATuDxqtoEPN79TbdtO3AFsBW4O8mK7lj3ADuATd2ydYLXIkkaYmjoV9Wxqnq6W38VOASsBbYBe7pme4CbuvVtwINV9VpVvQQcBrYkWQNcVFVPVFUB9/ftI0magnnN6SfZALwdeBK4rKqOQe8/DMClXbO1wMt9u812tbXd+tz6oPPsSDKTZObEiRPz6aIk6SxGDv0kbwK+Bny0qn51tqYDanWW+unFqt1VtbmqNq9evXrULkqShhgp9JO8gV7gP1BVX+/Kr3RTNnSvx7v6LLC+b/d1wNGuvm5AXZI0JaM8vRPgy8Chqvpc36Z9wC3d+i3AQ3317UkuSLKR3ge2T3VTQK8muaY75s19+0iSpmDlCG3eCbwfOJDkma72ceAOYG+SDwA/A94LUFUHk+wFnqf35M9tVfV6t9+twH3AhcCj3SJJmpKhoV9V32PwfDzAdWfYZxewa0B9BrhyPh2UJE2O38iVpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JasjKpe6ApPnbsPPhJTnvkTtuXJLzanK805ekhhj6ktQQQ1+SGuKcviZiqeaYJc2Pd/qS1BBDX5IaYuhLUkMMfUlqiKEvSQ0ZGvpJ7k1yPMlzfbVPJfl5kme65Ya+bbcnOZzkhSTX99WvTnKg23ZXkkz+ciRJZzPKnf59wNYB9c9X1VXd8ghAksuB7cAV3T53J1nRtb8H2AFs6pZBx5QkLaKhoV9V3wV+OeLxtgEPVtVrVfUScBjYkmQNcFFVPVFVBdwP3LTAPkuSFmicOf0PJ3m2m/65uKutBV7uazPb1dZ263PrAyXZkWQmycyJEyfG6KIkqd9CQ/8e4G3AVcAx4LNdfdA8fZ2lPlBV7a6qzVW1efXq1QvsoiRprgWFflW9UlWvV9WvgS8CW7pNs8D6vqbrgKNdfd2AuiRpihYU+t0c/SnvAU492bMP2J7kgiQb6X1g+1RVHQNeTXJN99TOzcBDY/RbkrQAQ39wLclXgWuBVUlmgU8C1ya5it4UzRHggwBVdTDJXuB54CRwW1W93h3qVnpPAl0IPNotkqQpGhr6VfW+AeUvn6X9LmDXgPoMcOW8eidJmii/kStJDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkOGhn6Se5McT/JcX+2SJI8lebF7vbhv2+1JDid5Icn1ffWrkxzott2VJJO/HEnS2Yxyp38fsHVObSfweFVtAh7v/ibJ5cB24Ipun7uTrOj2uQfYAWzqlrnHlCQtsqGhX1XfBX45p7wN2NOt7wFu6qs/WFWvVdVLwGFgS5I1wEVV9URVFXB/3z6SpClZ6Jz+ZVV1DKB7vbSrrwVe7ms329XWdutz6wMl2ZFkJsnMiRMnFthFSdJck/4gd9A8fZ2lPlBV7a6qzVW1efXq1RPrnCS1bqGh/0o3ZUP3eryrzwLr+9qtA4529XUD6pKkKVpo6O8DbunWbwEe6qtvT3JBko30PrB9qpsCejXJNd1TOzf37SNJmpKVwxok+SpwLbAqySzwSeAOYG+SDwA/A94LUFUHk+wFngdOArdV1evdoW6l9yTQhcCj3SJJmqKhoV9V7zvDpuvO0H4XsGtAfQa4cl69kyRNlN/IlaSGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNWTlUndAk7Vh58NL3QUtY0v1/jpyx41Lct7lyDt9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1JCxQj/JkSQHkjyTZKarXZLksSQvdq8X97W/PcnhJC8kuX7czkuS5mcSd/p/XlVXVdXm7u+dwONVtQl4vPubJJcD24ErgK3A3UlWTOD8kqQRLcb0zjZgT7e+B7ipr/5gVb1WVS8Bh4Eti3B+SdIZjBv6BXwryf4kO7raZVV1DKB7vbSrrwVe7tt3tqudJsmOJDNJZk6cODFmFyVJp4z7MwzvrKqjSS4FHkvyo7O0zYBaDWpYVbuB3QCbN28e2EaSNH9j3elX1dHu9TjwDXrTNa8kWQPQvR7vms8C6/t2XwccHef8kqT5WXDoJ/ndJG8+tQ68C3gO2Afc0jW7BXioW98HbE9yQZKNwCbgqYWeX5I0f+NM71wGfCPJqeN8par+LckPgL1JPgD8DHgvQFUdTLIXeB44CdxWVa+P1XtJ0rwsOPSr6ifAnwyo/zdw3Rn22QXsWug5JUnj8Ru5ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ1ZudQdkKRhNux8eGLHOjLPYx6548aJnftc4J2+JDXE0Jekhji9swgm+U9RSZok7/QlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ6b+yGaSrcA/ASuAL1XVHYt1Lh+dlKTfNNXQT7IC+GfgL4FZ4AdJ9lXV89PshySNaqluHhfr5x+mPb2zBThcVT+pqv8FHgS2TbkPktSsaU/vrAVe7vt7FvjTuY2S7AB2dH/+T5IXFqEvq4BfLMJxlxPHaDjHaLhzaoxyauXOdy9lN+Y6bYxy59jH/INBxWmHfgbU6rRC1W5g96J2JJmpqs2LeY7znWM0nGM0nGM03DTHaNrTO7PA+r6/1wFHp9wHSWrWtEP/B8CmJBuTvBHYDuybch8kqVlTnd6pqpNJPgz8O71HNu+tqoPT7EOfRZ0+WiYco+Eco+Eco+GmNkapOm1KXZK0TPmNXElqiKEvSQ1Z1qGf5JIkjyV5sXu9+AzttiZ5IcnhJDsHbP/7JJVk1eL3errGHaMkn0nyoyTPJvlGkrdMrfOLaIT3RJLc1W1/Nsk7Rt13uVjoGCVZn+Q7SQ4lOZjkI9Pv/fSM817qtq9I8h9JvjmRDlXVsl2ATwM7u/WdwJ0D2qwAfgz8IfBG4IfA5X3b19P74PmnwKqlvqZzbYyAdwEru/U7B+1/vi3D3hNdmxuAR+l99+Qa4MlR910Oy5hjtAZ4R7f+ZuA/l+MYjTtOfdv/DvgK8M1J9GlZ3+nT+4mHPd36HuCmAW2G/TTE54F/YMCXyJaJscaoqr5VVSe7dt+n992L890oPxeyDbi/er4PvCXJmhH3XQ4WPEZVdayqngaoqleBQ/S+rb8cjfNeIsk64EbgS5Pq0HIP/cuq6hhA93rpgDaDfhpiLUCSvwJ+XlU/XOyOLqGxxmiOv6V3x3K+G+V6z9Rm1LE6340zRv8vyQbg7cCTk+/iOWHccfoCvZvOX0+qQ1P/aeVJS/Jt4PcHbPrEqIcYUKskv9Md410L7du5YrHGaM45PgGcBB6YX+/OSaP8XMiZ2oz0UyPLwDhj1NuYvAn4GvDRqvrVBPt2LlnwOCV5N3C8qvYnuXZSHTrvQ7+q/uJM25K8cuqfk90/l44PaHamn4Z4G7AR+GGSU/Wnk2ypqv+a2AVMwSKO0alj3AK8G7iuuknI89woPxdypjZvHGHf5WCcMSLJG+gF/gNV9fVF7OdSG2ec/hr4qyQ3AL8NXJTkX6vqb8bq0VJ/0LHIH6J8ht/8kPLTA9qsBH5CL+BPfdByxYB2R1ieH+SONUbAVuB5YPVSX8sEx2Toe4LePGv/h29Pzef9dL4vY45RgPuBLyz1dZzL4zSnzbVM6IPcJR+URR7w3wMeB17sXi/p6m8FHulrdwO9Jwh+DHziDMdarqE/1hgBh+nNRz7TLf+y1Nc0oXE57XqBDwEf6tZD738I9GPgALB5Pu+n5bAsdIyAP6M3xfFs3/vmhqW+nnNtnOYcY2Kh788wSFJDlvvTO5KkPoa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1Jasj/AeJXYdlBt6E7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is the null distribution\n",
    "plt.hist(null_vals)\n",
    "# and this is where our observed statistic falls \n",
    "plt.axvline(obs_diff, color='r', linewidth=2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ccd6c383",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0059"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the p-value will then be all the null values that are more extreme than\n",
    "# our statistic in favor of the alternative\n",
    "(null_vals > obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ee23ef",
   "metadata": {},
   "source": [
    "With a p-value of approximately 0.6%, the difference in CTR for the control and the experiment group does appear to be significant. If we have, for instance, an $\\alpha$ of 1%, we can **reject $H_{0}$**. In our example, it certainly looks like Audacity should launch the new version of the home page. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18298ca5",
   "metadata": {},
   "source": [
    "### Experiment 2:\n",
    "\n",
    "The second change, Audacity is A/B testing is a more career-focused description on a course overview page. They hope that this change may encourage more users to enroll and complete this course. In this experiment, we’re going to analyze the following metrics:\n",
    "\n",
    "1. **Enrollment Rate:** Click-through rate for the Enroll button on the course overview page\n",
    "2. **Average Reading Duration:** Average number of seconds spent on the course overview page\n",
    "3. **Average Classroom Time:** Average number of days spent in the classroom for students enrolled in the course\n",
    "4. **Completion Rate:** Course completion rate for students enrolled in the course\n",
    "First, let's determine if the difference observed for each metric is statistically significant individually."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d930b335",
   "metadata": {},
   "source": [
    "### 1. Metric: Enrollment rate\n",
    "#### Question: Does one page lead users to more enrollments?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "9a355db6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>action</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-09-24 17:14:52.012145</td>\n",
       "      <td>261869</td>\n",
       "      <td>experiment</td>\n",
       "      <td>view</td>\n",
       "      <td>130.545004</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group action    duration\n",
       "0  2016-09-24 17:14:52.012145  261869  experiment   view  130.545004"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "df = pd.read_csv('support-files/13_Case-Study-AB-Tests/course_page_actions.csv')\n",
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1f82f02f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['view', 'enroll'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retriving the control group from the original dataset\n",
    "control_df = df.query('group == \"control\"')\n",
    "\n",
    "# checking to see the types of actions before calculating the click through rate (CTR)\n",
    "control_df['action'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7aceed33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2364438839848676"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the click through rate for the control group\n",
    "control_ctr = control_df.query('action == \"enroll\"')['id'].nunique() / control_df.query('action == \"view\"')['id'].nunique()\n",
    "\n",
    "# display ctr\n",
    "control_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ed851520",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2668693009118541"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retriving the experiment group from the original dataset\n",
    "experiment_df = df.query('group == \"experiment\"')\n",
    "\n",
    "# get the click through rate for the experiment group\n",
    "experiment_ctr = experiment_df.query('action == \"enroll\"')['id'].nunique() / experiment_df.query('action == \"view\"')['id'].nunique()\n",
    "\n",
    "# display ctr\n",
    "experiment_ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bfa41474",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.030425416926986526"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the observed difference in CTRs\n",
    "obs_diff = experiment_ctr - control_ctr\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "21f8edea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now, we have to see if this difference is significant, and not just due to chance\n",
    "# bootstrap the sample to simulate the sampling distribution for the difference in proportions\n",
    "\n",
    "diffs = []\n",
    "\n",
    "for _ in range(int(1e4)):\n",
    "    bootsample = df.sample(df.shape[0], replace=True) # bootstrap the WHOLE sample (df.shape[0])\n",
    "    control_df = bootsample.query('group == \"control\"')\n",
    "    experiment_df = bootsample.query('group == \"experiment\"')\n",
    "    control_ctr = control_df.query('action == \"enroll\"')['id'].nunique() / control_df.query('action == \"view\"')['id'].nunique()\n",
    "    experiment_ctr = experiment_df.query('action == \"enroll\"')['id'].nunique() / experiment_df.query('action == \"view\"')['id'].nunique()\n",
    "    diffs.append(experiment_ctr - control_ctr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "044d4a51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the p-value for our statistic, which is the observed difference in proportions \n",
    "# by simulating the distribution under the null hypothesis and then finding the probability\n",
    "# that our statistic came from this distribution \n",
    "\n",
    "# simulating from the null creating a normal distribution centered at zero\n",
    "# with the same standard deviation as our sampling distribution above \n",
    "\n",
    "diffs = np.array(diffs)\n",
    "null_vals = np.random.normal(0, np.std(diffs), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "52955c6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAASNklEQVR4nO3df6zd9X3f8eerJqGsCQLKhTm2M9PInWSQ6gTLY8o0sVEFD6KaSovkSCuumsotIlOidVpNo6nZH5ZIszYt0sLkJhlGTYusJhlWgLYOytRV4kevGcExDsMJbnDsYbddFfoPnZ33/jgfR6eXY99z77n3XJvP8yEdfb/n/f18v9/Px8d++Xs/53vOTVUhSerDj6x0ByRJ02PoS1JHDH1J6oihL0kdMfQlqSOXrXQH5nPttdfW+vXrV7obUp8OHhwsb755ZfuhBTt48OBfVtXM3PpFH/rr169ndnZ2pbsh9SkZLP03eMlJ8hej6k7vSFJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkXlDP8mPJnk2yTeSHE7yn1r9miQHkrzcllcP7XNfkqNJXkpy+1D95iSH2rYHknM3AUuSpmGcK/03gH9ZVT8FbAK2JrkF2AU8WVUbgCfbc5JsBLYDNwJbgc8mWdWO9SCwE9jQHluXbiiSpPnM+4ncGvyWlb9tT9/WHgVsA25t9b3A/wB+tdUfqao3gFeSHAW2JDkGXFlVTwEkeRi4C3hiaYaiXq3f9diKnPfY/XeuyHmlSYw1p59kVZLngVPAgap6Bri+qk4CtOV1rfka4NWh3Y+32pq2Prc+6nw7k8wmmT19+vQChiNJupCxQr+qzlbVJmAtg6v2my7QfNQ8fV2gPup8e6pqc1Vtnpl50/cFSZIWaUF371TV3zCYxtkKvJZkNUBbnmrNjgPrhnZbC5xo9bUj6pKkKRnn7p2ZJFe19SuAnwa+BewHdrRmO4BH2/p+YHuSy5PcwOAN22fbFNDrSW5pd+3cPbSPJGkKxvlq5dXA3nYHzo8A+6rqq0meAvYl+QjwXeBDAFV1OMk+4EXgDHBvVZ1tx7oHeAi4gsEbuL6JK0lTNM7dOy8A7x1R/yvgtvPssxvYPaI+C1zo/QBJ0jLyE7mS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JF5Qz/JuiRfT3IkyeEkH2v1Tyb5XpLn2+OOoX3uS3I0yUtJbh+q35zkUNv2QJIsz7AkSaNcNkabM8CvVNVzSd4JHExyoG37TFX95+HGSTYC24EbgXcBX0vyk1V1FngQ2Ak8DTwObAWeWJqhSJLmM++VflWdrKrn2vrrwBFgzQV22QY8UlVvVNUrwFFgS5LVwJVV9VRVFfAwcNekA5AkjW+cK/0fSrIeeC/wDPB+4KNJ7gZmGfw08H8Z/Ifw9NBux1vt/7X1ufVR59nJ4CcC3v3udy+ki1oh63c9ttJdkDSGsd/ITfIO4EvAx6vq+wymat4DbAJOAr95rumI3esC9TcXq/ZU1eaq2jwzMzNuFyVJ8xgr9JO8jUHgf7GqvgxQVa9V1dmq+gHwu8CW1vw4sG5o97XAiVZfO6IuSZqSce7eCfB54EhV/dZQffVQs58FvtnW9wPbk1ye5AZgA/BsVZ0EXk9ySzvm3cCjSzQOSdIYxpnTfz/wc8ChJM+32q8BH06yicEUzTHglwCq6nCSfcCLDO78ubfduQNwD/AQcAWDu3a8c0eSpmje0K+qP2P0fPzjF9hnN7B7RH0WuGkhHZQkLR0/kStJHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHZk39JOsS/L1JEeSHE7ysVa/JsmBJC+35dVD+9yX5GiSl5LcPlS/Ocmhtu2BJFmeYUmSRrlsjDZngF+pqueSvBM4mOQA8PPAk1V1f5JdwC7gV5NsBLYDNwLvAr6W5Cer6izwILATeBp4HNgKPLHUg5KmYf2ux1bs3Mfuv3PFzq1L27xX+lV1sqqea+uvA0eANcA2YG9rthe4q61vAx6pqjeq6hXgKLAlyWrgyqp6qqoKeHhoH0nSFCxoTj/JeuC9wDPA9VV1Egb/MQDXtWZrgFeHdjveamva+ty6JGlKxg79JO8AvgR8vKq+f6GmI2p1gfqoc+1MMptk9vTp0+N2UZI0j7FCP8nbGAT+F6vqy638WpuyoS1PtfpxYN3Q7muBE62+dkT9TapqT1VtrqrNMzMz445FkjSPce7eCfB54EhV/dbQpv3Ajra+A3h0qL49yeVJbgA2AM+2KaDXk9zSjnn30D6SpCkY5+6d9wM/BxxK8nyr/RpwP7AvyUeA7wIfAqiqw0n2AS8yuPPn3nbnDsA9wEPAFQzu2vHOHUmaonlDv6r+jNHz8QC3nWef3cDuEfVZ4KaFdFCStHT8RK4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOzBv6Sb6Q5FSSbw7VPpnke0meb487hrbdl+RokpeS3D5UvznJobbtgSRZ+uFIki5knCv9h4CtI+qfqapN7fE4QJKNwHbgxrbPZ5Osau0fBHYCG9pj1DElScto3tCvqj8F/nrM420DHqmqN6rqFeAosCXJauDKqnqqqgp4GLhrkX2WJC3SJHP6H03yQpv+ubrV1gCvDrU53mpr2vrc+khJdiaZTTJ7+vTpCbooSRq22NB/EHgPsAk4Cfxmq4+ap68L1Eeqqj1VtbmqNs/MzCyyi5KkuRYV+lX1WlWdraofAL8LbGmbjgPrhpquBU60+toRdUnSFC0q9Nsc/Tk/C5y7s2c/sD3J5UluYPCG7bNVdRJ4Pckt7a6du4FHJ+i3JGkRLpuvQZI/AG4Frk1yHPh14NYkmxhM0RwDfgmgqg4n2Qe8CJwB7q2qs+1Q9zC4E+gK4In2kCRN0byhX1UfHlH+/AXa7wZ2j6jPAjctqHeSpCXlJ3IlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR2Z95eo6NKyftdjK90FSRcxr/QlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHZk39JN8IcmpJN8cql2T5ECSl9vy6qFt9yU5muSlJLcP1W9OcqhteyBJln44kqQLGedK/yFg65zaLuDJqtoAPNmek2QjsB24se3z2SSr2j4PAjuBDe0x95iSpGU2b+hX1Z8Cfz2nvA3Y29b3AncN1R+pqjeq6hXgKLAlyWrgyqp6qqoKeHhoH0nSlCx2Tv/6qjoJ0JbXtfoa4NWhdsdbbU1bn1sfKcnOJLNJZk+fPr3ILkqS5lrqN3JHzdPXBeojVdWeqtpcVZtnZmaWrHOS1LvFhv5rbcqGtjzV6seBdUPt1gInWn3tiLokaYoWG/r7gR1tfQfw6FB9e5LLk9zA4A3bZ9sU0OtJbml37dw9tI8kaUrm/T79JH8A3Apcm+Q48OvA/cC+JB8Bvgt8CKCqDifZB7wInAHuraqz7VD3MLgT6ArgifaQJE3RvKFfVR8+z6bbztN+N7B7RH0WuGlBvZMkLSk/kStJHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/N+DYOki8/6XY9N5TzH5pzv2P13TuW8Wj5e6UtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjkwU+kmOJTmU5Pkks612TZIDSV5uy6uH2t+X5GiSl5LcPmnnJUkLsxRX+v+iqjZV1eb2fBfwZFVtAJ5sz0myEdgO3AhsBT6bZNUSnF+SNKblmN7ZBuxt63uBu4bqj1TVG1X1CnAU2LIM55ckncekoV/AnyQ5mGRnq11fVScB2vK6Vl8DvDq07/FWe5MkO5PMJpk9ffr0hF2UJJ0z6a9LfH9VnUhyHXAgybcu0DYjajWqYVXtAfYAbN68eWQbSdLCTXSlX1Un2vIU8BUG0zWvJVkN0JanWvPjwLqh3dcCJyY5vyRpYRYd+kl+LMk7z60DHwC+CewHdrRmO4BH2/p+YHuSy5PcAGwAnl3s+SVJCzfJ9M71wFeSnDvO71fVHyX5c2Bfko8A3wU+BFBVh5PsA14EzgD3VtXZiXovSVqQRYd+VX0H+KkR9b8CbjvPPruB3Ys9pyRpMn4iV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjLpt2xqhPW7HlvpLkjSSF7pS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSR/xErqSxrdSnzY/df+eKnPetyCt9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JGp37KZZCvwO8Aq4HNVdf9ynctfZiJJf99Ur/STrAL+C/CvgI3Ah5NsnGYfJKln077S3wIcrarvACR5BNgGvDjlfki6hKzkT+1vtQ+GTTv01wCvDj0/DvyTuY2S7AR2tqd/m+SlKfRtKVwL/OVKd2IJOZ6L11TGknMrn/rgcp/qon1t8qlF7XYxjOcfjSpOO/QzolZvKlTtAfYsf3eWVpLZqtq80v1YKo7n4vVWGgs4nmma9t07x4F1Q8/XAiem3AdJ6ta0Q//PgQ1JbkjydmA7sH/KfZCkbk11eqeqziT5KPDHDG7Z/EJVHZ5mH5bZJTclNQ/Hc/F6K40FHM/UpOpNU+qSpLcoP5ErSR0x9CWpI4b+AiW5JsmBJC+35dXnabc1yUtJjibZNWL7v09SSa5d/l6f36TjSfLpJN9K8kKSryS5amqdn6dvQ9uT5IG2/YUk7xt335Ww2PEkWZfk60mOJDmc5GPT7/2b+rro16ZtX5XkfyX56vR6fX4T/l27Kskftn8vR5L80+n2vqkqHwt4AL8B7Grru4BPjWizCvg28BPA24FvABuHtq9j8Gb2XwDXXsrjAT4AXNbWPzVq/2Xu/wX/rFubO4AnGHxO5BbgmXH3XYHXY5LxrAbe19bfCfzvlRzPJGMZ2v7vgN8HvrqSr8tSjAfYC/xiW387cNVKjMMr/YXbxuDFoy3vGtHmh183UVV/B5z7uolzPgP8B0Z8MG0FTDSeqvqTqjrT2j3N4LMX0zTfnzXt+cM18DRwVZLVY+47bYseT1WdrKrnAKrqdeAIg0/Br5RJXhuSrAXuBD43zU5fwKLHk+RK4J8Dnweoqr+rqr+ZYt9/yNBfuOur6iRAW143os2or5tYA5DkZ4DvVdU3lrujY5poPHP8AoOrnGkap2/nazPuuKZpkvH8UJL1wHuBZ5a+i2ObdCy/zeDi6AfL1L+FmmQ8PwGcBv5bm676XJIfW87Ons/Uv1r5UpDka8A/HLHpE+MeYkStkvyDdowPLLZvi7Fc45lzjk8AZ4AvLqx3Exvnqz3O12asrwWZsknGM9iYvAP4EvDxqvr+EvZtoRY9liQfBE5V1cEkty51xxZpktfmMuB9wL+tqmeS/A6D6dT/uLRdnJ+hP0JV/fT5tiV57dyP0u3H0FMjmp3v6ybeA9wAfCPJufpzSbZU1f9ZsgHMsYzjOXeMHcAHgduqTVhO0Thf7XG+Nm8fY99pm2Q8JHkbg8D/YlV9eRn7OY5JxvKvgZ9Jcgfwo8CVSX6vqv7NMvZ3PpOMp4DjVXXuJ68/ZBD607fSb45cag/g0/z9Nz5/Y0Sby4DvMAj4c2/43Dii3TFW/o3cicYDbGXw1dgzK9T/ef+sGcwLD7+59uxCXqdLaDwBHgZ+eyXHsBRjmdPmVi6ON3InGg/wP4F/3NY/CXx6Rcax0n+Ql9oD+HHgSeDltrym1d8FPD7U7g4Gd098G/jEeY51MYT+ROMBjjKYw3y+Pf7rCozhTX0Dfhn45bYeBr+859vAIWDzQl6nS2U8wD9jcEX5wtDrccelOJY5x7goQn8J/q5tAmbb6/PfgatXYgx+DYMkdcS7dySpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6sj/B/PZwy3oV4a4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is the null distribution\n",
    "plt.hist(null_vals)\n",
    "# and this is where our observed statistic falls \n",
    "plt.axvline(obs_diff, color='r', linewidth=2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bc727851",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0188"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the p-value will then be all the null values that are more extreme than\n",
    "# our statistic in favor of the alternative\n",
    "t1 = (null_vals > obs_diff).mean()\n",
    "t1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f80a5f42",
   "metadata": {},
   "source": [
    "With a p-value of approximately 2.07%, the difference in CTR for the control and the experiment group does appear to be significant. If we have a type I error rate ($\\alpha$) of 5%, we can **reject $H_{0}$**. In our example, it certainly looks like Audacity should use the experimental description on its overview page."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d26def35",
   "metadata": {},
   "source": [
    "### 2. Metric: Average Reading Duration\n",
    "#### Question: Does one page engage users for a longer period of time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "74fbae14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# with this metric, we'll be anaylyzing difference in means\n",
    "np.random.seed(42)\n",
    "df = pd.read_csv('support-files/13_Case-Study-AB-Tests/course_page_actions.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4d1deba2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# since we're comparing reading duration, we only care about view actions\n",
    "views = df.query('action == \"view\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "4c7127e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# count only each unique user once by finding their average reading duration\n",
    "# if they visited the site more than once\n",
    "reading_times = views.groupby(['id', 'group'])['duration'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "dc7ecf24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>183260</td>\n",
       "      <td>control</td>\n",
       "      <td>107.331484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>183615</td>\n",
       "      <td>experiment</td>\n",
       "      <td>24.627594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>184277</td>\n",
       "      <td>experiment</td>\n",
       "      <td>193.212489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>184360</td>\n",
       "      <td>experiment</td>\n",
       "      <td>226.586283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>184589</td>\n",
       "      <td>experiment</td>\n",
       "      <td>12.052097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id       group    duration\n",
       "0  183260     control  107.331484\n",
       "1  183615  experiment   24.627594\n",
       "2  184277  experiment  193.212489\n",
       "3  184360  experiment  226.586283\n",
       "4  184589  experiment   12.052097"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# resetting the indexes just so we keep the ID and group as column names\n",
    "reading_times = reading_times.reset_index()\n",
    "reading_times.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "04a7d982",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(115.38637100678449, 131.32084104717944)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# finding the average reading durations for each group\n",
    "control_mean = reading_times.query('group == \"control\"')['duration'].mean()\n",
    "experiment_mean = reading_times.query('group == \"experiment\"')['duration'].mean()\n",
    "control_mean, experiment_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "4ed56027",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15.934470040394956"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# finding the difference of means\n",
    "obs_diff = experiment_mean - control_mean\n",
    "obs_diff # users spent almost 16s more on the overview page in the experimental version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "737f6442",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now, we have to see if this difference is significant, and not just due to chance\n",
    "# bootstrap the sample to simulate the sampling distribution for the difference in means\n",
    "\n",
    "diffs = []\n",
    "\n",
    "for _ in range(int(1e4)):\n",
    "    bootsample = reading_times.sample(reading_times.shape[0], replace=True)\n",
    "    control_df = bootsample.query('group == \"control\"')\n",
    "    experiment_df = bootsample.query('group == \"experiment\"')\n",
    "    control_mean = control_df['duration'].mean()\n",
    "    experiment_mean = experiment_df['duration'].mean()\n",
    "    diffs.append(experiment_mean - control_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "21db8cca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the p-value for our statistic, which is the observed difference in proportions \n",
    "# by simulating the distribution under the null hypothesis and then finding the probability\n",
    "# that our statistic came from this distribution \n",
    "diffs = np.array(diffs)\n",
    "null_vals = np.random.normal(0, np.std(diffs), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "58d567cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPTUlEQVR4nO3dX4hc53nH8e+vdmpMEoON1kaRRCWCApUNVepFNRiKQ9pYjS9kF1zki1gXBgUj0wRyI+cmuRGoJX+o29qgNMYyJFEFibGo7SSOCJiAE2cdTPSvJiJW7Y2EtGkK0ZWKlKcXewTT9Wj/z8xq3+8HhnPmmXPOPC/D/HT87pnjVBWSpDb80agbkCQNj6EvSQ0x9CWpIYa+JDXE0Jekhtw46gbmsmbNmtq4ceOo25Ck4Xnzzenl3Xcv4RBv/raqxmbWV3zob9y4kYmJiVG3IUnDk0wvl5B9Sf6rX93pHUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JasiK/0WuVqaNe18ayHHP7H9gIMeVNM0zfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1Jaoi/yNWK4i99pcGa80w/yYYkP05yKsmJJJ/r6l9O8pskb3WPT/fs82SS00neTnJ/T/3uJMe6155Krv7ffyVJwzCfM/3LwBeq6hdJPgy8meTV7rWvV9VXejdOsgXYCdwJfAT4UZKPVdUV4BlgN/BT4GVgO/DK8gxFkjSXOc/0q+pcVf2iW78InALWzbLLDuBQVV2qqneA08C2JGuBW6rq9aoq4HngwaUOQJI0fwv6Q26SjcDHgZ91pSeS/DLJs0lu7WrrgPd6dpvsauu69Zn1fu+zO8lEkompqamFtChJmsW8Qz/Jh4DvAp+vqt8zPVXzUWArcA746tVN++xes9TfX6w6UFXjVTU+NjY23xYlSXOYV+gn+QDTgf+tqvoeQFWdr6orVfUH4BvAtm7zSWBDz+7rgbNdfX2fuiRpSOZz9U6AbwKnquprPfW1PZs9BBzv1o8AO5PclGQTsBl4o6rOAReT3NMd81HgxWUahyRpHuZz9c69wGeAY0ne6mpfBB5JspXpKZozwGcBqupEksPASaav/NnTXbkD8DjwHHAz01fteOWOJA3RnKFfVT+h/3z8y7Pssw/Y16c+Ady1kAYlScvH2zBIUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0JekhswZ+kk2JPlxklNJTiT5XFe/LcmrSX7VLW/t2efJJKeTvJ3k/p763UmOda89lSSDGZYkqZ/5nOlfBr5QVX8K3APsSbIF2AscrarNwNHuOd1rO4E7ge3A00lu6I71DLAb2Nw9ti/jWCRJc5gz9KvqXFX9olu/CJwC1gE7gIPdZgeBB7v1HcChqrpUVe8Ap4FtSdYCt1TV61VVwPM9+0iShmBBc/pJNgIfB34G3FFV52D6Hwbg9m6zdcB7PbtNdrV13frMer/32Z1kIsnE1NTUQlqUJM1i3qGf5EPAd4HPV9XvZ9u0T61mqb+/WHWgqsaranxsbGy+LUqS5jCv0E/yAaYD/1tV9b2ufL6bsqFbXujqk8CGnt3XA2e7+vo+dUnSkMzn6p0A3wROVdXXel46Auzq1ncBL/bUdya5Kckmpv9g+0Y3BXQxyT3dMR/t2UeSNAQ3zmObe4HPAMeSvNXVvgjsBw4neQx4F3gYoKpOJDkMnGT6yp89VXWl2+9x4DngZuCV7iFJGpI5Q7+qfkL/+XiAT15jn33Avj71CeCuhTQoSVo+/iJXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ15MZRN6DB27j3pVG3IGmF8Exfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNWTO0E/ybJILSY731L6c5DdJ3uoen+557ckkp5O8neT+nvrdSY51rz2VJMs/HEnSbOZznf5zwL8Az8+of72qvtJbSLIF2AncCXwE+FGSj1XVFeAZYDfwU+BlYDvwypK6l+ZpUL9VOLP/gYEcVxqUOc/0q+o14HfzPN4O4FBVXaqqd4DTwLYka4Fbqur1qiqm/wF5cJE9S5IWaSlz+k8k+WU3/XNrV1sHvNezzWRXW9etz6z3lWR3kokkE1NTU0toUZLUa7Gh/wzwUWArcA74alfvN09fs9T7qqoDVTVeVeNjY2OLbFGSNNOiQr+qzlfVlar6A/ANYFv30iSwoWfT9cDZrr6+T12SNESLCv1ujv6qh4CrV/YcAXYmuSnJJmAz8EZVnQMuJrmnu2rnUeDFJfQtSVqEOa/eSfId4D5gTZJJ4EvAfUm2Mj1Fcwb4LEBVnUhyGDgJXAb2dFfuADzO9JVANzN91Y5X7kjSkM0Z+lX1SJ/yN2fZfh+wr099ArhrQd1JkpaVv8iVpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDZkz9JM8m+RCkuM9tduSvJrkV93y1p7XnkxyOsnbSe7vqd+d5Fj32lNJsvzDkSTNZj5n+s8B22fU9gJHq2ozcLR7TpItwE7gzm6fp5Pc0O3zDLAb2Nw9Zh5TkjRgc4Z+Vb0G/G5GeQdwsFs/CDzYUz9UVZeq6h3gNLAtyVrglqp6vaoKeL5nH0nSkCx2Tv+OqjoH0C1v7+rrgPd6tpvsauu69Zn1vpLsTjKRZGJqamqRLUqSZlruP+T2m6evWep9VdWBqhqvqvGxsbFla06SWrfY0D/fTdnQLS909UlgQ89264GzXX19n7okaYgWG/pHgF3d+i7gxZ76ziQ3JdnE9B9s3+imgC4muae7aufRnn0kSUNy41wbJPkOcB+wJskk8CVgP3A4yWPAu8DDAFV1Islh4CRwGdhTVVe6Qz3O9JVANwOvdA9J0hDNGfpV9cg1XvrkNbbfB+zrU58A7lpQd5KkZeUvciWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNWTO/12ihmfj3pdG3YKkVc4zfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNWdJtGJKcAS4CV4DLVTWe5Dbg34GNwBng76rqf7rtnwQe67b/+6r6wVLeXxq1Qd0648z+BwZyXGk5zvQ/UVVbq2q8e74XOFpVm4Gj3XOSbAF2AncC24Gnk9ywDO8vSZqnQUzv7AAOdusHgQd76oeq6lJVvQOcBrYN4P0lSdew1NAv4IdJ3kyyu6vdUVXnALrl7V19HfBez76TXe19kuxOMpFkYmpqaoktSpKuWuqtle+tqrNJbgdeTfKfs2ybPrXqt2FVHQAOAIyPj/fdRpK0cEs606+qs93yAvAC09M155OsBeiWF7rNJ4ENPbuvB84u5f0lSQuz6NBP8sEkH766DnwKOA4cAXZ1m+0CXuzWjwA7k9yUZBOwGXhjse8vSVq4pUzv3AG8kOTqcb5dVd9P8nPgcJLHgHeBhwGq6kSSw8BJ4DKwp6quLKl7SdKCLDr0q+rXwJ/1qf838Mlr7LMP2LfY95QkLY2/yJWkhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqyI2jbuB6tHHvS6NuQZIWxdCXVqBBnVic2f/AQI6r64fTO5LUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNWTo1+kn2Q78E3AD8G9VtX9Q7+WPqCTp/xtq6Ce5AfhX4K+BSeDnSY5U1clh9iG1yh99adjTO9uA01X166r6X+AQsGPIPUhSs4Y9vbMOeK/n+STwFzM3SrIb2N09vZTk+BB6W2nWAL8ddRND1uKYYRWMO/+w4F2u+zEv0sLGnSzlvf6kX3HYod9vBPW+QtUB4ABAkomqGh90YytNi+NucczQ5rhbHDOsjHEPe3pnEtjQ83w9cHbIPUhSs4Yd+j8HNifZlOSPgZ3AkSH3IEnNGur0TlVdTvIE8AOmL9l8tqpOzLHbgcF3tiK1OO4WxwxtjrvFMcMKGHeq3jelLklapfxFriQ1xNCXpIas2NBP8nCSE0n+kGR8xmtPJjmd5O0k94+qx0FK8uUkv0nyVvf49Kh7GqQk27vP83SSvaPuZxiSnElyrPt8J0bdz6AkeTbJhd7f2yS5LcmrSX7VLW8dZY+DcI1xj/x7vWJDHzgO/C3wWm8xyRamr/q5E9gOPN3d3mE1+npVbe0eL4+6mUHpuT3H3wBbgEe6z7kFn+g+39V8zfpzTH9Xe+0FjlbVZuBo93y1eY73jxtG/L1esaFfVaeq6u0+L+0ADlXVpap6BzjN9O0ddP3y9hyrWFW9BvxuRnkHcLBbPwg8OMyehuEa4x65FRv6s+h3K4d1I+pl0J5I8svuPxNX3X/+9mjpM+1VwA+TvNndeqQld1TVOYBuefuI+xmmkX6vRxr6SX6U5Hifx2xnefO6lcP1YI7xPwN8FNgKnAO+OspeB2zVfKYLdG9V/TnT01p7kvzlqBvSwI38ez30++n3qqq/WsRuq+ZWDvMdf5JvAP8x4HZGadV8pgtRVWe75YUkLzA9zfXa7HutGueTrK2qc0nWAhdG3dAwVNX5q+uj+l5fj9M7R4CdSW5KsgnYDLwx4p6WXfdFuOohpv+wvVo1d3uOJB9M8uGr68CnWN2f8UxHgF3d+i7gxRH2MjQr4Xs90jP92SR5CPhnYAx4KclbVXV/VZ1Ichg4CVwG9lTVlVH2OiD/mGQr09McZ4DPjrSbAVrk7Tmud3cAL2T61rk3At+uqu+PtqXBSPId4D5gTZJJ4EvAfuBwkseAd4GHR9fhYFxj3PeN+nvtbRgkqSHX4/SOJGmRDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUkP8Dwf0XeshYBH8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is the null distribution\n",
    "plt.hist(null_vals)\n",
    "# and this is where our observed statistic falls \n",
    "plt.axvline(obs_diff, color ='r', linewidth=2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "05868ad1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the p-value will then be all the null values that are more extreme than\n",
    "# our statistic in favor of the alternative\n",
    "t2 = (null_vals > obs_diff).mean()\n",
    "t2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f9451a",
   "metadata": {},
   "source": [
    "With a p-value of 0%, the difference in means for the control and the experiment group does appear to be significant. If we have a type I error rate ($\\alpha$) of 5%, we can **reject $H_{0}$**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b0aa205",
   "metadata": {},
   "source": [
    "### 3. Metric: Average Classroom Time\n",
    "#### Question: Does one page engage users for a longer period of time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "1081c302",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>total_days</th>\n",
       "      <th>completed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-08-10 17:06:01.032740</td>\n",
       "      <td>610019</td>\n",
       "      <td>experiment</td>\n",
       "      <td>97</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-08-10 17:15:28.950975</td>\n",
       "      <td>690224</td>\n",
       "      <td>control</td>\n",
       "      <td>75</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-08-10 17:34:40.920384</td>\n",
       "      <td>564994</td>\n",
       "      <td>experiment</td>\n",
       "      <td>128</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-08-10 17:50:39.847374</td>\n",
       "      <td>849588</td>\n",
       "      <td>experiment</td>\n",
       "      <td>66</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-08-10 19:10:40.650599</td>\n",
       "      <td>849826</td>\n",
       "      <td>experiment</td>\n",
       "      <td>34</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group  total_days  completed\n",
       "0  2015-08-10 17:06:01.032740  610019  experiment          97       True\n",
       "1  2015-08-10 17:15:28.950975  690224     control          75      False\n",
       "2  2015-08-10 17:34:40.920384  564994  experiment         128       True\n",
       "3  2015-08-10 17:50:39.847374  849588  experiment          66      False\n",
       "4  2015-08-10 19:10:40.650599  849826  experiment          34      False"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('support-files/13_Case-Study-AB-Tests/classroom_actions.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "5ad528ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(73.36899038461539, 74.6715935334873)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the total_days column represents the total amount of time each student has spent in classroom.\n",
    "# get the average classroom time for control group\n",
    "control_mean = df.query('group == \"control\"')['total_days'].mean()\n",
    "\n",
    "# get the average classroom time for experiment group\n",
    "experiment_mean = df.query('group == \"experiment\"')['total_days'].mean()\n",
    "\n",
    "# display average classroom time for each group\n",
    "control_mean, experiment_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "4150057c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3026031488719099"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# observed difference in average classroom time \n",
    "obs_diff = experiment_mean - control_mean\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "cc538e15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking if this diference is significant by bootstraping\n",
    "\n",
    "diffs = []\n",
    "\n",
    "for _ in range(int(1e4)):\n",
    "    bootsample = df.sample(df.shape[0], replace=True)\n",
    "    control_df = bootsample.query('group == \"control\"')\n",
    "    experiment_df = bootsample.query('group == \"experiment\"')\n",
    "    control_mean = control_df['total_days'].mean()\n",
    "    experiment_mean = experiment_df['total_days'].mean()\n",
    "    diffs.append(experiment_mean - control_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "edf6bd04",
   "metadata": {},
   "outputs": [],
   "source": [
    "diffs = np.array(diffs)\n",
    "\n",
    "# we can find the p-value simulating the difference in means under the null hypothesis \n",
    "# and then finding the probability that our statistic come from this distribution\n",
    "null_vals = np.random.normal(0, np.std(diffs), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "1804f3e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAQxklEQVR4nO3df6jd9X3H8eer6py0lSpenU3iIl02qtKmeMmE7g87u5m1pbEDIf5RAxPSSoQW+seSCrNjBCxd281RBTvFCLYuYMVQdasNHa7gj94U1xhjZqip3iaYtKVrysA1+t4f95vtLJ7kntx7ck7u/TwfcDjf8z6f7/f7/nL1db/5nO/33FQVkqQ2vG3cDUiSRsfQl6SGGPqS1BBDX5IaYuhLUkMMfUlqyKyhn+S3kzyb5N+T7Ery1139/CRPJHmpez6vZ51NSfYm2ZPk2p76lUl2du/dkSSn5rAkSf0Mcqb/OvDHVfV+YCWwOslVwEZge1WtALZ3r0lyGbAWuBxYDdyZ5IxuW3cB64EV3WP18A5FkjSbWUO/Zvy6e3lW9yhgDbClq28BruuW1wAPVtXrVfUysBdYleRi4Nyqeqpm7gi7v2cdSdIInDnIoO5MfQfwe8DXquqZJBdV1QGAqjqQ5MJu+BLg6Z7Vp7vab7rlY+sndMEFF9Ty5csHaVPSQrNjx8zzlVeOt49FaMeOHT+rqolj6wOFflW9AaxM8i7g4SRXnGB4v3n6OkH9rRtI1jMzDcQll1zC1NTUIG1KWmiOfqzn/+NDl+Qn/eondfVOVf0S+Fdm5uJf66Zs6J4PdsOmgWU9qy0F9nf1pX3q/fZzd1VNVtXkxMRbflFJkuZokKt3JrozfJKcA3wYeBHYBqzrhq0DHumWtwFrk5yd5FJmPrB9tpsKOpzkqu6qnRt71pEkjcAg0zsXA1u6ef23AVur6ttJngK2JrkJeAW4HqCqdiXZCrwAHAE2dNNDADcD9wHnAI93D0nSiOR0/2rlycnJck5fWqSOzumf5jm0ECXZUVWTx9a9I1eSGmLoS1JDDH1JaoihL0kNMfQlqSED3ZEr6a2Wb3x0bPved/tHx7ZvLWye6UtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1ZNbQT7IsyfeS7E6yK8lnuvoXkvw0yXPd4yM962xKsjfJniTX9tSvTLKze++OJDk1hyVJ6ufMAcYcAT5XVT9M8k5gR5Inuve+WlV/2zs4yWXAWuBy4N3Ad5P8flW9AdwFrAeeBh4DVgOPD+dQJEmzmTX0q+oAcKBbPpxkN7DkBKusAR6sqteBl5PsBVYl2QecW1VPASS5H7gOQ1/ztHzjo+NuQVowTmpOP8ly4APAM13pliQ/SnJvkvO62hLg1Z7Vprvakm752LokaUQGDv0k7wAeAj5bVb9iZqrmPcBKZv4l8OWjQ/usXieo99vX+iRTSaYOHTo0aIuSpFkMFPpJzmIm8B+oqm8BVNVrVfVGVb0JfB1Y1Q2fBpb1rL4U2N/Vl/apv0VV3V1Vk1U1OTExcTLHI0k6gUGu3glwD7C7qr7SU7+4Z9gngOe75W3A2iRnJ7kUWAE82302cDjJVd02bwQeGdJxSJIGMMjVOx8EPgnsTPJcV/s8cEOSlcxM0ewDPgVQVbuSbAVeYObKnw3dlTsANwP3Aecw8wGuH+JK0ggNcvXO9+k/H//YCdbZDGzuU58CrjiZBiVJw+MduZLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ2YN/STLknwvye4ku5J8pqufn+SJJC91z+f1rLMpyd4ke5Jc21O/MsnO7r07kuTUHJYkqZ9BzvSPAJ+rqvcCVwEbklwGbAS2V9UKYHv3mu69tcDlwGrgziRndNu6C1gPrOgeq4d4LJKkWcwa+lV1oKp+2C0fBnYDS4A1wJZu2Bbgum55DfBgVb1eVS8De4FVSS4Gzq2qp6qqgPt71pEkjcCZJzM4yXLgA8AzwEVVdQBmfjEkubAbtgR4ume16a72m2752Lqkk7R846Nj2e++2z86lv1qeAb+IDfJO4CHgM9W1a9ONLRPrU5Q77ev9UmmkkwdOnRo0BYlSbMYKPSTnMVM4D9QVd/qyq91UzZ0zwe7+jSwrGf1pcD+rr60T/0tquruqpqsqsmJiYlBj0WSNItBrt4JcA+wu6q+0vPWNmBdt7wOeKSnvjbJ2UkuZeYD22e7qaDDSa7qtnljzzqSpBEYZE7/g8AngZ1JnutqnwduB7YmuQl4BbgeoKp2JdkKvMDMlT8bquqNbr2bgfuAc4DHu4ckaURmDf2q+j795+MBrjnOOpuBzX3qU8AVJ9OgJGl4vCNXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDZk19JPcm+Rgkud7al9I8tMkz3WPj/S8tynJ3iR7klzbU78yyc7uvTuSZPiHI0k6kUHO9O8DVvepf7WqVnaPxwCSXAasBS7v1rkzyRnd+LuA9cCK7tFvm5KkU2jW0K+qJ4FfDLi9NcCDVfV6Vb0M7AVWJbkYOLeqnqqqAu4Hrptjz5KkOZrPnP4tSX7UTf+c19WWAK/2jJnuaku65WPrfSVZn2QqydShQ4fm0aIkqddcQ/8u4D3ASuAA8OWu3m+evk5Q76uq7q6qyaqanJiYmGOLkqRjzSn0q+q1qnqjqt4Evg6s6t6aBpb1DF0K7O/qS/vUJUkjNKfQ7+boj/oEcPTKnm3A2iRnJ7mUmQ9sn62qA8DhJFd1V+3cCDwyj74lSXNw5mwDknwTuBq4IMk0cBtwdZKVzEzR7AM+BVBVu5JsBV4AjgAbquqNblM3M3Ml0DnA491DkjRCs4Z+Vd3Qp3zPCcZvBjb3qU8BV5xUd5KkofKOXElqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ2b9IyrSIJZvfHTcLUgagGf6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhoya+gnuTfJwSTP99TOT/JEkpe65/N63tuUZG+SPUmu7alfmWRn994dSTL8w5EkncggZ/r3AauPqW0EtlfVCmB795oklwFrgcu7de5Mcka3zl3AemBF9zh2m5KkU2zW0K+qJ4FfHFNeA2zplrcA1/XUH6yq16vqZWAvsCrJxcC5VfVUVRVwf886kqQRmeuc/kVVdQCge76wqy8BXu0ZN93VlnTLx9YlSSM07A9y+83T1wnq/TeSrE8ylWTq0KFDQ2tOklo319B/rZuyoXs+2NWngWU945YC+7v60j71vqrq7qqarKrJiYmJObYoSTrWXEN/G7CuW14HPNJTX5vk7CSXMvOB7bPdFNDhJFd1V+3c2LOOJGlEZv3LWUm+CVwNXJBkGrgNuB3YmuQm4BXgeoCq2pVkK/ACcATYUFVvdJu6mZkrgc4BHu8ekqQRmjX0q+qG47x1zXHGbwY296lPAVecVHeSpKHyjlxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhoy63X6knTU8o2PDnV7+wbc7r7bPzrU/bbMM31JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIfMK/ST7kuxM8lySqa52fpInkrzUPZ/XM35Tkr1J9iS5dr7NS5JOzjDO9D9UVSurarJ7vRHYXlUrgO3da5JcBqwFLgdWA3cmOWMI+5ckDehUTO+sAbZ0y1uA63rqD1bV61X1MrAXWHUK9i9JOo75hn4B30myI8n6rnZRVR0A6J4v7OpLgFd71p3uapKkETlznut/sKr2J7kQeCLJiycYmz616jtw5hfIeoBLLrlkni1Kko6a15l+Ve3vng8CDzMzXfNakosBuueD3fBpYFnP6kuB/cfZ7t1VNVlVkxMTE/NpUZLUY86hn+TtSd55dBn4U+B5YBuwrhu2DnikW94GrE1ydpJLgRXAs3PdvyTp5M1neuci4OEkR7fzjar65yQ/ALYmuQl4BbgeoKp2JdkKvAAcATZU1Rvz6l6SdFLmHPpV9WPg/X3qPweuOc46m4HNc92nJGl+vCNXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkPm+9XKOs0s3/jouFuQdBrzTF+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIZ4nb6k09447z/Zd/tHx7bvU8EzfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDRn5zVpLVwN8DZwD/WFW3j7qHU80/ZCLpdDXS0E9yBvA14E+AaeAHSbZV1Quj7EOSBjWuk7hTdSfwqKd3VgF7q+rHVfXfwIPAmhH3IEnNGvX0zhLg1Z7X08AfnqqdOc0iSf/fqEM/fWr1lkHJemB99/LXSfacxD4uAH42h94WAo9tYfLYjuN/A+GLHxtKM0M21p9bvjjvTfxuv+KoQ38aWNbzeimw/9hBVXU3cPdcdpBkqqom59be6c1jW5g8toVpsR7bqOf0fwCsSHJpkt8C1gLbRtyDJDVrpGf6VXUkyS3AvzBzyea9VbVrlD1IUstGfp1+VT0GPHYKdzGnaaEFwmNbmDy2hWlRHluq3vI5qiRpkfJrGCSpIYsy9JP8TZIfJXkuyXeSvHvcPQ1Lki8lebE7voeTvGvcPQ1LkuuT7EryZpIFf9VEktVJ9iTZm2TjuPsZpiT3JjmY5Plx9zJsSZYl+V6S3d1/j58Zd0/DtChDH/hSVb2vqlYC3wb+asz9DNMTwBVV9T7gP4BNY+5nmJ4H/hx4ctyNzFfPV478GXAZcEOSy8bb1VDdB6wedxOnyBHgc1X1XuAqYMNi+tktytCvql/1vHw7fW4AW6iq6jtVdaR7+TQz9zosClW1u6pO5ka809mi/sqRqnoS+MW4+zgVqupAVf2wWz4M7Gbm2wQWhZFfvTMqSTYDNwL/CXxozO2cKn8B/NO4m1BfI/3KEZ0aSZYDHwCeGXMrQ7NgQz/Jd4Hf6fPWrVX1SFXdCtyaZBNwC3DbSBuch9mOrRtzKzP/DH1glL3N1yDHtkgM9JUjOn0leQfwEPDZY2YPFrQFG/pV9eEBh34DeJQFFPqzHVuSdcDHgGtqgV1zexI/t4VuoK8c0ekpyVnMBP4DVfWtcfczTItyTj/Jip6XHwdeHFcvw9b9EZq/BD5eVf817n50XH7lyAKVJMA9wO6q+sq4+xm2RXlzVpKHgD8A3gR+Any6qn463q6GI8le4Gzg513p6ar69BhbGpoknwD+AZgAfgk8V1XXjrWpeUjyEeDv+L+vHNk83o6GJ8k3gauZ+SbK14DbquqesTY1JEn+CPg3YCczGQLw+e7bBBa8RRn6kqT+FuX0jiSpP0Nfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG/A80dIA72Ds6SAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is the null distribution\n",
    "plt.hist(null_vals)\n",
    "# this is where the observed difference in means falls on the distribution\n",
    "plt.axvline(obs_diff, color='r', linewidth=2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "c27f6b76",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0361"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# finally, finding the probability that our statistic came from this distribution (p-value)\n",
    "t3 = (null_vals > obs_diff).mean()\n",
    "t3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2757fda7",
   "metadata": {},
   "source": [
    "With a p-value of approximately 4%, the difference in means for the control and the experiment group does appear to be significant. If we have a type I error rate ($\\alpha$) of 5%, we can **reject $H_{0}$**. In order words, the hypothesis test above provides evidence that users spend more time in the classroom after seeing the experimental description in the course overview page.\n",
    "\n",
    "While the difference is *statistically significant*, it might not be *practically* so. Engaging students for 1.3 more days in the classroom, when they average around 74 days in total, doesn't seem to indicate a large enough value to launch this change from a practical perspective for Audacity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9703f48e",
   "metadata": {},
   "source": [
    "### 4. Metric: Completion Rate\n",
    "#### Question: Does one page influence more completion rate than the other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a00e5018",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>id</th>\n",
       "      <th>group</th>\n",
       "      <th>total_days</th>\n",
       "      <th>completed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-08-10 17:06:01.032740</td>\n",
       "      <td>610019</td>\n",
       "      <td>experiment</td>\n",
       "      <td>97</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-08-10 17:15:28.950975</td>\n",
       "      <td>690224</td>\n",
       "      <td>control</td>\n",
       "      <td>75</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-08-10 17:34:40.920384</td>\n",
       "      <td>564994</td>\n",
       "      <td>experiment</td>\n",
       "      <td>128</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-08-10 17:50:39.847374</td>\n",
       "      <td>849588</td>\n",
       "      <td>experiment</td>\n",
       "      <td>66</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-08-10 19:10:40.650599</td>\n",
       "      <td>849826</td>\n",
       "      <td>experiment</td>\n",
       "      <td>34</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timestamp      id       group  total_days  completed\n",
       "0  2015-08-10 17:06:01.032740  610019  experiment          97       True\n",
       "1  2015-08-10 17:15:28.950975  690224     control          75      False\n",
       "2  2015-08-10 17:34:40.920384  564994  experiment         128       True\n",
       "3  2015-08-10 17:50:39.847374  849588  experiment          66      False\n",
       "4  2015-08-10 19:10:40.650599  849826  experiment          34      False"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('support-files/13_Case-Study-AB-Tests/classroom_actions.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "a9d75fd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3719951923076923"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# control dataset\n",
    "control_df = df.query('group == \"control\"')\n",
    "\n",
    "# control completion rate\n",
    "control_cr = control_df['completed'].mean()\n",
    "control_cr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "cd27774f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3935334872979215"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# experiment dataset\n",
    "experiment_df = df.query('group == \"experiment\"')\n",
    "\n",
    "# experiment completion rate\n",
    "experiment_cr = experiment_df['completed'].mean()\n",
    "experiment_cr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "229bdd03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.02153829499022919"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# difference of completion rates means\n",
    "obs_diff = experiment_cr - control_cr\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "3380e68c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking if this difference is significant by bootstraping\n",
    "\n",
    "diffs = []\n",
    "\n",
    "for _ in range(int(1e4)):\n",
    "    bootsample = df.sample(df.shape[0], replace=True)\n",
    "    control_df = bootsample.query('group == \"control\"')\n",
    "    experiment_df = bootsample.query('group == \"experiment\"')\n",
    "    control_cr = control_df['completed'].mean()\n",
    "    experiment_cr = experiment_df['completed'].mean()\n",
    "    diffs.append(experiment_cr - control_cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "26110dc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "diffs = np.array(diffs)\n",
    "\n",
    "# we can find the p-value simulating the difference in means under the null hypothesis \n",
    "# and then finding the probability that our statistic come from this distribution\n",
    "null_vals = np.random.normal(0, np.std(diffs), diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "bb41edd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAARHElEQVR4nO3df6zddX3H8edrrTCmEspaWG2bFU1dVkiGUjsWtwXHIh0YwWQmJZmQzKVK0GjmshTNolnSBHT+GMlgwcEsESWN4iACm0jMnAmCtwwppTKqVKnt6HVmkf2DKbz3x/nWHMvpvefec+65bT/PR3Jyvuf9/Xy/38/nnvbV7/2c7/k2VYUkqQ2/stgdkCRNjqEvSQ0x9CWpIYa+JDXE0Jekhixd7A7MZvny5bV27drF7oZ0ctq5s/d8wQWL2w+N3c6dO39SVSuOrh/3ob927VqmpqYWuxvSySnpPft37KST5IeD6k7vSFJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ477b+RKs1m79d5FOe6+6y9blONKo/BMX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQv5wlzdNifSkM/GKY5s8zfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JasisoZ9kTZJvJNmTZHeSD3T1jyX5cZLHuselfdtcl2RvkqeSXNJXvyDJrm7djUmyMMOSJA0yzG0YDgMfqqpHk7wa2JnkgW7dp6vq7/obJ1kPbAbOBV4DfD3J66vqReBmYAvwbeA+YBNw/3iGIkmazaxn+lV1sKoe7ZafB/YAq2bY5HLgzqp6oaqeAfYCG5OsBE6vqoeqqoDbgStGHYAkaXhzmtNPshZ4A/BwV3pfkseT3JZkWVdbBTzbt9n+rraqWz66Pug4W5JMJZmanp6eSxclSTMYOvSTvAr4MvDBqvoZvama1wHnAweBTx5pOmDzmqH+8mLVLVW1oao2rFixYtguSpJmMVToJ3kFvcC/o6ruAqiq56rqxap6CfgssLFrvh9Y07f5auBAV189oC5JmpBhrt4JcCuwp6o+1Vdf2dfsHcAT3fI9wOYkpyY5B1gHPFJVB4Hnk1zY7fMq4O4xjUOSNIRhrt55M/AuYFeSx7rah4Erk5xPb4pmH/AegKranWQH8CS9K3+u7a7cAbgG+BxwGr2rdrxyR5ImaNbQr6pvMXg+/r4ZttkGbBtQnwLOm0sHJUnj4zdyJakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1JBZQz/JmiTfSLInye4kH+jqZyZ5IMnT3fOyvm2uS7I3yVNJLumrX5BkV7fuxiRZmGFJkgYZ5kz/MPChqvpt4ELg2iTrga3Ag1W1Dniwe023bjNwLrAJuCnJkm5fNwNbgHXdY9MYxyJJmsWsoV9VB6vq0W75eWAPsAq4HNjeNdsOXNEtXw7cWVUvVNUzwF5gY5KVwOlV9VBVFXB73zaSpAmY05x+krXAG4CHgbOr6iD0/mEAzuqarQKe7dtsf1db1S0fXR90nC1JppJMTU9Pz6WLkqQZDB36SV4FfBn4YFX9bKamA2o1Q/3lxapbqmpDVW1YsWLFsF2UJM1iqNBP8gp6gX9HVd3VlZ/rpmzong919f3Amr7NVwMHuvrqAXVJ0oQMc/VOgFuBPVX1qb5V9wBXd8tXA3f31TcnOTXJOfQ+sH2kmwJ6PsmF3T6v6ttGkjQBS4do82bgXcCuJI91tQ8D1wM7krwb+BHwToCq2p1kB/AkvSt/rq2qF7vtrgE+B5wG3N89JEkTMmvoV9W3GDwfD3DxMbbZBmwbUJ8CzptLByVJ4+M3ciWpIYa+JDXE0Jekhhj6ktSQYa7ekWa1duu9i90FSUPwTF+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhswa+kluS3IoyRN9tY8l+XGSx7rHpX3rrkuyN8lTSS7pq1+QZFe37sYkGf9wJEkzGeZM/3PApgH1T1fV+d3jPoAk64HNwLndNjclWdK1vxnYAqzrHoP2KUlaQLOGflV9E/jpkPu7HLizql6oqmeAvcDGJCuB06vqoaoq4Hbginn2WZI0T6PM6b8vyePd9M+yrrYKeLavzf6utqpbPro+UJItSaaSTE1PT4/QRUlSv/mG/s3A64DzgYPAJ7v6oHn6mqE+UFXdUlUbqmrDihUr5tlFSdLR5hX6VfVcVb1YVS8BnwU2dqv2A2v6mq4GDnT11QPqkqQJmlfod3P0R7wDOHJlzz3A5iSnJjmH3ge2j1TVQeD5JBd2V+1cBdw9Qr8lSfOwdLYGSb4IXAQsT7If+ChwUZLz6U3R7APeA1BVu5PsAJ4EDgPXVtWL3a6uoXcl0GnA/d1DkjRBs4Z+VV05oHzrDO23AdsG1KeA8+bUO0nSWPmNXElqyKxn+pKOP2u33juW/eyb4/72XX/ZWI6rxeOZviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIbMGvpJbktyKMkTfbUzkzyQ5OnueVnfuuuS7E3yVJJL+uoXJNnVrbsxScY/HEnSTIY50/8csOmo2lbgwapaBzzYvSbJemAzcG63zU1JlnTb3AxsAdZ1j6P3KUlaYLOGflV9E/jpUeXLge3d8nbgir76nVX1QlU9A+wFNiZZCZxeVQ9VVQG3920jSZqQ+c7pn11VBwG657O6+irg2b52+7vaqm756PpASbYkmUoyNT09Pc8uSpKONu4PcgfN09cM9YGq6paq2lBVG1asWDG2zklS6+Yb+s91UzZ0z4e6+n5gTV+71cCBrr56QF2SNEHzDf17gKu75auBu/vqm5OcmuQceh/YPtJNAT2f5MLuqp2r+raRJE3I0tkaJPkicBGwPMl+4KPA9cCOJO8GfgS8E6CqdifZATwJHAauraoXu11dQ+9KoNOA+7uHJGmCZg39qrryGKsuPkb7bcC2AfUp4Lw59U6SNFZ+I1eSGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGrJ0sTug8Vq79d7F7oKk45hn+pLUEENfkhoyUugn2ZdkV5LHkkx1tTOTPJDk6e55WV/765LsTfJUkktG7bwkaW7Gcab/lqo6v6o2dK+3Ag9W1Trgwe41SdYDm4FzgU3ATUmWjOH4kqQhLcT0zuXA9m55O3BFX/3Oqnqhqp4B9gIbF+D4kqRjGDX0C/hakp1JtnS1s6vqIED3fFZXXwU827ft/q72Mkm2JJlKMjU9PT1iFyVJR4x6yeabq+pAkrOAB5J8b4a2GVCrQQ2r6hbgFoANGzYMbCNJmruRzvSr6kD3fAj4Cr3pmueSrATong91zfcDa/o2Xw0cGOX4kqS5mXfoJ3llklcfWQbeCjwB3ANc3TW7Gri7W74H2Jzk1CTnAOuAR+Z7fEnS3I0yvXM28JUkR/bzhar61yTfAXYkeTfwI+CdAFW1O8kO4EngMHBtVb04Uu8lSXMy79Cvqh8AvzOg/j/AxcfYZhuwbb7HlCSNxnvvSBraYt3bad/1ly3KcU9G3oZBkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kN8T9GXwCL9Z9HS9JsPNOXpIYY+pLUEKd3JB33FnPKdN/1ly3asReCZ/qS1BBDX5IaYuhLUkMmHvpJNiV5KsneJFsnfXxJatlEQz/JEuAfgD8B1gNXJlk/yT5IUssmffXORmBvVf0AIMmdwOXAkwtxML8kJWlUi5UjC3XV0KRDfxXwbN/r/cDvHt0oyRZgS/fy/5I8NY9jLQd+Mo/tjjcnwzhOhjHASTiOHKnc8LZF68wITrr3o19uGHm/vzmoOOnQz4BavaxQdQtwy0gHSqaqasMo+zgenAzjOBnGAI7jeOM45mfSH+TuB9b0vV4NHJhwHySpWZMO/e8A65Kck+QUYDNwz4T7IEnNmuj0TlUdTvI+4N+AJcBtVbV7gQ430vTQceRkGMfJMAZwHMcbxzEPqXrZlLok6STlN3IlqSGGviQ15IQO/SRnJnkgydPd87JjtDvmrR+SvL9btzvJxyfT8186/shj6Nb/VZJKsnzhez2wfyONI8knknwvyeNJvpLkjIl1foZ+9a1Pkhu79Y8neeOw207KfMeQZE2SbyTZ0/09+MDke/9L/Zz3e9GtX5LkP5N8dXK9frkR/0ydkeRL3d+JPUl+b2wdq6oT9gF8HNjaLW8FbhjQZgnwfeC1wCnAd4H13bq3AF8HTu1en3WijaFbv4beh+M/BJafoO/FW4Gl3fINg7ZfwL7P+PPt2lwK3E/vuyYXAg8Pu+0JMIaVwBu75VcD/7UYYxh1HH3r/xL4AvDVxRjDOMYBbAf+ols+BThjXH07oc/06d3CYXu3vB24YkCbX9z6oap+Dhy59QPANcD1VfUCQFUdWtjuDjTqGAA+Dfw1A77oNkEjjaOqvlZVh7t236b3HY5Jme3nS/f69ur5NnBGkpVDbjsJ8x5DVR2sqkcBqup5YA+9b88vhlHeC5KsBi4D/mmSnR5g3uNIcjrwh8CtAFX186r633F17EQP/bOr6iBA93zWgDaDbv1w5A/064E/SPJwkn9P8qYF7e1gI40hyduBH1fVdxe6o7MY9b3o9+f0zoAmZZh+HavNsGNaaKOM4ReSrAXeADw8/i4OZdRxfIbeCdBLC9S/YY0yjtcC08A/d9NU/5TklePq2HH/3yUm+TrwGwNWfWTYXQyoHTkjXgoso/er1ZuAHUleW93vVOOyUGNI8mvdPt46377NxQK/F0eO8RHgMHDH3Ho3kmFuD3KsNkPdWmQCRhlDb2XyKuDLwAer6mdj7NtczHscSd4GHKqqnUkuGnfH5miU92Mp8Ebg/VX1cJK/pzdl+jfj6NhxH/pV9cfHWpfkuSO/nna/3g2anpnp1g/7gbu6kH8kyUv0bn40PZ7e9yzgGF4HnAN8N8mR+qNJNlbVf49tAJ0Ffi9IcjXwNuDicf/DO4thbg9yrDanDLHtJIwyBpK8gl7g31FVdy1gP2czyjj+FHh7kkuBXwVOT/L5qvqzBezvsYwyjgL2V9WR37a+RC/0x2OxPugYxwP4BL/84eHHB7RZCvyAXjge+UDl3G7de4G/7ZZfT+9XrZxIYziq3T4W74PcUd+LTfRusb1iEfo+68+X3jxx/4duj8zlvTnOxxDgduAzi/FnZ1zjOKrNRSzuB7kjjQP4D+C3uuWPAZ8YW98W+00e8Qf768CDwNPd85ld/TXAfX3tLqV3RcL3gY/01U8BPg88ATwK/NGJNoaj9rWPxQv9Ud+LvfT+0X2se/zjhPv/sn7ROyl4b7ccev8B0PeBXcCGubw3x/MYgN+nd3b5eN/P/9ITbRxH7eMiFjH0x/Bn6nxgqntP/gVYNq5+eRsGSWrIiX71jiRpDgx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1JD/B6Tp+zt+++76AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is the null distribution\n",
    "plt.hist(null_vals)\n",
    "# this is where the observed difference in completion rate means fall on the null distribution\n",
    "plt.axvline(obs_diff, color='r', linewidth=2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "eb8b8ba3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0886"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# finally, finding the probability that our statistic came from this distribution (p-value)\n",
    "t4 = (null_vals > obs_diff).mean()\n",
    "t4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395b77e2",
   "metadata": {},
   "source": [
    "With a p-value of approximately 9.6%, the difference in means for the control and the experiment group doesn't appear to be significant. If we have a type I error rate ($\\alpha$) of 5%, we **fail to reject $H_{0}$**. In order words, the hypothesis test above doesn't provide evidence that the experimental page is more effective in improving completion rate. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f28f43",
   "metadata": {},
   "source": [
    "### Anayzing Multiple Metrics\n",
    "The more metrics you evaluate, the more likely you are to observe significant differences just by chance.\n",
    "\n",
    "The [Bonferroni Correction](http://en.wikipedia.org/wiki/Bonferroni_correction) is one way we could handle experiments with multiple tests or metrics in this case. To compute the new Bonferroni correct alpha value, we need to divide the original alpha value by the number of tests.\n",
    "\n",
    "#### Question: If our original alpha value was 0.05, what would be our new, Bonferroni corrected alpha value, considering we had four tests?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "df4c1dc4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0125"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new alpha\n",
    "bonf_alpha = 0.05/4\n",
    "bonf_alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930cea6c",
   "metadata": {},
   "source": [
    "#### Question: What results are still statistically significant using the new alpha? \n",
    "\n",
    "1. **Enrollment Rate:** not significant anymore, we **fail to reject $H_{0}$**\n",
    "2. **Average Reading Duration:** still statistically significant, even under the corrected alpha we **reject $H_{0}$**\n",
    "3. **Average Classroom Time:** not significant anymore, we **fail to reject $H_{0}$**\n",
    "4. **Completion Rate:** still not statistically significant, we **fail to reject $H_{0}$**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "fd2de191",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0188, 0.0125, False)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# enrollment rate: not significant anymore, we fail to reject H0\n",
    "t1, bonf_alpha, t1 < bonf_alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "13deb509",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 0.0125, True)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# average reading duration: still statistically significant, even under the corrected alpha\n",
    "# we reject H0\n",
    "t2, bonf_alpha, t2 < bonf_alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "cd9d2b26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0361, 0.0125, False)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# average classroom time: not significant anymore, we fail to reject H0\n",
    "t3, bonf_alpha, t3 < bonf_alpha "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "578508cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0886, 0.0125, False)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# completion rate: still not statistically significant, we fail to reject H0\n",
    "t4, bonf_alpha, t4 < bonf_alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d406c6a4",
   "metadata": {},
   "source": [
    "### Drawing Conclusions\n",
    "Since the Bonferroni method is too conservative when we expect correlation among metrics, we can better approach this problem with more sophisticated methods, such as the [closed testing procedure](http://en.wikipedia.org/wiki/Closed_testing_procedure), [Boole-Bonferroni bound](http://en.wikipedia.org/wiki/Bonferroni_bound), and the [Holm-Bonferroni method](http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method). These are less conservative and take this correlation into account.\n",
    "\n",
    "If you choose to use a less conservative method, make sure the assumptions of that method are truly met in your situation and that you're not just trying to [cheat on a p-value](http://freakonometrics.hypotheses.org/19817). Choosing a poorly suited test to get significant results will only lead to misguided decisions that harm your company's performance in the long run.\n",
    "\n",
    "### Difficulties to consider when running A/B Tests\n",
    "\n",
    "- Novelty effect and change aversion when existing users first experience a change\n",
    "- Sufficient traffic and conversions to have significant and repeatable results\n",
    "- The best metric choice for making the ultimate decision (eg. measuring revenue vs. clicks)\n",
    "- Long enough run time for the experiment to account for changes in behavior based on time of day/week or seasonal events.\n",
    "- The practical significance of a conversion rate (the cost of launching a new feature vs. the gain from the increase in conversion)\n",
    "-  Consistency among test subjects in the control and experiment group (imbalance in the population represented in each group can lead to situations like Simpson's Paradox)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:dand] *",
   "language": "python",
   "name": "conda-env-dand-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}