FDS-S1-01-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Foundations of Data Science\n",
    "## Week S1 01: Introduction to Jupyter Notebooks and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning outcomes:** \n",
    "In this lab you will learn the very basics of the Python library \"pandas\", which is used for data management. By the end of the lab you should be able to: \n",
    "- use a Jupyter notebook,\n",
    "- load different data file types, \n",
    "- display data,\n",
    "- filter your data for specific values, and \n",
    "- apply basic statistical computations on the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Helpful background**  \n",
    "- Basic knowledge of Python is helpful for this course. If you haven't used Python before or need a refresher there is an introduction to Python data structures in Inf2-IADS.  We can also recommend the following [Python tutorial](http://bebi103.caltech.edu/2015/tutorials/t1a_intro_to_python.html) as a starting point. \n",
    "- Basic knowledge of NumPy is helpful for this course. NumPy provides data structures and functions for scientific computing on Python.  If you haven't used NumPy before or need a refresher, we can recommend the following [NumPy tutorial](http://cs231n.github.io/python-numpy-tutorial/#numpy). You may only need to do this when you get stuck."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to cover a different research question every week. This week we will take the position of a historian and try to answer the following question.\n",
    "\n",
    "**Research question:** Which passenger group had the worst survival rate on the [*Titanic*](https://en.wikipedia.org/wiki/Titanic)?\n",
    "\n",
    "**Data information:** We will use a well-known dataset in the machine learning community, often referred to as the [titanic dataset](https://www.kaggle.com/c/titanic/data). It contains a list of passengers, information about whether they survived or not, and some extra information, such as age, fare, gender and the class they travelled in. On the website, you will find a testing and training dataset. What training data and testing data means will be covered later in this course. In this lab we will not be doing machine learning; therefore the data is combined into a single dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.A IPython / Jupyter environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter Notebook is a web-based interactive computational environment, which enables code to be shared and documented. It supports Julia, Python and R (Ju-pyte-r). A notebook is a collection of *code* and *Markdown* (text) **cells**. We will only give a high-level introduction to Jupyter Notebooks, which will be enough to solve the labs for this course. If you are interested in creating your own notebooks, or you just generally want to get a better understanding, we recommend the following [tutorial](http://bebi103.caltech.edu/2015/tutorials/t0b_intro_to_jupyter_notebooks.html). \n",
    "\n",
    "Each code cell can be run separately, and the output is given below the cell. A number appears at the side of the *code* cell to indicate the order in which the cells were run. \n",
    "\n",
    "**Remarks**\n",
    "1. Code in one cell can run, even if there are errors in other cells.\n",
    "2. The order in which these cells are run is important, e.g. if you are calling a function in cell A, which is defined in cell B, cell B needs to be executed before cell A.\n",
    "3. This means that the state of the variables in the notebook can be difficult to see.\n",
    "\n",
    "All objects created by running cells are stored in the *kernel* running in the background. You can restart the kernel by using the Kernel menu at the top of the notebook. Because of the issues with the state of the system noted in the remarks, we recommend that when developing notebooks, you periodically select **Kernel->Restart & Run All** to check that your code really is reproducible. If you run into problems, it can also be a good idea to select **Kernel->Restart & Run All**; the problem might be due to the state of the variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.A.1 Basic operation and shortcuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two modes of selection when inside a Jupyter Notebook:\n",
    "1. Command Mode - When you hit up/down arrows you select different cells. Hit `<enter>` to enter edit mode.\n",
    "2. Edit Mode - You can edit the cell. Hit `<esc>` to enter Command Mode again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Command Mode (cell highlighted blue):\n",
    "```\n",
    "              <h> - bring up help window (contains full list of shortcuts!)\n",
    "          <enter> - Enter Edit Mode\n",
    "              <a> - create new cell above selected\n",
    "              <b> - create cell below selected\n",
    "          <d> <d> - delete selected cell (pressing 'd' twice)\n",
    "```\n",
    "\n",
    "In Edit Mode (cell highlighted green):\n",
    "```\n",
    "            <esc> - Enter Command Mode\n",
    "<shift> + <enter> - Run cell and move to cell below in Command Mode\n",
    " <ctrl> + <enter> - Run cell in place\n",
    "\n",
    "```\n",
    "\n",
    "Try running the following code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 1\n",
    "b = 2\n",
    "a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the notebook will try to display the last thing in the cell, even if you don't use a `print` statement. However, if you want to print multiple things from one cell, you need to use multiple `print` statements (or multiple cells)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_name = 'Jane'\n",
    "last_name = 'Doe'\n",
    "print(first_name)\n",
    "print(last_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Good Practice** \n",
    "- It is good practice to separate code into different cells. One cell should correspond to one task, similarly to functions. For example, use a cell for the `import` statements, one for loading data, one for preprocessing data, and one for each different operation you carry out on the data.\n",
    "- It's generally good practice to import all your packages at the top of a file. We will do so in future tutorials.\n",
    "\n",
    "Before we start, we need to import the packages that we will be using later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`os` stands for the standard Python operating system module, which we will use to access files. `pd` is an alias for the `pandas` module, to save typing later. Likewise `np` is an alias for NumPy module. `pd` and `np` are the standard aliases for `pandas` and NumPy.  Here is a more [in-depth tutorial on installing, importing and using modules](https://www.digitalocean.com/community/tutorials/how-to-import-modules-in-python-3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.B Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a library for data manipulation and analysis. There are two fundamental data structures in pandas: the **Series** and **DataFrame** structures which are built on top of NumPy arrays. (Again, if you need a refresher, you can check out this [numpy tutorial](http://cs231n.github.io/python-numpy-tutorial/#python).)\n",
    "\n",
    "Pandas is well documented and you will find good information about all methods and structures in the [API reference](http://pandas.pydata.org/pandas-docs/version/0.23.4/api.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.B.1 Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **Series** is a one-dimensional object (similar to a list). Each element has a corresponding *index*. By default the indices range from `0` to `N-1`, where `N` is the length of the Series. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passenger = pd.Series(['Mr. Owen Harris Braund', 22.0, False])\n",
    "passenger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to specify **meaningful labels for the index**, we can do so with the `index` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passenger = pd.Series(['Mr. Owen Harris Braund', 22.0, False], index=['Name', 'Age', 'Survived'])\n",
    "passenger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **access a Series** entry the same way as you access list entries, either using the assigned index labels, such as `'Name'`, or by using the numeric index, i.e. `0:(N-1)`, where `N` is the length of the Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(passenger[1]) # Careful: indexing starts at 0.\n",
    "print(passenger['Age']) # Remember to use quotes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.B.2 DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A DataFrame is a tabular data structure comprised of rows and columns. You can also think of the DataFrame as a collection of Series objects that share an index. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating DataFrame structures, adding rows, deleting rows and modifying entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create an empty DataFrame by specifying the column names. Then we can insert data row by row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers = pd.DataFrame(columns=['Gender', 'Age', 'Survived'])\n",
    "passengers # Careful, the dataframe is called passenger*s*, use meaningful variable names when coding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's start filling the dataframe. To specify the row of a data frame, we use the `.loc` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.loc[0] = ['Male', 22.0, False]  # Note how we used df.loc() to specify the index\n",
    "passengers "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, we said that a DataFrame is a collection of Series. Let's double check that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(passengers.loc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas DataFrames are quite flexible. Just as with Series, we can also use strings as index labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.loc['Mrs. John Bradley Cumings'] = ['Female', 38.0, 'Yes']\n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark** It is generally bad practice to mix different kinds of indices. So let's remove the first entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_passengers = passengers.drop(0)\n",
    "print(cleaned_passengers)\n",
    "print('\\n') # Empty line between DataFrames\n",
    "print(passengers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark**: By default, `df.drop(index)` creates a copy of the DataFrame without modifying the original DataFrame, which is why if you want to drop a row without creating a new DataFrame, you need to write `df = df.drop(0)`, or set the optional `inplace` argument to `True`. You can see the difference between `cleaned_passengers` and `passengers` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.drop(0, inplace=True) # Remove the 0th passenger without creating a copy \n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also populate a DataFrame using a dictionary which allows you to do things in a nonstandard order. Let's get our first entry back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.loc['Mr. Owen Harris Braund'] = dict(Survived=False, Age=22.0, Gender='Male') # Remark that the attributes are assigned in a different order\n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just made a mess. In the first row, we used a string to denote whether the passenger survived, and in the second a Boolean value. Let's clean this up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.loc['Mrs. John Bradley Cumings', 'Survived'] = True\n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating DataFrame from other structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create a DataFrame from:\n",
    "* A dict of 1D ndarrays, lists, dicts, or Series\n",
    "* 2-D numpy.ndarray\n",
    "* Structured or record ndarray\n",
    "* A Series\n",
    "* Another DataFrame\n",
    "\n",
    "Let's recreate the same DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame from a list\n",
    "passengers_list = [['Male', 22.0, False], ['Female', 38.0, True]]\n",
    "passengers = pd.DataFrame(passengers_list, index=['Mr. Owen Harris Braund', 'Mrs. John Bradley Cumings'],\n",
    "                          columns=['Gender', 'Age', 'Survived'])\n",
    "passengers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame from a dictionary where keys are column values\n",
    "column_key_dict = {\n",
    "    'Gender': ['Male', 'Female'],\n",
    "    'Age': [22.0, 38.0],\n",
    "    'Survived': [False, True]\n",
    "}\n",
    "passengers = pd.DataFrame.from_dict(column_key_dict, orient='columns')\n",
    "passengers.index = ['Mr. Owen Harris Braund', 'Mrs. John Bradley Cumings']\n",
    "passengers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame from a dictionary where keys are index values\n",
    "index_key_dict = {'Mr. Owen Harris Braund':['Male', 22.0, False],\n",
    "                  'Mrs. John Bradley Cumings':['Female', 38.0, True]}\n",
    "passengers = pd.DataFrame.from_dict(index_key_dict, orient='index')\n",
    "passengers.columns = ['Gender', 'Age', 'Survived']\n",
    "passengers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the DataFrame call, keys are assumed to be column headers\n",
    "passengers = pd.DataFrame({'Mr. Owen Harris Braund':['Male', 22.0, False],\n",
    "                           'Mrs. John Bradley Cumings':['Female', 38.0, True]}, \n",
    "                   index=['Gender', 'Age', 'Survived'])\n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "However, now the rows have become columns and vice versa. We could rewrite the code above, assigning the passenger names to the `index` argument, and using the Gender, Age and Survived attributes as the dict keys. However, there is a more elegant solution: we can use the transpose method `df.T`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers = passengers.T\n",
    "passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark** Again, the transpose method creates a copy. Thus, if you want to actually apply the changes to the DataFrame, you need to save it to the variable, as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine a few things we have learned so far.\n",
    "\n",
    "**Exercise 01 a:** \n",
    "- Delete the markdown cell below, which says 'We don't need this cell.'.\n",
    "- Insert a new code cell below this cell.\n",
    "- Create a 'passengers' DataFrame with one of the options we have presented above, which contains the following entries: \n",
    "    - Miss. Laina Heikkinen, Female, aged 26, survived; \n",
    "    - Mrs. Jacques Heath Futrelle, Female, 35 years old, survived;\n",
    "    - Mr. William Henry Allen, male, 35.0, did not survive;\n",
    "- Make sure that you use a consistent notation in your DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*We don't need this cell. Delete it.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 01 b:**\n",
    "- Append the two passengers mentioned previously (Mr. Own Harris Braund and Mrs. John Bradley Cumings) to the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.B.3 Dataset operations in Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__At this point we suggest driver and navigator swap over again.__ The current driver should:\n",
    "1. Select __File->Save and Checkpoint__ or click on the __Save__ icon or `Ctrl-S`\n",
    "2. Close this tab (but not the Jupyter file browswer)\n",
    "The new driver should open `FDS-S1-01-Introduction.ipynb` from their Jupyter file browser. You can then both continue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most commonly we create DataFrame structures by reading csv files. We store the datasets you need in the first lab in `datasets`. The next labs will have the datasets stored accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers_filepath = os.path.join(os.getcwd(), 'datasets', 'titanic.csv')\n",
    "passengers_filepath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers = pd.read_csv(passengers_filepath)\n",
    "passengers.head() # Head shows the first five elements (unless specified otherwise) of the DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the five passengers you should have added to your dataframe in Exercise 01. Now, we have some extra information. Using `df.head()` we can get an impression of how the data looks like in our dataset. Note that the information in the Survived column is stored as 0/1 instead of `False/True`. Pandas doesn't care how Boolean values are stored. If a column only contains 0s and 1s, Pandas treats the column as Boolean, and works with it in the same way as a column of `False` and `True`.\n",
    "\n",
    "However, `df.head()` only shows the first five entries. How many entries are there in total? You can use the Python native `len()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(passengers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tab completion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tab completion is a powerful method for viewing object attributes and available methods.\n",
    "\n",
    "We have just seen the `df.head()` method above. Let's see what other functions we can call on a DataFrame. You can see what methods are available by typing the DataFrame name followed by `.` and then hitting the `<tab>` key. Then you can access any method's help documentation by hitting the method's name followed by `?`; this opens a 'pager' at the bottom of the screen, you can hit `<esc>` to exit it.\n",
    "\n",
    "For example, to find the last few entries of the DataFrame, first type `passengers.t`, then hit `<tab>` and then choose `df.tail()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's get the documentation\n",
    "passengers.tail?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to get more than five entries you can do so by specifying `N` entries with `df.head(N)` or `df.tail(N)`. Give it a try above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Row selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As already mentioned, you can think of a DataFrame as a group of Series that share an index (*either* the column headers or the row id). This makes it easy to select specific **observations (i.e. rows)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(passengers.loc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already talked about `df.loc[label]`, which selects a row based on the label of the index, e.g. 'Mr. Owen Harris Braund'. If we want the N-th row, we can use `df.iloc[N]`. In the loaded Titanic dataset, it so happens that the indices also run from 0 to 886, and thus `df.loc[N]` and `df.iloc[N]` return the same observation.\n",
    "\n",
    "Technically, there are three options to select rows:\n",
    "* `df.loc[label]`: works on labels in the index\n",
    "* `df.iloc[N]`: works on the position in the index (so it only takes integers)\n",
    "* `df[N]`: works the same way as `df.iloc[N]`\n",
    "\n",
    "It is often safest to use the first two methods (rather than just using square brackets) to index into pandas DataFrames. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select several rows; the resulting structure is a DataFrame. This operation is called **slicing**.\n",
    "\n",
    "**Remark:** Python slicing might seem a bit confusing. When you specify a range to slice with `i:j` the slice returned runs from the `i`-th entry to the `j-1`-th entry. This can be very helpful at times, when you want the last few entries, and you don't know how long the object is (e.g. a DataFrame or list). In that case, you can slice with `k:len(object)`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(passengers.iloc[0:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.iloc[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This is equivalent to using .iloc\n",
    "passengers[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Column Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As already mentioned, you can think of a DataFrame as a group of Series that share an index (*either* the column headers or the row id). This makes it as easy to select specific **columns** as it is to select rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(passengers['Name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers['Name'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select multiple columns we simply need to pass a list of column names. We said above that we were interested in who survived, so let's check that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Remark the double brackets, because we passed a list of names\n",
    "passengers[['Name', 'Survived']].head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 02:** What do you expect what the type of `passengers[['Name', 'Survived']].head(7)` is? Check it! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your Code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can combine row and column selection, as we already did above, when selecting a specific entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.iloc[0]['Survived']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 03:** Look at the dataframe you obtained in Exercise 01. Reconstruct the same dataframe using the newly loaded dataframe, i.e. select the correct rows and columns, such that the output is the same as in Exercise 01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your Code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose that you want to select all the observations of minors (i.e. people under the age of 18 in the UK). It is easy to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers[passengers['Age'] <= 17]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or equivalently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers[passengers.Age <= 17]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concept is called *filtering*, and `passengers.Age <= 17` is called a **mask**, which hides/masks all entries that don't fit the criteria; i.e. it only returns the observations for which the mask returns `True`. You can also filter the data by using multiple attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "young_passengers = passengers[(passengers.Age <= 17) & (passengers.Survived)] #Remark that we were able to drop 'Survived == 1'\n",
    "young_passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark** The first row has index 9. As mentioned earlier: the label of the index does not have to coincide with the position of the data entry. Here, `df.iloc[9]` and `df.loc[9]` return different values!.\n",
    "\n",
    "**You may wish to swap driver and navigator at this point.**\n",
    "\n",
    "**Exercise 04:** What will be the return values of `young_passengers.iloc[8:10]` and `young_passengers.loc[8:10]`. Check the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your Code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can also index into columns using `loc`. We just have to specify the second dimension (much as we would do with NumPy arrays). Let's give it a try. We want to get the list of minors (children) aboard the *Titanic*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "young_passengers.loc[:, 'Name']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "young_passengers.iloc[:, 2] # And now using column indexing **Remark** this would not work with .loc[:, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we try the following we will get an empty DataFrame because there are no rows with labels 0 and 1.\n",
    "young_passengers.loc[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The result is still a DataFrame\n",
    "type(young_passengers.loc[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more, check out [Advanced Indexing](http://pandas.pydata.org/pandas-docs/version/0.23.4/advanced.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.B.4 Summary and final exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now know how to \n",
    "- load a dataset from a csv file `pd.read_csv()`, \n",
    "- get the first few entries `df.head()`, \n",
    "- select certain rows or columns based on their position `df.iloc[]` or index label `df.loc[]`,\n",
    "- and finally how to filter the dataset for entries that satisfy certain conditions `df[condition]`.\n",
    "\n",
    "However, at the beginning of the lab, we wanted to figure out the passenger group worst hit. Using filtering, we can already split the dataset into different subsets. Now let's apply some operations.\n",
    "\n",
    "First let's get the age spread, and figure out the oldest and youngest passenger. The passengers in the dataset are not sorted by age. Looking at each entry individually is not an option. Thankfully pandas dataframes have a method: `df.sort_values(by='column', ascending={True,False})`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.sort_values(by='Age').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "passengers.sort_values(by='Age', ascending=False).head() # Returns the data in descending order of the sorted value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two results above already show that young passengers had a better survival rate than senior passengers. Let's try, tentatively, to answer the research question.  \n",
    "\n",
    "**Discussion:** We will consider the different travel classes, genders, and age groups. Discuss with your lab partner who you would expect had the best and who had the worst survival rate. \n",
    "\n",
    "**Exercise 05:**\n",
    "\n",
    "a. Compute the survival rate of: minors (0-17 years old), adults (18-65), and seniors (66+). (Hint: All you need is `len()`.)\n",
    "b. Compute the survival rate of: women and men.\n",
    "c. Compute the survival rate of the travel classes (1, 2, and 3).\n",
    "d. Does the gender have an influence on the survival rates of minors? (What is the survival rate of girls vs boys?)\n",
    "e. Compute the survival rate of all combinations of class, age group, and gender, and print them out as \"Age group, gender, class: percentage\", e.g. \"Minors, Male, First Class: 0.3\". (Hint: These are quite a few computations, think about how to automate it, e.g. `conditions = condition1 & condition2` might help.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your Code\n",
    "\n",
    "# Exercise a\n",
    "\n",
    "# Exercise b\n",
    "    \n",
    "# Exercise c\n",
    "    \n",
    "# Exercise d\n",
    "\n",
    "# Exercise e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pair programmers: How to share your files with each other\n",
    "\n",
    "\n",
    "At the end of the session you will probably want to share the actual file (not just access to each other's Noteable). A simple (but not elegant) method is to download the file and email it to each other. To download a file from this  notebook, select: __File > Download as > Notebook (.ipynb)__.\n",
    "\n",
    "When you share access to your Noteable, as in this notebook, you and your partner will be logged in with your account. You should both \"log out\" when you stop working together. To do this, you can stop the Jupyter server by:\n",
    "1. Going back to the Jupyter file browser\n",
    "2. Clicking on the __Noteable__ icon at the top right\n",
    "3. On the next screen, clicking on __Shutdown__\n",
    "You can then close the browser tab."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}