Module_2a_Basic_Stat_Metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Metrics\n",
    "\n",
    "When we think about summarizing data, what are the metrics that we look at?\n",
    "\n",
    "In this notebook, we will look at the car dataset\n",
    "\n",
    "To read how the data was acquired, please read [this repo](https://github.com/amitkaps/cars) to get more information\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Import the required libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from datetime import datetime as dt\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cars = pd.read_csv(\"cars_v1.csv\", encoding = \"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Warm up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LE 8-STR (Diesel)</td>\n",
       "      <td>750</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 8-STR (Diesel)</td>\n",
       "      <td>800</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 8-STR (Diesel)</td>\n",
       "      <td>830</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR (Diesel)</td>\n",
       "      <td>850</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR Alloy (Diesel)</td>\n",
       "      <td>880</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Make                                        Model  Price  \\\n",
       "0  Ashok Leyland Stile        Ashok Leyland Stile LE 8-STR (Diesel)    750   \n",
       "1  Ashok Leyland Stile        Ashok Leyland Stile LS 8-STR (Diesel)    800   \n",
       "2  Ashok Leyland Stile        Ashok Leyland Stile LX 8-STR (Diesel)    830   \n",
       "3  Ashok Leyland Stile        Ashok Leyland Stile LS 7-STR (Diesel)    850   \n",
       "4  Ashok Leyland Stile  Ashok Leyland Stile LS 7-STR Alloy (Diesel)    880   \n",
       "\n",
       "  Type  ABS  BootSpace GearType AirBag  Engine  FuelCapacity  Mileage  \n",
       "0  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "1  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "2  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "3  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "4  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LE 8-STR (Diesel)</td>\n",
       "      <td>750</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 8-STR (Diesel)</td>\n",
       "      <td>800</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 8-STR (Diesel)</td>\n",
       "      <td>830</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR (Diesel)</td>\n",
       "      <td>850</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR Alloy (Diesel)</td>\n",
       "      <td>880</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 7-STR (Diesel)</td>\n",
       "      <td>900</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 7-STR Alloy (Diesel)</td>\n",
       "      <td>930</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Aston Martin Rapide</td>\n",
       "      <td>Aston Martin Rapide LUXE (Petrol)</td>\n",
       "      <td>35000</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>90.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Aston Martin Rapide S</td>\n",
       "      <td>Aston Martin Rapide S (Petrol)</td>\n",
       "      <td>44000</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Aston Martin V12 Vantage</td>\n",
       "      <td>Aston Martin V12 Vantage Coupe (Petrol)</td>\n",
       "      <td>35000</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Make                                        Model  \\\n",
       "0       Ashok Leyland Stile        Ashok Leyland Stile LE 8-STR (Diesel)   \n",
       "1       Ashok Leyland Stile        Ashok Leyland Stile LS 8-STR (Diesel)   \n",
       "2       Ashok Leyland Stile        Ashok Leyland Stile LX 8-STR (Diesel)   \n",
       "3       Ashok Leyland Stile        Ashok Leyland Stile LS 7-STR (Diesel)   \n",
       "4       Ashok Leyland Stile  Ashok Leyland Stile LS 7-STR Alloy (Diesel)   \n",
       "5       Ashok Leyland Stile        Ashok Leyland Stile LX 7-STR (Diesel)   \n",
       "6       Ashok Leyland Stile  Ashok Leyland Stile LX 7-STR Alloy (Diesel)   \n",
       "7       Aston Martin Rapide            Aston Martin Rapide LUXE (Petrol)   \n",
       "8     Aston Martin Rapide S               Aston Martin Rapide S (Petrol)   \n",
       "9  Aston Martin V12 Vantage      Aston Martin V12 Vantage Coupe (Petrol)   \n",
       "\n",
       "   Price   Type   ABS  BootSpace    GearType AirBag  Engine  FuelCapacity  \\\n",
       "0    750    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "1    800    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "2    830    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "3    850    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "4    880    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "5    900    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "6    930    MPV    No      500.0      Manual     No  1461.0          50.0   \n",
       "7  35000  Sedan   Yes      300.0   Automatic    Yes  5935.0          90.5   \n",
       "8  44000  Sedan   Yes        NaN   Automatic    Yes  5935.0          90.0   \n",
       "9  35000  Coupe   Yes      300.0   Automatic    Yes  5935.0          80.0   \n",
       "\n",
       "   Mileage  \n",
       "0     20.7  \n",
       "1     20.7  \n",
       "2     20.7  \n",
       "3     20.7  \n",
       "4     20.7  \n",
       "5     20.7  \n",
       "6     20.7  \n",
       "7      7.0  \n",
       "8     11.9  \n",
       "9      9.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Display the first 10 records\n",
    "cars.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>828</th>\n",
       "      <td>Volvo V40</td>\n",
       "      <td>Volvo V40 D3 R-Design (Diesel)</td>\n",
       "      <td>2770</td>\n",
       "      <td>Hatchback</td>\n",
       "      <td>Yes</td>\n",
       "      <td>324.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>27.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>829</th>\n",
       "      <td>Volvo V40 Cross Country</td>\n",
       "      <td>Volvo V40 D3 Cross Country (Diesel)</td>\n",
       "      <td>3180</td>\n",
       "      <td>Hatchback</td>\n",
       "      <td>Yes</td>\n",
       "      <td>335.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>16.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>830</th>\n",
       "      <td>Volvo V40 Cross Country</td>\n",
       "      <td>Volvo V40 T4 Cross Country (Petrol)</td>\n",
       "      <td>3240</td>\n",
       "      <td>Hatchback</td>\n",
       "      <td>Yes</td>\n",
       "      <td>324.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1596.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>831</th>\n",
       "      <td>Volvo XC90</td>\n",
       "      <td>Volvo XC 90 D5 Momentum(Diesel)</td>\n",
       "      <td>6490</td>\n",
       "      <td>SUV</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1969.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>832</th>\n",
       "      <td>Volvo XC90</td>\n",
       "      <td>Volvo XC 90 D5 Inscription(Diesel)</td>\n",
       "      <td>7790</td>\n",
       "      <td>SUV</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>1969.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Make                                Model  Price  \\\n",
       "828                Volvo V40       Volvo V40 D3 R-Design (Diesel)   2770   \n",
       "829  Volvo V40 Cross Country  Volvo V40 D3 Cross Country (Diesel)   3180   \n",
       "830  Volvo V40 Cross Country  Volvo V40 T4 Cross Country (Petrol)   3240   \n",
       "831               Volvo XC90      Volvo XC 90 D5 Momentum(Diesel)   6490   \n",
       "832               Volvo XC90   Volvo XC 90 D5 Inscription(Diesel)   7790   \n",
       "\n",
       "          Type   ABS  BootSpace    GearType AirBag  Engine  FuelCapacity  \\\n",
       "828  Hatchback   Yes      324.0   Automatic    Yes  1984.0          62.0   \n",
       "829  Hatchback   Yes      335.0   Automatic    Yes  1984.0          60.0   \n",
       "830  Hatchback   Yes      324.0   Automatic    Yes  1596.0          50.0   \n",
       "831        SUV   Yes        NaN   Automatic    Yes  1969.0          68.0   \n",
       "832        SUV   Yes        NaN   Automatic    Yes  1969.0          68.0   \n",
       "\n",
       "     Mileage  \n",
       "828     27.8  \n",
       "829     16.8  \n",
       "830     16.0  \n",
       "831     11.1  \n",
       "832     11.1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Display the last 5 records\n",
    "cars.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(833, 11)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find the number of rows and columns in the dataset\n",
    "cars.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Make', 'Model', 'Price', 'Type', 'ABS', 'BootSpace', 'GearType',\n",
       "       'AirBag', 'Engine', 'FuelCapacity', 'Mileage'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#What are the column names in the dataset?\n",
    "cars.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make             object\n",
       "Model            object\n",
       "Price             int64\n",
       "Type             object\n",
       "ABS              object\n",
       "BootSpace       float64\n",
       "GearType         object\n",
       "AirBag           object\n",
       "Engine          float64\n",
       "FuelCapacity    float64\n",
       "Mileage         float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#What are the types of those columns ? \n",
    "cars.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cars.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              0\n",
       "Model             0\n",
       "Price             0\n",
       "Type              0\n",
       "ABS              16\n",
       "BootSpace       179\n",
       "GearType         16\n",
       "AirBag           21\n",
       "Engine            7\n",
       "FuelCapacity      0\n",
       "Mileage         171\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#How to check if there are null values in any of the columns?\n",
    "\n",
    "#Hint: use the isnull() function  (how about using sum or values/any with it?)\n",
    "cars.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Make              0\n",
       "Model             0\n",
       "Price             0\n",
       "Type              0\n",
       "ABS              16\n",
       "BootSpace       179\n",
       "GearType         16\n",
       "AirBag           21\n",
       "Engine            7\n",
       "FuelCapacity      0\n",
       "Mileage         171\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**How to handle missing values?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#fillna function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mean, Median, Variance, Standard Deviation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mean\n",
    "\n",
    "arithmetic average of a range of values or quantities, computed by dividing the total of all values by the number of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3159.4957983193276"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find mean of price\n",
    "cars.Price.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17.480407854984893"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find mean of Mileage\n",
    "cars.Mileage.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do something fancier.\n",
    "Let's find mean mileage of every make. \n",
    "\n",
    "*Hint*: need to use `groupby`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>20.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aston Martin Rapide</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aston Martin Rapide S</td>\n",
       "      <td>11.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aston Martin V12 Vantage</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aston Martin V8 Vantage</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aston Martin Vanquish</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Audi A3 Cabriolet</td>\n",
       "      <td>19.186667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Audi A4</td>\n",
       "      <td>14.804000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Audi A6</td>\n",
       "      <td>15.260000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Audi A7</td>\n",
       "      <td>14.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Audi A8 L</td>\n",
       "      <td>12.575000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Audi Q3</td>\n",
       "      <td>16.260000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Audi Q5</td>\n",
       "      <td>13.063333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Audi Q7</td>\n",
       "      <td>12.466667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Audi R8</td>\n",
       "      <td>9.515000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Audi RS5</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Audi RS6</td>\n",
       "      <td>10.410000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Audi RS7</td>\n",
       "      <td>13.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Audi S6</td>\n",
       "      <td>10.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Audi TT</td>\n",
       "      <td>8.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>BMW 1 Series</td>\n",
       "      <td>20.580000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>BMW 3 Series</td>\n",
       "      <td>17.857500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>BMW 3 Series GT</td>\n",
       "      <td>19.590000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>BMW 5 Series</td>\n",
       "      <td>13.982500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>BMW 5 Series GT</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>BMW 6 Series</td>\n",
       "      <td>17.540000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>BMW 7 Series</td>\n",
       "      <td>11.962500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>BMW M3</td>\n",
       "      <td>10.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>BMW M5</td>\n",
       "      <td>13.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>BMW M6 Gran Coupe</td>\n",
       "      <td>10.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>Tata Movus</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>Tata Nano</td>\n",
       "      <td>24.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>Tata Safari DICOR</td>\n",
       "      <td>13.930000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>Tata Safari Storme</td>\n",
       "      <td>13.875000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>Tata Sumo Gold</td>\n",
       "      <td>14.612500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>Tata Sumo Grande</td>\n",
       "      <td>13.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>Tata Venture</td>\n",
       "      <td>15.420000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>Tata Winger</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>Tata Xenon</td>\n",
       "      <td>12.745000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>Tata Zest</td>\n",
       "      <td>20.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>Toyota Corolla Altis 2014</td>\n",
       "      <td>21.322500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>Toyota Etios</td>\n",
       "      <td>20.185000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>Toyota Etios Cross</td>\n",
       "      <td>13.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>Toyota Etios Liva</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>Toyota Etios Xclusive</td>\n",
       "      <td>16.780000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>Toyota Fortuner</td>\n",
       "      <td>12.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>189</th>\n",
       "      <td>Volkswagen Cross Polo</td>\n",
       "      <td>19.285000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>Volkswagen New JETTA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>Volkswagen New Phaeton</td>\n",
       "      <td>12.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>Volkswagen Passat</td>\n",
       "      <td>17.286667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>Volkswagen Polo</td>\n",
       "      <td>18.305000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>Volkswagen Polo GT</td>\n",
       "      <td>18.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>Volkswagen Taigun</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>Volkswagen Up</td>\n",
       "      <td>21.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>Volkswagen Vento</td>\n",
       "      <td>18.957000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>Volvo S60</td>\n",
       "      <td>19.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>Volvo S80</td>\n",
       "      <td>16.660000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>Volvo V40</td>\n",
       "      <td>27.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Volvo V40 Cross Country</td>\n",
       "      <td>16.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>Volvo XC90</td>\n",
       "      <td>11.100000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>203 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Make    Mileage\n",
       "0          Ashok Leyland Stile  20.700000\n",
       "1          Aston Martin Rapide   7.000000\n",
       "2        Aston Martin Rapide S  11.900000\n",
       "3     Aston Martin V12 Vantage   9.000000\n",
       "4      Aston Martin V8 Vantage   5.000000\n",
       "5        Aston Martin Vanquish   8.000000\n",
       "6            Audi A3 Cabriolet  19.186667\n",
       "7                      Audi A4  14.804000\n",
       "8                      Audi A6  15.260000\n",
       "9                      Audi A7  14.400000\n",
       "10                   Audi A8 L  12.575000\n",
       "11                     Audi Q3  16.260000\n",
       "12                     Audi Q5  13.063333\n",
       "13                     Audi Q7  12.466667\n",
       "14                     Audi R8   9.515000\n",
       "15                    Audi RS5   9.000000\n",
       "16                    Audi RS6  10.410000\n",
       "17                    Audi RS7  13.300000\n",
       "18                     Audi S6  10.300000\n",
       "19                     Audi TT   8.300000\n",
       "20                BMW 1 Series  20.580000\n",
       "21                BMW 3 Series  17.857500\n",
       "22             BMW 3 Series GT  19.590000\n",
       "23                BMW 5 Series  13.982500\n",
       "24             BMW 5 Series GT        NaN\n",
       "25                BMW 6 Series  17.540000\n",
       "26                BMW 7 Series  11.962500\n",
       "27                      BMW M3  10.750000\n",
       "28                      BMW M5  13.250000\n",
       "29           BMW M6 Gran Coupe  10.100000\n",
       "..                         ...        ...\n",
       "173                 Tata Movus        NaN\n",
       "174                  Tata Nano  24.083333\n",
       "175          Tata Safari DICOR  13.930000\n",
       "176         Tata Safari Storme  13.875000\n",
       "177             Tata Sumo Gold  14.612500\n",
       "178           Tata Sumo Grande  13.550000\n",
       "179               Tata Venture  15.420000\n",
       "180                Tata Winger        NaN\n",
       "181                 Tata Xenon  12.745000\n",
       "182                  Tata Zest  20.600000\n",
       "183  Toyota Corolla Altis 2014  21.322500\n",
       "184               Toyota Etios  20.185000\n",
       "185         Toyota Etios Cross  13.300000\n",
       "186          Toyota Etios Liva        NaN\n",
       "187      Toyota Etios Xclusive  16.780000\n",
       "188            Toyota Fortuner  12.550000\n",
       "189      Volkswagen Cross Polo  19.285000\n",
       "190       Volkswagen New JETTA        NaN\n",
       "191     Volkswagen New Phaeton  12.800000\n",
       "192          Volkswagen Passat  17.286667\n",
       "193            Volkswagen Polo  18.305000\n",
       "194         Volkswagen Polo GT  18.450000\n",
       "195          Volkswagen Taigun        NaN\n",
       "196              Volkswagen Up  21.000000\n",
       "197           Volkswagen Vento  18.957000\n",
       "198                  Volvo S60  19.133333\n",
       "199                  Volvo S80  16.660000\n",
       "200                  Volvo V40  27.800000\n",
       "201    Volvo V40 Cross Country  16.400000\n",
       "202                 Volvo XC90  11.100000\n",
       "\n",
       "[203 rows x 2 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#cars.groupby('Make') : Finish the code\n",
    "cars.groupby('Make').Mileage.mean().reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "**How about finding the average mileage for every `Type-GearType` combination?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>GearType</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Convertible</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>12.147143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Coupe</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>9.746071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hatchback</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>19.446111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hatchback</td>\n",
       "      <td>Manual</td>\n",
       "      <td>20.804878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hatchback</td>\n",
       "      <td>No</td>\n",
       "      <td>21.748000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>MPV</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>20.434286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>MPV</td>\n",
       "      <td>Manual</td>\n",
       "      <td>18.825862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MUV</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>12.466667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>MUV</td>\n",
       "      <td>Manual</td>\n",
       "      <td>15.634091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>MUV</td>\n",
       "      <td>No</td>\n",
       "      <td>13.912000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SUV</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>14.603176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>SUV</td>\n",
       "      <td>Manual</td>\n",
       "      <td>15.772308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SUV</td>\n",
       "      <td>No</td>\n",
       "      <td>11.973333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Sedan</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>16.670000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Sedan</td>\n",
       "      <td>Manual</td>\n",
       "      <td>20.143626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Sedan</td>\n",
       "      <td>No</td>\n",
       "      <td>19.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Type    GearType    Mileage\n",
       "0   Convertible   Automatic  12.147143\n",
       "1         Coupe   Automatic   9.746071\n",
       "2     Hatchback   Automatic  19.446111\n",
       "3     Hatchback      Manual  20.804878\n",
       "4     Hatchback          No  21.748000\n",
       "5           MPV   Automatic  20.434286\n",
       "6           MPV      Manual  18.825862\n",
       "7           MUV   Automatic  12.466667\n",
       "8           MUV      Manual  15.634091\n",
       "9           MUV          No  13.912000\n",
       "10          SUV   Automatic  14.603176\n",
       "11          SUV      Manual  15.772308\n",
       "12          SUV          No  11.973333\n",
       "13        Sedan   Automatic  16.670000\n",
       "14        Sedan      Manual  20.143626\n",
       "15        Sedan          No  19.500000"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Median\n",
    "\n",
    "Denotes value or quantity lying at the midpoint of a frequency distribution of observed values or quantities, such that there is an equal probability of falling above or below it. Simply put, it is the *middle* value in the list of numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If count is odd, the median is the value at (n+1)/2,\n",
    "\n",
    "else it is the average of n/2 and (n+1)/2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Find median of mileage**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17.985"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Mileage.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mode\n",
    "\n",
    "It is the number which appears most often in a set of numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Find the mode of `Type` of cars**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Let's first find count of each of the car Types\n",
    "#Hint: use value_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sedan          294\n",
       "Hatchback      222\n",
       "SUV            186\n",
       "MPV             47\n",
       "MUV             40\n",
       "Coupe           33\n",
       "Convertible     11\n",
       "Name: Type, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Type.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Mode of cars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0              MPV\n",
       "1              MPV\n",
       "2              MPV\n",
       "3              MPV\n",
       "4              MPV\n",
       "5              MPV\n",
       "6              MPV\n",
       "7            Sedan\n",
       "8            Sedan\n",
       "9            Coupe\n",
       "10           Coupe\n",
       "11           Coupe\n",
       "12           Coupe\n",
       "13           Sedan\n",
       "14           Sedan\n",
       "15           Sedan\n",
       "16           Sedan\n",
       "17           Sedan\n",
       "18           Sedan\n",
       "19     Convertible\n",
       "20           Sedan\n",
       "21           Sedan\n",
       "22           Sedan\n",
       "23           Sedan\n",
       "24           Sedan\n",
       "25           Sedan\n",
       "26           Sedan\n",
       "27           Sedan\n",
       "28           Sedan\n",
       "29           Sedan\n",
       "          ...     \n",
       "803      Hatchback\n",
       "804      Hatchback\n",
       "805      Hatchback\n",
       "806      Hatchback\n",
       "807      Hatchback\n",
       "808      Hatchback\n",
       "809            SUV\n",
       "810      Hatchback\n",
       "811          Sedan\n",
       "812          Sedan\n",
       "813          Sedan\n",
       "814          Sedan\n",
       "815          Sedan\n",
       "816          Sedan\n",
       "817          Sedan\n",
       "818          Sedan\n",
       "819          Sedan\n",
       "820          Sedan\n",
       "821          Sedan\n",
       "822          Sedan\n",
       "823          Sedan\n",
       "824          Sedan\n",
       "825          Sedan\n",
       "826          Sedan\n",
       "827      Hatchback\n",
       "828      Hatchback\n",
       "829      Hatchback\n",
       "830      Hatchback\n",
       "831            SUV\n",
       "832            SUV\n",
       "Name: Type, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Sedan\n",
       "dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Type.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LE 8-STR (Diesel)</td>\n",
       "      <td>750</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 8-STR (Diesel)</td>\n",
       "      <td>800</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 8-STR (Diesel)</td>\n",
       "      <td>830</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR (Diesel)</td>\n",
       "      <td>850</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR Alloy (Diesel)</td>\n",
       "      <td>880</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Make                                        Model  Price  \\\n",
       "0  Ashok Leyland Stile        Ashok Leyland Stile LE 8-STR (Diesel)    750   \n",
       "1  Ashok Leyland Stile        Ashok Leyland Stile LS 8-STR (Diesel)    800   \n",
       "2  Ashok Leyland Stile        Ashok Leyland Stile LX 8-STR (Diesel)    830   \n",
       "3  Ashok Leyland Stile        Ashok Leyland Stile LS 7-STR (Diesel)    850   \n",
       "4  Ashok Leyland Stile  Ashok Leyland Stile LS 7-STR Alloy (Diesel)    880   \n",
       "\n",
       "  Type  ABS  BootSpace GearType AirBag  Engine  FuelCapacity  Mileage  \n",
       "0  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "1  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "2  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "3  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "4  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variance\n",
    "\n",
    "> Once two statistician of height 4 feet and 5 feet have to cross a river of AVERAGE depth 3 feet. Meanwhile, a third person comes and said, \"what are you waiting for? You can easily cross the river\"\n",
    "\n",
    "It's the average distance of the data values from the *mean*\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/variance.png\" height=\"320\" width=\"320\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Find variance of mileage**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21.018811179847432"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Mileage.var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Standard Deviation\n",
    "\n",
    "It is the square root of variance. This will have the same units as the data and mean. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**Find standard deviation of mileage**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.5846277035161132"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.Mileage.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Pandas built-in function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/amitkaps/miniconda3/lib/python3.5/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile\n",
      "  RuntimeWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>833.000000</td>\n",
       "      <td>654.000000</td>\n",
       "      <td>826.000000</td>\n",
       "      <td>833.000000</td>\n",
       "      <td>662.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3159.495798</td>\n",
       "      <td>409.536697</td>\n",
       "      <td>1900.533898</td>\n",
       "      <td>51.341693</td>\n",
       "      <td>17.480408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>6272.519112</td>\n",
       "      <td>175.351525</td>\n",
       "      <td>1065.149758</td>\n",
       "      <td>17.643132</td>\n",
       "      <td>4.584628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>199.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>624.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>619.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>880.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2329.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>6592.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>30.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Price    BootSpace       Engine  FuelCapacity     Mileage\n",
       "count    833.000000   654.000000   826.000000    833.000000  662.000000\n",
       "mean    3159.495798   409.536697  1900.533898     51.341693   17.480408\n",
       "std     6272.519112   175.351525  1065.149758     17.643132    4.584628\n",
       "min      199.000000    80.000000   624.000000      0.000000    5.000000\n",
       "25%      619.000000          NaN          NaN     42.000000         NaN\n",
       "50%      880.000000          NaN          NaN     50.000000         NaN\n",
       "75%     2329.000000          NaN          NaN     64.000000         NaN\n",
       "max    50000.000000  1000.000000  6592.000000    120.000000   30.000000"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Co-variance \n",
    "\n",
    "covariance as a measure of the (average) co-variation between two variables, say x and y. Covariance describes both how far the variables are spread out, and the nature of their relationship, Covariance is a measure of how much two variables change together. Compare this to Variance, which is just the range over which one measure (or variable) varies.\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/covariance.png\" height=\"270\" width=\"270\">\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Co-variance of mileage of Automatic and Manual Gear Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' Manual', ' Automatic', nan, ' No'], dtype=object)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.unique(cars.GearType)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cars_Automatic = cars[cars.GearType==' Automatic'].copy().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cars_Manual = cars[cars.GearType==' Manual'].copy().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>Aston Martin Rapide</td>\n",
       "      <td>Aston Martin Rapide LUXE (Petrol)</td>\n",
       "      <td>35000</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>90.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8</td>\n",
       "      <td>Aston Martin Rapide S</td>\n",
       "      <td>Aston Martin Rapide S (Petrol)</td>\n",
       "      <td>44000</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>Aston Martin V12 Vantage</td>\n",
       "      <td>Aston Martin V12 Vantage Coupe (Petrol)</td>\n",
       "      <td>35000</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5935.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>Aston Martin V8 Vantage</td>\n",
       "      <td>Aston Martin V8 Vantage Coupe (Petrol)</td>\n",
       "      <td>13500</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>4735.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11</td>\n",
       "      <td>Aston Martin V8 Vantage</td>\n",
       "      <td>Aston Martin V8 Vantage S Coupe (Petrol)</td>\n",
       "      <td>25500</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>Yes</td>\n",
       "      <td>300.0</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>4735.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                      Make                                     Model  \\\n",
       "0      7       Aston Martin Rapide         Aston Martin Rapide LUXE (Petrol)   \n",
       "1      8     Aston Martin Rapide S            Aston Martin Rapide S (Petrol)   \n",
       "2      9  Aston Martin V12 Vantage   Aston Martin V12 Vantage Coupe (Petrol)   \n",
       "3     10   Aston Martin V8 Vantage    Aston Martin V8 Vantage Coupe (Petrol)   \n",
       "4     11   Aston Martin V8 Vantage  Aston Martin V8 Vantage S Coupe (Petrol)   \n",
       "\n",
       "   Price   Type   ABS  BootSpace    GearType AirBag  Engine  FuelCapacity  \\\n",
       "0  35000  Sedan   Yes      300.0   Automatic    Yes  5935.0          90.5   \n",
       "1  44000  Sedan   Yes        NaN   Automatic    Yes  5935.0          90.0   \n",
       "2  35000  Coupe   Yes      300.0   Automatic    Yes  5935.0          80.0   \n",
       "3  13500  Coupe   Yes      300.0   Automatic    Yes  4735.0          80.0   \n",
       "4  25500  Coupe   Yes      300.0   Automatic    Yes  4735.0          80.0   \n",
       "\n",
       "   Mileage  \n",
       "0      7.0  \n",
       "1     11.9  \n",
       "2      9.0  \n",
       "3      5.0  \n",
       "4      5.0  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Automatic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Price</th>\n",
       "      <th>Type</th>\n",
       "      <th>ABS</th>\n",
       "      <th>BootSpace</th>\n",
       "      <th>GearType</th>\n",
       "      <th>AirBag</th>\n",
       "      <th>Engine</th>\n",
       "      <th>FuelCapacity</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LE 8-STR (Diesel)</td>\n",
       "      <td>750</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 8-STR (Diesel)</td>\n",
       "      <td>800</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LX 8-STR (Diesel)</td>\n",
       "      <td>830</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR (Diesel)</td>\n",
       "      <td>850</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Ashok Leyland Stile</td>\n",
       "      <td>Ashok Leyland Stile LS 7-STR Alloy (Diesel)</td>\n",
       "      <td>880</td>\n",
       "      <td>MPV</td>\n",
       "      <td>No</td>\n",
       "      <td>500.0</td>\n",
       "      <td>Manual</td>\n",
       "      <td>No</td>\n",
       "      <td>1461.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                 Make                                        Model  \\\n",
       "0      0  Ashok Leyland Stile        Ashok Leyland Stile LE 8-STR (Diesel)   \n",
       "1      1  Ashok Leyland Stile        Ashok Leyland Stile LS 8-STR (Diesel)   \n",
       "2      2  Ashok Leyland Stile        Ashok Leyland Stile LX 8-STR (Diesel)   \n",
       "3      3  Ashok Leyland Stile        Ashok Leyland Stile LS 7-STR (Diesel)   \n",
       "4      4  Ashok Leyland Stile  Ashok Leyland Stile LS 7-STR Alloy (Diesel)   \n",
       "\n",
       "   Price Type  ABS  BootSpace GearType AirBag  Engine  FuelCapacity  Mileage  \n",
       "0    750  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "1    800  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "2    830  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "3    850  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  \n",
       "4    880  MPV   No      500.0   Manual     No  1461.0          50.0     20.7  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Manual.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(421, 12)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Manual.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(372, 12)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Automatic.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of observations have to be same. For the current exercise, let's take the first 300 observations in both the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cars_Automatic = cars_Automatic.ix[:299,:]\n",
    "cars_Manual = cars_Manual.ix[:299,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300, 12)"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Automatic.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300, 12)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_Manual.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cars_manual_automatic = pd.DataFrame([cars_Automatic.Mileage, cars_Manual.Mileage])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mileage</th>\n",
       "      <td>7.0</td>\n",
       "      <td>11.9</td>\n",
       "      <td>9.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>16.60</td>\n",
       "      <td>17.0</td>\n",
       "      <td>16.55</td>\n",
       "      <td>...</td>\n",
       "      <td>13.1</td>\n",
       "      <td>13.10</td>\n",
       "      <td>17.2</td>\n",
       "      <td>13.7</td>\n",
       "      <td>17.20</td>\n",
       "      <td>14.80</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mileage</th>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>20.7</td>\n",
       "      <td>17.32</td>\n",
       "      <td>13.7</td>\n",
       "      <td>13.70</td>\n",
       "      <td>...</td>\n",
       "      <td>20.5</td>\n",
       "      <td>20.45</td>\n",
       "      <td>20.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>19.01</td>\n",
       "      <td>19.01</td>\n",
       "      <td>19.01</td>\n",
       "      <td>13.05</td>\n",
       "      <td>19.87</td>\n",
       "      <td>19.87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1     2     3     4     5     6      7     8      9    ...    \\\n",
       "Mileage   7.0  11.9   9.0   5.0   5.0   5.0   8.0  16.60  17.0  16.55  ...     \n",
       "Mileage  20.7  20.7  20.7  20.7  20.7  20.7  20.7  17.32  13.7  13.70  ...     \n",
       "\n",
       "          290    291   292   293    294    295    296    297    298    299  \n",
       "Mileage  13.1  13.10  17.2  13.7  17.20  14.80    NaN    NaN    NaN  19.30  \n",
       "Mileage  20.5  20.45  20.5  19.0  19.01  19.01  19.01  13.05  19.87  19.87  \n",
       "\n",
       "[2 rows x 300 columns]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_manual_automatic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cars_manual_automatic = cars_manual_automatic.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mileage</th>\n",
       "      <th>Mileage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11.9</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Mileage  Mileage\n",
       "0      7.0     20.7\n",
       "1     11.9     20.7\n",
       "2      9.0     20.7\n",
       "3      5.0     20.7\n",
       "4      5.0     20.7"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_manual_automatic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cars_manual_automatic.columns = ['Mileage_Automatic', 'Mileage_Manual']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mileage_Automatic</th>\n",
       "      <th>Mileage_Manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11.9</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>20.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Mileage_Automatic  Mileage_Manual\n",
       "0                7.0            20.7\n",
       "1               11.9            20.7\n",
       "2                9.0            20.7\n",
       "3                5.0            20.7\n",
       "4                5.0            20.7"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_manual_automatic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mileage_Automatic</th>\n",
       "      <th>Mileage_Manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mileage_Automatic</th>\n",
       "      <td>22.375515</td>\n",
       "      <td>0.446292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mileage_Manual</th>\n",
       "      <td>0.446292</td>\n",
       "      <td>12.776373</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Mileage_Automatic  Mileage_Manual\n",
       "Mileage_Automatic          22.375515        0.446292\n",
       "Mileage_Manual              0.446292       12.776373"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Co-variance matrix between the mileages of automatic and manual:\n",
    "cars_manual_automatic.cov()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation\n",
    "\n",
    "Extent to which two or more variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases.\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/correlation.gif\" height=\"270\" width=\"270\">\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#### Find the correlation between the mileages of automatic and manual in the above dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mileage_Automatic</th>\n",
       "      <th>Mileage_Manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mileage_Automatic</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.026011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mileage_Manual</th>\n",
       "      <td>0.026011</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Mileage_Automatic  Mileage_Manual\n",
       "Mileage_Automatic           1.000000        0.026011\n",
       "Mileage_Manual              0.026011        1.000000"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_manual_automatic.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cars_manual_automatic.corrwith?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Correlation != Causation\n",
    "\n",
    "correlation between two variables does not necessarily imply that one causes the other.\n",
    "\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/correlation_not_causation.gif\" height=\"570\" width=\"570\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}