PA4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [CPSC 310](https://github.com/GonzagaCPSC310) Data Mining\n",
    "[Gonzaga University](https://www.gonzaga.edu/)\n",
    "\n",
    "[Gina Sprint](http://cs.gonzaga.edu/faculty/sprint/)\n",
    "# PA4 Simple Classifiers (100 pts)\n",
    "\n",
    "## Learner Objectives\n",
    "At the conclusion of this programming assignment, participants should be able to:\n",
    "* Implement a linear regression classifier\n",
    "* Implement a kNN classifier\n",
    "* Evaluate classifiers using train/test sets sampled using\n",
    "    * Random sub-sampling\n",
    "    * Stratified k-fold cross validation\n",
    "\n",
    "## Prerequisites\n",
    "Before starting this programming assignment, participants should be able to:\n",
    "* Calculate a least squares linear regression line\n",
    "\n",
    "## Acknowledgments\n",
    "Content used in this assignment is based upon information in the following sources:\n",
    "* Dr. Shawn Bowers' Data Mining HW3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Github Classroom Setup\n",
    "For this assignment, you will use GitHub Classroom to create a private code repositories to track code changes and submit your assignment. Open this PA4 link to accept the assignment and create a private repository for your assignment in Github classroom: https://classroom.github.com/a/P8AFayKL\n",
    "\n",
    "Your repo, for example, will be named GonzagaCPSC310/pa4-yourusername (where yourusername is your Github username). I highly recommend committing/pushing regularly so your work is always backed up. We will grade your most recent commit, even if that commit is after the due date (your work will be marked late if this is the case)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview and Requirements\n",
    "Write a program (`pa4.py`) that builds simple classifiers for the \"pre-processed\" automobile dataset (auto-data.txt) you created for PA1. Pick one method to deal with missing values for this assignment (e.g., eliminate rows with missing values, use means or medians, etc.).\n",
    "\n",
    "For this assignment you will need to perform the following steps and turn in your source code and a log of any assumptions and/or issues you had in doing the steps. Your log needs to be written separately from your .py file and may be written in a .txt or a .md (markdown) file.\n",
    "\n",
    "Note: as you write solutions for the following steps, I highly encourage you to design functions that are generic and re-usable for future programming assignments and data mining tasks.\n",
    "\n",
    "Note: we are learning data mining from scratch! The only libraries you should need to use for this assignment include numpy (sparingly), csv (if you'd like to use it for file I/O), and tabulate (if you'd like to use it for pretty printing). This means no pandas..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 Random Instances: Linear Regression\n",
    "Create a classifier that predicts mpg values using (least squares) linear regression based on vehicle\n",
    "weight. Your classifier should take one or more instances, compute the predicted MPG values, and then\n",
    "map these to the DOE classification/ranking (given in PA2) for each corresponding instance. Test your\n",
    "classifier by selecting 5 random instances (via your script) from the dataset, predict their corresponding mpg ranking, and then show their actual mpg ranking as follows:\n",
    "\n",
    "```\n",
    "===========================================\n",
    "STEP 1: Linear Regression MPG Classifier\n",
    "===========================================\n",
    "instance: 15.0, 8, 400.0, 150.0, 3761, 9.5, 70, 1, chevrolet monte carlo, 3123\n",
    "class: 4, actual: 3\n",
    "instance: 17.0, 8, 302.0, 140.0, 3449, 10.5, 70, 1, ford torino, 2778\n",
    "class: 5, actual: 4\n",
    "instance: 28.4, 4, 151.0, 90.00, 2670, 16.0, 79, 1, buick skylark limited, 4462\n",
    "class: 6, actual: 7\n",
    "instance: 20.0, 6, 232.0, 100.0, 2914, 16.0, 75, 1, amc gremlin, 2798\n",
    "class: 5, actual: 5\n",
    "instance: 16.2, 6, 163.0, 133.0, 3410, 15.8, 78, 2, peugeot 604sl, 10990\n",
    "class: 5, actual: 4\n",
    "```\n",
    "\n",
    "Note: you should run your program enough times to check that the approach is working correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 Random Instances: kNN\n",
    "Create a nearest neighbor classifier for mpg that uses the number of cylinders, weight, and acceleration\n",
    "attributes to predict mpg for k = 5. Be sure to normalize the three attribute values and also use the Euclidean\n",
    "distance metric. Similar to Step 1, test your classifier by selecting random instances from the dataset, predict\n",
    "their corresponding mpg ranking, and then show their actual mpg ranking:\n",
    "\n",
    "```\n",
    "===========================================\n",
    "STEP 2: k=5 Nearest Neighbor MPG Classifier\n",
    "===========================================\n",
    "instance: 15.0, 8, 400.0, 150.0, 3761, 9.5, 70, 1, chevrolet monte carlo, 3123\n",
    "class: 7, actual: 3\n",
    "instance: 17.0, 8, 302.0, 140.0, 3449, 10.5, 70, 1, ford torino, 2778\n",
    "class: 7, actual: 4\n",
    "instance: 28.4, 4, 151.0, 90.00, 2670, 16.0, 79, 1, buick skylark limited, 4462\n",
    "class: 1, actual: 7\n",
    "instance: 20.0, 6, 232.0, 100.0, 2914, 16.0, 75, 1, amc gremlin, 2798\n",
    "class: 1, actual: 5\n",
    "instance: 16.2, 6, 163.0, 133.0, 3410, 15.8, 78, 2, peugeot 604sl, 10990\n",
    "class: 7, actual: 4\n",
    "```\n",
    "\n",
    "Note: make sure a test instance can't be its own nearest neighbor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 Train/Test Sets: Random Sub-sampling\n",
    "Compute the (multi-class) predictive accuracy and error rate of the two classifiers using separate training and test sets. You should use use random sub-sampling with k = 10. Your output should look something like this (where the ??'s should be replaced by actual values):\n",
    "\n",
    "```\n",
    "===========================================\n",
    "STEP 3: Predictive Accuracy\n",
    "===========================================\n",
    "Random Subsample (k=10, 2:1 Train/Test)\n",
    "Linear Regression: accuracy = 0.??, error rate = 0.??\n",
    "k Nearest Neighbors: accuracy = 0.??, error rate = 0.??\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 Train/Test Sets: Stratified Cross Validation\n",
    "Compute the (multi-class) predictive accuracy and error rate of the two classifiers using separate training and test sets. You should use mpg rankings for stratified k-fold cross validation with k = 10. Your output should look something like this (where the ??'s should be replaced by actual values):\n",
    "\n",
    "```\n",
    "===========================================\n",
    "STEP 4: Predictive Accuracy\n",
    "===========================================\n",
    "Stratified 10-Fold Cross Validation\n",
    "Linear Regression: accuracy = 0.??, error rate = 0.??\n",
    "k Nearest Neighbors: accuracy = 0.??, error rate = 0.??\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 Confusion Matrices\n",
    "Create confusion matrices for each classifier based on the stratified 10-fold cross validation results.\n",
    "You can use the tabulate package to display your confusion matrices (it is also okay to format the table\n",
    "manually). Here is an example:\n",
    "\n",
    "```\n",
    "===========================================\n",
    "STEP 5: Confusion Matrices\n",
    "===========================================\n",
    "```\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/GonzagaCPSC310/PAs/master/figures/stratified_mpg_confusion_matrix.png\" width=\"700\">\n",
    "\n",
    "Note: your output should have columns properly aligned for readability. I took a screen shot for the table so you could see the alignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting Assignments\n",
    "1. Use Github classroom to submit your assignment via a Github repo. See the \"Github Classroom Setup\" section at the beginning of this document for details on how to do this. You must commit your solution by the due date and time.\n",
    "1. Your repo should contain only your .py file(s), your input .csv/.txt file(s), and your log file (.txt or .md) file(s). Double check that this is the case by cloning (or downloading a zip) your submission repo and running your code from command line like we will when we grade your code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grading Guidelines\n",
    "This assignment is worth 100 points. Your assignment will be evaluated based on a successful compilation from command line (using the Anaconda Python Distribution v3.7) and adherence to the program requirements. We will grade according to the following criteria:\n",
    "* 15 pts for correct step 1\n",
    "* 20 pts for correct step 2\n",
    "* 20 pts for correct step 3\n",
    "* 20 pts for correct step 4\n",
    "* 15 pts for correct step 5\n",
    "* 2 pts for quantity and quality of Github commit messages\n",
    "* 8 pts for adherence to course [coding standard](https://nbviewer.jupyter.org/github/GonzagaCPSC310/PAs/blob/master/Coding%20Standard.ipynb)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}