Copy_of_Copy_of_Copy_of_fashion_mnist_keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Copy of Copy of fashion_mnist_keras.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLMRPLVCFwEc",
        "colab_type": "text"
      },
      "source": [
        "## Learning CNNs using Fashion-MNIST\n",
        "\n",
        "This assignment will bulld CNNS using the Fashion-MINST dataset. In particular, we will\n",
        "*   Use MNIST Fashion to study CNN architectures and parameters.\n",
        "\n",
        "Read more about the Fashion-MINST dataset in this paper [here](https://arxiv.org/abs/1708.07747) (**Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms**)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ixyte299ZZgk",
        "colab_type": "text"
      },
      "source": [
        "## Assignment Overview\n",
        "\n",
        "\n",
        "This assignment focuses on how to classify **fashion_mnist** data with a simple **Convolutional Neural Network** in Keras. \n",
        "Keras is part of the core TensorFlow library, in addition to being an independent open source project. \n",
        "\n",
        "The [fashion_mnist](https://github.com/zalandoresearch/fashion-mnist) dataset consists of \n",
        "60,000 train and 10,000 test instances. The aim is to classify an image into one of 10 categories. The possible categories/labals are:\n",
        "\n",
        "<br> **Label**\t**Description**\n",
        "<br> 0 T-shirt/top\n",
        "<br> 1 Trouser\n",
        "<br> 2 Pullover\n",
        "<br> 3 Dress\n",
        "<br> 4 Coat\n",
        "<br> 5 Sandal\n",
        "<br> 6 Shirt\n",
        "<br> 7 Sneaker\n",
        "<br> 8 Bag\n",
        "<br> 9 Ankle boot\n",
        "\n",
        "Each gray-scale input is a 28x28 low-resolution image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jQGPl2l7kF7",
        "colab_type": "toc"
      },
      "source": [
        ">>[Learning CNNs using Fashion-MNIST](#scrollTo=XLMRPLVCFwEc)\n",
        "\n",
        ">>[Notebook Overview](#scrollTo=Ixyte299ZZgk)\n",
        "\n",
        ">>[Download the fashion_mnist data](#scrollTo=LbCigZtNZZgl)\n",
        "\n",
        ">>[Visualize the data](#scrollTo=tWORMSC8FDR4)\n",
        "\n",
        ">>[Data normalization](#scrollTo=Zx-Ee6LHZZgt)\n",
        "\n",
        ">>[Split the data into train/validation/test data sets](#scrollTo=CFlNHktHBtru)\n",
        "\n",
        ">>[Create the basic model architecture](#scrollTo=HhalcO03ZZg3)\n",
        "\n",
        ">>[Compile the model](#scrollTo=FhxJ5dinZZg8)\n",
        "\n",
        ">>[Train the model](#scrollTo=DtOvh3YVZZg_)\n",
        "\n",
        ">>[Load Model with the best validation accuracy](#scrollTo=e-MGLwZQy05d)\n",
        "\n",
        ">>[Test Accuracy](#scrollTo=9RTRkan4yq5H)\n",
        "\n",
        ">>[Visualize prediction](#scrollTo=oJv7XEk10bOv)\n",
        "\n",
        ">>[Congragulations!](#scrollTo=8AehWdRAVKN5)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LbCigZtNZZgl",
        "colab_type": "text"
      },
      "source": [
        "## Download the fashion_mnist data\n",
        "First let's install TensorFlow version 1.8.0 and import Tensorflow. Then we download fashion-mnist which is one of the Keras datasets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d44TznbgZZgm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q -U tensorflow>=1.8.0\n",
        "import tensorflow as tf\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Load the fashion-mnist pre-shuffled train data and test data\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()\n",
        "\n",
        "print(\"x_train shape:\", x_train.shape, \"y_train shape:\", y_train.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWORMSC8FDR4",
        "colab_type": "text"
      },
      "source": [
        "## Visualize the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aFe4wHGRFKle",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Print training set shape - note there are 60,000 training data of image size of 28x28, 60,000 train labels)\n",
        "print(\"x_train shape:\", x_train.shape, \"y_train shape:\", y_train.shape)\n",
        "\n",
        "# Print the number of training and test datasets\n",
        "print(x_train.shape[0], 'train set')\n",
        "print(x_test.shape[0], 'test set')\n",
        "\n",
        "# Define the text labels\n",
        "fashion_mnist_labels = [\"T-shirt/top\",  # index 0\n",
        "                        \"Trouser\",      # index 1\n",
        "                        \"Pullover\",     # index 2 \n",
        "                        \"Dress\",        # index 3 \n",
        "                        \"Coat\",         # index 4\n",
        "                        \"Sandal\",       # index 5\n",
        "                        \"Shirt\",        # index 6 \n",
        "                        \"Sneaker\",      # index 7 \n",
        "                        \"Bag\",          # index 8 \n",
        "                        \"Ankle boot\"]   # index 9\n",
        "\n",
        "# Image index, you can pick any number between 0 and 59,999\n",
        "img_index = 5\n",
        "# y_train contains the lables, ranging from 0 to 9\n",
        "label_index = y_train[img_index]\n",
        "# Print the label, for example 2 Pullover\n",
        "print (\"y = \" + str(label_index) + \" \" +(fashion_mnist_labels[label_index]))\n",
        "# # Show one of the images from the training dataset\n",
        "plt.imshow(x_train[img_index])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zx-Ee6LHZZgt",
        "colab_type": "text"
      },
      "source": [
        "## Data normalization\n",
        "Normalize the data dimensions so that they are of approximately the same scale."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XNh5NIckZZgu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train = x_train.astype('float32') / 255\n",
        "x_test = x_test.astype('float32') / 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LMSg53fiZZgx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Number of train data - \" + str(len(x_train)))\n",
        "print(\"Number of test data - \" + str(len(x_test)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFlNHktHBtru",
        "colab_type": "text"
      },
      "source": [
        "## Split the data into train/validation/test data sets\n",
        "\n",
        "\n",
        "*   Training data - used for training the model\n",
        "*   Validation data - used for tuning the hyperparameters and evaluate the models\n",
        "*   Test data - used to test the model after the model has gone through initial vetting by the validation set.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ShU787gZZg0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Further break training data into train / validation sets (# put 5000 into validation set and keep remaining 55,000 for train)\n",
        "(x_train, x_valid) = x_train[5000:], x_train[:5000] \n",
        "(y_train, y_valid) = y_train[5000:], y_train[:5000]\n",
        "\n",
        "# Reshape input data from (28, 28) to (28, 28, 1)\n",
        "w, h = 28, 28\n",
        "x_train = x_train.reshape(x_train.shape[0], w, h, 1)\n",
        "x_valid = x_valid.reshape(x_valid.shape[0], w, h, 1)\n",
        "x_test = x_test.reshape(x_test.shape[0], w, h, 1)\n",
        "\n",
        "# One-hot encode the labels\n",
        "y_train = tf.keras.utils.to_categorical(y_train, 10)\n",
        "y_valid = tf.keras.utils.to_categorical(y_valid, 10)\n",
        "y_test = tf.keras.utils.to_categorical(y_test, 10)\n",
        "\n",
        "# Print training set shape\n",
        "print(\"x_train shape:\", x_train.shape, \"y_train shape:\", y_train.shape)\n",
        "\n",
        "# Print the number of training, validation, and test datasets\n",
        "print(x_train.shape[0], 'train set')\n",
        "print(x_valid.shape[0], 'validation set')\n",
        "print(x_test.shape[0], 'test set')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HhalcO03ZZg3",
        "colab_type": "text"
      },
      "source": [
        "## Create the basic model architecture\n",
        "\n",
        "There are two APIs for defining a model in Keras:\n",
        "1. [Sequential model API](https://keras.io/models/sequential/)\n",
        "2. [Functional API](https://keras.io/models/model/)\n",
        "\n",
        "In this assignment we are using the Sequential model API. \n",
        "If you are interested in a tutorial using the Functional API, checkout Sara Robinson's blog [Predicting the price of wine with the Keras Functional API and TensorFlow](https://medium.com/tensorflow/predicting-the-price-of-wine-with-the-keras-functional-api-and-tensorflow-a95d1c2c1b03).\n",
        "\n",
        "In defining the model we will be using some of these Keras APIs:\n",
        "*   Conv2D() [link text](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D/) - create a convolutional layer \n",
        "*   Pooling() [link text](https://keras.io/layers/pooling/) - create a pooling layer \n",
        "*   Dropout() [link text](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) - apply drop out \n",
        "\n",
        "In the basic architecture we create two convolutional layers, and then an inference layer. \n",
        "\n",
        "The key parameters for a convolution layer are: \n",
        "for convolution--(a) number of convolution filters; (b) kernel size;\n",
        "for pooling--kernel for pooling-size;\n",
        "for dropout--dropout probability. We will refer to these as the convolution-layer parameters.\n",
        "\n",
        "The inference-layer parameters are:\n",
        "(a) hidden layer vector-size and activation;\n",
        "(b) dropout probability;\n",
        "(c) output-layer activation.\n",
        "\n",
        "When we design a network we can specify the CNN model using these parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QgTZ47SsZZg4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential()\n",
        "\n",
        "# Must define the input shape in the first layer of the neural network\n",
        "model.add(tf.keras.layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1))) \n",
        "model.add(tf.keras.layers.MaxPooling2D(pool_size=2))\n",
        "model.add(tf.keras.layers.Dropout(0.3))\n",
        "\n",
        "model.add(tf.keras.layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))\n",
        "model.add(tf.keras.layers.MaxPooling2D(pool_size=2))\n",
        "model.add(tf.keras.layers.Dropout(0.3))\n",
        "\n",
        "model.add(tf.keras.layers.Flatten())\n",
        "model.add(tf.keras.layers.Dense(256, activation='relu'))\n",
        "model.add(tf.keras.layers.Dropout(0.5))\n",
        "model.add(tf.keras.layers.Dense(10, activation='softmax'))\n",
        "\n",
        "# Take a look at the model summary\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhxJ5dinZZg8",
        "colab_type": "text"
      },
      "source": [
        "## Compile the model\n",
        "Configure the learning process with compile() API before training the model. It receives three arguments:\n",
        "\n",
        "*   An optimizer \n",
        "*   A loss function \n",
        "*   A list of metrics \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CQUlOa8cZZg9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "             optimizer='adam',\n",
        "             metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DtOvh3YVZZg_",
        "colab_type": "text"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "Now let's train the model with fit() API.\n",
        "\n",
        "We use  the [ModelCheckpoint](https://keras.io/callbacks/#modelcheckpoint) API to save the model after every epoch. Set \"save_best_only = True\" to save only when the validation accuracy improves.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZTmapAttZZhA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import ModelCheckpoint\n",
        "\n",
        "checkpointer = ModelCheckpoint(filepath='model.weights.best.hdf5', verbose = 1, save_best_only=True)\n",
        "model.fit(x_train,\n",
        "         y_train,\n",
        "         batch_size=64,\n",
        "         epochs=10,\n",
        "         validation_data=(x_valid, y_valid),\n",
        "         callbacks=[checkpointer])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-MGLwZQy05d",
        "colab_type": "text"
      },
      "source": [
        "## Load Model with the best validation accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UD1tecxUZZhE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load the weights with the best validation accuracy\n",
        "model.load_weights('model.weights.best.hdf5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9RTRkan4yq5H",
        "colab_type": "text"
      },
      "source": [
        "## Test Accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VZtqBqFFy62R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Evaluate the model on test set\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "\n",
        "# Print test accuracy\n",
        "print('\\n', 'Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJv7XEk10bOv",
        "colab_type": "text"
      },
      "source": [
        "## Visualize prediction\n",
        "Now let's visualize the prediction using the model you just trained. \n",
        "First we get the predictions with the model from the test data.\n",
        "Then we print out 15 images from the test data set, and set the titles with the prediction (and the groud truth label).\n",
        "If the prediction matches the true label, the title will be green; otherwise it's displayed in red."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QwNmlfIC0YxM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_hat = model.predict(x_test)\n",
        "\n",
        "# Plot a random sample of 10 test images, their predicted labels and ground truth\n",
        "figure = plt.figure(figsize=(20, 8))\n",
        "for i, index in enumerate(np.random.choice(x_test.shape[0], size=15, replace=False)):\n",
        "    ax = figure.add_subplot(3, 5, i + 1, xticks=[], yticks=[])\n",
        "    # Display each image\n",
        "    ax.imshow(np.squeeze(x_test[index]))\n",
        "    predict_index = np.argmax(y_hat[index])\n",
        "    true_index = np.argmax(y_test[index])\n",
        "    # Set the title for each image\n",
        "    ax.set_title(\"{} ({})\".format(fashion_mnist_labels[predict_index], \n",
        "                                  fashion_mnist_labels[true_index]),\n",
        "                                  color=(\"green\" if predict_index == true_index else \"red\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C2ZWPOMglvmA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PI00Mflfl0pI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AehWdRAVKN5",
        "colab_type": "text"
      },
      "source": [
        "## Task 1: Architecture Comparison\n",
        "The code given above is called the baseline architecture, and will successfully train a CNN to classify fashion-MNIST with near 90% accuracy.\n",
        "We now want to compare other different architectures.\n",
        "\n",
        "Each group must run 10 additional architectures:\n",
        "\n",
        "\n",
        "*   2 different combination of number of filters\n",
        "*   2 different combinations of kernels for (convolution, pooling)\n",
        "*   2 different dropout probability values\n",
        "*   1 new architecture with more layers\n",
        "*   3 experiments with combined changes\n",
        "\n",
        "Report the results in a table with the first column describing the architecture and the second column the predictive accuracy of the model on the data.\n",
        "\n",
        "Describe how the changes to architecture influence the change in accuracy, if there is any significant change. Specifically, describe:\n",
        "* impact of number of filters\n",
        "* impact of kernel size\n",
        "* impact of dropout probability\n",
        "* impact of number of layers\n",
        "* impact of combining multiple changes.\n",
        "\n",
        "## Task 2: TensorFlow coding of model\n",
        "The second main task is to rewrite the high-level Keras code for the model using TensorFlow. The necessary code fragments are noted below. Please fill in the outstanding code and replace the Keras model with your TensorFlow model. Then compile the code to run the basic model configuration (baseline model).\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n7Pr5WsFJTix",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cnn_model():\n",
        "  #def cnn_model(features, labels, mode) is more complex than is needed for this assignment",
        "    #Reshapinng the input\n",
        "    input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n",
        "    \n",
        "# TO DO: WRITE A FUNCTION TO INPUT A CONVOLUTION LAYER,\n",
        "#   denoting inputs=input_layer,filters, kernel_size, padding=\"same\",\n",
        "#        activation\n",
        "\n",
        "# TO DO: WRITE A FUNCTION TO INPUT A POOLING LAYER, \n",
        "#    denoting inputs pool_size, strides\n",
        "\n",
        "# DEFINE THE MODEL USING THE CODE YOU JUST WROTE\n",
        "\n",
        "     # Convolutional Layer #1 and Pooling Layer #1\n",
        "    conv1 = \n",
        "    \n",
        "    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
        "    \n",
        "    # Convolutional Layer #2 and Pooling Layer #2\n",
        "\n",
        "    \n",
        "    dropout_1 = tf.layers.dropout(inputs=pool2, rate=0.25,training=mode == tf.estimator.ModeKeys.TRAIN )\n",
        "    \n",
        "    # Convolutional Layer #2 Pooling Layer #2   Dropout #2\n",
        "\n",
        "    # TO DO: IMPLEMENT CODE FOR THE DECISION AND OUTPUT LAYERS\n",
        "\n",
        " # Take a look at the model summary\n",
        "model.summary()\n",
        "\n",
        "  #  make sure the model matches the baseline model  \n",
        "    \n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}