Data_Training_And_Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Training\n",
    "\n",
    "## Detection Model\n",
    "\n",
    "The classifier used is support vector machines. The classifier is trained on the HOG descriptors of the image and not on the image directly. This helps in identifying the distinct shapes of the traffic signs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split,GridSearchCV,StratifiedKFold\n",
    "from sklearn import svm\n",
    "from sklearn.decomposition import PCA\n",
    "from skimage.feature import hog\n",
    "import numpy as np\n",
    "import cv2\n",
    "from sklearn.externals import joblib\n",
    "import random\n",
    "from sklearn.metrics import precision_score,recall_score,f1_score,classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper functions for preprocessing data\n",
    "\n",
    "**extract_hog(image)** function extracts HOG features from image. The HOG features are extracted with the parameters of 8x8 pixel cells and 1 cell per block. There are 8 directions for the gradients as mentioned by the parameter orientations.\n",
    "\n",
    "The images are then converted to grayscale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_hog(image):\n",
    "    return hog(image, orientations=8, pixels_per_cell=(8, 8),\n",
    "                    cells_per_block=(1, 1),visualise=False)\n",
    "\n",
    "def convert_grayscale(image):\n",
    "    return cv2.cvtColor(image,cv2.COLOR_RGB2GRAY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Preprocessing\n",
    "\n",
    "The images are loaded from the data files. The images are resized to a uniform 32x32. and converted to grayscale. The HOG features are extracted from all the images and stored in **X_hog**. Due to imbalance in the data, additional positive samples are created from recognition dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8000, 128)\n"
     ]
    }
   ],
   "source": [
    "X = np.load('storage/detect_data_file.npy')\n",
    "y = np.load('storage/detect_labels_file.npy')\n",
    "X,test_x = X[:6000],X[6000:]\n",
    "y,test_y = y[:6000],y[6000:]\n",
    "\n",
    "X_extra = np.load('storage/data_file.npy')\n",
    "X_extra_format = []\n",
    "\n",
    "for i in xrange(2000):\n",
    "    index = random.randint(0,X_extra.shape[0])\n",
    "    image = cv2.cvtColor(X_extra[index],cv2.COLOR_RGB2GRAY)\n",
    "    image = cv2.resize(image,(32,32))\n",
    "    X_extra_format.append(image)\n",
    "    y = np.append(y,1)\n",
    "\n",
    "\n",
    "X = [convert_grayscale(i) for i in X]\n",
    "X = X+X_extra_format\n",
    "X = [extract_hog(i) for i in X]\n",
    "y = np.array(y)\n",
    "test_x = [convert_grayscale(i) for i in test_x]\n",
    "test_x = np.array([extract_hog(i) for i in test_x])\n",
    "X_hog = np.array(X)\n",
    "print X_hog.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the classifier\n",
    "An SVM is trained using Gridsearch to obtain the best parameters for the classifier.\n",
    "Crossvalidation is done using stratified Kfold because of the imbalance of the positive and negative examples. The SVM is trained with **probability** equal to true, as probability of prediction is needed in searching for best prediction in the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detection Training Score: \n",
      "0.990125\n"
     ]
    }
   ],
   "source": [
    "\n",
    "clf = svm.SVC(random_state=42,probability=True)\n",
    "cv = StratifiedKFold(n_splits=5)\n",
    "param_grid = {'kernel':['rbf','linear'],'C':[0.1,0.5,0.001,1]}\n",
    "grid = GridSearchCV(clf,param_grid,cv=cv)\n",
    "grid.fit(X_hog,y)\n",
    "best_clf = grid.best_estimator_\n",
    "print 'Detection Training Score: '\n",
    "print grid.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving the trained SVM classifier in a .sav file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier Saved Successfully\n"
     ]
    }
   ],
   "source": [
    "clf_filename = 'storage/clf.sav'\n",
    "joblib.dump(best_clf, clf_filename)\n",
    "print 'Classifier Saved Successfully'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detection Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Recall Score: 0.985875706215\n",
      "\n",
      "Precision Score: 0.961432506887\n",
      "\n",
      "F1 Score: 0.97350069735\n",
      "\n",
      "Classification Report:\n",
      "\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.99      1.00      2130\n",
      "          1       0.96      0.99      0.97       354\n",
      "\n",
      "avg / total       0.99      0.99      0.99      2484\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions = best_clf.predict(test_x)\n",
    "\n",
    "print '\\nRecall Score: ' + str(recall_score(test_y,predictions))\n",
    "\n",
    "print '\\nPrecision Score: '+ str(precision_score(test_y,predictions))\n",
    "\n",
    "print '\\nF1 Score: '+ str(f1_score(test_y,predictions))\n",
    "\n",
    "print '\\nClassification Report:\\n'\n",
    "print classification_report(test_y,predictions)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recognition Model\n",
    "\n",
    "A multi-scale convolutional neural network is used for recognition of the traffic signs.\n",
    "\n",
    "The CNN has a total of 2 convolutional stages. Each stage has two convolutional filters. The first has 32 filters while the second has 64 filters. Unlike traditional CNN's, the output of the 1st stage is branched out and fed to the classifier in addition to being inputs for the 2nd stage. The branched 1st-stage outputs are then subsampled once more so that it undergoes the same amount of subsampling (4x4) as the 2nd stage outputs.\n",
    "\n",
    "The final classifier comprises of three layers:\n",
    "1. Hidden fully connected layer with 256 neurons.\n",
    "2. Dropout layer to avoid overfitting\n",
    "3. Fully connected output layer with 43 neurons (Because there are 43 categories of traffic signs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "import numpy\n",
    "\n",
    "from keras.layers import Dense, Dropout, Flatten, merge\n",
    "from keras.layers import Convolution2D, MaxPooling2D\n",
    "from keras.utils import np_utils\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "from keras.optimizers import SGD\n",
    "from keras.models import model_from_json\n",
    "\n",
    "import time\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Data Shape: \n",
      "(39209, 32, 32)\n",
      "Testing Data Shape: \n",
      "(12630, 32, 32)\n"
     ]
    }
   ],
   "source": [
    "testing_labels = 'datasets/carnd_dataset/test.p'\n",
    "\n",
    "with open(testing_labels, mode='rb') as f:\n",
    "    test = pickle.load(f)\n",
    "\n",
    "X_train = numpy.load('storage/data_file.npy')\n",
    "y_train = numpy.load('storage/labels_file.npy')\n",
    "X_test = numpy.load('storage/test_data_file.npy')\n",
    "y_test = test['labels']\n",
    "\n",
    "X_train = numpy.array([convert_grayscale(i) for i in X_train])\n",
    "X_test = numpy.array([convert_grayscale(i) for i in X_test])\n",
    "\n",
    "\n",
    "print 'Training Data Shape: '\n",
    "print X_train.shape\n",
    "print 'Testing Data Shape: '\n",
    "print X_test.shape\n",
    "X_test = X_test.reshape(X_test.shape[0],1,32,32)\n",
    "Y_test = np_utils.to_categorical(y_test,43)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(39209, 2)\n",
      "Shuffling of data Done\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train.reshape(X_train.shape[0],1,32,32)\n",
    "X_test = X_test.reshape(X_test.shape[0],1,32,32)\n",
    "Y_train = np_utils.to_categorical(y_train,43)\n",
    "Y_test = np_utils.to_categorical(y_test,43)\n",
    "\n",
    "shuffle_data = [(i,j) for i,j in zip(X_train,Y_train)]\n",
    "\n",
    "random.shuffle(shuffle_data)\n",
    "X = []\n",
    "y = []\n",
    "shuffle_data = numpy.array(shuffle_data)\n",
    "print shuffle_data.shape\n",
    "for i in xrange(len(shuffle_data)):\n",
    "    X.append(shuffle_data[i][0])\n",
    "    y.append(shuffle_data[i][1])\n",
    "\n",
    "X_train = numpy.array(X)\n",
    "Y_train = numpy.array(y)\n",
    "\n",
    "print 'Shuffling of data Done'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 31367 samples, validate on 7842 samples\n",
      "Epoch 1/30\n",
      "158s - loss: 2.6437 - categorical_accuracy: 0.3334 - val_loss: 1.2158 - val_categorical_accuracy: 0.7335\n",
      "Epoch 2/30\n",
      "156s - loss: 0.9633 - categorical_accuracy: 0.7697 - val_loss: 0.5565 - val_categorical_accuracy: 0.8870\n",
      "Epoch 3/30\n",
      "158s - loss: 0.5547 - categorical_accuracy: 0.8765 - val_loss: 0.3287 - val_categorical_accuracy: 0.9480\n",
      "Epoch 4/30\n",
      "159s - loss: 0.3764 - categorical_accuracy: 0.9228 - val_loss: 0.2476 - val_categorical_accuracy: 0.9651\n",
      "Epoch 5/30\n",
      "159s - loss: 0.2864 - categorical_accuracy: 0.9447 - val_loss: 0.1732 - val_categorical_accuracy: 0.9737\n",
      "Epoch 6/30\n",
      "558s - loss: 0.2298 - categorical_accuracy: 0.9546 - val_loss: 0.1444 - val_categorical_accuracy: 0.9769\n",
      "Epoch 7/30\n",
      "160s - loss: 0.1936 - categorical_accuracy: 0.9633 - val_loss: 0.1226 - val_categorical_accuracy: 0.9784\n",
      "Epoch 8/30\n",
      "161s - loss: 0.1639 - categorical_accuracy: 0.9700 - val_loss: 0.0987 - val_categorical_accuracy: 0.9839\n",
      "Epoch 9/30\n",
      "161s - loss: 0.1415 - categorical_accuracy: 0.9746 - val_loss: 0.0924 - val_categorical_accuracy: 0.9846\n",
      "Epoch 10/30\n",
      "161s - loss: 0.1276 - categorical_accuracy: 0.9770 - val_loss: 0.0810 - val_categorical_accuracy: 0.9866\n",
      "Epoch 11/30\n",
      "162s - loss: 0.1150 - categorical_accuracy: 0.9795 - val_loss: 0.0770 - val_categorical_accuracy: 0.9862\n",
      "Epoch 12/30\n",
      "769s - loss: 0.1028 - categorical_accuracy: 0.9816 - val_loss: 0.0705 - val_categorical_accuracy: 0.9876\n",
      "Epoch 13/30\n",
      "162s - loss: 0.0957 - categorical_accuracy: 0.9819 - val_loss: 0.0647 - val_categorical_accuracy: 0.9870\n",
      "Epoch 14/30\n",
      "162s - loss: 0.0855 - categorical_accuracy: 0.9850 - val_loss: 0.0581 - val_categorical_accuracy: 0.9894\n",
      "Epoch 15/30\n",
      "164s - loss: 0.0786 - categorical_accuracy: 0.9864 - val_loss: 0.0566 - val_categorical_accuracy: 0.9901\n",
      "Epoch 16/30\n",
      "163s - loss: 0.0742 - categorical_accuracy: 0.9865 - val_loss: 0.0538 - val_categorical_accuracy: 0.9898\n",
      "Epoch 17/30\n",
      "191s - loss: 0.0673 - categorical_accuracy: 0.9887 - val_loss: 0.0519 - val_categorical_accuracy: 0.9902\n",
      "Epoch 18/30\n",
      "187s - loss: 0.0663 - categorical_accuracy: 0.9886 - val_loss: 0.0478 - val_categorical_accuracy: 0.9904\n",
      "Epoch 19/30\n",
      "181s - loss: 0.0599 - categorical_accuracy: 0.9903 - val_loss: 0.0456 - val_categorical_accuracy: 0.9912\n",
      "Epoch 20/30\n",
      "182s - loss: 0.0559 - categorical_accuracy: 0.9905 - val_loss: 0.0459 - val_categorical_accuracy: 0.9909\n",
      "Epoch 21/30\n",
      "181s - loss: 0.0529 - categorical_accuracy: 0.9903 - val_loss: 0.0438 - val_categorical_accuracy: 0.9909\n",
      "Epoch 22/30\n",
      "182s - loss: 0.0504 - categorical_accuracy: 0.9918 - val_loss: 0.0429 - val_categorical_accuracy: 0.9909\n",
      "Epoch 23/30\n",
      "182s - loss: 0.0490 - categorical_accuracy: 0.9922 - val_loss: 0.0486 - val_categorical_accuracy: 0.9899\n",
      "Epoch 24/30\n",
      "182s - loss: 0.0463 - categorical_accuracy: 0.9921 - val_loss: 0.0417 - val_categorical_accuracy: 0.9917\n",
      "Epoch 25/30\n",
      "182s - loss: 0.0434 - categorical_accuracy: 0.9928 - val_loss: 0.0397 - val_categorical_accuracy: 0.9923\n",
      "Epoch 26/30\n",
      "189s - loss: 0.0431 - categorical_accuracy: 0.9924 - val_loss: 0.0391 - val_categorical_accuracy: 0.9925\n",
      "Epoch 27/30\n",
      "184s - loss: 0.0406 - categorical_accuracy: 0.9928 - val_loss: 0.0371 - val_categorical_accuracy: 0.9930\n",
      "Epoch 28/30\n",
      "183s - loss: 0.0396 - categorical_accuracy: 0.9928 - val_loss: 0.0373 - val_categorical_accuracy: 0.9917\n",
      "Epoch 29/30\n",
      "183s - loss: 0.0372 - categorical_accuracy: 0.9930 - val_loss: 0.0361 - val_categorical_accuracy: 0.9923\n",
      "Epoch 30/30\n",
      "182s - loss: 0.0347 - categorical_accuracy: 0.9942 - val_loss: 0.0346 - val_categorical_accuracy: 0.9931\n"
     ]
    }
   ],
   "source": [
    "inputs = Input(shape=(1,32,32))\n",
    "\n",
    "first_layer = Convolution2D(32,3,3,activation='relu')(inputs)\n",
    "first_layer = Convolution2D(32,3,3,activation='relu')(first_layer)\n",
    "\n",
    "first_p_layer = MaxPooling2D(pool_size=(2, 2))(first_layer)\n",
    "drop_1 = Dropout(0.2)(first_p_layer)\n",
    "\n",
    "second_p_layer = MaxPooling2D(pool_size=(2, 2))(drop_1)\n",
    "\n",
    "first_input_layer = Flatten()(second_p_layer)\n",
    "\n",
    "second_layer = Convolution2D(64,3,3,activation='relu')(drop_1)\n",
    "second_layer = Convolution2D(64,3,3,activation='relu')(second_layer)\n",
    "\n",
    "third_p_layer = MaxPooling2D(pool_size=(2, 2))(second_layer)\n",
    "drop_2 = Dropout(0.2)(third_p_layer)\n",
    "\n",
    "second_input_layer = Flatten()(drop_2)\n",
    "\n",
    "input_layer = merge([first_input_layer,second_input_layer],mode='concat',concat_axis=1)\n",
    "hidden_layer = Dense(256,activation='sigmoid')(input_layer)\n",
    "drop = Dropout(0.5)(hidden_layer)\n",
    "predictions = Dense(43,activation='softmax')(drop)\n",
    "\n",
    "model = Model(input=inputs,output=predictions)\n",
    "\n",
    "sgd = SGD(lr=0.01,decay=1e-6)\n",
    "\n",
    "model.compile(optimizer=sgd,\n",
    "                metrics=['categorical_accuracy'],\n",
    "                loss='categorical_crossentropy')\n",
    "\n",
    "\n",
    "\n",
    "model.fit(X_train,Y_train,nb_epoch=30,batch_size=32,verbose=2,validation_split=0.2)\n",
    "time.sleep(0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving...\n",
      "Saved model to disk\n"
     ]
    }
   ],
   "source": [
    "print \"Saving...\"\n",
    "model_json = model.to_json()\n",
    "with open(\"storage/model.json\", \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "model.save_weights(\"storage/model.h5\")\n",
    "print(\"Saved model to disk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recognition Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded model from disk\n",
      "\n",
      "Testing: \n",
      "12630/12630 [==============================] - 21s    \n",
      "\n",
      "Categorical Accuracy Score : 0.971654790154\n",
      "Precision Score : 0.984920771275\n",
      "Recall Score : 0.962549485371\n",
      "F1 Score : 0.973406097706\n"
     ]
    }
   ],
   "source": [
    "json_file = open('storage/model.json', 'r')\n",
    "loaded_model_json = json_file.read()\n",
    "json_file.close()\n",
    "loaded_model = model_from_json(loaded_model_json)\n",
    "loaded_model.load_weights(\"storage/model.h5\")\n",
    "print(\"Loaded model from disk\\n\")\n",
    "print (\"Testing: \")\n",
    "sgd = SGD(lr=0.01,decay=1e-6)\n",
    "\n",
    "loaded_model.compile(optimizer=sgd,\n",
    "                metrics=['categorical_accuracy','precision','recall','fbeta_score'],\n",
    "                loss='categorical_crossentropy')\n",
    "\n",
    "score = loaded_model.evaluate(X_test,Y_test,verbose=1)\n",
    "print '\\nCategorical Accuracy Score : '+str(score[1])\n",
    "print 'Precision Score : '+str(score[2])\n",
    "print 'Recall Score : '+str(score[3])\n",
    "print 'F1 Score : '+str(score[4])\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}