notebooks/Step2_Data_Augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, we are going to build a Convolution Neural Network model to classify images into 5 different severity levels of acne: Clear, Almost Clear, Mild, Moderate, and Severe. Since CNN models (or most of the deep learning models) are spatial sensitive, unless we have enough images which cover almost all possible locations of acne lesions in the training data, the CNN models trained using a limited set of images will not generalize well if the testing images have new locations of acne lesions that have never been seen during the training process. \n",
    "\n",
    "To overcome this problem, we roll each image patch for some number of pixels. The rolling action is taken on all image patches, no matter whether the patches are from the dominating class (mild), or other minor classes. \n",
    "\n",
    "For image patches from the mild class, we roll each patch 2 times. For image patches from other minor classes, the number of rolling is determined by the ratio between the numbers of mild class images and the minor class images. Therefore, after rolling, the numbers of images of the 5 classes are almost balanced. \n",
    "\n",
    "It is important to roll the images of the dominating class since we need to increase the coverage of acne lesion locations of this class as well in the training data. \n",
    "\n",
    "Forehead image patches are rolled from right to left. Cheeks and chin image patches are rolled bottom up. \n",
    "\n",
    "For instance, for a forehead patch with width 1000 pixels, assuming that the rolling step size is 200 pixels, then, \n",
    "image_after_rolling[:, 0:800, :] = image_before_rolling[:, 200:1000, :]\n",
    "image_after_rolling[:, 800:1000, :] = image_before_rolling[:, 0:200, :]\n",
    "\n",
    "For a cheek patch with height 500 pixels, assuming that the rolling step size is 100 pixels, then,\n",
    "image_after_rolling[0:400, :, :] = image_before_rolling[100:500, :, :]\n",
    "image_after_rolling[400:500, :, :] = image_before_rolling[0:100, :, :]\n",
    "\n",
    "The rolling step size is determined by the size of the dimension that the rolling is on, and the number of rolling times, i.e., \n",
    "***step_size = int(rolling dimension size/(num\\_rolling\\_times+1))***\n",
    "\n",
    "The image patches without rolling (rolling pixels = 0) are also saved to the destination directories. \n",
    "\n",
    "The image patches after rolling are also allocated to different subdirectories under the destination directory, based on the labels the original images received from the dermatologists. In this Jupyter notebook, we assume that the labels of the non-golden set images have been saved in a csv file. Therefore, we can be sure that images allocated based on this csv file will not have any golden-set images. \n",
    "\n",
    "During allocating image patches to different subdirectories, we also used a random number generator to decide whether an image patch goes to the training data or the validation data, based on whether the random number is greater than a pre-defined threshold. Two mapping files are created to record which files are in training data, which files are in validation data. These two mapping files will be used by CNTK to train CNN models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "### Skin patches extracted from the original images\n",
    "You need to run ***Step 1. Extract Forehead, cheeks, and chin skin patches from raw images using facial landmark model and One Eye model*** to extract the skin patches and save them in a single directory. It is OK that at this step, you extract the skin patches of the golden set images. However, in this Jupyter notebook, the skin patches from the golden set images will not be allocated to the subdirectories since the golden set image patches will not exist in the csv file with the image labels. \n",
    "\n",
    "### Save the non-golden set image labels in a csv file\n",
    "The image labels are stored in a database. Run the following query to retrieve the latest labels (final) of all non-golden set images. In the following query, shared_image = 0 stands for non-golden set images, and shared_image = 1 stands for golden set images. After getting the results, save them to a csv file, and upload it to the server where this Jupyter notebook runs. The csv file has two columns and a header line.: \n",
    "\n",
    "- Col1: Image Name (e.g., 0001.jpg)\n",
    "- Col2: Image label (e.g., 1-Clear, 2-Almost Clear)\n",
    "\n",
    "\n",
    "    select a.image_name as image_name, b.label as label\n",
    "    from\n",
    "    (\n",
    "        select labeler, image_name, max(label_at) as latest_time\n",
    "        from image_labels_new\n",
    "        where label is not null and shared_image = 0\n",
    "        group by labeler, image_name\n",
    "    )a\n",
    "    left outer join\n",
    "    (\n",
    "        select labeler, image_name, label, label_at \n",
    "        from image_labels_new\n",
    "        where label is not null and shared_image = 0\n",
    "    )b\n",
    "    on a.labeler = b.labeler and a.image_name = b.image_name and a.latest_time = b.label_at\n",
    "    order by image_name\n",
    "    \n",
    "### Python and Python libraries\n",
    "    - Python 3.5 and later version\n",
    "    - shutil, PIL, random, cv2, scipy copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "training_ratio = 0.7 # this ratio will be used to generate the mapping files, which will be used for CNTK models later on\n",
    "root_dir = \"<source data dir>\" # root_dir/source_dir will be the directory with the image patches\n",
    "dirs = [\"0-Not Acne\", \"1-Clear\", \"2-Almost Clear\", \"3-Mild\", \"4-Moderate\", \"5-Severe\"] #The subdirectory names have to be consistent \n",
    "                                                                                       #with the image label names in database\n",
    "source_dir = \"Landmarks_Frontal_Faces_Selected_fh_cheeks_patches\"\n",
    "dest_dir = \"0926_Landmarks_Frontal_Faces_Selected_fh_cheeks_patches\" # root_dir/dest_dir/dirs[i] will be the destination \n",
    "                                                                     # directory for rolled images belonging to the ith label\n",
    "image_label_file_name = \"0915_image_labels.csv\" #image label csv file name. Assuming it is in root_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create directories if not existing and mapping files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from shutil import copyfile\n",
    "from os import listdir\n",
    "from os.path import join, isfile, splitext, basename\n",
    "from PIL import Image\n",
    "from random import randint\n",
    "import numpy as np\n",
    "import cv2\n",
    "from scipy import misc\n",
    "import copy\n",
    "\n",
    "\n",
    "mapping_train = os.path.join(root_dir, dest_dir, \"mapping_train.txt\") #mapping file of the training images\n",
    "mapping_valid = os.path.join(root_dir, dest_dir, \"mapping_valid.txt\") #mapping file of the validation images\n",
    "train_fp = open(mapping_train, 'w')\n",
    "valid_fp = open(mapping_valid, 'w')\n",
    "for dir in dirs: #create directories for classes of image patches if not existing\n",
    "    path = os.path.join(root_dir, dest_dir, dir)\n",
    "    if not os.path.exists(path):\n",
    "        os.makedirs(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read file names in the skin patch original directory into a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 5479 files in the source dir /mnt/Nestle/data/Landmarks_Frontal_Faces_Selected_fh_cheeks_patches\n"
     ]
    }
   ],
   "source": [
    "imageFiles = [f for f in listdir(join(root_dir, source_dir)) if isfile(join(root_dir, source_dir, f))]\n",
    "print(\"There are %d files in the source dir %s\"%(len(imageFiles), join(root_dir,source_dir)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a function to get the indices of image patches in _imageFiles_ that belong to the same image ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_index_of_images(imageFiles, imagename):\n",
    "    num_images = len(imageFiles)\n",
    "    index = [i for i in range(num_images) if imagename in imageFiles[i]]\n",
    "    return index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Allocate image patches into destination subdirectories based on labels given by dermatologists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'1-Clear': 244, '4-Moderate': 614, '0-Not Acne': 38, '2-Almost Clear': 1156, '5-Severe': 160, '3-Mild': 1822}\n"
     ]
    }
   ],
   "source": [
    "label_result_file = join(root_dir, image_label_file_name) # Assuming that the image label file is in the root_dir\n",
    "fp = open(label_result_file, 'r')\n",
    "fp.readline() # skip the headerline\n",
    "label_count = {}\n",
    "max_count = 0\n",
    "\n",
    "# There is a bug in this handling. We are counting the number of images in each class, from the image label file. \n",
    "# However, the image patches we are rolling and allocating are from the selected images. The distribution of the \n",
    "# classes in image patches is different from the distribution in the labeled images, which including all non-golden set images.\n",
    "# That is the reason why after rolling and balancing, the classes of images are still not balanced. \n",
    "# Correcting this should have positive impact on the model performance. \n",
    "# For now, let's keep as it is. But we need to fix this later on when we retrain the model. \n",
    "for row in fp: #Read the count of images in the image label file, and get the number of images of the dominating class\n",
    "    row = row.strip().split(\",\")\n",
    "    label = row[1]\n",
    "    label_count[label] = label_count.get(label, 0) + 1\n",
    "    if max_count < label_count[label]: # Get the count of images in the dominating class\n",
    "        max_count = label_count[label]\n",
    "fp.close()\n",
    "print(label_count) \n",
    "\n",
    "fp = open(label_result_file, 'r') # Read the image label file again for allocating purpose\n",
    "fp.readline()\n",
    "random.seed(98052) # Set the random seed in order to reproduce the result. \n",
    "\n",
    "# This is the function that rolls an image patch, and saves the rolled image patch as a jpg file on the destination directory\n",
    "# img: image data frame before rolling\n",
    "# dest_path: destination directory to save the rolled image patch\n",
    "# file_name_wo_ext: file name without extension, i.e., just the image ID\n",
    "# image_names: a list of image names and path. The new image name and path will be appended to it and returns as an output\n",
    "# x_or_y: 'x' or 'y'. It specifies whether it is rolling the images right to left, or bottom to top.\n",
    "# pixels: number of pixels to roll in the direction specified by x_or_y.\n",
    "# returns: image_names\n",
    "def roll_and_save(img, dest_path, file_name_wo_ext, image_names, x_or_y, pixels):\n",
    "    img_height, img_width = img.shape[0:2]\n",
    "    img2 = copy.copy(img)\n",
    "    if x_or_y == 'x':\n",
    "        img2[:, 0:(img_width-pixels),:] = img[:,pixels:img_width,:]\n",
    "        img2[:,(img_width-pixels):img_width,:] = img[:,0:pixels,:]\n",
    "    else:\n",
    "        img2[0:(img_height-pixels), :, :] = img[pixels:img_height, :, :]\n",
    "        img2[(img_height-pixels):img_height, :,:] = img[0:pixels,:, :]\n",
    "    img2 = cv2.cvtColor(img2, cv2.COLOR_BGR2RGB)        \n",
    "    dest = join(dest_path, file_name_wo_ext+\"_roll_\"+x_or_y+\"_\"+str(pixels)+\".jpg\") #rolled image file name e.g., 0001_roll_x_112.jpg\n",
    "    misc.imsave(dest, img2) \n",
    "    image_names.append(dest)\n",
    "    return image_names\n",
    "\n",
    "minimal_roll_times = 2 #Even the dominating class images need to roll twice\n",
    "\n",
    "for row in fp: # go over for each row in the image label file\n",
    "    rn = random.uniform(0, 1) # a random number determining whether this file goes to training or validation\n",
    "    row = row.strip().split(\",\")\n",
    "    file_name = row[0] \n",
    "    label = row[1]\n",
    "    file_name_wo_ext = splitext(file_name)[0] #get the image ID\n",
    "    index = find_index_of_images(imageFiles, file_name_wo_ext) #find the image patches that belong to this image ID\n",
    "    num_files_found = len(index) #number of image patches found. If the image is not in the selected image set, num_files_found=0\n",
    "    image_names = []\n",
    "    for i in range(num_files_found):\n",
    "        source = join(root_dir, source_dir, imageFiles[index[i]])\n",
    "        image_name_no_ext = splitext(imageFiles[index[i]])[0] #get the image patch name, e.g., 0001_fh, 0003_lc, etc.\n",
    "        img = cv2.imread(source)\n",
    "        img_height, img_width = img.shape[0:2]\n",
    "        if 'fh' in imageFiles[index[i]]: # forehead image patches, rolling right to left\n",
    "            x_or_y = 'x'\n",
    "        else: #if cheeks, or chins, rolling bottom to top\n",
    "            x_or_y = 'y'\n",
    "        roll_ratio = float(max_count)/float(label_count[label]) # determining how many times to roll, in order to balance\n",
    "        dest_path = join(root_dir, dest_dir, label) #destination path at the image class level\n",
    "        \n",
    "        image_names = roll_and_save(img, dest_path, image_name_no_ext, image_names, x_or_y, 0) #save the image without rolling \n",
    "        if roll_ratio > 1: # of this is not the dominating class, we need to roll in order to balance\n",
    "            num_times = int(np.floor(roll_ratio) - 1)\n",
    "        else:\n",
    "            num_times = 0\n",
    "        num_times += minimal_roll_times # adding the number of times that the dominating class is also rolling. \n",
    "        if num_times > 0: # determining the step size based on number of times to roll. We want the constant step size for each image\n",
    "            if x_or_y == 'x':\n",
    "                step_size = int(np.floor(np.float(img_width)/np.float(num_times+1))) \n",
    "            else:\n",
    "                step_size = int(np.floor(np.float(img_height)/np.float(num_times+1)))\n",
    "            for j in range(num_times):\n",
    "                image_names = roll_and_save(img, dest_path, image_name_no_ext, image_names, x_or_y, step_size*(j+1))\n",
    "        # The following lines of writing image names to the mapping file have some problem. The image path and name list image_names \n",
    "        # is accumulating over the image patches of the same image ID. However, the following lines is writing to the mapping file for\n",
    "        # every image patch. There will be duplicates in the mapping files. \n",
    "        # However, it does not affect the tensorflow models we built since tensorflow was not using the mapping file. \n",
    "        # It should not affect the CNTK models either since CNTK models were not using the mapping files I created. \n",
    "        # A simple fix of this is to move the following lines to the outer for loop\n",
    "        label_index = [i for i,x in enumerate(dirs) if x == label][0] # Determining the label index. dirs has 0-Not Acne in the list. \n",
    "                                                                      # So, for 1-Clear images, the label index in dirs is 1.\n",
    "        if label_index >= 1: # We do not model 0-Not Acne, where label_index = 0\n",
    "            label_index -= 1 #writing the image path and names of the entire rolled image\n",
    "                             #set for a skin patch to the training mapping file\n",
    "            if rn <= training_ratio:\n",
    "                for image_name in image_names:\n",
    "                    train_fp.write(\"%s\\t%d\\n\"%(image_name, label_index)) \n",
    "            else:\n",
    "                for image_name in image_names:\n",
    "                    valid_fp.write(\"%s\\t%d\\n\"%(image_name, label_index))\n",
    "fp.close()\n",
    "train_fp.close()\n",
    "valid_fp.close()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}