notebooks/Step1_Data_RemoveNoise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the first step we take to preprocess the raw images to extract the skin patches from the face. Before you run this Jupyter Notebook, please complete the ***Prerequisites***, and change the parameters in section ***Parameters***.\n",
    "\n",
    "Two pre-trained models are used in this step. The facial landmark model _shape\\_predictor\\_68\\_face\\_landmarks_ is capable of detecting landmarks of frontal faces or faces slightly facing off the camera, as long as two open eyes can be observed in the image. Then skin patches of forehead, cheeks, and chin will be extracted from the original images and saved as image files on the specified destination directory. \n",
    "\n",
    "If a face is facing too much off the camera such that the camera cannot detected both eyes, such as those profile faces, facial landmark model will not be able to detect the faces. If this happens, One Eye model will be used to detect the location of the eye. Based on the detected one eye, the forehead and cheek skin patches will be inferred, extracted, and saved as image files on the specified destination directory. \n",
    "\n",
    "The skin patch file names will have \\_fh, \\_lc, \\_rc, and \\_chin to represent forehead, left cheek, right cheek, and chin, respectively. \n",
    "\n",
    "If neither facial landmark model, nor the One Eye model works, the original image will be copied to the destination directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the prerequisites to run this Jupyter notebook:\n",
    "\n",
    "### Raw images\n",
    "    You need to have the raw images downloaded to the machine where the Jupyter notebook runs on.\n",
    "    \n",
    "### Python \n",
    "    You need to have Python 3.5 or later version installed on the machine. \n",
    "\n",
    "### Python libraries\n",
    "    You need to have the following Python libraries installed on the machine. \n",
    "    - numpy\n",
    "    - OpenCV\n",
    "    - dlib\n",
    "    - matplotlib\n",
    "    - Scipy\n",
    "    \n",
    "### Pretrained models\n",
    "\n",
    "You need to download the pretrained _shape\\_predictor\\_68\\_face\\_landmarks_ and _One Eye Haarcascade model_ to the ../models directory. \n",
    "- [shape\\_predictor\\_68\\_face\\_landmarks](https://github.com/AKSHAYUBHAT/TensorFace/blob/master/openface/models/dlib/shape_predictor_68_face_landmarks.dat)\n",
    "- [One Eye model](https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_eye.xml)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters\n",
    "You need to set the following parameters to let the script know where is the directory of raw images, where are the facial landmark and one eye models, how to determine the sizes and locations of forehead and cheeks, and where to save the skin patch images. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "originalFile_Dir = \"<directory of the original images>\" # Location of the raw images. Assuming that all images are in the same directory\n",
    "                                                   # subdirectories will not be scanned, and this directory only contains image files. \n",
    "croppedFaces_Dir = \"<directory of the destination to hold image patches>\" # Location to store the skin patch images.\n",
    "Eye_Cascade_Path = '../models/haarcascade_eye.xml'\n",
    "PREDICTOR_PATH = '../models/shape_predictor_68_face_landmarks.dat'\n",
    "\n",
    "verb = False # If verb = False, extracted skin patches will not be displayed on the Jupyter notebook. \n",
    "             # Otherwise, extracted skin patches for each raw image will be displayed on the Jupyter notebook. \n",
    "             # Set verb = True if you want to inspect the result of the skin patch extraction to make sure you are getting the \n",
    "             # skin patches as expected. Do this on a small subset of your images, and set it to False and run it on the entire\n",
    "             # image set after you are comfortable with the results. \n",
    "width_ratio = 1.5 # This parameter controls the width of the forehead if One Eye model is used to infer the forehead. If left cheek\n",
    "                  # is determined to be facing the camera, in x direction, extracted forehead skin patch starts at the left edge\n",
    "                  # of the detected eye, and ends at min(image width, eye's left edge + width\\_ratio *  eye's width)\n",
    "                  # If right cheek is determed to be facing the camera, in x direction, forehead starts at \n",
    "                  # max(0, eye's left edge - width\\_ratio * eye's width), and ends at \n",
    "                  # min(image width, eye's left edge + width\\_ratio * eye's width) \n",
    "top_ratio = 1.5   # This parameter controls the height of the forehead if One Eye model is used to infer the forehead. \n",
    "                  # In y direction, forehead starts at max(0, eye's top edge - top\\_ratio * eye's height), and ends at \n",
    "                  # max(0, eye's top edge - 0.5 * eye's height). 0.5 here is to skip the eyebrow. \n",
    "down_ratio = 4.5  # This parameter controls how much we are going down from the detected eye's by the One Eye model in order to get \n",
    "                  # the cheek skin patch. In y direction, cheek starts at the lower edge of the detected eye, and ends \n",
    "                  # at min(image height, eye's top edge + down\\_ratio * eye's height)\n",
    "cheek_width_ratio = 2.8 # This parameter controls the width of the cheek skin patch if One Eye model is used. \n",
    "                        # If left cheek is determined to be facing the camera, in x direction, cheek starts at \n",
    "                        # the right edge of the eye, and ends at \n",
    "                        # min(image width, (eye's left edge + cheek\\_width\\_ratio + 0.5) * eye's width).\n",
    "                        # If right cheek is determed to be facing the camera, in x direction, cheek starts at \n",
    "                        # max(0, eye's left edge - cheek\\_width\\_ratio * eye's width), and ends at \n",
    "                        # eye's left edge + 0.5*eye's width\n",
    "forehead_ratio = 0.3    # This parameter controls the height of forehead when facial landmark model is working. \n",
    "                        # The extracted forehead height will be forehead\\_ratio * face\\_height, where face\\_height is \n",
    "                        # the distance in y direction between the highest eyebrow and the lowest landmark points (the bottom point)\n",
    "                        # of the chin. Then, the forehead skin patch, in y direction, starts at \n",
    "                        # max(0, highest eyebrow - forehead\\_height), and ends at \n",
    "                        # min(highest eyebrow, forehead top edge + forehead\\_height)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define functions and initialize landmark and one eye models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "from os import listdir\n",
    "from os.path import join, isfile, splitext\n",
    "from scipy import misc\n",
    "import sys\n",
    "import dlib\n",
    "import os\n",
    "%matplotlib inline\n",
    "\n",
    "eye_cascade = cv2.CascadeClassifier(Eye_Cascade_Path) # Initialize the Eye cascade model\n",
    "\n",
    "#####################################################\n",
    "# These are the default parameters for landmark model\n",
    "# Do not change them.\n",
    "#####################################################\n",
    "SCALE_FACTOR = 1 \n",
    "FEATHER_AMOUNT = 11\n",
    "\n",
    "FACE_POINTS = list(range(17, 68))\n",
    "MOUTH_POINTS = list(range(48, 61))\n",
    "RIGHT_BROW_POINTS = list(range(17, 22))\n",
    "LEFT_BROW_POINTS = list(range(22, 27))\n",
    "RIGHT_EYE_POINTS = list(range(36, 42))\n",
    "LEFT_EYE_POINTS = list(range(42, 48))\n",
    "NOSE_POINTS = list(range(27, 35))\n",
    "JAW_POINTS = list(range(0, 17))\n",
    "######################################################\n",
    "# Above are the default parameters for landmark model.\n",
    "######################################################\n",
    "\n",
    "# Points used to line up the images.\n",
    "ALIGN_POINTS = (LEFT_BROW_POINTS + RIGHT_EYE_POINTS + LEFT_EYE_POINTS +\n",
    "                               RIGHT_BROW_POINTS + NOSE_POINTS + MOUTH_POINTS)\n",
    "\n",
    "# Points from the second image to overlay on the first. The convex hull of each\n",
    "# element will be overlaid.\n",
    "# Do not need it since it is not an overlay problem\n",
    "OVERLAY_POINTS = [\n",
    "    LEFT_EYE_POINTS + RIGHT_EYE_POINTS + LEFT_BROW_POINTS + RIGHT_BROW_POINTS,\n",
    "    NOSE_POINTS + MOUTH_POINTS,\n",
    "]\n",
    "\n",
    "# Amount of blur to use during colour correction, as a fraction of the\n",
    "# pupillary distance.\n",
    "#COLOUR_CORRECT_BLUR_FRAC = 0.6\n",
    "\n",
    "detector = dlib.get_frontal_face_detector()\n",
    "predictor = dlib.shape_predictor(PREDICTOR_PATH) # Initialize the landmark model\n",
    "\n",
    "class TooManyFaces(Exception):\n",
    "    pass\n",
    "\n",
    "class NoFaces(Exception):\n",
    "    pass\n",
    "\n",
    "def get_landmarks(im):\n",
    "    rects = detector(im, 1)\n",
    "    \n",
    "    if len(rects) > 1:\n",
    "        raise TooManyFaces\n",
    "    if len(rects) == 0:\n",
    "        raise NoFaces\n",
    "\n",
    "    return np.matrix([[p.x, p.y] for p in predictor(im, rects[0]).parts()])\n",
    "\n",
    "def annotate_landmarks(im, landmarks):\n",
    "    im = im.copy()\n",
    "    for idx, point in enumerate(landmarks):\n",
    "        pos = (point[0, 0], point[0, 1])\n",
    "        cv2.putText(im, str(idx), pos,\n",
    "                    fontFace=cv2.FONT_HERSHEY_SCRIPT_SIMPLEX,\n",
    "                    fontScale=0.4,\n",
    "                    color=(0, 0, 255))\n",
    "        cv2.circle(im, pos, 3, color=(0, 255, 255))\n",
    "    return im\n",
    "\n",
    "def draw_convex_hull(im, points, color):\n",
    "    points = cv2.convexHull(points)\n",
    "    cv2.fillConvexPoly(im, points, color=color)\n",
    "\n",
    "def get_face_mask(im, landmarks):\n",
    "    im = np.zeros(im.shape[:2], dtype=np.float64)\n",
    "\n",
    "    for group in OVERLAY_POINTS:\n",
    "        draw_convex_hull(im,\n",
    "                         landmarks[group],\n",
    "                         color=1)\n",
    "\n",
    "    im = np.array([im, im, im]).transpose((1, 2, 0))\n",
    "\n",
    "    im = (cv2.GaussianBlur(im, (FEATHER_AMOUNT, FEATHER_AMOUNT), 0) > 0) * 1.0\n",
    "    im = cv2.GaussianBlur(im, (FEATHER_AMOUNT, FEATHER_AMOUNT), 0)\n",
    "\n",
    "    return im\n",
    "    \n",
    "def transformation_from_points(points1, points2):\n",
    "    \"\"\"\n",
    "    Return an affine transformation [s * R | T] such that:\n",
    "        sum ||s*R*p1,i + T - p2,i||^2\n",
    "    is minimized.\n",
    "    \"\"\"\n",
    "    # Solve the procrustes problem by subtracting centroids, scaling by the\n",
    "    # standard deviation, and then using the SVD to calculate the rotation. See\n",
    "    # the following for more details:\n",
    "    #   https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem\n",
    "\n",
    "    points1 = points1.astype(np.float64)\n",
    "    points2 = points2.astype(np.float64)\n",
    "\n",
    "    c1 = np.mean(points1, axis=0)\n",
    "    c2 = np.mean(points2, axis=0)\n",
    "    points1 -= c1\n",
    "    points2 -= c2\n",
    "\n",
    "    s1 = np.std(points1)\n",
    "    s2 = np.std(points2)\n",
    "    points1 /= s1\n",
    "    points2 /= s2\n",
    "\n",
    "    U, S, Vt = np.linalg.svd(points1.T * points2)\n",
    "\n",
    "    # The R we seek is in fact the transpose of the one given by U * Vt. This\n",
    "    # is because the above formulation assumes the matrix goes on the right\n",
    "    # (with row vectors) where as our solution requires the matrix to be on the\n",
    "    # left (with column vectors).\n",
    "    R = (U * Vt).T\n",
    "\n",
    "    return np.vstack([np.hstack(((s2 / s1) * R,\n",
    "                                       c2.T - (s2 / s1) * R * c1.T)),\n",
    "                         np.matrix([0., 0., 1.])])\n",
    "\n",
    "def read_im_and_landmarks(fname):\n",
    "    im = cv2.imread(fname, cv2.IMREAD_COLOR)\n",
    "    im = cv2.resize(im, (im.shape[1] * SCALE_FACTOR,\n",
    "                         im.shape[0] * SCALE_FACTOR))\n",
    "    s = get_landmarks(im)\n",
    "\n",
    "    return im, s\n",
    "\n",
    "def warp_im(im, M, dshape):\n",
    "    output_im = np.zeros(dshape, dtype=im.dtype)\n",
    "    cv2.warpAffine(im,\n",
    "                   M[:2],\n",
    "                   (dshape[1], dshape[0]),\n",
    "                   dst=output_im,\n",
    "                   borderMode=cv2.BORDER_TRANSPARENT,\n",
    "                   flags=cv2.WARP_INVERSE_MAP)\n",
    "    return output_im\n",
    "\n",
    "def correct_colours(im1, im2, landmarks1):\n",
    "    blur_amount = COLOUR_CORRECT_BLUR_FRAC * np.linalg.norm(\n",
    "                              np.mean(landmarks1[LEFT_EYE_POINTS], axis=0) -\n",
    "                              np.mean(landmarks1[RIGHT_EYE_POINTS], axis=0))\n",
    "    blur_amount = int(blur_amount)\n",
    "    if blur_amount % 2 == 0:\n",
    "        blur_amount += 1\n",
    "    im1_blur = cv2.GaussianBlur(im1, (blur_amount, blur_amount), 0)\n",
    "    im2_blur = cv2.GaussianBlur(im2, (blur_amount, blur_amount), 0)\n",
    "\n",
    "    # Avoid divide-by-zero errors.\n",
    "    im2_blur += (128 * (im2_blur <= 1.0)).astype(im2_blur.dtype)\n",
    "\n",
    "    return (im2.astype(np.float64) * im1_blur.astype(np.float64) /\n",
    "                                                im2_blur.astype(np.float64))\n",
    "\n",
    "# This function infers the region of either left or right cheek. The inferred cheek region will be used to \n",
    "# determine whether it is a left or a right cheek. \n",
    "def infer_cheek_region(eye, width_ratio, down_ratio, left_or_right):\n",
    "    region1 = [0] * 4\n",
    "    if left_or_right == 'right': #assuming it is the absolute right chin\n",
    "        region1[0] = int(max(0, int(eye[0] - 0.5 * eye[2]))) #cheek region should go lefwards\n",
    "        region1[2] = int(0.5 * eye[2])\n",
    "    else: # assuming it is the absolute left cheek\n",
    "        region1[0] = int(eye[0] + eye[2]) # cheek region should go rightwards\n",
    "        region1[2] = int(0.5 * eye[2])\n",
    "    region1[1] = int(eye[1] + eye[3])\n",
    "    region1[3] = int(1.5 * eye[3])\n",
    "    return region1\n",
    "\n",
    "# This function determines whether it is a left or a right cheek. The determination is pretty heuristic. \n",
    "# We assume that when the left cheek is facing the camera, the patch on the left side of the region under the eye should \n",
    "# have higher variation than the right side since the left side will be more background, and the right side will have \n",
    "# lower variation since the right side will be mostly face skin.\n",
    "# On the other side, if the right cheek is facing the camera, the patch on the left side will be mostly skin, and right side \n",
    "# will be mostly background. Therefore right side will have higher variation than the left side. \n",
    "\n",
    "def detect_face_direction(gray, face, eye, down_ratio, cheek_width_ratio):  \n",
    "    region1 = [0] * 4 # assuming this is the left eye, forhead should go rightward\n",
    "    region2 = [0] * 4 # assuming this is the right eye, forhead should go leftward\n",
    "    region1 = infer_cheek_region(eye[0], cheek_width_ratio, down_ratio, 'left') #region1 is from eye to right\n",
    "    region2 = infer_cheek_region(eye[0], cheek_width_ratio, down_ratio, 'right') # region2 is from eye to left\n",
    "    std1 = np.std(gray[region1[1]:(region1[1]+region1[3]), region1[0]:(region1[0]+region1[2])])\n",
    "    std2 = np.std(gray[region2[1]:(region2[1]+region2[3]), region2[0]:(region2[0]+region2[2])])\n",
    "    face_direction = \"\"\n",
    "    if std1 > std2:  #eye right has higher variance than eye left\n",
    "        face_direction = \"right\"\n",
    "    else:\n",
    "        face_direction = \"left\"\n",
    "    return face_direction\n",
    "\n",
    "# Extract cheek patches based on face landmarks and eye landmarks, and whether it is left cheek or right cheek\n",
    "def extract_cheek_region(face_x_min, face_x_max, face_y_max, eye_landmarks, left_or_right):\n",
    "    if left_or_right == \"Left\":\n",
    "        cheek_region_min_x = eye_landmarks[0,0] # left cheek, x direction, cheek starts at the most inner point of left eye,\n",
    "                                                # and ends with the face_x_max - a margin value. The margin value is 0.05 * \n",
    "                                                # (face_x_max - most inner point of left eye). This margin is set to avoid hair \n",
    "                                                # or other backgrounds\n",
    "        cheek_region_max_x = int(face_x_max - 0.05 * (face_x_max - min(eye_landmarks[:,0])))\n",
    "    else:\n",
    "        cheek_region_max_x = eye_landmarks[-1, 0] # right cheek, x direction, cheek starts at the face_x_min + a margin, \n",
    "                                                  # where the margin is 0.1 * (cheek_region_max_x - face_x_min). This margin is set \n",
    "                                                  # to avoid hair or other backgrounds, and ends at the most inner point of right eye\n",
    "        cheek_region_min_x = int(face_x_min + 0.1 * (cheek_region_max_x - face_x_min)) \n",
    "    # y direction, cheek starts at lowest eye landmarks + 0.2 * eye height. 0.2 * eye height is to avoid the skin that might be \n",
    "    # dominated by eyelash\n",
    "    cheek_region_min_y = int(max(eye_landmarks[:,1]) + 0.2 * (max(eye_landmarks[:,1])  - min(eye_landmarks[:,1])))\n",
    "    cheek_region_max_y = int(face_y_max - 0.1 * (face_y_max - max(eye_landmarks[:,1])))\n",
    "    return [cheek_region_min_x, cheek_region_min_y, cheek_region_max_x, cheek_region_max_y]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start processing images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the list of images in the directory of the original images     \n",
    "imageFiles = [join(originalFile_Dir, f) for f in listdir(originalFile_Dir) if isfile(join(originalFile_Dir, f))]\n",
    "num_images = len(imageFiles)\n",
    "image_counter = 0\n",
    "\n",
    "# If you want to only work on a small subset of the images, change the following parameters\n",
    "start_index = 0\n",
    "end_index = num_images\n",
    "\n",
    "for imagefile in imageFiles[start_index:end_index]:\n",
    "    image_counter += 1\n",
    "    imageName = splitext(os.path.basename(imagefile))[0] # Get the root name of the image (without extension). This variable will be \n",
    "                                                         # used to generate the skin patch file name.\n",
    "    face_detected = False\n",
    "    try:\n",
    "        img, landmarks = read_im_and_landmarks(imagefile) # Try if landmark model works. If it works, image is already read into img\n",
    "        face_detected = True\n",
    "    except:\n",
    "        img = cv2.imread(imagefile) # if landmark model does not work, read the image \n",
    "        face_detected = False\n",
    "    img_height, img_width = img.shape[0:2] #get the image height and width. Image data is in the format of [height, width, channel]\n",
    "    min_dim = min(img_height, img_width)\n",
    "    min_face_size = min_dim * 0.2 # Specify the minimal face size. Heuristic. \n",
    "    min_eye = min_face_size * 0.2 # specify the minimal eye size. \n",
    "    min_eye_area = min_eye ** 2 # specify the miniaml area of the eye. This is used screen detected eyes by the OneEye model.\n",
    "                                # Keep in mind, the One Eye model will identify whatever looks like an eye. We need to screen out\n",
    "                                # those that are too small. \n",
    "    \n",
    "    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) #Convert image from BGR to GRAY. OpenCV reads images in as BGR. One Eye model \n",
    "                                                 # works on gray scale images. \n",
    "    \n",
    "    if face_detected: # if face is detected by landmark model\n",
    "        mask = get_face_mask(img, landmarks) \n",
    "        face_x_min = int(max(0, np.asarray(min(landmarks[:,0])).flatten()[0])) #Get the minimal value of the detected landmarks in x\n",
    "        face_x_max = int(min(img_width, np.asarray(max(landmarks[:,0])).flatten()[0])) # Get the maximal value of the detected landmarks in x\n",
    "        face_y_min = int(max(0, np.asarray(min(landmarks[:,1])).flatten()[0])) # Get the minimal value of the detected landmarks in y\n",
    "        face_y_max = int(min(img_height, np.asarray(max(landmarks[:,1])).flatten()[0])) # Get the maximal value of the detected landmarks in y\n",
    "        face_height = face_y_max - face_y_min # Get the height of face\n",
    "        forehead_height = int(face_height * forehead_ratio) # Ideally, forehead height should be 1/2 of the height between eyebrow and bottom of chin\n",
    "                                                            # We choose forehead_ratio = 0.3 to avoid hairs on the forehead.\n",
    "        new_face_y_min = max(0, face_y_min - forehead_height) # new_face_y_min is the top edge of the forehead. \n",
    "        right_brow_landmarks = landmarks[RIGHT_BROW_POINTS,:]\n",
    "        left_brow_landmarks = landmarks[LEFT_BROW_POINTS,:]\n",
    "        right_eye_landmarks = landmarks[RIGHT_EYE_POINTS,:]\n",
    "        left_eye_landmarks = landmarks[LEFT_EYE_POINTS,:]\n",
    "        mouse_landmarks = landmarks[MOUTH_POINTS,:]\n",
    "        ########################\n",
    "        # Get the forehead patch\n",
    "        ########################\n",
    "        [right_brow_min_x, left_brow_max_x] = \\\n",
    "            [max(0, np.min(np.array(right_brow_landmarks[:,0]))), min(img_width, np.max(np.array(left_brow_landmarks[:,0])))]\n",
    "        brow_min_y = min(np.min(np.array(right_brow_landmarks[:,1])),np.min(np.array(left_brow_landmarks[:,1])))\n",
    "        forehead_x_min = right_brow_min_x # forehead starts at the left landmark of the right eye brow\n",
    "        forehead_x_max = left_brow_max_x\n",
    "        forehead_y_min = max(0, brow_min_y - forehead_height)\n",
    "        forehead_y_max = min(brow_min_y, forehead_y_min + forehead_height)\n",
    "        forehead_region = img[forehead_y_min:forehead_y_max, forehead_x_min:forehead_x_max, :]\n",
    "        forehead_file_name = join(croppedFaces_Dir, imageName+\"_fh.jpg\")\n",
    "        # BGR image needs to be converted to RGB before saving as image file\n",
    "        forehead_region = cv2.cvtColor(forehead_region, cv2.COLOR_BGR2RGB)\n",
    "        misc.imsave(forehead_file_name, forehead_region)\n",
    "        \n",
    "        chin_x_min = np.max(np.array(right_eye_landmarks[:,0])) #In x direction, chin patch will be between the two most inner\n",
    "                                                                #points of eyebrows\n",
    "        chin_x_max = np.min(np.array(left_eye_landmarks[:,0]))\n",
    "        chin_y_min = np.max(np.array(mouse_landmarks[:,1])) #In y direction, chin patch starts at the lowest point of mouse landmarks\n",
    "        chin_y_max = face_y_max # In y direction, chin patch ends at the lowest point of face\n",
    "        chin_region = img[chin_y_min:chin_y_max, chin_x_min:chin_x_max, :]\n",
    "        chin_file_name = join(croppedFaces_Dir, imageName+\"_chin.jpg\")\n",
    "        chin_region = cv2.cvtColor(chin_region, cv2.COLOR_BGR2RGB)\n",
    "        misc.imsave(chin_file_name, chin_region)\n",
    "        \n",
    "        ##########################\n",
    "        # Get the cheeks patch\n",
    "        ##########################\n",
    "        # Decide whether it is a side view or not. If the the left eye is 15% wider than the right eye, we determine\n",
    "        # that it is the left cheek facing the camera. \n",
    "        # If the right eye is 15% wider than the left eye, we determine that it is the right cheek facing the camera. \n",
    "        # Under such situation, we only extract one cheek patch. \n",
    "        left_eye_width = np.max(np.array(left_eye_landmarks[:,0])) - np.min(np.array(left_eye_landmarks[:,0]))\n",
    "        right_eye_width = np.max(np.array(right_eye_landmarks[:,0])) - np.min(np.array(right_eye_landmarks[:,0]))\n",
    "        right_face = True\n",
    "        left_face = True\n",
    "        if float(right_eye_width) / float(left_eye_width) >= 1.15: # right eye is bigger than left eye, showing the right face\n",
    "            left_face = False\n",
    "        elif float(left_eye_width) / float(right_eye_width) >= 1.15: # left eye is bigger than right eye, showing the left face\n",
    "            right_face = False\n",
    "        \n",
    "        if right_face:\n",
    "            right_cheek_region = extract_cheek_region(face_x_min, face_x_max, face_y_max, right_eye_landmarks, \"Right\")\n",
    "            cheek_region = img[right_cheek_region[1]:right_cheek_region[3], right_cheek_region[0]:right_cheek_region[2], :]\n",
    "            cheek_file_name = join(croppedFaces_Dir, imageName+\"_rc.jpg\")\n",
    "            cheek_region = cv2.cvtColor(cheek_region, cv2.COLOR_BGR2RGB)\n",
    "            misc.imsave(cheek_file_name, cheek_region)\n",
    "        if left_face:\n",
    "            left_cheek_region = extract_cheek_region(face_x_min, face_x_max, face_y_max, left_eye_landmarks, \"Left\")\n",
    "            cheek_region = img[left_cheek_region[1]:left_cheek_region[3], left_cheek_region[0]:left_cheek_region[2], :]\n",
    "            cheek_file_name = join(croppedFaces_Dir, imageName+\"_lc.jpg\")\n",
    "            cheek_region = cv2.cvtColor(cheek_region, cv2.COLOR_BGR2RGB)\n",
    "            misc.imsave(cheek_file_name, cheek_region)\n",
    "        # if verb == True, display the detected skin patches on the original image, using rectangle to highlight the skin patches\n",
    "        if verb:\n",
    "            img_tmp = img\n",
    "            img_tmp = cv2.cvtColor(img_tmp, cv2.COLOR_BGR2RGB)\n",
    "            cv2.rectangle(img_tmp, (forehead_x_min, forehead_y_min), (forehead_x_max, forehead_y_max), (0, 255, 0), 10)\n",
    "            if right_face:\n",
    "                cv2.rectangle(img_tmp, (right_cheek_region[0], right_cheek_region[1]), \\\n",
    "                              (right_cheek_region[2], right_cheek_region[3]), (255, 255, 0), 10)\n",
    "            if left_face:\n",
    "                cv2.rectangle(img_tmp, (left_cheek_region[0], left_cheek_region[1]), \\\n",
    "                              (left_cheek_region[2], left_cheek_region[3]), (255, 255, 0), 10)\n",
    "            plt.imshow(img_tmp)\n",
    "                \n",
    "    if not face_detected:\n",
    "        print(\"Face not detected by landmarks model...\")\n",
    "        # Use the OneEye model to detect one eye, and infer the face region based on the eye location\n",
    "        eye_detected = False\n",
    "        roi_gray = gray\n",
    "        roi_color = img\n",
    "        roi_color = cv2.cvtColor(roi_color, cv2.COLOR_BGR2RGB)\n",
    "        eyes = eye_cascade.detectMultiScale(roi_gray, 1.1, 5)\n",
    "        max_area = 0\n",
    "        eye_count = 0\n",
    "        max_index = 0\n",
    "        \n",
    "        for (ex,ey,ew,eh) in eyes: # there might be multiple eyes detected. Choose the biggest one\n",
    "            if ew*eh >= max_area and ex >= img_width * 0.1 and ex <= img_width * 0.9:\n",
    "                max_area = ew*eh\n",
    "                max_index = eye_count\n",
    "            eye_count += 1\n",
    "        if max_area >= min_eye_area: # if area of maximal eye is greater than the eye area threshold, take it as a real eye\n",
    "            eye_detected = True\n",
    "            (ex, ey, ew, eh) = eyes[max_index]\n",
    "            if float(ew) / float(img_width) > 0.15 or float(eh) / float(img_height) > 0.15: # detected eye too large\n",
    "                # resize the detected eye\n",
    "                center_x = ex + ew/2\n",
    "                center_y = ey + eh/2\n",
    "                resized_w = min(img_width * 0.15, img_height * 0.15) # resize the eye\n",
    "                ex = int(center_x - resized_w/2)\n",
    "                ey = int(center_y - resized_w/2)\n",
    "                ew = int(resized_w)\n",
    "                eh = int(resized_w)\n",
    "                eyes1 = np.array([ex, ey, resized_w, resized_w]).reshape((1,4))\n",
    "            else:\n",
    "                eyes1 = np.array(eyes[max_index]).reshape((1,4))\n",
    "            face1 = np.array(())\n",
    "            face_direction = detect_face_direction(gray, face1, eyes1, down_ratio, cheek_width_ratio)\n",
    "            if face_direction == \"left\":\n",
    "                print(\"Left eye detected\")\n",
    "                face_min_x = eyes1[0, 0]\n",
    "                face_max_x = min(img_width, int(eyes1[0,0] + (cheek_width_ratio + 0.5) * eyes1[0, 2]))\n",
    "                forehead_max_x = min(img_width, int(eyes1[0,0] + width_ratio * eyes1[0, 2]))\n",
    "                forehead_min_x = face_min_x\n",
    "                cheek_min_x = int(eyes1[0, 0] + 0.5 * eyes1[0,2])\n",
    "                cheek_max_x = face_max_x\n",
    "            else:\n",
    "                print(\"Right eye detected\")\n",
    "                face_min_x = max(0, int(eyes1[0, 0] - cheek_width_ratio * eyes1[0, 2]))\n",
    "                face_max_x = eyes1[0, 0] + eyes1[0, 2]\n",
    "                forehead_min_x = max(0, int(eyes1[0, 0] - width_ratio * eyes1[0, 2]))\n",
    "                forehead_max_x = min(img_width, int(eyes1[0, 0] + width_ratio * eyes1[0, 2]))   \n",
    "                cheek_max_x = int(eyes1[0,0] + 0.5*eyes1[0,2])\n",
    "                cheek_min_x = face_min_x\n",
    "            forehead_min_y = max(0, int(eyes1[0, 1] - top_ratio * eyes1[0,3]))\n",
    "            forehead_max_y = max(0, int(eyes1[0, 1] - 0.5 * eyes1[0, 3]))\n",
    "            forehead_ok = False\n",
    "            # Get the forehead region\n",
    "            if forehead_max_y - forehead_min_y >= 0.7 * eyes1[0, 3]:\n",
    "                forehead_ok = True\n",
    "                forehead_region = img[forehead_min_y:forehead_max_y, forehead_min_x: forehead_max_x, :]\n",
    "                forehead_region = cv2.cvtColor(forehead_region, cv2.COLOR_BGR2RGB)\n",
    "                forehead_file_name = join(croppedFaces_Dir, imageName+\"_fh.jpg\")\n",
    "                misc.imsave(forehead_file_name, forehead_region)\n",
    "            # Get the cheek region\n",
    "            cheek_min_y = int(eyes1[0, 1] + eyes1[0, 3])\n",
    "            cheek_max_y = min(img_height, int(eyes1[0, 1] + down_ratio * eyes1[0, 3]))\n",
    "            cheek_region = img[cheek_min_y: cheek_max_y, cheek_min_x: cheek_max_x, :]\n",
    "            cheek_region = cv2.cvtColor(cheek_region, cv2.COLOR_BGR2RGB)\n",
    "            if face_direction == \"left\":\n",
    "                cheek_file_name = join(croppedFaces_Dir, imageName+\"_lc.jpg\")\n",
    "            elif face_direction == \"right\":\n",
    "                cheek_file_name = join(croppedFaces_Dir, imageName+\"_rc.jpg\")\n",
    "            else:\n",
    "                cheek_file_name = join(croppedFaces_Dir, imageName+\"_c.jpg\")\n",
    "            misc.imsave(cheek_file_name, cheek_region)\n",
    "            if verb:\n",
    "                image = img\n",
    "                image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "                if forehead_ok:\n",
    "                    cv2.rectangle(image, (forehead_min_x, forehead_min_y), \\\n",
    "                                  (forehead_max_x, forehead_max_y), (0, 255, 0), 5)\n",
    "                cv2.rectangle(image, (cheek_min_x, cheek_min_y), \\\n",
    "                              (cheek_max_x, cheek_max_y), (255, 255, 0), 5)\n",
    "                cv2.rectangle(image,(ex,ey),(ex+ew,ey+eh), (0,255,0), 5)\n",
    "                               \n",
    "                plt.imshow(image)\n",
    "                #plt.imshow(roi_color)\n",
    "    if (not face_detected) and (not eye_detected): # no face detected, nor eye detected, save the entire image and write to dest\n",
    "        print(\"No cheeks or forehead detected, output the original file %s.jpg\"%imageName)\n",
    "        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "        if verb:\n",
    "            plt.imshow(img)\n",
    "        outfile = join(croppedFaces_Dir, imageName+\".jpg\")\n",
    "        misc.imsave(outfile, img)\n",
    "\n",
    "    if image_counter % 500 == 0: # Report the progress on image processing every 500 images processed\n",
    "        print(\"%d images have been processed.\"%image_counter)\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}