160306-Readme.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In [Second Annual Data Science Bowl Kaggle competition](https://www.kaggle.com/c/second-annual-data-science-bowl) you estimate the heart volume at maximal expansion and contraction using an MRI study.\n",
    "\n",
    "My solution localizes the heart in the horizontal (sax) slice images of a study and use it to crop the images. It also converts the time sequence to channels (dc and first two DFT with phase.)\n",
    "\n",
    "Each horizontal cropped slices are feed into a CNN which predicts the volume contribution of each slice to the entire volume of the heart. When predicting, the results from the same study are added up.\n",
    "When training a special arrangement is used in which all slices from the same study appear in the same batch\n",
    "and the loss function sums all slices from the same study before computing loss.\n",
    "\n",
    "CNN predicts both the volume and the error of the prediction and the loss is negative log liklihood of a normal distribution.\n",
    "\n",
    "The final submission is made from ensembly of many predictions, each based on a different sub set of the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The solution is made from several steps, each a jupyter notebook, which you execute one after the other.\n",
    "Results of each step can be stored on S3 allowing for parallel execution in some of the steps on several AWS EC2 instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "## hardware\n",
    "All steps are run on AWS EC2 `g2.2xlarge` instance running `Ubuntu 14.04.3 LTS`. But many other Linux / OS X will also work.\n",
    "\n",
    "## software\n",
    "Code is running on `python 2.7`\n",
    "\n",
    "Install [my fork of Keras](https://github.com/udibr/keras/tree/validate_batch_size)\n",
    "\n",
    "```bash\n",
    "pip install git+git://github.com/udibr/keras.git#validate_batch_size\n",
    "```\n",
    "\n",
    "## configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"TRAIN_DATA_PATH\": \"/vol1/data/train\",\r\n",
      "  \"VALID_DATA_PATH\": \"/vol1/data/validate\",\r\n",
      "  \"TEST_DATA_PATH\": \"/vol1/data/test\",\r\n",
      "  \"OUT_DATA_PATH\": \"s3://udikaggle/dsb.2\",\r\n",
      "  \"TEMP_DATA_PATH\": \"/mnt/data\"\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!cat SETTINGS.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each step stores its results in `OUT_DATA_PATH`.\n",
    "Modify this field to an S3 bucket and folder for which you have access.\n",
    "You can change it to a regular file system directory.\n",
    "\n",
    "    Some notebooks require a large disk space to store temporary results in `TEMP_DATA_PATH` make sure the path you are chossing exists (and/or mounted) and that you have enough disk space.\n",
    "\n",
    "## data\n",
    "[Download the data](./160308-download.ipynb)\n",
    "\n",
    "`train.csv` should be in `TRAIN_DATA_PATH` and should have exactly `Nt=500` studies.\n",
    "This directory should also contain all the 500 train DICOM study sub directories named `1` $\\ldots$ `500`\n",
    "\n",
    "`validate.csv` should be in `VALID_DATA_PATH` and should have exactly `Nv=200` studies.\n",
    "It could be missing if first stage run is performed.\n",
    "This directory should contain all the 200 valdiation study sub directories named `501` $\\ldots$ `700`\n",
    "\n",
    "`TEST_DATA_PATH` should contain all the `Ns=440` test study sub directories named `701` $\\ldots$ `1140`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "[Patient information is read](./160306-patient.ipynb) from the CSV file(s) and from DICOM meta data and placed in a\n",
    "Pandas Data Frame.\n",
    "\n",
    "## localization\n",
    "    you can perform the localization step in parallel to the preprocessing step\n",
    "    \n",
    "Both modeling and prediction require the approximate location and size of the LV in each image slice.\n",
    "\n",
    "The next two notebooks are almost identical to the [deep learning tutorial](https://www.kaggle.com/c/second-annual-data-science-bowl/details/deep-learning-tutorial)\n",
    "and you should follow the installation process of that tutorial in order for the notebooks below to work.\n",
    "\n",
    "You first need to [build a pixel-model](./160206-FCN_model.ipynb) which detemrines if each indvidual pixel belongs to LV.\n",
    "The building of the pixel model uses the Sunnybrook dataset\n",
    "(see instructions at start of notebook on how to download and open these file.)\n",
    "\n",
    "    This dataset does not change in the different stages of the competition and you can use my precomputed model which is downloaded for you in the next notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next you need to predict the [pixel value](./160306-FCN_tutorial.ipynb) for the entire competition data set which is stored in \"masks\".\n",
    "\n",
    "Next run a [fourier analysis](./160306-segment.ipynb) which performs the localization based on the pixel level predictions read from the masks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cropping\n",
    "[crop, zoom and rotate](./160306-crop.ipynb) the area around the LV from every slice and store the reuslt in single file. This step also collapses the time sequence of images into 5 channels. DC and the first two frequencies of a DFT with phase.\n",
    "\n",
    "At the bottom of this notebook you have the option to review samples from 120 studies at a time, validating that the croped images cover the LV and that it appears in about the same size and orientation in all studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "[Build a model and use it to predict the heart volume on test data](./160306-train.ipynb).\n",
    "The modeling looks on only $2/3$ of the training data based on the value assigned to the\n",
    "`seed` and `isplit` variables at the top of the notebook.\n",
    "\n",
    "An ensembly of predictions is built by running the notebook many times, each with different `seed` and `isplit` values which you have to manipulate, by hand, before every run.\n",
    "For stage 2 of the competiton generate 60 models.\n",
    "The `seed` values ranging from `1000` to `1019` (20 different values) and for each `seed` value\n",
    "run on 3 different `isplit` values: `0`, `1` and `2`.\n",
    "    \n",
    "    You can run in parallel on many `g2.2xlarge` AWS EC2 instances. After starting each instance you can \"forget\" it since it will self terminate once the final prediction is made.\n",
    "\n",
    "# Building submission\n",
    "Finally, [fuse the ensembly](./160313-fuse.ipynb) to a single submission file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}