5 Random forest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "56733ad6-2e01-440a-b1f2-ab5fdb36a2c2",
   "metadata": {},
   "source": [
    "# Random forest model\n",
    "Author: Roddy Jaques <br>\n",
    "*NHS Blood and Transplant*\n",
    "***\n",
    "\n",
    "\n",
    "## Assessing the predictive ability of a random forest model\n",
    "In this notebook random forest models will be fit to the DBD and DCD cohorts.\n",
    "\n",
    "First load in the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "349c70cd-a4aa-4d91-b5ba-ffeb4c7594af",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "import sklearn.metrics as mets\n",
    "import time\n",
    "%matplotlib inline\n",
    "\n",
    "# Function to print confusion matrix, balanced accuracy and accuracy for a set of actual and predicted labels\n",
    "def show_metrics(actual,predict):\n",
    "    \"\"\" Prints the confusion matrix, balanced accuracy and accuracy given datasets of actual and predicted labels\n",
    "    \n",
    "    Arguments:\n",
    "        actual - Dataset of actual labels\n",
    "        predict - Dataset of predicted labels\n",
    "     \"\"\"\n",
    "    cm = mets.confusion_matrix(actual, predict)\n",
    "    \n",
    "    print(\"********* MODEL METRIC REPORT *********\\n\\nConfusion matrix:\\n\")\n",
    "\n",
    "    print(\"TP  FN\\nFP  TN\\n\") #this is a reminder of what each part of the confusion matrix means e.g. TP = True Positive\n",
    "    \n",
    "    # print the confusion matrix\n",
    "    print(str(int(cm[0,0])) + \"    \" + str(int(cm[0,1])))\n",
    "    print(str(int(cm[1,0])) + \"    \" + str(int(cm[1,1])) + \"\\n\") \n",
    "\n",
    "    # classification report for DBD model\n",
    "    print(\"Classification report:\\n\")\n",
    "    print(mets.classification_report(actual, predict))\n",
    "\n",
    "    print(\"Balanced accuracy: \" + str(round(mets.balanced_accuracy_score(actual, predict),2)))\n",
    "\n",
    "    print(\"Accuracy: \" + str(round(mets.accuracy_score(actual, predict),2)))\n",
    "    \n",
    "    # Predicted vs actual consent rates\n",
    "    cons_rate = int(100 * len(actual[actual==\"Consent\"]) / len(actual) )\n",
    "    print(\"\\nActual consent rate: \" + str(cons_rate))\n",
    "    \n",
    "    pred_rate = int(100 * len(predict[predict==\"Consent\"]) / len(predict) )\n",
    "    print(\"Predicted consent rate: \" + str(pred_rate))\n",
    "    \n",
    "    pass\n",
    " \n",
    "# Function to format consent column from integer code to text\n",
    "def format_consent(x):\n",
    "    if x == 2:\n",
    "        return \"Consent\"\n",
    "    if x == 1:\n",
    "        return \"Non-consent\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2971600f-bfdd-4da6-84a6-99da4e72fd10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in datasets \n",
    "dbd_model_data = pd.read_csv(\"Data/dbd_model_data.csv\")\n",
    "dcd_model_data = pd.read_csv(\"Data/dcd_model_data.csv\")\n",
    "\n",
    "# Columns used to create DBD model\n",
    "dbd_cols = [\"wish\", \"FORMAL_APR_WHEN\", \"donation_mentioned\", \"app_nature\", \"eth_grp\", \"religion_grp\", \"GENDER\", \"FAMILY_WITNESS_BSDT\", \"DTC_PRESENT_BSD_CONV\", \n",
    "            \"acorn_new\", \"adult\",\"FAMILY_CONSENT\"]\n",
    "\n",
    "dbd_model_data2 = pd.get_dummies(data=dbd_model_data,columns=dbd_cols[:-1],drop_first=True)\n",
    "\n",
    "dbd_features = dbd_model_data2.drop(\"FAMILY_CONSENT\",axis=1)\n",
    "dbd_consents = dbd_model_data2[\"FAMILY_CONSENT\"].apply(format_consent)\n",
    "\n",
    "# Columns used to create DCD model in paper\n",
    "dcd_cols = [\"wish\", \"donation_mentioned\", \n",
    "            \"app_nature\", \"eth_grp\", \"religion_grp\", \"GENDER\", \"DTC_WD_TRTMENT_PRESENT\", \n",
    "            \"acorn_new\", \"adult\",\"cod_neuro\",\"FAMILY_CONSENT\"]\n",
    "\n",
    "dcd_model_data2 = pd.get_dummies(data=dcd_model_data,columns=dcd_cols[:-1],drop_first=True)\n",
    "\n",
    "dcd_features = dcd_model_data2.drop(\"FAMILY_CONSENT\",axis=1)\n",
    "dcd_consents = dcd_model_data2[\"FAMILY_CONSENT\"].apply(format_consent)\n",
    "\n",
    "# creating a train and testing dataset for DBD and DCD approaches\n",
    "DBD_X_train, DBD_X_test, DBD_y_train, DBD_y_test = train_test_split(dbd_features,dbd_consents, test_size=0.33, random_state=10)\n",
    "\n",
    "DCD_X_train, DCD_X_test, DCD_y_train, DCD_y_test = train_test_split(dcd_features,dcd_consents, test_size=0.33, random_state=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e668f1bc-565f-4b79-bae9-43245b4d2229",
   "metadata": {},
   "source": [
    "Fit random forest models with default hyperparameters to the DBD and DCD cohorts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "67b8d836-2fa2-419a-b68d-345ec189ab18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# fitting untuned random forest to training data \n",
    "tree_model = RandomForestClassifier(n_estimators=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5c6ada3a-8786-4e64-93b7-ce465e4dad2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1122    260\n",
      "314    304\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.78      0.81      0.80      1382\n",
      " Non-consent       0.54      0.49      0.51       618\n",
      "\n",
      "    accuracy                           0.71      2000\n",
      "   macro avg       0.66      0.65      0.66      2000\n",
      "weighted avg       0.71      0.71      0.71      2000\n",
      "\n",
      "Balanced accuracy: 0.65\n",
      "Accuracy: 0.71\n",
      "\n",
      "Actual consent rate: 69\n",
      "Predicted consent rate: 71\n"
     ]
    }
   ],
   "source": [
    "DBD_tree = tree_model.fit(DBD_X_train,DBD_y_train)\n",
    "\n",
    "DBD_preds = DBD_tree.predict(DBD_X_test)\n",
    "\n",
    "show_metrics(DBD_y_test,DBD_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6cb3d982-8a27-4319-a4de-1c89f23b4760",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1388    477\n",
      "463    776\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.75      0.74      0.75      1865\n",
      " Non-consent       0.62      0.63      0.62      1239\n",
      "\n",
      "    accuracy                           0.70      3104\n",
      "   macro avg       0.68      0.69      0.68      3104\n",
      "weighted avg       0.70      0.70      0.70      3104\n",
      "\n",
      "Balanced accuracy: 0.69\n",
      "Accuracy: 0.7\n",
      "\n",
      "Actual consent rate: 60\n",
      "Predicted consent rate: 59\n"
     ]
    }
   ],
   "source": [
    "DCD_tree = tree_model.fit(DCD_X_train,DCD_y_train)\n",
    "\n",
    "DCD_preds = DCD_tree.predict(DCD_X_test)\n",
    "\n",
    "dcd_cm = mets.confusion_matrix(DCD_y_test, DCD_preds)\n",
    "\n",
    "show_metrics(DCD_y_test,DCD_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f5347f-cbdd-48c8-a446-5933099aed3c",
   "metadata": {},
   "source": [
    "***\n",
    "#### DBD model\n",
    "The balanced accuracy for the DBD model is 0.65, 0.02 lower than the logistic regression model. The non-consent recall is 0.49 (0.05 more than the logistic regression), but the recall for consents is 0.81 which is 0.09 lower than the logistic regression model. \n",
    "\n",
    "The predicted consent rate is 71%, closer to the actual consent rate than the logistic regression predicts.\n",
    "\n",
    "#### DCD model\n",
    "The DCD model has a balanced accuracy of 0.69, 0.02 lower than the logistic regression model. The recall for consents is 0.75, 0.05 lower than the logistic regression. The recall for non-consents is 0.01 higher than the logistic regression model.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fd598cd-2a0e-48ba-8063-7a4463df615c",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning\n",
    "\n",
    "As before, a 5-fold cross validated grid search will be used to tune hyperparameters, optimising for balanced accuracy.<br>\n",
    "For both models the hyper parameters and range of values to be explored are: <br>\n",
    "* max_depth - the maximum tree depth. From 1 to 200 in increments of 25.\n",
    "* min_samples_split - the minimum number of samples needed in a leaf. From 2 to 200 in increments of 25.\n",
    "* class weight - a weighting parameter to members of the non-consent class so consents aren't overfit. From 1 to 4 in increments of 0.25.\n",
    "* n_estimators - the number of trees in the forest. From 10 to 100 in steps of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e45d53a0-0e94-4848-a18c-83cd0dbaa171",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime = 11.5minutes\n",
      "{'class_weight': {'Non-consent': 2.75, 'Consent': 1}, 'max_depth': 26, 'min_samples_split': 177, 'n_estimators': 50}\n",
      "0.7497479664748189\n"
     ]
    }
   ],
   "source": [
    "cv_forest_model = RandomForestClassifier(random_state=66)\n",
    "\n",
    "# create list of dictionaries with weighting on Non-consent class\n",
    "weights = []\n",
    "for w in np.arange(2,4,step=0.25):\n",
    "    w_dic = {\"Non-consent\":w,\"Consent\":1}\n",
    "    weights.append(w_dic)\n",
    "\n",
    "# dictionary of hyperparameter values to be explored\n",
    "params = {'max_depth':np.arange(1,200,step=25),'min_samples_split':np.arange(2,200,step=25),'class_weight':weights,'n_estimators':np.arange(10,100,step=10)}\n",
    "\n",
    "# to caclulate run time\n",
    "start_time = time.time()\n",
    "\n",
    "# train and fit model using cross validated grid search\n",
    "dbd_gs_forest_model = GridSearchCV(cv_forest_model, param_grid=params, scoring=\"balanced_accuracy\",cv=5,n_jobs=3)\n",
    "\n",
    "dbd_gs_forest_model.fit(DBD_X_train,DBD_y_train)\n",
    "\n",
    "# calculate and print run time\n",
    "runtime = time.time() - start_time\n",
    "print(\"Runtime = {}minutes\".format(round(runtime/60,1)))\n",
    "\n",
    "# show hyperparams and balanced accuracy for model with best balanced accuracy\n",
    "dbd_gs_forest_model.score(DBD_X_train,DBD_y_train)\n",
    "\n",
    "print(dbd_gs_forest_model.best_params_)\n",
    "print(dbd_gs_forest_model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5ccf8ecd-e838-49ff-85db-bff5fccc1760",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "858    524\n",
      "77    541\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.92      0.62      0.74      1382\n",
      " Non-consent       0.51      0.88      0.64       618\n",
      "\n",
      "    accuracy                           0.70      2000\n",
      "   macro avg       0.71      0.75      0.69      2000\n",
      "weighted avg       0.79      0.70      0.71      2000\n",
      "\n",
      "Balanced accuracy: 0.75\n",
      "Accuracy: 0.7\n",
      "\n",
      "Actual consent rate: 69\n",
      "Predicted consent rate: 46\n"
     ]
    }
   ],
   "source": [
    "DBD_preds = dbd_gs_forest_model.predict(DBD_X_test)\n",
    "\n",
    "show_metrics(DBD_y_test,DBD_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4eb9376b-8e47-4a23-9ebd-127ea2c910ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime = 15.1minutes\n",
      "{'class_weight': {'Non-consent': 2.0, 'Consent': 1}, 'max_depth': 26, 'min_samples_split': 77, 'n_estimators': 50}\n",
      "0.7325624705327677\n"
     ]
    }
   ],
   "source": [
    "# repeat for DCD cohort\n",
    "start_time = time.time()\n",
    "\n",
    "dcd_gs_forest_model = GridSearchCV(cv_forest_model, param_grid=params, scoring=\"balanced_accuracy\",cv=5,n_jobs=3)\n",
    "\n",
    "dcd_gs_forest_model.fit(DCD_X_train,DCD_y_train)\n",
    "\n",
    "runtime = time.time() - start_time\n",
    "print(\"Runtime = {}minutes\".format(round(runtime/60,1)))\n",
    "\n",
    "dcd_gs_forest_model.score(DCD_X_train,DCD_y_train)\n",
    "\n",
    "print(dcd_gs_forest_model.best_params_)\n",
    "print(dcd_gs_forest_model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "45088b20-6ded-4845-8d96-5833477a3736",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1029    836\n",
      "142    1097\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.88      0.55      0.68      1865\n",
      " Non-consent       0.57      0.89      0.69      1239\n",
      "\n",
      "    accuracy                           0.68      3104\n",
      "   macro avg       0.72      0.72      0.68      3104\n",
      "weighted avg       0.75      0.68      0.68      3104\n",
      "\n",
      "Balanced accuracy: 0.72\n",
      "Accuracy: 0.68\n",
      "\n",
      "Actual consent rate: 60\n",
      "Predicted consent rate: 37\n"
     ]
    }
   ],
   "source": [
    "DCD_preds = dcd_gs_forest_model.predict(DCD_X_test)\n",
    "\n",
    "show_metrics(DCD_y_test,DCD_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb24f7e-aeb2-41ef-b267-a5ee7bb56bc6",
   "metadata": {},
   "source": [
    "***\n",
    "#### DBD model\n",
    "The balanced accuracy for this model is 0.75, 0.08 higher than the logistic regression model and an improvement on the untuned random forest model.\n",
    "As with the tuned decision tree model the recall for non-consents increased at the expense of the recall for non-consents. \n",
    "\n",
    "The predicted consent rate is much lower than the actual consent rate, as it was for the tuned decision tree.\n",
    "\n",
    "#### DCD model\n",
    "The 0.72 balanced accuracy for this model is only a 0.01% increase on the logistic regression model. The non-consent recall is 0.89, 0.37 higher than the logistic regression model. The consent recall is 0.55, a 0.24 decrease compared to the logistic regression.\n",
    "\n",
    "The predicted consent rate is much lower than the actual consent rate.\n",
    "\n",
    "***\n",
    "\n",
    "After fitting two models it has become apparent that untuned models are much better at identifying consents than non-consents. Tuning models with a weighting hyperparameter for non-consents and optimising for balanced accuracy reverses this; the recall for non-consents is much higher than the recall for consents in tuned models. This trade off is undesirable as useful models should be good at identifying both consents and non-consents.<br> \n",
    "The untuned models overestimate the consent rate, and the tuned models underestimate the consent rate. So all models are relying on over estimation of one class to correctly identify members of that class, no model has improved the ability of the model to correctly classify both classes.<br><br>\n",
    "The untuned models overestimate consent, this is because of the wish variable in the dataset. This variable shows the potential donor's prior decision to donate. The highest odds ratio in the logistic regression model for any variable is the variable representing patients who have previously expressed a wish to donate, and intuitively people who have expressed a wish to donate organs are more likely to donate organs.<br><br>\n",
    "\n",
    "In the next notebook a random forest will be fit to datasets without the wish variable in an attempt to create a model that is more balanced in its ability to  predict consents and non-consents.\n",
    "***\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}