Case Study 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MSDS 7331 - Case Study 1 - Superconducting Materials\n",
    "Daniel Crouthamel,\n",
    "Sophia Wu,\n",
    "Fabio Savorgnan,\n",
    "Bo Yun\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this study, we will build a linear regression model using L1 or L2 regularization to predict the critical temperature. There are total 6 sections in this case analysis:\n",
    "\n",
    "1. Business Understanding\n",
    "2. Data Evaluation / Engineering\n",
    "3. Modeling Preparations\n",
    "4. Model Building & Evaluation\n",
    "5. Model Interpretability & Explainability\n",
    "6. Case Conclusions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing libraries and reading in file\r\n",
    "import numpy as np\r\n",
    "import pandas as pd\r\n",
    "import matplotlib.pyplot as plt\r\n",
    "import seaborn as sns\r\n",
    "\r\n",
    "#general sklearn libraries\r\n",
    "from sklearn.model_selection import train_test_split\r\n",
    "from sklearn.model_selection import GridSearchCV\r\n",
    "from sklearn.preprocessing import StandardScaler\r\n",
    "from sklearn.preprocessing import RobustScaler\r\n",
    "from sklearn.preprocessing import MinMaxScaler\r\n",
    "from sklearn.pipeline import make_pipeline\r\n",
    "\r\n",
    "#logistic regression\r\n",
    "from sklearn.linear_model import Lasso\r\n",
    "from sklearn.linear_model import LinearRegression\r\n",
    "from sklearn.linear_model import ElasticNet\r\n",
    "from sklearn.linear_model import Ridge\r\n",
    "from sklearn.pipeline import make_pipeline\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\r\n",
    "## Business Understanding\r\n",
    "\r\n",
    "**Objective:** \r\n",
    "The objective of this case study is to explore Linerar Regression with L1 and L2 regularization, and the impact to predicting the critical temperature of a superconductor. Additionally, feature importance is also investigated with the best model. Three models will be considered.\r\n",
    "\r\n",
    "* Lasso (L1)\r\n",
    "* Ridge (L2)\r\n",
    "* Elastic Net (L1 and L2).\r\n",
    "\r\n",
    "Grid search will be performed on each model to find the optimal hyperparameters. We'll then use the optimal hyperparameters for each model and then determine which model performs the best. Along the way will explore the importance of normalization and scaling and condlude with a summary of the most important features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Evaluation and Engineering\n",
    "\n",
    "\n",
    "Below we'll load our supoerconductor data and then perform a quick data exploration. Our data consists of two files which were merged together. Some initial obervations are:\n",
    "\n",
    "* There is no missing data\n",
    "* We have features with constant values\n",
    "* There is one string object in the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Data\r\n",
    "data_train = pd.read_csv('data/train.csv')\r\n",
    "data_materials = pd.read_csv('data/unique_m.csv')\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Addressing missing Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21263 entries, 0 to 21262\n",
      "Columns: 169 entries, number_of_elements to material\n",
      "dtypes: float64(156), int64(12), object(1)\n",
      "memory usage: 27.4+ MB\n",
      "\n",
      "Missing Data? Index([], dtype='object')\n",
      "\n",
      "Columns that have the same value ['He' 'Ne' 'Ar' 'Kr' 'Xe' 'Pm' 'Po' 'At' 'Rn']\n"
     ]
    }
   ],
   "source": [
    "\r\n",
    "# Drop the duplicate column 'critical_temp' in the first frame\r\n",
    "data_train = data_train.drop(['critical_temp'], axis=1)\r\n",
    "\r\n",
    "# Merge the two frames\r\n",
    "data = pd.merge(data_train, data_materials, left_index=True, right_index=True)\r\n",
    "\r\n",
    "# Data frame to csv drop index\r\n",
    "#data.to_csv('data/super_conducter_data.csv', index=False)\r\n",
    "\r\n",
    "# Print out some typicaly descriptive statistics\r\n",
    "print(\"\")\r\n",
    "data.info()\r\n",
    "data.describe()\r\n",
    "\r\n",
    "# Do we have missing data?\r\n",
    "print(\"\")\r\n",
    "print(\"Missing Data?\", data.columns[data.isnull().any().values])\r\n",
    "\r\n",
    "# Columns with a Constant value\r\n",
    "print(\"\")\r\n",
    "print(\"Columns that have the same value\", data.columns[data.nunique() <= 1].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Cleaning - Duplicates\n",
    "Now we will work on cleaning up any duplicates in our dataset.The string feature will be removed from our data set. This appears to be a name, and some values are duplicated. We felt it was safe to remove. Addtionally, the features identified above with constant values will be removed.\n",
    "After doing a quick profiling on all of 159 columns, we don't see any missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(21263, 159)\n"
     ]
    }
   ],
   "source": [
    "# Drop columns with constant values\r\n",
    "data.drop(columns=['material', 'He', 'Ne', 'Ar', 'Kr', 'Xe', 'Pm', 'Po', 'At', 'Rn'], inplace=True)\r\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Statistics\n",
    "We will next review some simple statistics on our dataset.This will be primarily focused on our main expected variable of 'temperature'. First, we'll check the range, mode, mean, median, variance and counts for the variable of 'temperature'. Then we'll check the relationships between temperature and other attributes.\n",
    "#### Median, Standard Deviation, Mean of  'critical temperature'\n",
    "We can see from observing the simple statistic is that the median critical temperature is around 20. However, what's interesting to note here is that the median and the mean are very different with the big gap of 14.42. There is a standard deviation of about 34.254. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    index  critical_temp\n",
      "0  median      20.000000\n",
      "1     std      34.254362\n",
      "2    mean      34.421219\n",
      "min: 0.00021\n",
      "min: 185.0\n"
     ]
    }
   ],
   "source": [
    "print(data['critical_temp'].aggregate([np.median, np.std, np.mean]).reset_index())\r\n",
    "print('min: '+str(data['critical_temp'].min()))\r\n",
    "print('min: '+str(data['critical_temp'].max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Distribution of critical temperature\n",
    "As our main focus variable is critical temperature, we will look deeper into this variable. From a simple IQR districution, we can see the majority of the data is between 5 and 63."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25%     5.365\n",
      "50%    20.000\n",
      "75%    63.000\n",
      "Name: critical_temp, dtype: float64\n",
      "185.0\n"
     ]
    }
   ],
   "source": [
    "print(data.describe()['critical_temp'][['25%', '50%', '75%']])\r\n",
    "print(data.max()['critical_temp'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we want to visualize the distribution for our primary variable critical temperature with a boxplot and histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dancr\\miniconda3\\envs\\NLP\\lib\\site-packages\\seaborn\\_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAJNCAYAAACBe1nxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhIklEQVR4nO3dfbRld13f8c83MwRQwUAyKwsnkw6aqEVawTUCglLLY6SWoPLkcpmpYoMWYsD6hLZiUVq0KkJaH1LJMnEhD6IuIqVgENDVUh4mgGB4KFcgJkMIIcGABoKT/PrH2YN3Mvdm7p3Mued773291rprzvmdfc79zc6+c9/Z++yza4wRAAD6OWnREwAAYGVCDQCgKaEGANCUUAMAaEqoAQA0tXPRE5iH0047bezdu3fR0wAAOKYrr7zy02OMXSs9tiVDbe/evTlw4MCipwEAcExVdfVqjzn0CQDQlFADAGhKqAEANCXUAACaEmoAAE0JNQCApoQaAEBTQg0AoCmhBgDQlFADAGhKqAEANCXUAACaEmoAAE0JNQCApoQaAEBTQg0AoCmhBgDQlFADAGhKqAEANCXUAACaEmoAAE0JNQCApnYuegLbzUUXXZSlpaVFT+NOHTx4MEmye/fuBc+EE+mss87KBRdcsOhpALAOQm2DLS0t5b1/9cHc9mX3XfRUVrXjlpuTJJ+81eaxVey45aZFTwGA4+A38QLc9mX3zee//gmLnsaq7vmh1ydJ6zmyPof/mwKwuXiPGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqx+miiy7KRRddtOhpANCM3w+cSDsXPYHNamlpadFTAKAhvx84kexRAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgDACm688cb86I/+aG688caFzUGoAQCs4NJLL8373//+XHbZZQubg1ADALiDG2+8MW94wxsyxsgb3vCGhe1V27mQ77oFHDx4MJ///Odz4YUXrut5S0tLOemLY06zgpWd9IXPZmnpc+veXoH1W1payj3vec9FT4O76NJLL83tt9+eJLntttty2WWX5bnPfe6Gz2PL7FGrqvOr6kBVHbjhhhsWPR0AYBN705velEOHDiVJDh06lCuuuGIh89gye9TGGBcnuThJ9u3bN/ddVrt3706SvOQlL1nX8y688MJc+dHr5zElWNXt97h3zvrq09e9vQLrZ8/11vCYxzwmr3/963Po0KHs3Lkzj33sYxcyjy2zRw0A4ETZv39/Tjpplkk7duzIeeedt5B5CDUAgDs49dRTc84556Sqcs455+TUU09dyDy2zKFPAIATaf/+/fn4xz++sL1piVADAFjRqaeempe+9KULnYNDnwAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFM7Fz2Bzeqss85a9BQAaMjvB04koXacLrjggkVPAYCG/H7gRHLoEwCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmdi56AtvRjltuyj0/9PpFT2NVO265MUlaz5H12XHLTUlOX/Q0AFgnobbBzjrrrEVP4ZgOHjyUJNm92y/2reP0TbHtAXAkobbBLrjggkVPAQDYJLxHDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATdUYY9FzOOGq6oYkV2/Atzotyac34PtsFtbH0ayTI1kfR7NOjmR9HM06OdpWWyf/ZIyxa6UHtmSobZSqOjDG2LfoeXRhfRzNOjmS9XE06+RI1sfRrJOjbad14tAnAEBTQg0AoCmhdtdcvOgJNGN9HM06OZL1cTTr5EjWx9Gsk6Ntm3XiPWoAAE3ZowYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmtq56AnMw2mnnTb27t276GkAABzTlVde+ekxxq6VHtuSobZ3794cOHBg0dMAADimqrp6tccc+gQAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdTugt17zkxVrelr954zFz1dAGCT2ZLX+twon7j2mjztt9+2pmVf9cyHz3k2AMBWY48aAEBTQg0AoCmhBgDQlFADAGhKqAEANCXUAACaEmoAAE0JNQCApoQaAEBTQg0AoCmhBgDQlFADAGhKqAEANDX3UKuqHVX1nqp63XT//lX1jqpaqqpXVdXJ0/jdp/tL0+N7l73G86bxD1fV4+c9ZwCADjZij9qFST647P4vJXnxGOOsJJ9J8oxp/BlJPjONv3haLlX1gCRPT/INSc5J8htVtWMD5g0AsFBzDbWqOiPJv0ryO9P9SvKoJK+ZFrk0yZOm2+dO9zM9/uhp+XOTvHKMcesY42NJlpI8ZJ7zBgDoYN571H49yU8muX26f2qSvx1jHJruX5tk93R7d5JrkmR6/OZp+S+Nr/CcL6mq86vqQFUduOGGG07wXwMAYOPNLdSq6juTfGqMceW8vsdyY4yLxxj7xhj7du3atRHfEgBgrnbO8bUfkeSJVfWEJPdIcu8kL0lySlXtnPaanZHk4LT8wSR7klxbVTuTfGWSG5eNH7b8OQAAW9bc9qiNMZ43xjhjjLE3s5MB3jzG+L4kb0ny5Gmx/UleO92+fLqf6fE3jzHGNP706azQ+yc5O8k75zVvAIAu5rlHbTU/leSVVfWLSd6T5GXT+MuS/F5VLSW5KbO4yxjjqqp6dZIPJDmU5FljjNs2ftoAABtrQ0JtjPHWJG+dbn80K5y1Ocb4QpKnrPL8FyZ54fxmCADQjysTAAA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNzC7WqukdVvbOq/rKqrqqq/zSN37+q3lFVS1X1qqo6eRq/+3R/aXp877LXet40/uGqevy85gwA0Mk896jdmuRRY4xvTPKgJOdU1cOS/FKSF48xzkrymSTPmJZ/RpLPTOMvnpZLVT0gydOTfEOSc5L8RlXtmOO8AQBamFuojZm/m+7ebfoaSR6V5DXT+KVJnjTdPne6n+nxR1dVTeOvHGPcOsb4WJKlJA+Z17wBALqY63vUqmpHVb03yaeSXJHkr5P87Rjj0LTItUl2T7d3J7kmSabHb05y6vLxFZ6z/HudX1UHqurADTfcMIe/DQDAxpprqI0xbhtjPCjJGZntBfv6OX6vi8cY+8YY+3bt2jWvbwMAsGE25KzPMcbfJnlLkm9JckpV7ZweOiPJwen2wSR7kmR6/CuT3Lh8fIXnAABsWfM863NXVZ0y3b5nkscm+WBmwfbkabH9SV473b58up/p8TePMcY0/vTprND7Jzk7yTvnNW8AgC52HnuR43a/JJdOZ2ielOTVY4zXVdUHkryyqn4xyXuSvGxa/mVJfq+qlpLclNmZnhljXFVVr07ygSSHkjxrjHHbHOcNANDC3EJtjPG+JA9eYfyjWeGszTHGF5I8ZZXXemGSF57oOQIAdObKBAAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaG2UU7amapa09fuPWcuerYAQAM7Fz2BbeP2Q3nab79tTYu+6pkPn/NkAIDNwB41AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJpaU6hV1SPWMgYAwImz1j1qF61xDACAE+ROr0xQVd+S5OFJdlXVjy176N5JdsxzYgAA292xLiF1cpKvmJa717LxzyZ58rwmBQDAMUJtjPHnSf68qn53jHH1Bs0JAICs/aLsd6+qi5PsXf6cMcaj5jEpAADWHmp/kOS3kvxOktvmNx0AAA5ba6gdGmP85lxnAgDAEdb68Rx/UlX/rqruV1X3Pfw115kBAGxza92jtn/68yeWjY0kX31ipwMAwGFrCrUxxv3nPREAAI60plCrqvNWGh9jXHZipwMAwGFrPfT5zctu3yPJo5O8O4lQAwCYk7Ue+rxg+f2qOiXJK+cxIQAAZtZ61ucd/X0S71sDAJijtb5H7U8yO8szmV2M/Z8mefW8JgUAwNrfo/Yry24fSnL1GOPaOcwHAIDJmg59Thdn/1CSeyW5T5IvznNSAACsMdSq6qlJ3pnkKUmemuQdVfXkeU4MAGC7W+uhz59N8s1jjE8lSVXtSvKmJK+Z18QAALa7tZ71edLhSJvcuI7nAgBwHNa6R+0NVfXGJK+Y7j8tyevnMyUAAJJjhFpVnZXk9DHGT1TVdyf51umh/5vk5fOeHADAdnasPWq/nuR5STLG+KMkf5QkVfXPpsf+9RznBgCwrR3rfWanjzHef8fBaWzvnT2xqvZU1Vuq6gNVdVVVXTiN37eqrqiqj0x/3mcar6p6aVUtVdX7quqblr3W/mn5j1TV/nX/LQEANqFjhdopd/LYPY/x3ENJ/v0Y4wFJHpbkWVX1gCQ/neTPxhhnJ/mz6X6SfEeSs6ev85P8ZjILuyTPT/LQJA9J8vzDcQcAsJUdK9QOVNW/veNgVf1Qkivv7IljjOvGGO+ebn8uyQeT7E5ybpJLp8UuTfKk6fa5SS4bM29PckpV3S/J45NcMca4aYzxmSRXJDlnLX85AIDN7FjvUXtOkj+uqu/LP4bZviQnJ/mutX6Tqtqb5MFJ3pHZ4dTrpoc+meT06fbuJNcse9q109hq43f8HudnticuZ5555lqnBgDQ1p2G2hjj+iQPr6p/meSB0/D/HGO8ea3foKq+IskfJnnOGOOzVbX89UdVjVWfvA5jjIuTXJwk+/btOyGvCQCwSGv6HLUxxluSvGW9L15Vd8ss0l4+nTWaJNdX1f3GGNdNhzYPf5DuwSR7lj39jGnsYJJvv8P4W9c7FwCAzWZuVxeo2a6zlyX54Bjj15Y9dHmSw2du7k/y2mXj501nfz4syc3TIdI3JnlcVd1nOongcdMYAMCWttYrExyPRyT5/iTvr6r3TmM/k+RFSV5dVc9IcnVmF3lPZlc6eEKSpSS3JPmBJBlj3FRVv5DkXdNyLxhj3DTHeQMAtDC3UBtj/O8ktcrDj15h+ZHkWau81iVJLjlxswMA6M+F1QEAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0INAKApoQYA0JRQAwBoSqgBADQl1Do6aWeqas1fu/ecuegZAwBzsHPRE2AFtx/K0377bWte/FXPfPgcJwMALIo9agAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdQAAJoSagAATQk1AICmhBoAQFNCDQCgKaEGANCUUAMAaEqoAQA0JdS2gpN2pqrW9LV7z5mLni0AsEY7Fz0BToDbD+Vpv/22NS36qmc+fM6TAQBOFHvUAACaEmoAAE0JNQCApuYWalV1SVV9qqr+atnYfavqiqr6yPTnfabxqqqXVtVSVb2vqr5p2XP2T8t/pKr2z2u+AADdzHOP2u8mOecOYz+d5M/GGGcn+bPpfpJ8R5Kzp6/zk/xmMgu7JM9P8tAkD0ny/MNxx3FyhigAbBpzO+tzjPEXVbX3DsPnJvn26falSd6a5Kem8cvGGCPJ26vqlKq637TsFWOMm5Kkqq7ILP5eMa95b3nOEAWATWOj36N2+hjjuun2J5OcPt3eneSaZctdO42tNn6Uqjq/qg5U1YEbbrjhxM4aAGABFnYywbT3bJzA17t4jLFvjLFv165dJ+plAQAWZqND7frpkGamPz81jR9MsmfZcmdMY6uNAwBseRsdapcnOXzm5v4kr102ft509ufDktw8HSJ9Y5LHVdV9ppMIHjeNAQBseXM7maCqXpHZyQCnVdW1mZ29+aIkr66qZyS5OslTp8Vfn+QJSZaS3JLkB5JkjHFTVf1CkndNy73g8IkFAABb3TzP+vzeVR569ArLjiTPWuV1LklyyQmcGgDApuDKBAAATQk1AICmhBoAQFNCDQCgKaEGsILde850XVxg4eZ21ifAZvaJa69xXVxg4exRYyHsrQCAY7NHjYWwtwIAjs0eNQCApoQaqztp55oPT+48+R5rXraq5jYPh0lZzXoOt697GwWYE4c+Wd3th9Z1eHKtyx5efl7zgJWs53B7YlsCerBHDQCgKaEGANCUUAMAaEqoAa34jD2Af+RkAqAVn7EH8I+EGrB5TR/dArBVCTVg8/LRLcAW5z1qbC0+HBeALcQeNbYWe1gA2ELsUQPWbb2XY1rPJcYA+Ef2qAHrdjyXY7KnE2D97FEDAGhKqAEANCXUAACaEmoAAE0JNQCApoQaAEBTQg0AoCmhxva1jstNrfeSU+v5QFiXsgJgNT7wlu1rHZebStb3Qazr+UDY9X7A6+49Z+YT116zpmV33O3uue0fbl3Tsl91xp4cvOZv1jUXAOZLqMGiTXv21mM9EbjmZX/kkS7hBNCMUINFm+OevXnNw2WeADaGUIO1Oo49XwBwVwg1WCt7nADYYM76BABoSqgBADQl1ADuqnV8Jt88PzfP5/fB1uM9agB3VZP3L87z8/uAxbBHDQCgKaEGANCUUANobD3vOwO2Hu9RA2jM+85ge7NHDWAjreMM0S57ydazV88ZpXBi2aMGsJG6XNt1nZdEW9ecf+SRa37trzpjTw5e8zdrfu212r3nzHzi2mvWvPy85gF31aYJtao6J8lLkuxI8jtjjBcteEoAm9c8P1JkPa+9jqjbcbe757Z/uHXN02gRxHAXbYpQq6odSf57kscmuTbJu6rq8jHGBxY7MwDuknUGo/frsd1slveoPSTJ0hjjo2OMLyZ5ZZJzFzwnAIC5qjHGoudwTFX15CTnjDF+aLr//UkeOsZ49rJlzk9y/nT365J8eAOmdlqST2/A99ksrI+jWSdHsj6OZp0cyfo4mnVytK22Tv7JGGPXSg9sikOfazHGuDjJxRv5PavqwBhj30Z+z86sj6NZJ0eyPo5mnRzJ+jiadXK07bRONsuhz4NJ9iy7f8Y0BgCwZW2WUHtXkrOr6v5VdXKSpye5fMFzAgCYq01x6HOMcaiqnp3kjZl9PMclY4yrFjytZIMPtW4C1sfRrJMjWR9Hs06OZH0czTo52rZZJ5viZAIAgO1osxz6BADYdoQaAEBTQu04VNU5VfXhqlqqqp9e9HwWoar2VNVbquoDVXVVVV04jf98VR2sqvdOX09Y9Fw3SlV9vKreP/29D0xj962qK6rqI9Of91n0PDdKVX3dsu3gvVX12ap6znbbRqrqkqr6VFX91bKxFbeLmnnp9G/L+6rqmxY38/lYZX3816r60PR3/uOqOmUa31tVn1+2rfzWwiY+R6usk1V/TqrqedM28uGqevxiZj0/q6yPVy1bFx+vqvdO41t+G/EetXWaLmf1/7LsclZJvne7Xc6qqu6X5H5jjHdX1b2SXJnkSUmemuTvxhi/ssj5LUJVfTzJvjHGp5eN/XKSm8YYL5qi/j5jjJ9a1BwXZfq5OZjkoUl+INtoG6mqRyb5uySXjTEeOI2tuF1Mv4wvSPKEzNbVS8YYD13U3OdhlfXxuCRvnk4c+6UkmdbH3iSvO7zcVrXKOvn5rPBzUlUPSPKKzK7Y81VJ3pTka8cYt23opOdopfVxh8d/NcnNY4wXbIdtxB619XM5qyRjjOvGGO+ebn8uyQeT7F7srFo6N8ml0+1LM4vZ7ejRSf56jHH1oiey0cYYf5HkpjsMr7ZdnJvZL6cxxnh7klOm/ynaMlZaH2OMPx1jHJruvj2zz8rcNlbZRlZzbpJXjjFuHWN8LMlSZr+Xtow7Wx9VVZntEHjFhk5qgYTa+u1Ocs2y+9dmmwfK9H80D07yjmno2dMhjEu206G+JCPJn1bVlTW7pFmSnD7GuG66/ckkpy9magv39Bz5D+t23UYOW2278O9L8oNJ/tey+/evqvdU1Z9X1bctalILstLPyXbfRr4tyfVjjI8sG9vS24hQ4y6pqq9I8odJnjPG+GyS30zyNUkelOS6JL+6uNltuG8dY3xTku9I8qxp9/2XjNn7DLbdew1q9iHVT0zyB9PQdt5GjrJdt4uVVNXPJjmU5OXT0HVJzhxjPDjJjyX5/aq696Lmt8H8nKzse3Pk//Rt+W1EqK2fy1lNqupumUXay8cYf5QkY4zrxxi3jTFuT/I/ssV2yd+ZMcbB6c9PJfnjzP7u1x8+dDX9+anFzXBhviPJu8cY1yfbextZZrXtYtv++1JV/ybJdyb5vileMx3eu3G6fWWSv07ytQub5Aa6k5+T7byN7Ezy3UledXhsO2wjQm39XM4qX3qfwMuSfHCM8WvLxpe/n+a7kvzVHZ+7FVXVl08nVaSqvjzJ4zL7u1+eZP+02P4kr13MDBfqiP8D3q7byB2stl1cnuS86ezPh2X2hunrVnqBraSqzknyk0meOMa4Zdn4rulElFTVVyc5O8lHFzPLjXUnPyeXJ3l6Vd29qu6f2Tp550bPb0Eek+RDY4xrDw9sh21kU1xCqpPGl7PaaI9I8v1J3n/4NOkkP5Pke6vqQZkdyvl4kmcuYnILcHqSP571a3Ym+f0xxhuq6l1JXl1Vz0hydWZvgt02pmh9bI7cDn55O20jVfWKJN+e5LSqujbJ85O8KCtvF6/P7IzPpSS3ZHaG7Jayyvp4XpK7J7li+hl6+xjjh5M8MskLquofktye5IfHGGt90/2msco6+faVfk7GGFdV1auTfCCzw8TP2kpnfCYrr48xxsty9Htdk22wjfh4DgCAphz6BABoSqgBADQl1AAAmhJqAABNCTUAgKaEGgBAU0IN2HKq6olV9dPT7SdV1QOWPfaCqnrMcbzm3qpa9cN5q+pBVfWE45sxwMqEGrClVNXOMcblY4wXTUNPSvKlUBtj/NwY401z+NYPyuzDagFOGB94C2w6VXVekh/P7FPb35fktiRfSPLgJP9nGtuX5PeTvC7JzdPX9yT5j0leN8Z4TVV9c5KXJPnyJLcmeXSSU5P83jSWJM8eY7ytqvZOz3vgCvM5ObOrCdwzs+su/pfp+16U5IFJ7pbk58cYr52uafmk6fXPTvIrSU7O7EoftyZ5whjjpqp6a5K/TPIvMrvaxQ+OMbbLpYKAiUtIAZtKVX1Dkv+Q5OFjjE9X1X2T/FpmF6d++BjjtimGMgXW5ZnCbHr+4dc5ObOLOz9tjPGuqrp3ks9ndoH0x44xvlBVZ2d2yZp9dzanMcYXq+rnkuwbYzx7ev3/nOTNY4wfrKpTkryzqg7vyXtgZlF5j8wC76fGGA+uqhcnOS/Jr0/LfdkY40FV9cgkl0zPA7YRoQZsNo9K8gdjjE8nybT3KdPYeq55+HVJrhtjvGt6nc8mX7o+6X+brrN4W5KvPc55Pi7JE6vqx6f790hy5nT7LWOMzyX5XFXdnORPpvH3J/nny17jFdPc/qKq7l1Vp4wx/vY45wNsQkIN2Cr+/gS9znOTXJ/kGzN7H+8XjvN1Ksn3jDE+fMRg1UMzO8R52O3L7t+eI/9dvuN7U7xXBbYZJxMAm82bkzylqk5NkunQ5535XJJ7rTD+4ST3m96nlqq6V1XtTPKVme1puz2z943tWOO87vh93pjkgpp291XVg9f4Oss9bXrutya5eYxx83G8BrCJCTVgUxljXJXkhUn+vKr+MrP3p92ZVyb5iap6T1V9zbLX+WJmIXTR9DpXZHZ48jeS7J/Gvj5r31P3liQPqKr3VtXTkvxCZicRvK+qrprur9cXquo9SX4ryTOO4/nAJuesT4CGprM+f3yMcWDRcwEWxx41AICm7FEDWIeqenySX7rD8MfGGN+1iPkAW5tQAwBoyqFPAICmhBoAQFNCDQCgKaEGANDU/wd6iM0eHEGpKgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, figsize=(10,10))\r\n",
    " \r\n",
    "sns.boxplot(data[\"critical_temp\"], x=\"critical_temp\", ax=ax_box)\r\n",
    "sns.histplot(data, x=\"critical_temp\", ax=ax_hist)\r\n",
    " \r\n",
    "# Remove x axis name for the boxplot\r\n",
    "ax_box.set(xlabel='')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above there seems to be one strange outlier to the right at around 180. Since we are not domain experts here, we are going to keep it in our data set. \r\n",
    "\r\n",
    "We took a look the simple statistics of the continuous variables to see if they make sense and has reasonable data distribution. Outliers could be extreme minimum and maximum that resulted from data entry error or the nature of the nature itself.  \r\n",
    "\r\n",
    "*  The 25% and 75% interquartile range (IQR) was utilized in order to identify and remove the outliers from the dataset.\r\n",
    "*  Outliers that are positioned at above and below the 1.5 times the upper and lower limit of IQR are removed\r\n",
    "*  gmean_atomic_mass is the variable with the most reduction (Down to 17949 from 21263 entries, 15% reduction)\r\n",
    "*  Other variables also have shown significant reduction after outlier removal. Thus outlier removal is not recommended.\r\n",
    "*  Additionally, we do not have domain expertise on this dataset to determine the requirement of what it takes to be outliers. Thus we assume that there is no outliers for this study.\r\n",
    "\r\n",
    "Perhaps this what the scientists want. A room temperature superconductor!\r\n",
    "\r\n",
    "The code below is used to create a pandas profile, which is an easy way to see summary statistics for each feature. The html file will be provided as part of the case study submission. Some findings from that report are:\r\n",
    "\r\n",
    "* Confirms no missing values\r\n",
    "* There are no negative values\r\n",
    "* Varying distrubutions for the features, some bell shaped, others poisson, etc.\r\n",
    "* We have features with skewness and outliers.\r\n",
    "\r\n",
    "To run the code, uncomment the last 3 lines of code below and run the cell. The first two lines indicate what needs to be installed. You will need to use an older version of pandas. After dropping the duplicates from our dataset, we still have 159 columns. For now, we assume the temperature is significant correlated with all of other 158 variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "## install pandas 1.2.4\r\n",
    "## pip install pandas-profiling==2.8.0\r\n",
    "\r\n",
    "# from pandas_profiling import ProfileReport\r\n",
    "# profile = ProfileReport(data, title=\"Pandas Profiling Report\", minimal=True)\r\n",
    "# profile.to_file(output_file=\"PandasProfile.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling Preparations\n",
    "\n",
    "### Methods\n",
    "In a dataset that has a large number of features, there is a strong tendency for a linear regression model to overfit. We will be using three different models: Lasso(L1), Ridge(L2) and Elastic Net to overcome the issue of overfitting and to optimize this linear regression problem. We will also apply GridSearch to tune the hyper-parameters for these models. The followings are brief descriptions of how each algorithm works and how they apply to this problem. \n",
    "\n",
    "__`LASSO(L1)`__\n",
    "* It reduces some coefficients to absolute zero while it keeps some in the model. This property is known as feature selection and is absent in RIDGE. \n",
    "* This algorithm is useful in models with a large number of features like this case study(159 features) \n",
    "* In case of multicollinearity, LASSO will randomly pick one of its correlated variables and set the rest to zero. This can lead to some loss of important information and result in lower accuracy. Thus we will proceed with caution. \n",
    "\n",
    "__`RIDGE(L2)`__\n",
    "* It shrinks the coefficients, therefore it reduces model complexity and multicollinearity. \n",
    "* Even after the magnitude of the coefficients are reduced, it still retains all the features, which may lead to poor performance.  \n",
    "\n",
    "__`Elastic Net`__\n",
    "* Combination of LASSO and RIDGE and uses both L1 and L2 penalty terms. \n",
    "* Generally works well with large dataset. \n",
    "\n",
    "\n",
    "### Evaluation Metrics\n",
    "- For each model we'll use __R²__ as a cross validation metric, which can explain the variance of the dependent variable explained by the independent variables of the model. It measures the strength of the relationship between your model and the dependent variable. However, R² alone cannot be used for comparing the models as the value of R² increases with the increase in the number of predictors. This is important to note since we will have different number of predictors in our models due to LASSO performing feature reduction by shrinking coefficients to absolute zero. \n",
    "- We'll then incorporate __MAE__ (Mean Absolute Error), which is simply the average of the absolute difference between predicted and test values, to compare Lasso, Ridge, or Elastic Net. \n",
    "- In conclusion, we will select the best performing model based on combinations of __largest R² value and smallest MAE__. We will also look into the top coefficients that contribute the most to the chosen model and utilize them to interpret the model. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Building and Evaluation\r\n",
    "\r\n",
    "Below we create three models for Lasso, Ridge and Elastic Net. We'll creaete a pipeline for each model that will scale the data using RobustScaler and then use Grid Search with cross validation. Our data is split in 80% training and 20% test sets. We'll then use the test sets to evaluate each of the models at the end.\r\n",
    "\r\n",
    "Note that we'll be using the Pipeline approach in SKlearn. This allows us to define a sequence of steps to be performed on the data. First, we'll scale the data using RobustScaler. Because of the skewness and outliers of some features, we decided to use the RobustScaler which scales features using statistics that are robust to outliers. Next we'll define the model to be used (Lasso vs Ridge vs ElasticNet). Finally, we'll use Grid Search to find the optimal hyperparameters. By using a Pipeline with GridSearchCV we only scale the data in the training set. The test sets in each fold are then scaled using the same scaler. We are trying to eliminate data leakage. Please see references below for more information.\r\n",
    "\r\n",
    "For Lasso and Ridge, we'll vary the hyperparameter alpha. For Elastic Net, we'll vary the hyperparameters alpha, which is a constant that multiplies the penalty terms. Additionally, we'll vary the hyperparameter l1_ratio which controls the amount of mixing between L1 and L2. If l1_ratio is 1, then we'll only use L1 regularization. If l1_ratio is 0, then we'll only use L2 regularization.\r\n",
    "\r\n",
    "**Hyperparameters Used**\r\n",
    "param_range = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 10, 100, 1000, 10000]\r\n",
    "param_l1_ratio = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\r\n",
    "\r\n",
    "https://towardsdatascience.com/pre-process-data-with-pipeline-to-prevent-data-leakage-during-cross-validation-e3442cca7fdc  \r\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html  \r\n",
    "https://scikit-learn.org/stable/modules/cross_validation.html  \r\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html  \r\n",
    "https://stats.stackexchange.com/questions/445259/combining-pca-feature-scaling-and-cross-validation-without-training-test-data  \r\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html?highlight=elasticnet#sklearn.linear_model.ElasticNet  \r\n",
    "https://machinelearningmastery.com/data-preparation-without-data-leakage/  \r\n",
    "https://stackoverflow.com/questions/35388647/how-to-use-gridsearchcv-output-for-a-scikit-prediction  \r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lasso\n",
      "0.7129520975572087\n",
      "{'lasso__alpha': 0.3}\n",
      "\n",
      "Ridge\n",
      "0.706531275524331\n",
      "{'ridge__alpha': 1000}\n",
      "\n",
      "Elastic\n",
      "0.7083853891920866\n",
      "{'elasticnet__alpha': 0.3, 'elasticnet__l1_ratio': 0.9}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\r\n",
    "X = data.drop(columns=['critical_temp']).copy(deep=True)\r\n",
    "y = data.loc[:,'critical_temp'].copy(deep=True)\r\n",
    "\r\n",
    "X_train, X_test, y_train, y_test =\\\r\n",
    "    train_test_split(X, y,\r\n",
    "    test_size=0.2,\r\n",
    "    random_state=1)\r\n",
    "\r\n",
    "lasso_pipe_svc = make_pipeline(RobustScaler(), Lasso(random_state=1))\r\n",
    "ridge_pipe_svc = make_pipeline(RobustScaler(), Ridge(random_state=1))\r\n",
    "elastic_pipe_svc = make_pipeline(RobustScaler(), ElasticNet(random_state=1))\r\n",
    "\r\n",
    "param_range = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 10, 100, 1000, 10000]\r\n",
    "param_l1_ratio = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\r\n",
    "\r\n",
    "param_grid_lasso = [{'lasso__alpha': param_range}]\r\n",
    "param_grid_ridge = [{'ridge__alpha': param_range}]\r\n",
    "param_grid_elastic = [{'elasticnet__alpha': param_range, 'elasticnet__l1_ratio': param_l1_ratio}]\r\n",
    "\r\n",
    "gs_lasso = GridSearchCV(estimator=lasso_pipe_svc, param_grid=param_grid_lasso, scoring='r2', cv=5, n_jobs=-1)\r\n",
    "gs_lasso.fit(X_train, y_train)\r\n",
    "\r\n",
    "gs_ridge = GridSearchCV(estimator=ridge_pipe_svc, param_grid=param_grid_ridge, scoring='r2', cv=5, n_jobs=-1)\r\n",
    "gs_ridge.fit(X_train, y_train)\r\n",
    "\r\n",
    "gs_elastic = GridSearchCV(estimator=elastic_pipe_svc, param_grid=param_grid_elastic, scoring='r2', cv=5, n_jobs=-1)\r\n",
    "gs_elastic.fit(X_train, y_train)\r\n",
    "\r\n",
    "print(\"Lasso\")\r\n",
    "print(gs_lasso.best_score_)\r\n",
    "print(gs_lasso.best_params_)\r\n",
    "print(\"\")\r\n",
    "\r\n",
    "print(\"Ridge\")\r\n",
    "print(gs_ridge.best_score_)\r\n",
    "print(gs_ridge.best_params_)\r\n",
    "print(\"\")\r\n",
    "\r\n",
    "print(\"Elastic\")\r\n",
    "print(gs_elastic.best_score_)\r\n",
    "print(gs_elastic.best_params_)\r\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we see that each model performs almost identically on the training data. Recall that L1 is used for Lasso, and L2 is used for Ridget. Elastic Net uses both. The optimal l1_ration hyperparameter found for ElasticNet is 0.9. It's moving into the direction of Lasso. If the ratio is 1, it's Lasso, if it's 0, it's Ridge.\r\n",
    "\r\n",
    "Additionally, the optimal alpha found for Ridge is 1000, which is quite high. It's trying to penalize cofficients harder, which means we'll see more coefficients that are close to zero.\r\n",
    "\r\n",
    "Next we'll perform predictions on our test data. Note that because we are using a pipeline, the scaler is only applied to the test data. This ensures that we have no leakage of information from the training data.\r\n",
    "https://stackoverflow.com/questions/35388647/how-to-use-gridsearchcv-output-for-a-scikit-prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lasso\n",
      "R2 -> 0.7197254814065868\n",
      "MAE -> 13.660714491348584\n",
      "Ridge\n",
      "R2 -> 0.7264906957630273\n",
      "MAE -> 13.359929361601823\n",
      "\n",
      "Elastic\n",
      "R2 -> 0.7146988481193569\n",
      "MAE -> 13.811785860925973\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\r\n",
    "\r\n",
    "# Note the X_test gets run through the pipeline above! Very important, it means that the scaler is also run on the test data\r\n",
    "y_lasso_pred = gs_lasso.predict(X_test)\r\n",
    "y_ridge_pred = gs_ridge.predict(X_test)\r\n",
    "y_elastic_pred = gs_elastic.predict(X_test)\r\n",
    "\r\n",
    "print(\"Lasso\")\r\n",
    "print(\"R2 ->\", metrics.r2_score(y_test, y_lasso_pred))\r\n",
    "print(\"MAE ->\", metrics.mean_absolute_error(y_test, y_lasso_pred))\r\n",
    "\r\n",
    "print(\"Ridge\")\r\n",
    "print(\"R2 ->\", metrics.r2_score(y_test, y_ridge_pred))\r\n",
    "print(\"MAE ->\", metrics.mean_absolute_error(y_test, y_ridge_pred))\r\n",
    "print(\"\")\r\n",
    "\r\n",
    "print(\"Elastic\")\r\n",
    "print(\"R2 ->\", metrics.r2_score(y_test, y_elastic_pred))\r\n",
    "print(\"MAE ->\", metrics.mean_absolute_error(y_test, y_elasti\n",
    "c_pred))\r\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we see that the R2 values are consistent with what we found on the training data. However, the Ridge model performans marginally better. In this case I'd be inclined to use the Lasso model because of the reduced number of features. Below we output the ABSOLUTE value of each coefficient, in sorted fashion. This helps to give an indication of which features are important. The sign doesn't matter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Ba': 12.747910288530562,\n",
       " 'wtd_gmean_ThermalConductivity': 12.364521222651708,\n",
       " 'wtd_mean_ThermalConductivity': 11.947603500510482,\n",
       " 'range_atomic_mass': 6.955103730407498,\n",
       " 'wtd_std_Valence': 6.5603440594602995,\n",
       " 'wtd_gmean_ElectronAffinity': 4.328328539587897,\n",
       " 'Bi': 3.97490881456744,\n",
       " 'wtd_entropy_atomic_mass': 3.9508095199980304,\n",
       " 'wtd_entropy_ThermalConductivity': 3.924730652055882,\n",
       " 'Ca': 3.3178161344471087,\n",
       " 'wtd_entropy_FusionHeat': 2.8062532004771885,\n",
       " 'wtd_entropy_ElectronAffinity': 2.5435733987843343,\n",
       " 'mean_Density': 1.7386569847400537,\n",
       " 'Si': 1.6972961033719114,\n",
       " 'mean_fie': 1.5836528543493764,\n",
       " 'Sr': 1.3418330269172292,\n",
       " 'wtd_range_atomic_mass': 1.2028148462719683,\n",
       " 'range_atomic_radius': 1.1880683928528302,\n",
       " 'As': 1.1522532857677543,\n",
       " 'S': 1.0526391079045863,\n",
       " 'wtd_std_FusionHeat': 1.046997414247526,\n",
       " 'mean_ThermalConductivity': 0.9839999275894176,\n",
       " 'wtd_std_atomic_mass': 0.7363630703449164,\n",
       " 'wtd_range_FusionHeat': 0.5820119169251454,\n",
       " 'wtd_range_Density': 0.5578557645139761,\n",
       " 'Ge': 0.5532431718577415,\n",
       " 'std_ElectronAffinity': 0.419474827926435,\n",
       " 'B': 0.3381522940299983,\n",
       " 'Al': 0.32446796220072877,\n",
       " 'wtd_std_Density': 0.2711147428607582,\n",
       " 'Ti': 0.16585668759745884,\n",
       " 'Tl': 0.16003493811249433,\n",
       " 'Nb': 0.1480172271062835,\n",
       " 'C': 0.12906196232215852,\n",
       " 'V': 0.12640116643325944,\n",
       " 'La': 0.12478981428672298,\n",
       " 'Zr': 0.05164513864755047,\n",
       " 'Pd': 0.03141891595180431,\n",
       " 'Mo': 0.021814294515427206,\n",
       " 'number_of_elements': 0.0,\n",
       " 'mean_atomic_mass': 0.0,\n",
       " 'wtd_mean_atomic_mass': 0.0,\n",
       " 'gmean_atomic_mass': 0.0,\n",
       " 'wtd_gmean_atomic_mass': 0.0,\n",
       " 'entropy_atomic_mass': 0.0,\n",
       " 'std_atomic_mass': 0.0,\n",
       " 'wtd_mean_fie': 0.0,\n",
       " 'gmean_fie': 0.0,\n",
       " 'wtd_gmean_fie': 0.0,\n",
       " 'entropy_fie': 0.0,\n",
       " 'wtd_entropy_fie': 0.0,\n",
       " 'range_fie': 0.0,\n",
       " 'wtd_range_fie': 0.0,\n",
       " 'std_fie': 0.0,\n",
       " 'wtd_std_fie': 0.0,\n",
       " 'mean_atomic_radius': 0.0,\n",
       " 'wtd_mean_atomic_radius': 0.0,\n",
       " 'gmean_atomic_radius': 0.0,\n",
       " 'wtd_gmean_atomic_radius': 0.0,\n",
       " 'entropy_atomic_radius': 0.0,\n",
       " 'wtd_entropy_atomic_radius': 0.0,\n",
       " 'wtd_range_atomic_radius': 0.0,\n",
       " 'std_atomic_radius': 0.0,\n",
       " 'wtd_std_atomic_radius': 0.0,\n",
       " 'wtd_mean_Density': 0.0,\n",
       " 'gmean_Density': 0.0,\n",
       " 'wtd_gmean_Density': 0.0,\n",
       " 'entropy_Density': 0.0,\n",
       " 'wtd_entropy_Density': 0.0,\n",
       " 'range_Density': 0.0,\n",
       " 'std_Density': 0.0,\n",
       " 'mean_ElectronAffinity': 0.0,\n",
       " 'wtd_mean_ElectronAffinity': 0.0,\n",
       " 'gmean_ElectronAffinity': 0.0,\n",
       " 'entropy_ElectronAffinity': 0.0,\n",
       " 'range_ElectronAffinity': 0.0,\n",
       " 'wtd_range_ElectronAffinity': 0.0,\n",
       " 'wtd_std_ElectronAffinity': 0.0,\n",
       " 'mean_FusionHeat': 0.0,\n",
       " 'wtd_mean_FusionHeat': 0.0,\n",
       " 'gmean_FusionHeat': 0.0,\n",
       " 'wtd_gmean_FusionHeat': 0.0,\n",
       " 'entropy_FusionHeat': 0.0,\n",
       " 'range_FusionHeat': 0.0,\n",
       " 'std_FusionHeat': 0.0,\n",
       " 'gmean_ThermalConductivity': 0.0,\n",
       " 'entropy_ThermalConductivity': 0.0,\n",
       " 'range_ThermalConductivity': 0.0,\n",
       " 'wtd_range_ThermalConductivity': 0.0,\n",
       " 'std_ThermalConductivity': 0.0,\n",
       " 'wtd_std_ThermalConductivity': 0.0,\n",
       " 'mean_Valence': 0.0,\n",
       " 'wtd_mean_Valence': 0.0,\n",
       " 'gmean_Valence': 0.0,\n",
       " 'wtd_gmean_Valence': 0.0,\n",
       " 'entropy_Valence': 0.0,\n",
       " 'wtd_entropy_Valence': 0.0,\n",
       " 'range_Valence': 0.0,\n",
       " 'wtd_range_Valence': 0.0,\n",
       " 'std_Valence': 0.0,\n",
       " 'H': 0.0,\n",
       " 'Li': 0.0,\n",
       " 'Be': 0.0,\n",
       " 'N': 0.0,\n",
       " 'O': 0.0,\n",
       " 'F': 0.0,\n",
       " 'Na': 0.0,\n",
       " 'Mg': 0.0,\n",
       " 'P': 0.0,\n",
       " 'Cl': 0.0,\n",
       " 'K': 0.0,\n",
       " 'Sc': 0.0,\n",
       " 'Cr': 0.0,\n",
       " 'Mn': 0.0,\n",
       " 'Fe': 0.0,\n",
       " 'Co': 0.0,\n",
       " 'Ni': 0.0,\n",
       " 'Cu': 0.0,\n",
       " 'Zn': 0.0,\n",
       " 'Ga': 0.0,\n",
       " 'Se': 0.0,\n",
       " 'Br': 0.0,\n",
       " 'Rb': 0.0,\n",
       " 'Y': 0.0,\n",
       " 'Tc': 0.0,\n",
       " 'Ru': 0.0,\n",
       " 'Rh': 0.0,\n",
       " 'Ag': 0.0,\n",
       " 'Cd': 0.0,\n",
       " 'In': 0.0,\n",
       " 'Sn': 0.0,\n",
       " 'Sb': 0.0,\n",
       " 'Te': 0.0,\n",
       " 'I': 0.0,\n",
       " 'Cs': 0.0,\n",
       " 'Ce': 0.0,\n",
       " 'Pr': 0.0,\n",
       " 'Nd': 0.0,\n",
       " 'Sm': 0.0,\n",
       " 'Eu': 0.0,\n",
       " 'Gd': 0.0,\n",
       " 'Tb': 0.0,\n",
       " 'Dy': 0.0,\n",
       " 'Ho': 0.0,\n",
       " 'Er': 0.0,\n",
       " 'Tm': 0.0,\n",
       " 'Yb': 0.0,\n",
       " 'Lu': 0.0,\n",
       " 'Hf': 0.0,\n",
       " 'Ta': 0.0,\n",
       " 'W': 0.0,\n",
       " 'Re': 0.0,\n",
       " 'Os': 0.0,\n",
       " 'Ir': 0.0,\n",
       " 'Pt': 0.0,\n",
       " 'Au': 0.0,\n",
       " 'Hg': 0.0,\n",
       " 'Pb': 0.0}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_weights = {data.columns[k]:abs(v) for k, v in enumerate(gs_lasso.best_estimator_['lasso'].coef_)}\r\n",
    "dict(sorted(lasso_weights.items(), key=lambda item: item[1], reverse=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Ba': 10.370112488683707,\n",
       " 'wtd_mean_ThermalConductivity': 6.483779782189918,\n",
       " 'wtd_std_Valence': 6.0513734789108,\n",
       " 'wtd_gmean_ThermalConductivity': 5.803254805998128,\n",
       " 'range_atomic_mass': 4.462011897968567,\n",
       " 'Bi': 4.206544355861212,\n",
       " 'wtd_std_ThermalConductivity': 3.828515214405895,\n",
       " 'wtd_gmean_ElectronAffinity': 3.5891168955214394,\n",
       " 'Ca': 3.410551989821315,\n",
       " 'wtd_entropy_ElectronAffinity': 3.394887646825798,\n",
       " 'wtd_std_atomic_mass': 3.196081179425659,\n",
       " 'wtd_entropy_atomic_mass': 2.8317915478141638,\n",
       " 'mean_ThermalConductivity': 2.7609890505040484,\n",
       " 'wtd_entropy_FusionHeat': 2.5880223369523963,\n",
       " 'gmean_ThermalConductivity': 2.5244734431093048,\n",
       " 'range_atomic_radius': 2.52409341126252,\n",
       " 'Hg': 2.5220927025032776,\n",
       " 'wtd_range_ElectronAffinity': 2.521911168615366,\n",
       " 'std_ElectronAffinity': 2.4642131470561,\n",
       " 'Tl': 2.454638635844284,\n",
       " 'Ag': 2.3027845073780266,\n",
       " 'std_atomic_mass': 2.274435755236311,\n",
       " 'wtd_mean_atomic_radius': 2.2379734200628874,\n",
       " 'wtd_std_atomic_radius': 2.225423834408331,\n",
       " 'number_of_elements': 2.122047583186745,\n",
       " 'wtd_range_atomic_mass': 2.0528071077964136,\n",
       " 'range_ElectronAffinity': 2.0053606678417855,\n",
       " 'wtd_entropy_ThermalConductivity': 2.000837021011655,\n",
       " 'Cl': 1.9297673269011075,\n",
       " 'wtd_entropy_atomic_radius': 1.9121479085863473,\n",
       " 'Ce': 1.8901282417445275,\n",
       " 'wtd_std_Density': 1.8754816171782929,\n",
       " 'entropy_ThermalConductivity': 1.868334488881393,\n",
       " 'As': 1.841577308169385,\n",
       " 'Nd': 1.8397797864841345,\n",
       " 'wtd_range_Density': 1.76051799078602,\n",
       " 'mean_Density': 1.7353459958360702,\n",
       " 'wtd_entropy_fie': 1.7224005390501085,\n",
       " 'wtd_range_Valence': 1.6559624987557906,\n",
       " 'gmean_Density': 1.5920937181678754,\n",
       " 'wtd_std_FusionHeat': 1.5733640928211254,\n",
       " 'Si': 1.5648684463522984,\n",
       " 'wtd_range_atomic_radius': 1.560302361819028,\n",
       " 'range_ThermalConductivity': 1.5006385187436602,\n",
       " 'Eu': 1.4879310575091205,\n",
       " 'entropy_atomic_mass': 1.4004485747790711,\n",
       " 'range_fie': 1.3949199273322916,\n",
       " 'mean_atomic_mass': 1.3902674557297248,\n",
       " 'wtd_mean_atomic_mass': 1.374116100908515,\n",
       " 'std_ThermalConductivity': 1.3479725189697243,\n",
       " 'B': 1.267638502967452,\n",
       " 'Y': 1.2673991509755147,\n",
       " 'mean_FusionHeat': 1.2397777333138973,\n",
       " 'entropy_ElectronAffinity': 1.228590658016926,\n",
       " 'mean_ElectronAffinity': 1.1613942978971574,\n",
       " 'N': 1.1523885336052448,\n",
       " 'gmean_atomic_radius': 1.0789354008968335,\n",
       " 'Lu': 1.0502294812614594,\n",
       " 'S': 1.0185770550728528,\n",
       " 'wtd_entropy_Valence': 0.9689201386204973,\n",
       " 'Mg': 0.9488433421154072,\n",
       " 'wtd_range_FusionHeat': 0.9288042033723658,\n",
       " 'wtd_std_ElectronAffinity': 0.9243360421323548,\n",
       " 'gmean_fie': 0.8945877730999212,\n",
       " 'entropy_Density': 0.8838734917344984,\n",
       " 'Ge': 0.8828450746666208,\n",
       " 'std_fie': 0.8737810087448808,\n",
       " 'std_Valence': 0.8431055330812637,\n",
       " 'Hf': 0.7984913127493142,\n",
       " 'Li': 0.796634524993234,\n",
       " 'O': 0.7911594370653655,\n",
       " 'entropy_fie': 0.7670593506228472,\n",
       " 'Co': 0.7484383138135032,\n",
       " 'Pt': 0.7179778930779598,\n",
       " 'range_FusionHeat': 0.6993648955432503,\n",
       " 'Gd': 0.6783795504250749,\n",
       " 'Zn': 0.6659617240307146,\n",
       " 'std_atomic_radius': 0.6456580872403753,\n",
       " 'wtd_mean_Density': 0.6394065002973661,\n",
       " 'wtd_gmean_atomic_radius': 0.6349529219528366,\n",
       " 'range_Valence': 0.6296974141258311,\n",
       " 'Al': 0.6143174974849577,\n",
       " 'P': 0.6104434335937698,\n",
       " 'Yb': 0.6080235920245325,\n",
       " 'wtd_mean_Valence': 0.593962899908165,\n",
       " 'Be': 0.592345397108514,\n",
       " 'Sr': 0.5875302542168738,\n",
       " 'std_FusionHeat': 0.58598022598591,\n",
       " 'Os': 0.5725433309843191,\n",
       " 'Pb': 0.5626411211616261,\n",
       " 'wtd_entropy_Density': 0.5364630842177518,\n",
       " 'entropy_Valence': 0.5236726357001019,\n",
       " 'Cu': 0.4991546792018265,\n",
       " 'mean_fie': 0.4918038098409988,\n",
       " 'Rb': 0.4773993180614367,\n",
       " 'entropy_atomic_radius': 0.4767926026980457,\n",
       " 'range_Density': 0.46736777500558535,\n",
       " 'K': 0.45837800225386877,\n",
       " 'I': 0.43450304821194285,\n",
       " 'gmean_ElectronAffinity': 0.43211638269764685,\n",
       " 'Er': 0.42785929791818406,\n",
       " 'mean_Valence': 0.4241711056344406,\n",
       " 'Ni': 0.421455224947681,\n",
       " 'wtd_std_fie': 0.4066037516336181,\n",
       " 'std_Density': 0.35452227779509865,\n",
       " 'Se': 0.3414538426893445,\n",
       " 'gmean_Valence': 0.3230833254656482,\n",
       " 'F': 0.3208537195673984,\n",
       " 'H': 0.29538031696653255,\n",
       " 'mean_atomic_radius': 0.2807679308097941,\n",
       " 'Tc': 0.2712961013666125,\n",
       " 'Dy': 0.26367612222767717,\n",
       " 'wtd_gmean_FusionHeat': 0.25240899697080105,\n",
       " 'gmean_atomic_mass': 0.2514192058843516,\n",
       " 'Mn': 0.2502606972919306,\n",
       " 'W': 0.21782865620125683,\n",
       " 'Pd': 0.21323581881772546,\n",
       " 'wtd_gmean_Density': 0.20869211754232894,\n",
       " 'Ho': 0.2043372529766807,\n",
       " 'wtd_mean_fie': 0.19818411312443474,\n",
       " 'Au': 0.19735543241464792,\n",
       " 'Ti': 0.192919562062863,\n",
       " 'wtd_gmean_fie': 0.19234590189022066,\n",
       " 'Pr': 0.1670285897712909,\n",
       " 'wtd_gmean_Valence': 0.16432341792085112,\n",
       " 'Tb': 0.1594766931104036,\n",
       " 'La': 0.15544066245255522,\n",
       " 'Cd': 0.1550548446835377,\n",
       " 'Nb': 0.13505317534318065,\n",
       " 'V': 0.11956580487924762,\n",
       " 'Tm': 0.11811228993226197,\n",
       " 'Rh': 0.10230559755799028,\n",
       " 'Br': 0.09999983242006538,\n",
       " 'wtd_mean_ElectronAffinity': 0.09068023240866165,\n",
       " 'wtd_range_fie': 0.0903801142805733,\n",
       " 'entropy_FusionHeat': 0.08893691361159375,\n",
       " 'Ta': 0.08348592631428255,\n",
       " 'Sm': 0.0829435316009302,\n",
       " 'wtd_mean_FusionHeat': 0.08217011496939779,\n",
       " 'Cs': 0.07812043275290355,\n",
       " 'Fe': 0.06758399352255462,\n",
       " 'Sn': 0.0614433197300692,\n",
       " 'wtd_range_ThermalConductivity': 0.061299030651588204,\n",
       " 'Ru': 0.04617699829343605,\n",
       " 'Ga': 0.039355760232121506,\n",
       " 'Zr': 0.03887823108024104,\n",
       " 'In': 0.03319234984059206,\n",
       " 'Te': 0.03318108200549427,\n",
       " 'Cr': 0.02691993043988254,\n",
       " 'Sb': 0.024082020339040115,\n",
       " 'C': 0.021407473314099756,\n",
       " 'Sc': 0.020594831716504336,\n",
       " 'Na': 0.016942450459675514,\n",
       " 'Ir': 0.01337156203657521,\n",
       " 'wtd_gmean_atomic_mass': 0.013143277576503206,\n",
       " 'Re': 0.011506132232889477,\n",
       " 'Mo': 0.004963541124364872,\n",
       " 'gmean_FusionHeat': 0.002024485315775404}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_weights = {data.columns[k]:abs(v) for k, v in enumerate(gs_ridge.best_estimator_['ridge'].coef_)}\r\n",
    "dict(sorted(ridge_weights.items(), key=lambda item: item[1], reverse=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Interpretability & Explainability\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ba</th>\n",
       "      <th>wtd_gmean_ThermalConductivity</th>\n",
       "      <th>wtd_mean_ThermalConductivity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.20</td>\n",
       "      <td>0.621979</td>\n",
       "      <td>61.015189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.10</td>\n",
       "      <td>0.619735</td>\n",
       "      <td>61.372331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.10</td>\n",
       "      <td>0.619095</td>\n",
       "      <td>60.943760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.15</td>\n",
       "      <td>0.620535</td>\n",
       "      <td>60.979474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.30</td>\n",
       "      <td>0.624878</td>\n",
       "      <td>61.086617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21258</th>\n",
       "      <td>0.00</td>\n",
       "      <td>95.001493</td>\n",
       "      <td>111.537778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21259</th>\n",
       "      <td>2.00</td>\n",
       "      <td>1.577047</td>\n",
       "      <td>108.680590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21260</th>\n",
       "      <td>0.00</td>\n",
       "      <td>57.038314</td>\n",
       "      <td>57.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21261</th>\n",
       "      <td>0.00</td>\n",
       "      <td>58.781651</td>\n",
       "      <td>59.270000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21262</th>\n",
       "      <td>0.00</td>\n",
       "      <td>12.919996</td>\n",
       "      <td>40.752000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21263 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Ba  wtd_gmean_ThermalConductivity  wtd_mean_ThermalConductivity\n",
       "0      0.20                       0.621979                     61.015189\n",
       "1      0.10                       0.619735                     61.372331\n",
       "2      0.10                       0.619095                     60.943760\n",
       "3      0.15                       0.620535                     60.979474\n",
       "4      0.30                       0.624878                     61.086617\n",
       "...     ...                            ...                           ...\n",
       "21258  0.00                      95.001493                    111.537778\n",
       "21259  2.00                       1.577047                    108.680590\n",
       "21260  0.00                      57.038314                     57.400000\n",
       "21261  0.00                      58.781651                     59.270000\n",
       "21262  0.00                      12.919996                     40.752000\n",
       "\n",
       "[21263 rows x 3 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data\r\n",
    "col= data[['Ba','wtd_gmean_ThermalConductivity','wtd_mean_ThermalConductivity' ]]\r\n",
    "col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.35"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#calculate interquartile range for Ba\r\n",
    "q3, q1 = np.percentile(data['Ba'], [75 ,25])\r\n",
    "iqr_BA = q3 - q1\r\n",
    "#display interquartile range \r\n",
    "iqr_BA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46.22075786432602"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#calculate interquartile range for Ba\r\n",
    "q3, q1 = np.percentile(data['wtd_gmean_ThermalConductivity'], [75 ,25])\r\n",
    "iqr_Ther_gmean = q3 - q1\r\n",
    "#display interquartile range \r\n",
    "iqr_Ther_gmean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44.881957804581404"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#calculate interquartile range for Ba\r\n",
    "q3, q1 = np.percentile(data['wtd_mean_ThermalConductivity'], [75 ,25])\r\n",
    "iqr_Ther_mean = q3 - q1\r\n",
    "#display interquartile range \r\n",
    "iqr_Ther_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We decided to use the Lasso model because the performance as explained above, and the reduction of colinearity. The most important variable in the lasso model were the following:\n",
    "\n",
    "'Ba': 12.747910288530564,\n",
    "\n",
    "'wtd_gmean_ThermalConductivity': 12.364521222651712,\n",
    "\n",
    "'wtd_mean_ThermalConductivity': 11.947603500510484\n",
    "\n",
    "As you can see these 3 variables are the only variables among the 159 variables choosen to run the model that have a weight above 10. Therefore, the 3 mentioned variables are the most important variable to predict critical temperature by the lasso model.\n",
    "\n",
    "Because we had the necessity to normalize our data due to different distribution of the values in the variables. In order to provide appropriate variables to compare in the model, we interpret the variables accounting for our normalized data. We used the robust scalar for normalization. This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).\n",
    "Therefore, the interpretation of the most important 3 variables mentioned above are the following:\n",
    "BA= for any increase in 1.35 of the BA variables we will have an increase in the critical temperature of 12.75, when we keep constant the othe variables.\n",
    "Wtd_gmean_ThermalConductivity= for any increase in 46.22 of the wtd_gmean_ThermalConductivity variable we will have an increase in the critical temperature of 12.36, when we keep constant the othe variables.\n",
    "Wtd_mean_ThermalConductivity= for any increase in 44.89 of the wtd_mean_ThermalConductivity variable we will have an increase in the critical temperature of 11.95, when we keep constant the othe variables.\n",
    "\n",
    "The lasso model has MAE= 13.6; the general interpretation of this model for the user is that any prediction with this model will have a mean error of plus/minus 13.6.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will recommend to our audience to keep in mind that the most important variables to relate to critical high temperature are Ba,  wtd_gmean_ThermalConductivity, wtd_mean_ThermalConductivity. Because those variables turned out to have the highest weight in the lasso regularization model. So, moving forward the audience should concentrate in these 3 variables in the control or manipulation of critical high temperature. Furthermore, it is very reassurance that the same variables came out in the other models that we run as the variables with higher weight of importance. For the future, it can be done a prospective study looking the relation with of these 3 variables and critical high temperature. "
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "eaa798b471aa1a0109429a408b0faab53065248ef7f5a5989f90756771672ef6"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}