pa_pynb_sess4_LASSO.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/danielbauer1979/CAS_PredMod/blob/main/pa_pynb_sess4_LASSO.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Regularized Regression\n",
        "\n",
        "In this tutorial, we will discuss regularized regression approaches, particularly least absolute shrinkage and selection operator -- or, in short, the [LASSO](http://statweb.stanford.edu/~tibs/lasso.html/) -- in the context of predicting claim sizes."
      ],
      "metadata": {
        "id": "vUjV7JbXHmjU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's install relevant packages. Again, we're going to rely on the statistical learning toolkit ski-cit learn, which provides LASSO regression but also has many other predictive models. As before, it is less comfortable to use than some of the other packages and, unlike R, does not support formulas. But it is versatile and fast, and therefore one of the most popular predictive modeling toolkits."
      ],
      "metadata": {
        "id": "cKPJ-NiKInO2"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ignevWfjW8Vq"
      },
      "source": [
        "import numpy as np \n",
        "import pandas as pd \n",
        "import matplotlib.pyplot as plt \n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LinearRegression, Lasso, LassoCV\n",
        "from sklearn.metrics import mean_squared_error"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Ridge and LASSO Regression\n",
        "\n",
        "Ridge and LASSO regression are both examples of *regularized* regression approaches.  In what follows, we will first briefly review the corresponding approaches, and particularly highlight how they differ from their unregularized counterparts.   We then will work through a simulated example to become familiar with the impact of the *tuning parameter* on the resulting coefficient estimates.  We will also determine the in- and out-of-sample fit depending on the choice of the tuning parameter, uncovering a familiar relationship.\n",
        "\n",
        "## Review of Concepts and Maths\n",
        "\n",
        "In a conventional (linear) regression problem with independent variables $x_i$ and depedent variables $y_i$, we are determining the best fit in the least-squares sense:\n",
        "$$\n",
        "\\hat{\\beta}^{\\text{OLS}} = \\text{argmin}_{\\beta}\\left\\{\\sum_{i=1}^n \\left(y_i - \\left(\\beta_0 + \\sum_{j=1}^p \\beta_j\\,x_{i,j}\\right)\\right)^2 \\right\\}.\n",
        "$$\n",
        "Within a *regularized* approach, we now include penalties for choosing many or large parameters:\n",
        "$$\n",
        "\\hat{\\beta}^{\\text{REG}}_\\lambda = \\text{argmin}_{\\beta}\\left\\{\\sum_{i=1}^n \\left(y_i - \\left(\\beta_0 + \\sum_{j=1}^p \\beta_j\\,x_{i,j}\\right)\\right)^2 + \\lambda \\times R(\\beta) \\right\\}.\n",
        "$$\n",
        "Here, $R(\\beta)$ is a so-called *regularization* term that imposes a penalty on the complexity of the regression equation.  In particular, within Rigde regression the penalty term is *quadratic*, $R(\\beta) = \\sum_{j=1}^p \\beta_j^2,$ wheras the LASSO uses an L1 penalty, $R(\\beta) = \\sum_{j=1}^p |\\beta_j|.$  \n",
        "\n",
        "We call $\\lambda$ the *tuning parameter*, and it governs how sizable the complexity penalty will be.  In particular, note that for $\\lambda=0$ we are back to the unregularized problem, whereas for large lambda the penalty will be severe -- so this will lead to *shrinkage* of the coefficient estimates.  As $\\lambda$ becomes large and larger, the prediction will more and more closely resemble a *constant* prediction, $\\hat{y}_i = \\beta_0.$  Thus, the choice of the tuning parameter will directly be related to trading off a reduction in variance (due to shrinkage) with an increase in bias (due to the less flexible model fit).  Again, we will explore these aspects in more detail in the context of our example below.\n"
      ],
      "metadata": {
        "id": "JF4aC9I0IAo8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Practical Application: Predicting Claim Sizes\n",
        "\n",
        "We consider a data set of claim sizes (severities) from Allstate, that was used in a [Kaggle competition](https://www.kaggle.com/c/allstate-claims-severity) and is now available from various repositories, e.g. [here](https://github.com/Architectshwet/Allstate-Claims-Severity-Data/blob/master/Datasets).\n",
        "\n",
        "Let's load it, and take a look:"
      ],
      "metadata": {
        "id": "E4SOe62bIt8g"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-CKUbcD-ZKHh"
      },
      "source": [
        "!git clone https://github.com/danielbauer1979/CAS_PredMod.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YNLJ23WBZVhK"
      },
      "source": [
        "dat_1 = pd.read_csv('CAS_PredMod/pa_data_Allstate_train1.csv')\n",
        "dat_2 = pd.read_csv('CAS_PredMod/pa_data_Allstate_train2.csv')\n",
        "dat_3 = pd.read_csv('CAS_PredMod/pa_data_Allstate_train3.csv')\n",
        "df = pd.concat([dat_1,dat_2,dat_3])"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8198XlahW8Vs"
      },
      "source": [
        "df.info()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEOBwID9b_Gg"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJwUTJVIbqdx"
      },
      "source": [
        "df.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "So it is a large data set, and it is particularly large in the $p$ direction -- that is, there are many co-variates. So possibly shrinkage and selection will come in handy here.  One quick comment about the dataset and many Kaggle competitions more generally:  We don't really know what the variables `catx' and 'contx' stand for, so it is difficult to use experience/intuition in building a model -- which is an important aspect in real-world applications.  So performing well in a kaggle competition does not necessarily qualify a data scientist to work in the insurance industry."
      ],
      "metadata": {
        "id": "IXlm-EAtJfYT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Preparing the data\n",
        "\n",
        "There are a few very small losses that are outliers.  We thus disregard losses that are very small and  keep only records with loss greater or equal to $\\$100$, also because we are interested in these in actual settings."
      ],
      "metadata": {
        "id": "evPFU8RwJpLA"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohBtee79W8Vt"
      },
      "source": [
        "df = df[df['loss']>100]"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We convert categoricals into dummies:"
      ],
      "metadata": {
        "id": "qdXL1JySKASA"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2M_UC8hdW8Vu"
      },
      "source": [
        "del df['id']\n",
        "objects = []\n",
        "for c in df.columns:\n",
        "    if str(df[c].dtype) == 'object':\n",
        "        objects.append(c)\n",
        "X_ = df.drop(objects, axis = 1).astype('float64')\n",
        "X_ = X_.drop(['loss'], axis = 1)\n",
        "dummies = pd.get_dummies(df[objects], drop_first=True)\n",
        "X = pd.concat([X_, dummies], axis = 1)\n",
        "y = df.loss"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's look at out features:"
      ],
      "metadata": {
        "id": "gZjiAnGRKMw4"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p-Qt-7U7W8Vv"
      },
      "source": [
        "X"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We split data into training and test sets:"
      ],
      "metadata": {
        "id": "YgiRVq6NKQRV"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4_XIc50uW8Vw"
      },
      "source": [
        "X_train, X_test , y_train, y_test = train_test_split(X, y, test_size=0.8, random_state=1)\n",
        "X_val, X_test, y_val, y_test = train_test_split(X,y,test_size = 0.6, random_state=2)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We go to lods. The log-transformation makes the data much more amenable to regression."
      ],
      "metadata": {
        "id": "_aRwof99KfX4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_train = np.log(y_train)\n",
        "y_val = np.log(y_val)\n",
        "y_test = np.log(y_test)"
      ],
      "metadata": {
        "id": "OTOtlbO2KbC0"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Run Models\n",
        "\n",
        "We start with OLS:"
      ],
      "metadata": {
        "id": "rNRSECSmKnIM"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k5QU1v2GcdLA"
      },
      "source": [
        "model_ols = LinearRegression(fit_intercept=True)\n",
        "model_ols.fit(X_train, y_train)\n",
        "print(model_ols.intercept_)\n",
        "print(model_ols.coef_)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The RMSE is"
      ],
      "metadata": {
        "id": "yZs-E1yjKvdB"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O38_xW8WdKYS"
      },
      "source": [
        "TestRMSE_ols = np.sqrt(mean_squared_error(y_test,model_ols.predict(X_test)))\n",
        "print(TestRMSE_ols)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NHDXR7DyfzMC"
      },
      "source": [
        "tmp = np.abs(model_ols.predict(X_test)-y_test)\n",
        "print(np.median(tmp))\n",
        "print(np.quantile(tmp,0.9))\n",
        "print(np.quantile(tmp,0.99))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And let's run a LASSO regression, with some predefined values of lambda:"
      ],
      "metadata": {
        "id": "Sd7-ZLMmLMVd"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0hm8qn66W8Vy"
      },
      "source": [
        "alphas = np.array([0.000001, 0.000003, 0.000007, 0.00001, 0.0001, 0.001])\n",
        "model_lasso = Lasso(max_iter = 10000, normalize = True)\n",
        "coefs = []\n",
        "MSE = []\n",
        "for a in alphas:\n",
        "    model_lasso.set_params(alpha=a)\n",
        "    model_lasso.fit(X_train, y_train)\n",
        "    coefs.append(model_lasso.coef_)\n",
        "    MSE.append(mean_squared_error(y_val, model_lasso.predict(X_val)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3HmYsAH0W8Vz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "922ce046-c891-487f-bfac-7a5b40722015"
      },
      "source": [
        "RMSE = np.sqrt(MSE)\n",
        "RMSE"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.56483571, 0.56444628, 0.56407866, 0.56411633, 0.58435664,\n",
              "       0.72678256])"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ojM-VhvFW8V0"
      },
      "source": [
        "plt.plot(RMSE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kTxSvTU5Abmz"
      },
      "source": [
        "model_lasso.set_params(alpha=0.000007)\n",
        "model_lasso.fit(X_test, y_test)\n",
        "TestRMSE_lasso = np.sqrt(mean_squared_error(y_test,model_lasso.predict(X_test)))\n",
        "print(TestRMSE_lasso)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9tJbhmu7A63H"
      },
      "source": [
        "tmp = np.abs(model_lasso.predict(X_test)-y_test)\n",
        "print(np.median(tmp))\n",
        "print(np.quantile(tmp,0.9))\n",
        "print(np.quantile(tmp,0.99))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}