examples/00_stacking_concept_pictures_code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacking concept + Pictures + Code\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacking concept\n",
    "***\n",
    "\n",
    "1. We want to predict train set and test set with some 1st level model(s), and then use these predictions as features for 2nd level model(s).  \n",
    "2. Any model can be used as 1st level model or 2nd level model.\n",
    "3. To avoid overfitting (for train set) we use cross-validation technique and in each fold we predict out-of-fold (OOF) part of train set.\n",
    "4. The common practice is to use from 3 to 10 folds.\n",
    "5. Predict test set:\n",
    "   * **Variant A:** In each fold we predict test set, so after completion of all folds we need to find mean (mode) of all temporary test set predictions made in each fold. \n",
    "   * **Variant B:** We do not predict test set during cross-validation cycle. After completion of all folds we perform additional step: fit model on full train set and predict test set once. This approach takes more time because we need to perform one additional fitting.\n",
    "6. As an example we look at stacking implemented with single 1st level model and 3-fold cross-validation.\n",
    "7. Pictures:\n",
    "   * **Variant A:** Three pictures describe three folds of cross-validation. After completion of all three folds we get single train feature and single test feature to use with 2nd level model. \n",
    "   * **Variant B:** First three pictures describe three folds of cross-validation (like in Variant A) to get single train feature and fourth picture describes additional step to get single test feature.\n",
    "8. We can repeat this cycle using other 1st level models to get more features for 2nd level model.\n",
    "9. You can also look at animation of Variant A and Variant B."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Variant A\n",
    "***\n",
    "![Fold 1 of 3](https://github.com/vecxoz/vecstack/raw/master/pic/dia1.png \"Fold 1 of 3\")\n",
    "***\n",
    "![Fold 2 of 3](https://github.com/vecxoz/vecstack/raw/master/pic/dia2.png \"Fold 2 of 3\")\n",
    "***\n",
    "![Fold 3 of 3](https://github.com/vecxoz/vecstack/raw/master/pic/dia3.png \"Fold 3 of 3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Variant A. Animation\n",
    "***\n",
    "\n",
    "![Variant A. Animation](https://github.com/vecxoz/vecstack/raw/master/pic/animation1.gif \"Variant A. Animation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Variant B\n",
    "***\n",
    "![Step 1 of 4](https://github.com/vecxoz/vecstack/raw/master/pic/dia4.png \"Step 1 of 4\")\n",
    "***\n",
    "![Step 2 of 4](https://github.com/vecxoz/vecstack/raw/master/pic/dia5.png \"Step 2 of 4\")\n",
    "***\n",
    "![Step 3 of 4](https://github.com/vecxoz/vecstack/raw/master/pic/dia6.png \"Step 3 of 4\")\n",
    "***\n",
    "![Step 4 of 4](https://github.com/vecxoz/vecstack/raw/master/pic/dia7.png \"Step 4 of 4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Variant B. Animation\n",
    "***\n",
    "\n",
    "![Variant B. Animation](https://github.com/vecxoz/vecstack/raw/master/pic/animation2.gif \"Variant B. Animation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Code\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from vecstack import stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Prepare data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "boston = load_boston()\n",
    "X, y = boston.data, boston.target\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacking. Variant A. Regression task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Implementation A from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold 0: [3.38044832]\n",
      "fold 1: [3.21036931]\n",
      "fold 2: [3.49229353]\n",
      "\n",
      "MEAN:   [3.36103705] + [0.11591064]\n",
      "FULL:   [3.36071216]\n"
     ]
    }
   ],
   "source": [
    "# 1st level model\n",
    "model = LinearRegression()\n",
    "# Number of folds\n",
    "n_folds = 3\n",
    "# Empty array to store out-of-fold predictions (single column)\n",
    "S_train_A_scratch = np.zeros((X_train.shape[0], 1))\n",
    "# Empty array to store temporary test set predictions made in each fold\n",
    "S_test_temp = np.zeros((X_test.shape[0], n_folds))\n",
    "# Empty list to store scores from each fold\n",
    "scores = []\n",
    "# Split initialization\n",
    "kf = KFold(n_splits=n_folds, shuffle=False, random_state=0)\n",
    "\n",
    "# Loop across folds\n",
    "for fold_counter, (tr_index, te_index) in enumerate(kf.split(X_train, y_train)):\n",
    "    \n",
    "    # Split data and target\n",
    "    X_tr = X_train[tr_index]\n",
    "    y_tr = y_train[tr_index]\n",
    "    X_te = X_train[te_index]\n",
    "    y_te = y_train[te_index]\n",
    "    \n",
    "    # Fit\n",
    "    _ = model.fit(X_tr, y_tr)\n",
    "    \n",
    "    # Predict out-of-fold part of train set\n",
    "    S_train_A_scratch[te_index, :] = model.predict(X_te).reshape(-1, 1)\n",
    "    \n",
    "    # Predict test set\n",
    "    S_test_temp[:, fold_counter] = model.predict(X_test)\n",
    "    \n",
    "    # Print score of current fold\n",
    "    score = mean_absolute_error(y_te, S_train_A_scratch[te_index, :])\n",
    "    scores.append(score)\n",
    "    print('fold %d: [%.8f]' % (fold_counter, score))\n",
    "    \n",
    "# Compute mean of temporary test set predictions to get final test set prediction\n",
    "S_test_A_scratch = np.mean(S_test_temp, axis=1).reshape(-1, 1)\n",
    "\n",
    "# Mean OOF score + std\n",
    "print('\\nMEAN:   [%.8f] + [%.8f]' % (np.mean(scores), np.std(scores)))\n",
    "\n",
    "# Full OOF score\n",
    "# !!! FULL score slightly differs from MEAN score because folds contain\n",
    "# different number of examples (404 can't be divided by 3)\n",
    "# If we set n_folds=4 scores will be identical for given metric\n",
    "print('FULL:   [%.8f]' % (mean_absolute_error(y_train, S_train_A_scratch)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2. Implementation A using standard Scikit-learn tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are no suitable standard Scikit-learn tools for Variant A.  \n",
    "Please see corresponding implementation for Variant B below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3. Implementation A using vecstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task:       [regression]\n",
      "metric:     [mean_absolute_error]\n",
      "mode:       [oof_pred_bag]\n",
      "n_models:   [1]\n",
      "\n",
      "model 0:    [LinearRegression]\n",
      "    fold 0: [3.38044832]\n",
      "    fold 1: [3.21036931]\n",
      "    fold 2: [3.49229353]\n",
      "    ----\n",
      "    MEAN:   [3.36103705] + [0.11591064]\n",
      "    FULL:   [3.36071216]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "models = [LinearRegression()]\n",
    "S_train_A_vecstack, S_test_A_vecstack = stacking(models, \n",
    "                                                 X_train, y_train, X_test, \n",
    "                                                 regression=True, \n",
    "                                                 n_folds=n_folds,\n",
    "                                                 mode='oof_pred_bag', \n",
    "                                                 random_state=0, \n",
    "                                                 verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 32.87287178]\n",
      " [ 22.02957522]\n",
      " [ 27.16855956]\n",
      " [ 23.77791521]\n",
      " [  7.70569251]]\n",
      "\n",
      "[[ 32.87287178]\n",
      " [ 22.02957522]\n",
      " [ 27.16855956]\n",
      " [ 23.77791521]\n",
      " [  7.70569251]]\n"
     ]
    }
   ],
   "source": [
    "print('%s\\n\\n%s' % (S_train_A_scratch[:5], S_train_A_vecstack[:5]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 24.95478501]\n",
      " [ 23.63277494]\n",
      " [ 29.34879363]\n",
      " [ 12.0744784 ]\n",
      " [ 21.46079309]]\n",
      "\n",
      "[[ 24.95478501]\n",
      " [ 23.63277494]\n",
      " [ 29.34879363]\n",
      " [ 12.0744784 ]\n",
      " [ 21.46079309]]\n"
     ]
    }
   ],
   "source": [
    "print('%s\\n\\n%s' % (S_test_A_scratch[:5], S_test_A_vecstack[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Results from two implementations above are identical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "\n",
      "True\n",
      "\n",
      "Test:\n",
      "\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print('Train:\\n')\n",
    "print(all(S_train_A_scratch == S_train_A_vecstack))\n",
    "\n",
    "print('\\nTest:\\n')\n",
    "print(all(S_test_A_scratch == S_test_A_vecstack))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacking. Variant B. Regression task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Implementation B from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold 0: [3.38044832]\n",
      "fold 1: [3.21036931]\n",
      "fold 2: [3.49229353]\n",
      "\n",
      "MEAN:   [3.36103705] + [0.11591064]\n",
      "FULL:   [3.36071216]\n"
     ]
    }
   ],
   "source": [
    "# 1st level model\n",
    "model = LinearRegression()\n",
    "# Number of folds\n",
    "n_folds = 3\n",
    "# Empty array to store out-of-fold predictions (single column)\n",
    "S_train_B_scratch = np.zeros((X_train.shape[0], 1))\n",
    "# Empty list to store scores from each fold\n",
    "scores = []\n",
    "# Split initialization\n",
    "kf = KFold(n_splits=n_folds, shuffle=False, random_state=0)\n",
    "\n",
    "# Loop across folds\n",
    "for fold_counter, (tr_index, te_index) in enumerate(kf.split(X_train, y_train)):\n",
    "    \n",
    "    # Split data and target\n",
    "    X_tr = X_train[tr_index]\n",
    "    y_tr = y_train[tr_index]\n",
    "    X_te = X_train[te_index]\n",
    "    y_te = y_train[te_index]\n",
    "    \n",
    "    # Fit\n",
    "    _ = model.fit(X_tr, y_tr)\n",
    "    \n",
    "    # Predict out-of-fold part of train set\n",
    "    S_train_B_scratch[te_index, :] = model.predict(X_te).reshape(-1, 1)\n",
    "    \n",
    "    # Print score of current fold\n",
    "    score = mean_absolute_error(y_te, S_train_B_scratch[te_index, :])\n",
    "    scores.append(score)\n",
    "    print('fold %d: [%.8f]' % (fold_counter, score))\n",
    "    \n",
    "# Fit on full train set and predict test set once\n",
    "_ = model.fit(X_train, y_train)\n",
    "S_test_B_scratch = model.predict(X_test).reshape(-1, 1)\n",
    "\n",
    "# Mean OOF score + std\n",
    "print('\\nMEAN:   [%.8f] + [%.8f]' % (np.mean(scores), np.std(scores)))\n",
    "\n",
    "# Full OOF score\n",
    "# !!! FULL score slightly differs from MEAN score because folds contain\n",
    "# different number of examples (404 can't be divided by 3)\n",
    "# If we set n_folds=4 scores will be identical for given metric\n",
    "print('FULL:   [%.8f]' % (mean_absolute_error(y_train, S_train_B_scratch)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2. Implementation B using standard Scikit-learn tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold 0: [3.38044832]\n",
      "fold 1: [3.21036931]\n",
      "fold 2: [3.49229353]\n",
      "\n",
      "MEAN:   [3.36103705] + [0.11591064]\n",
      "FULL:   [3.36071216]\n"
     ]
    }
   ],
   "source": [
    "# 1st level model\n",
    "model = LinearRegression()\n",
    "\n",
    "# Scorer for cross_val_score\n",
    "scorer = make_scorer(mean_absolute_error)\n",
    "\n",
    "# Fit and predict out-of-fold parts of train set\n",
    "S_train_B_sklearn = cross_val_predict(model, \n",
    "                                      X_train, y=y_train, \n",
    "                                      cv=n_folds, n_jobs=1, \n",
    "                                      verbose=0).reshape(-1, 1)\n",
    "\n",
    "# Fit on full train set and predict test set once\n",
    "_ = model.fit(X_train, y_train)\n",
    "S_test_B_sklearn = model.predict(X_test).reshape(-1, 1)\n",
    "\n",
    "# Compute scores\n",
    "# !!! We need additional run of cross_val_score, because at current point \n",
    "# cross_val_predict hasn't ability to show scores\n",
    "scores = cross_val_score(model, \n",
    "                         X_train, y=y_train,                          \n",
    "                         cv=n_folds, n_jobs=1, \n",
    "                         scoring=scorer, verbose=0)\n",
    "\n",
    "# Print score of each fold\n",
    "for fold_counter, score in enumerate(scores):\n",
    "    print('fold %d: [%.8f]' % (fold_counter, score))\n",
    "    \n",
    "# Mean OOF score + std\n",
    "print('\\nMEAN:   [%.8f] + [%.8f]' % (np.mean(scores), np.std(scores)))\n",
    "\n",
    "# Full OOF score\n",
    "# !!! FULL score slightly differs from MEAN score because folds contain\n",
    "# different number of examples (404 can't be divided by 3)\n",
    "# If we set n_folds=4 scores will be identical for given metric\n",
    "print('FULL:   [%.8f]' % (mean_absolute_error(y_train, S_train_B_sklearn)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3. Implementation B using vecstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task:       [regression]\n",
      "metric:     [mean_absolute_error]\n",
      "mode:       [oof_pred]\n",
      "n_models:   [1]\n",
      "\n",
      "model 0:    [LinearRegression]\n",
      "    fold 0: [3.38044832]\n",
      "    fold 1: [3.21036931]\n",
      "    fold 2: [3.49229353]\n",
      "    ----\n",
      "    MEAN:   [3.36103705] + [0.11591064]\n",
      "    FULL:   [3.36071216]\n",
      "\n",
      "    Fitting on full train set...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "models = [LinearRegression()]\n",
    "S_train_B_vecstack, S_test_B_vecstack = stacking(models, \n",
    "                                                 X_train, y_train, X_test, \n",
    "                                                 regression=True, \n",
    "                                                 n_folds=n_folds,\n",
    "                                                 mode='oof_pred', \n",
    "                                                 random_state=0, \n",
    "                                                 verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 32.87287178]\n",
      " [ 22.02957522]\n",
      " [ 27.16855956]\n",
      " [ 23.77791521]\n",
      " [  7.70569251]]\n",
      "\n",
      "[[ 32.87287178]\n",
      " [ 22.02957522]\n",
      " [ 27.16855956]\n",
      " [ 23.77791521]\n",
      " [  7.70569251]]\n",
      "\n",
      "[[ 32.87287178]\n",
      " [ 22.02957522]\n",
      " [ 27.16855956]\n",
      " [ 23.77791521]\n",
      " [  7.70569251]]\n"
     ]
    }
   ],
   "source": [
    "print('%s\\n\\n%s\\n\\n%s' % (S_train_B_scratch[:5], S_train_B_sklearn[:5], S_train_B_vecstack[:5]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 24.89012999]\n",
      " [ 23.72488246]\n",
      " [ 29.37213304]\n",
      " [ 12.14010251]\n",
      " [ 21.4468654 ]]\n",
      "\n",
      "[[ 24.89012999]\n",
      " [ 23.72488246]\n",
      " [ 29.37213304]\n",
      " [ 12.14010251]\n",
      " [ 21.4468654 ]]\n",
      "\n",
      "[[ 24.89012999]\n",
      " [ 23.72488246]\n",
      " [ 29.37213304]\n",
      " [ 12.14010251]\n",
      " [ 21.4468654 ]]\n"
     ]
    }
   ],
   "source": [
    "print('%s\\n\\n%s\\n\\n%s' % (S_test_B_scratch[:5], S_test_B_sklearn[:5], S_test_B_vecstack[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Results from three implementations above are identical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "\n",
      "True\n",
      "True\n",
      "True\n",
      "\n",
      "Test:\n",
      "\n",
      "True\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print('Train:\\n')\n",
    "print(all(S_train_B_scratch == S_train_B_sklearn))\n",
    "print(all(S_train_B_scratch == S_train_B_vecstack))\n",
    "print(all(S_train_B_sklearn == S_train_B_vecstack))\n",
    "\n",
    "print('\\nTest:\\n')\n",
    "print(all(S_test_B_scratch == S_test_B_sklearn))\n",
    "print(all(S_test_B_scratch == S_test_B_vecstack))\n",
    "print(all(S_test_B_sklearn == S_test_B_vecstack))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}