notebooks/individual-dashboards/erroranalysis-dashboard/erroranalysis-interpretability-dashboard-breast-cancer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Analyze Errors and Explore Interpretability of Models\n",
    "\n",
    "This notebook demonstrates how to use the Responsible AI Widget's Error Analysis dashboard to understand a model trained on the Breast Cancer dataset. The goal of this sample notebook is to classify breast cancer diagnosis with scikit-learn and explore model errors and explanations:\n",
    "\n",
    "1. Train a LightGBM classification model using Scikit-learn\n",
    "2. Run Interpret-Community's 'explain_model' globally and locally to generate model explanations.\n",
    "3. Visualize model errors and global and local explanations with the Error Analysis visualization dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Required Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install --upgrade interpret-community\n",
    "# %pip install --upgrade raiwidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explain\n",
    "\n",
    "### Run model explainer at training time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from lightgbm import LGBMClassifier\n",
    "\n",
    "# Explainers:\n",
    "# SHAP Tabular Explainer\n",
    "from interpret.ext.blackbox import TabularExplainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the breast cancer diagnosis data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "breast_cancer_data = load_breast_cancer()\n",
    "classes = breast_cancer_data.target_names.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data into train and test\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a LightGBM classification model, which you want to explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = LGBMClassifier(n_estimators=1)\n",
    "model = clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load simple ErrorAnalysis view without explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from raiwidgets import ErrorAnalysisDashboard\n",
    "predictions = clf.predict(X_test)\n",
    "features = breast_cancer_data.feature_names\n",
    "ErrorAnalysisDashboard(dataset=X_test, true_y=y_test, features=features, pred_y=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explain predictions on your local machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Using SHAP TabularExplainer\n",
    "explainer = TabularExplainer(model, \n",
    "                             X_train, \n",
    "                             features=breast_cancer_data.feature_names, \n",
    "                             classes=classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate global explanations\n",
    "Explain overall model predictions (global explanation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
    "# X_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
    "global_explanation = explainer.explain_global(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sorted SHAP values\n",
    "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
    "# Corresponding feature names\n",
    "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
    "# Feature ranks (based on original order of features)\n",
    "print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
    "\n",
    "# Note: Do not run this cell if using PFIExplainer, it does not support per class explanations\n",
    "# Per class feature names\n",
    "print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
    "# Per class feature importance values\n",
    "print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out a dictionary that holds the sorted feature importance names and values\n",
    "print('global importance rank: {}'.format(global_explanation.get_feature_importance_dict()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explain overall model predictions as a collection of local (instance-level) explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feature shap values for all features and all data points in the training data\n",
    "print('local importance values: {}'.format(global_explanation.local_importance_values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate local explanations\n",
    "Explain local data points (individual instances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can pass a specific data point or a group of data points to the explain_local function\n",
    "\n",
    "# E.g., Explain the first data point in the test set\n",
    "instance_num = 1\n",
    "local_explanation = explainer.explain_local(X_test[:instance_num])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the prediction for the first member of the test set and explain why model made that prediction\n",
    "prediction_value = clf.predict(X_test)[instance_num]\n",
    "\n",
    "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
    "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
    "\n",
    "print('local importance values: {}'.format(sorted_local_importance_values))\n",
    "print('local importance names: {}'.format(sorted_local_importance_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize\n",
    "### [Optional] Load the interpretability visualization dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from raiwidgets import ExplanationDashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ExplanationDashboard(global_explanation, model, dataset=X_test, true_y=y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze model errors and explanations using Error Analysis dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ErrorAnalysisDashboard(global_explanation, model, dataset=X_test, true_y=y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "mesameki"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}