notebooks/responsibleaidashboard/text/responsibleaidashboard-question-answering-model-debugging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a50648b4",
   "metadata": {},
   "source": [
    "# Assess predictions on Stanford Question Answering Dataset (SQuAD) with a huggingface question answering model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72398086",
   "metadata": {},
   "source": [
    "This notebook demonstrates the use of the `responsibleai` API to assess a huggingface question answering model on the SQuAD dataset (see https://huggingface.co/datasets/squad for more information about the dataset). It walks through the API calls necessary to create a widget with model analysis insights, then guides a visual analysis of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "739385c6",
   "metadata": {},
   "source": [
    "* [Launch Responsible AI Toolbox](#Launch-Responsible-AI-Toolbox)\n",
    "    * [Load Model and Data](#Load-Model-and-Data)\n",
    "    * [Create Model and Data Insights](#Create-Model-and-Data-Insights)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1343e9b0",
   "metadata": {},
   "source": [
    "## Launch Responsible AI Toolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea121102",
   "metadata": {},
   "source": [
    "The following section examines the code necessary to create datasets and a model. It then generates insights using the `responsibleai` API that can be visually analyzed."
   ]
  },
{
   "cell_type": "markdown",
   "id": "6174bcad",
   "metadata": {},
   "source": [
    "### Prepare\n",
    "\n",
    "To run this notebook, we need to install the following packages:\n",
    "\n",
    "```\n",
    "raiutils\n",
    "raiwidgets\n",
    "datasets\n",
    "transformers\n",
    "responsibleai_text\n",
    "torch\n",
    "```\n",
    "\n",
    "Run the following command to load the spacy pipeline:\n",
    "\n",
    "```bash\n",
    "python -m spacy download en_core_web_sm\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40739025",
   "metadata": {},
   "source": [
    "### Load Model and Data\n",
    "*The following section can be skipped. It loads a dataset and trains a model for illustrative purposes.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac9d0df6",
   "metadata": {},
   "source": [
    "First we import all necessary dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75ef9e91",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "import pandas as pd\n",
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2132e2f",
   "metadata": {},
   "source": [
    "Next we load the SQuAD dataset from huggingface datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ae1bf09",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = datasets.load_dataset(\"squad\", split=\"train\")\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42786ee8",
   "metadata": {},
   "source": [
    "Reformat the dataset to be a pandas dataframe with three columns: context, questions and answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88cd5fed",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = []\n",
    "context = []\n",
    "answers = []\n",
    "for row in dataset:\n",
    "    context.append(row['context'])\n",
    "    questions.append(row['question'])\n",
    "    answers.append(row['answers']['text'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "051d4017",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({'context': context, 'questions': questions, 'answers': answers})\n",
    "data = data.sample(frac=1.0, random_state=42).reset_index(drop=True)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e694f20",
   "metadata": {},
   "source": [
    "Fetch a huggingface question answering model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2236e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the question-answering model\n",
    "pipeline_model = pipeline('question-answering')\n",
    "test_size = 5\n",
    "\n",
    "train_data = data\n",
    "test_data = data[:test_size]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cf8327b",
   "metadata": {},
   "source": [
    "See an example of the model's predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce087699",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_answer(dataset, idx):\n",
    "    model_output = pipeline_model(question=dataset['questions'][idx], \n",
    "                                  context=dataset['context'][idx])\n",
    "    pred = model_output['answer']\n",
    "    return pred\n",
    "\n",
    "def check_answer(dataset, idx):\n",
    "    pred = get_answer(dataset, idx)\n",
    "    print('Question  : ', dataset['questions'][idx])\n",
    "    print('Answer    : ', dataset['answers'][idx])\n",
    "    print('Predicted : ', pred)\n",
    "    print('Correct   : ', pred == dataset['answers'][idx])\n",
    "\n",
    "check_answer(test_data, 0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eec0c5c",
   "metadata": {},
   "source": [
    "### Create Model and Data Insights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dd2b169",
   "metadata": {},
   "outputs": [],
   "source": [
    "from responsibleai_text import RAITextInsights, ModelTask\n",
    "from raiwidgets import ResponsibleAIDashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d97b2c",
   "metadata": {},
   "source": [
    "To use Responsible AI Dashboard, initialize a RAITextInsights object upon which different components can be loaded.\n",
    "\n",
    "RAITextInsights accepts the model, the test dataset, the classes and the task type as its arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc8021d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "rai_insights = RAITextInsights(pipeline_model, test_data, \"answers\",\n",
    "                               task_type=ModelTask.QUESTION_ANSWERING)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0331491",
   "metadata": {},
   "source": [
    "Add the components of the toolbox for model assessment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "526aca04",
   "metadata": {},
   "outputs": [],
   "source": [
    "rai_insights.error_analysis.add()\n",
    "rai_insights.explainer.add()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "210aaaf6",
   "metadata": {},
   "source": [
    "Once all the desired components have been loaded, compute insights on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "901b1863",
   "metadata": {},
   "outputs": [],
   "source": [
    "rai_insights.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f844646",
   "metadata": {},
   "source": [
    "Finally, visualize and explore the model insights. Use the resulting widget or follow the link to view this in a new tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b63dbfa2",
   "metadata": {},
   "outputs": [],
   "source": [
    "ResponsibleAIDashboard(rai_insights)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}