python-sdk/tutorials/automl-with-azureml/automl-nlp-multilabel/automl-nlp-text-classification-multilabel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/automl-nlp-multilabel/automl-nlp-text-classification-multilabel.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**Multilabel Text Classification Using AutoML NLP**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Inference](#Inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "This notebook demonstrates multilabel classification with text data using AutoML NLP.\n",
    "\n",
    "AutoML highlights here include using end to end deep learning for NLP tasks like multilabel text classification.\n",
    "\n",
    "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
    "\n",
    "Notebook synopsis:\n",
    "\n",
    "1. Creating an Experiment in an existing Workspace\n",
    "2. Configuration and remote run of AutoML for a multilabel text classification dataset from [Kaggle](www.kaggle.com), [arXiv Paper Abstracts](https://www.kaggle.com/spsayakpaul/arxiv-paper-abstracts). \n",
    "3. Evaluating the trained model on a test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ast\n",
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "from azureml.automl.dnn.nlp.classification.io.read.read_utils import load_model_wrapper\n",
    "from azureml.automl.runtime.shared.score.scoring import score_classification\n",
    "import azureml.core\n",
    "from azureml.core import Dataset\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.dataset import Dataset\n",
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "from azureml.data.datapath import DataPath\n",
    "from azureml.core.run import Run\n",
    "from azureml.core.script_run_config import ScriptRunConfig\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"This notebook was created using version 1.39.0 of the Azure ML SDK\")\n",
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# Choose an experiment name.\n",
    "experiment_name = \"automl-nlp-text-classification-multilabel\"\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace Name\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Experiment Name\"] = experiment.name\n",
    "pd.set_option(\"display.max_colwidth\", -1)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up a compute cluster\n",
    "This section uses a user-provided compute cluster (named \"parallel-2\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_nodes = 2\n",
    "\n",
    "# Choose a name for your cluster.\n",
    "amlcompute_cluster_name = \"parallel-{}\".format(num_nodes)\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_NC6\", max_nodes=num_nodes  # Use GPU Nodes\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "Since the original dataset is very large, we leverage a subsampled dataset to allow for faster training for the purposes of running this example notebook. To run the full dataset (50K+ samples and 1k+ labels) you might need a GPU instance with larger memory and it may take longer to finish training.\n",
    "\n",
    "To run the code below, please first download `arxiv_data.csv` from [this link](https://www.kaggle.com/spsayakpaul/arxiv-paper-abstracts) and save it under the same directory as this notebook, and then run `preprocessing.py` to create a subset of the data for training, evaluation and test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we register train and valid for training purpose. We will register the test part later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload dataset to datastore\n",
    "data_dir = \"data\"  # Local directory to store data\n",
    "blobstore_datadir = data_dir  # Blob store directory to store data in\n",
    "\n",
    "datastore = ws.get_default_datastore()\n",
    "target = DataPath(datastore=datastore, path_on_datastore=blobstore_datadir)\n",
    "Dataset.File.upload_directory(\n",
    "    src_dir=data_dir, target=target, overwrite=True, show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain training data as a Tabular dataset to pass into AutoMLConfig\n",
    "train_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/arxiv_abstract_train.csv\")]\n",
    ")\n",
    "valid_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/arxiv_abstract_valid.csv\")]\n",
    ")\n",
    "test_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=[(datastore, blobstore_datadir + \"/arxiv_abstract_test.csv\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = train_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"arxiv_abstract_train\",\n",
    "    description=\"Multilabel train dataset\",\n",
    "    create_new_version=True,\n",
    ")\n",
    "\n",
    "valid_dataset = valid_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"arxiv_abstract_valid\",\n",
    "    description=\"Multilabel validation dataset\",\n",
    "    create_new_version=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train\n",
    "\n",
    "## Submit AutoML run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the run with the prepared compute resource and datasets. On a `STANDARD_NC6` compute instance with one node, the training would take around 25 minutes, excluding activating nodes in the compute instance. Here, to make training faster, we will use a `STANDARD_NC6` instance with 2 nodes and enable parallel training.\n",
    "\n",
    "To use distributed training, we need to set `enable_distributed_dnn_training = True` and `max_concurrent_iterations` to be the number of nodes available in you compute instance.\n",
    "\n",
    "Here we do not set `primary_metric` parameter as we only train one model and we do not need to rank trained models. The run will use default primary metrics, `accuracy`. But it is only for reporting purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"max_concurrent_iterations\": num_nodes,\n",
    "    \"enable_distributed_dnn_training\": True,\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "target_column_name = \"terms\"\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"text-classification-multilabel\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=train_dataset,\n",
    "    validation_data=valid_dataset,\n",
    "    label_column_name=target_column_name,\n",
    "    **automl_settings,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_run = experiment.submit(\n",
    "    automl_config, show_output=False\n",
    ")  # You might see a warning about \"enable_distributed_dnn_training\". Please simply ignore.\n",
    "_ = automl_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Metrics\n",
    "\n",
    "These metrics logged with the training run are computed with the trained model on validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_metrics = automl_run.get_metrics()\n",
    "pd.DataFrame(\n",
    "    {\"metric_name\": validation_metrics.keys(), \"value\": validation_metrics.values()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get the best run and the best model with `get_output` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    best_run,\n",
    "    best_model,\n",
    ") = (\n",
    "    automl_run.get_output()\n",
    ")  # You might see a warning about \"enable_distributed_dnn_training\". Please simply ignore.\n",
    "best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "Now you can use the trained model to do inference on unseen data. We use a `ScriptRun` to do this, with script that we provide. The following blocks will register the test dataset, download the inference script and trigger the inference run. Our inference run do not directly log the metrics. So we need to download the results and calculate the metrics offline\n",
    "\n",
    "## Submit Inference Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = test_dataset.register(\n",
    "    workspace=ws,\n",
    "    name=\"arxiv_abstract_test\",\n",
    "    description=\"Multilabel text dataset\",\n",
    "    create_new_version=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load training script run corresponding to AutoML run above.\n",
    "training_run_id = best_run.id\n",
    "training_run = Run(experiment, training_run_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference script run arguments\n",
    "arguments = [\n",
    "    \"--run_id\",\n",
    "    training_run_id,\n",
    "    \"--experiment_name\",\n",
    "    experiment.name,\n",
    "    \"--input_dataset_id\",\n",
    "    test_dataset.as_named_input(\"test_data\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_args = arguments\n",
    "with tempfile.TemporaryDirectory() as tmpdir:\n",
    "    # Download required files from training run into temp folder.\n",
    "    entry_script_name = \"score_script.py\"\n",
    "    output_path = os.path.join(tmpdir, entry_script_name)\n",
    "    training_run.download_file(\n",
    "        \"outputs/\" + entry_script_name, os.path.join(tmpdir, entry_script_name)\n",
    "    )\n",
    "\n",
    "    script_run_config = ScriptRunConfig(\n",
    "        source_directory=tmpdir,\n",
    "        script=entry_script_name,\n",
    "        compute_target=compute_target,\n",
    "        environment=training_run.get_environment(),\n",
    "        arguments=scoring_args,\n",
    "    )\n",
    "    scoring_run = experiment.submit(script_run_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = scoring_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_prediction_file = \"./preds_multilabel.csv\"\n",
    "scoring_run.download_file(\n",
    "    \"outputs/predictions.csv\", output_file_path=output_prediction_file\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data_df = test_dataset.to_pandas_dataframe()\n",
    "test_set_predictions_df = pd.read_csv(\"preds_multilabel.csv\")\n",
    "test_set_predictions_df[\"label_confidence\"] = test_set_predictions_df[\n",
    "    \"label_confidence\"\n",
    "].apply(lambda x: [float(num) for num in x.split(\",\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install this package to run the following block\n",
    "# !pip install azureml-automl-dnn-nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_transformer = load_model_wrapper(training_run).y_transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offline Evaluation\n",
    "\n",
    "We will use the evaluation module within AzureML to calculate the metrics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_y = y_transformer.transform(\n",
    "    test_data_df[target_column_name].apply(ast.literal_eval)\n",
    ").toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred_probs = []\n",
    "for i in range(test_set_predictions_df.shape[0]):\n",
    "    test_pred_probs.append(test_set_predictions_df.loc[i, \"label_confidence\"])\n",
    "test_pred_probs = np.array(test_pred_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L = len(y_transformer.classes_)\n",
    "test_metrics = score_classification(\n",
    "    test_y,\n",
    "    test_pred_probs,\n",
    "    list(validation_metrics.keys()),\n",
    "    np.arange(L),\n",
    "    np.arange(L),\n",
    "    y_transformer=y_transformer,\n",
    "    multilabel=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame({\"metric_name\": test_metrics.keys(), \"value\": test_metrics.values()})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Report\n",
    "\n",
    "We also provide the following function, which enables you to evaluate the trained model, for each class and average among classes, with any value of threshold you would like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classification_report_multilabel(\n",
    "    test_df, pred_df, label_col, y_transformer, threshold=0.5\n",
    "):\n",
    "\n",
    "    message = (\n",
    "        \"test_df and pred_df should have the same number of rows, but get {} and {}\"\n",
    "    )\n",
    "    assert test_df.shape[0] == pred_df.shape[0], message.format(\n",
    "        test_df.shape[0], pred_df.shape[0]\n",
    "    )\n",
    "\n",
    "    label_set = y_transformer.classes_\n",
    "    n = len(label_set)\n",
    "\n",
    "    y_true = []\n",
    "    y_pred = []\n",
    "\n",
    "    for row in range(test_df.shape[0]):\n",
    "        true_labels = y_transformer.transform(\n",
    "            [ast.literal_eval(test_df.loc[row, label_col])]\n",
    "        ).toarray()[0]\n",
    "        pred_labels = pred_df.loc[row, \"label_confidence\"]\n",
    "        for ind, (label, prob) in enumerate(zip(true_labels, pred_labels)):\n",
    "            predict_positive = prob >= threshold\n",
    "            if label or predict_positive:\n",
    "                y_true.append(label_set[ind] if label else \"\")\n",
    "                y_pred.append(label_set[ind] if predict_positive else \"\")\n",
    "\n",
    "    print(classification_report(y_true, y_pred, label_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_report_multilabel(\n",
    "    test_data_df,\n",
    "    test_set_predictions_df,\n",
    "    target_column_name,\n",
    "    y_transformer,\n",
    "    threshold=0.1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_report_multilabel(\n",
    "    test_data_df,\n",
    "    test_set_predictions_df,\n",
    "    target_column_name,\n",
    "    y_transformer,\n",
    "    threshold=0.5,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_report_multilabel(\n",
    "    test_data_df,\n",
    "    test_set_predictions_df,\n",
    "    target_column_name,\n",
    "    y_transformer,\n",
    "    threshold=0.9,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "anshirga"
   }
  ],
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "None"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "None"
  ],
  "friendly_name": "DNN Text Featurization",
  "index_order": 2,
  "interpreter": {
   "hash": "cc0892e042a269bcf4aec58f0c86eb5e2be478ff7be4e5f6b2680e2af1718f2e"
  },
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "tags": [
   "None"
  ],
  "task": "Text featurization using DNNs for classification"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}