notebooks/thoth-performance-dataset/PerformanceTensorFlow2.1.0SoftwareStackCombinations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance of sampled the state space of all the possible TensorFlow==2.1.0\n",
    "\n",
    "The purpose of this notebook is to show performance of all possible combinations of the software stack for TensorFlow==2.1.0. These results obtained using Amun service explain some of the knowledge stored into Thoth that is used to provide Advises on AI software stacks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Inputs for Dataset created using Amun\n",
    "\n",
    "All data have been gathered using [Amun Service](https://github.com/thoth-station/amun-api) and [Performance Indicators](https://github.com/thoth-station/performance) evaluated by Thoth Team.\n",
    "\n",
    "\n",
    "## TensorFlow builds\n",
    "\n",
    "Tensorflow builds have been created considering combinations of the following parameters:\n",
    "\n",
    "**Software stacks and native dependencies**\n",
    "\n",
    "All inspections use a combination of all stacks from the dependencies of TensorFlow in version 2.1.0. \n",
    "\n",
    "\n",
    "  * `upstream TensorFlow` - `tensorflow==2.1.0` available on PyPI (inspections prefixed with `tf`)\n",
    "\n",
    "**OS images**\n",
    "\n",
    "  * `rhel-8` \n",
    "\n",
    "**Python Interpreters**\n",
    "\n",
    "  * `3.6` \n",
    "  \n",
    "**Hardware**\n",
    "\n",
    "No node pinning used, any hardware available on OCP is used. No GPU was used. \n",
    "Analysis across inspection run will show which hardware have been identified.\n",
    "\n",
    "`Number of CPUs` used to run is selected a priori as input to Amun: 1\n",
    "\n",
    "## Performance indicators\n",
    "Performance Indicators (PI) used for performance analysis:\n",
    "\n",
    "  * [matrix multiplication](https://github.com/thoth-station/performance/blob/master/tensorflow/matmul.py)\n",
    "\n",
    "Each performance indicator was run `1 times` per inspection run (`batch size == 1`), performance indicators reported median of inspections to be further compared.\n",
    "\n",
    "## Dataset content\n",
    "\n",
    "Inspection specification, build logs, job logs, hardware information of the node where the performance indicator was run and the actual inspection job result are included in the dataset.\n",
    "\n",
    "No buildtime errors spotted with the tested stack.\n",
    "\n",
    "There are some runtime errors spotted with specific stack.\n",
    "\n",
    "\n",
    "## Analysis\n",
    "\n",
    "Results of performance are shown in terms of Elapsed time [ms].\n",
    "\n",
    "The analysis performed in this notebook are defined as:\n",
    "\n",
    "- Performance analysis across different Tf stacks (Python packages) (fixed Hardware, OS image, Python Interpreter, number of CPUs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set environment variables to access the datasets on Ceph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more detail on the Operate First Ceph public bucket used here, visit https://www.operate-first.cloud/apps/content/odh/trino/access_public_bucket.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env THOTH_CEPH_KEY_ID=LLEzCoxu7pvjzO4inoL8\n",
    "%env THOTH_CEPH_SECRET_KEY=1HnDVoIS2jt3h3xEpgeQlCX5+FeOUH0wOrvWVvZP\n",
    "%env THOTH_CEPH_BUCKET_PREFIX=thoth\n",
    "%env THOTH_S3_ENDPOINT_URL=https://s3-openshift-storage.apps.smaug.na.operate-first.cloud\n",
    "%env THOTH_CEPH_BUCKET=opf-datacatalog\n",
    "%env THOTH_DEPLOYMENT_NAME=datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from thoth.report_processing.components.inspection import AmunInspections\n",
    "from thoth.report_processing.components.inspection import AmunInspectionsSummary\n",
    "from thoth.report_processing.components.inspection import AmunInspectionsStatistics\n",
    "from thoth.report_processing.components.inspection import AmunInspectionsFailedSummary\n",
    "\n",
    "inspection = AmunInspections()\n",
    "inspection_runs_summary = AmunInspectionsSummary()\n",
    "inspection_statistics = AmunInspectionsStatistics()\n",
    "inspections_failed_sumary = AmunInspectionsFailedSummary()\n",
    "\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_rows', 500)\n",
    "pd.set_option('display.max_columns', 1000)\n",
    "pd.set_option('display.width', 1500)\n",
    "pd.options.plotting.backend = \"plotly\"  # Convert to matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_identifiers = [\"tf-dm-rw\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspection_runs = inspection.aggregate_thoth_inspections_results(\n",
    "    inspections_identifiers=inspections_identifiers\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "processed_inspection_runs, failed_inspection_runs = inspection.process_inspection_runs(\n",
    "    inspection_runs\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_df = inspection.create_inspections_dataframe(\n",
    "    processed_inspection_runs=processed_inspection_runs,\n",
    "    include_statistics=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_failed_df = inspection.create_inspections_dataframe(\n",
    "    processed_inspection_runs=failed_inspection_runs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Failed Inspections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_failed_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_failed_df['stderr'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create inspection results summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed, _ = inspection_runs_summary.produce_summary_report(inspections_df=inspections_failed_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hardware"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"hardware\"]['platform'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"hardware\"]['processor']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"hardware\"]['flags']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"hardware\"]['ncpus']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"hardware\"]['info']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operating System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"base_image\"]['base_image']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"base_image\"]['number_cpus_run']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Indicators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"pi\"]['pi']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Software Stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results_failed[\"software_stack\"]['requirements_locked'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_packages_dataframe_failed, _ = inspection.create_python_package_df(inspections_df=inspections_failed_df)\n",
    "python_packages_dataframe_failed.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Successfull inspections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspections_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create inspection results summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results, _ = inspection_runs_summary.produce_summary_report(inspections_df=inspections_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hardware"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "report_results[\"hardware\"]['platform'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"hardware\"]['processor']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"hardware\"]['flags']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"hardware\"]['ncpus']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"hardware\"]['info']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operating System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"base_image\"]['base_image']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"base_image\"]['number_cpus_run']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Indicators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"pi\"]['pi']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Software Stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_results[\"software_stack\"]['requirements_locked'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_packages_dataframe, _ = inspection.create_python_package_df(inspections_df=inspections_df)\n",
    "python_packages_dataframe.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_dataframe = inspection.create_final_dataframe(\n",
    "    inspections_df=inspections_df,\n",
    "    include_statistics=True\n",
    ")\n",
    "final_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from typing import List\n",
    "\n",
    "import pandas as pd\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "from typing import Optional, Dict\n",
    "\n",
    "pd.options.plotting.backend = \"plotly\"\n",
    "\n",
    "from kaleido.scopes.plotly import PlotlyScope\n",
    "scope = PlotlyScope()\n",
    "\n",
    "if not os.path.exists(\"images\"):\n",
    "    os.mkdir(\"images\")\n",
    "\n",
    "_PERFORMANCE_QUANTITY = [\"elapsed_time\", \"rate\"]\n",
    "\n",
    "_PERFORMANCE_QUANTITY_MAP = {\"elapsed_time\": \"Elapsed Time [ms]\", \"rate\": \"Rate [GFLOPS]\"}\n",
    "\n",
    "\n",
    "class AmunInspectionsVisualization:\n",
    "    \"\"\"Class of methods used to create statistics from Amun Inspections Runs.\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def create_inspection_3d_plot(\n",
    "        plot_df: pd.DataFrame,\n",
    "        varying_package: str,\n",
    "        title_plot: str,\n",
    "        quantity: Optional[str] = 'elapsed_time',\n",
    "        image_name: Optional[str] = None,\n",
    "    ):\n",
    "        \"\"\"Create inspection performance parameters plot in 3D.\n",
    "\n",
    "        :param final_inspections_df: df for plots provided by `create_final_dataframe` or its subset.\n",
    "        :param \n",
    "        \"\"\"\n",
    "        if quantity not in _PERFORMANCE_QUANTITY:\n",
    "            _LOGGER.info(f\"Only {_PERFORMANCE_QUANTITY} are accepted as quantity\")\n",
    "            return\n",
    "\n",
    "        x_vector = [x[0] for x in plot_df[[\"solver_string\"]].values]\n",
    "\n",
    "        integer_y_encoded = [y[0] for y in plot_df[[\"sws_hash_id_encoded\"]].values]\n",
    "\n",
    "        z_vector = [z[0] for z in plot_df[[quantity]].values]\n",
    "\n",
    "        trace1 = go.Scatter3d(\n",
    "            x=x_vector,\n",
    "            y=integer_y_encoded,\n",
    "            z=z_vector,\n",
    "            mode=\"markers\",\n",
    "            hovertext=[yc[0] for yc in plot_df[[\"sws_string\"]].values],\n",
    "            hoverinfo=\"text\",\n",
    "            marker=dict(\n",
    "                size=8,\n",
    "                color=z_vector,  # set color to an array/list of desired values\n",
    "                colorscale=\"Viridis\",  # choose a colorscale\n",
    "                opacity=0.8,\n",
    "                showscale=True,\n",
    "            ),\n",
    "        )\n",
    "\n",
    "        data = [trace1]\n",
    "\n",
    "        annotations = []\n",
    "        c = 0\n",
    "\n",
    "        for (x, y, z) in zip(x_vector, integer_y_encoded, z_vector):\n",
    "            annotations.append(\n",
    "                dict(\n",
    "                    showarrow=False,\n",
    "                    x=x,\n",
    "                    y=y,\n",
    "                    z=z,\n",
    "                    text=\"\".join(plot_df[varying_package].values[c]),\n",
    "                    xanchor=\"left\",\n",
    "                    xshift=15,\n",
    "                    opacity=0.7,\n",
    "                )\n",
    "            )\n",
    "            c += 1\n",
    "\n",
    "        margin = {\"l\": 200, \"r\": 100, \"b\": 100, \"t\": 100}\n",
    "\n",
    "        layout = go.Layout(\n",
    "            title=title_plot,\n",
    "#             margin=margin,\n",
    "            scene=dict(\n",
    "                xaxis=dict(\n",
    "                    title=\"Runtime Environment\",\n",
    "                    backgroundcolor=\"rgb(200, 200, 230)\",\n",
    "                    gridcolor=\"white\",\n",
    "                    showbackground=True,\n",
    "                    zerolinecolor=\"white\",\n",
    "                    ),\n",
    "                yaxis=dict(title=\"Software Stack ID\",\n",
    "                    backgroundcolor=\"rgb(230, 200,230)\",\n",
    "                    gridcolor=\"white\",\n",
    "                    showbackground=True,\n",
    "                    zerolinecolor=\"white\",\n",
    "                    ),\n",
    "                zaxis=dict(title=_PERFORMANCE_QUANTITY_MAP[quantity],\n",
    "                    backgroundcolor=\"rgb(230, 230,200)\",\n",
    "                    gridcolor=\"white\",\n",
    "                    showbackground=True,\n",
    "                    zerolinecolor=\"white\"),\n",
    "#                 annotations=annotations\n",
    "            ),\n",
    "            showlegend=True,\n",
    "            legend=dict(orientation=\"h\"),\n",
    "        )\n",
    "        fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "        if not image_name:\n",
    "            image_name = title_plot\n",
    "\n",
    "        with open(f\"images/{image_name}.png\", \"wb\") as f:\n",
    "            f.write(scope.transform(fig, format=\"png\"))\n",
    "\n",
    "        return fig\n",
    "    \n",
    "    \n",
    "    @staticmethod\n",
    "    def create_inspection_2d_plot(\n",
    "        plot_df: pd.DataFrame,\n",
    "        varying_package: str,\n",
    "        title_plot: str,\n",
    "        quantity: Optional[str] = 'elapsed_time',\n",
    "        have_annotations: bool = False,\n",
    "        image_name: Optional[str] = None,\n",
    "    ):\n",
    "        \"\"\"Create inspection performance parameters plot in 2D.\n",
    "\n",
    "        :param final_inspections_df: df for plots provided by `create_final_dataframe` or its subset.\n",
    "        \"\"\"\n",
    "        integer_y_encoded = [y[0] for y in plot_df[[\"sws_hash_id_encoded\"]].values]\n",
    "\n",
    "        data = []\n",
    "        annotations = []\n",
    "        colour_counter = 0\n",
    "        distance: float = 0\n",
    "        name_component = varying_package\n",
    "\n",
    "        subset_df = plot_df[plot_df[\"pi_component\"] == varying_package]\n",
    "        z_vector = [z[0] for z in subset_df[[quantity]].values]\n",
    "\n",
    "        trace = go.Scatter(\n",
    "            x=integer_y_encoded,\n",
    "            y=z_vector,\n",
    "            mode=\"markers\",\n",
    "            hovertext=[y[0] for y in subset_df[[\"sws_string\"]].values],\n",
    "            hoverinfo=\"text\",\n",
    "            marker=dict(\n",
    "                size=12,\n",
    "                color=z_vector,  # set color to an array/list of desired values\n",
    "#                 colorscale=color_scales[colour_counter],  # choose a colorscale\n",
    "                opacity=0.8,\n",
    "                showscale=True,\n",
    "                colorbar={\"x\": 1 + distance},\n",
    "            ),\n",
    "            name=f\"solver=={plot_df['solver_string'].unique()[0]}\",\n",
    "        )\n",
    "\n",
    "        data.append(trace)\n",
    "        colour_counter += 1\n",
    "        distance += 0.2\n",
    "\n",
    "        layout = go.Layout(\n",
    "            title=title_plot,\n",
    "            xaxis=dict(title=\"Software Stack ID integer encoded\"),\n",
    "            yaxis=dict(title=_PERFORMANCE_QUANTITY_MAP[quantity]),\n",
    "            showlegend=True,\n",
    "            legend=dict(orientation=\"h\", y=-0.3, yanchor=\"top\"),\n",
    "        )\n",
    "        fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "        if not image_name:\n",
    "            image_name = title_plot\n",
    "\n",
    "        with open(f\"images/{image_name}.png\", \"wb\") as f:\n",
    "            f.write(scope.transform(fig, format=\"png\"))\n",
    "\n",
    "        return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AmunInspectionsVisualization.create_inspection_3d_plot(plot_df=final_dataframe, varying_package=\"tensorflow\", title_plot=\"TF==2.1.0 performance (3D plot)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AmunInspectionsVisualization.create_inspection_2d_plot(plot_df=final_dataframe, varying_package=\"tensorflow\", title_plot=\"TF==2.1.0 performance (2D plot)\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}