GCN_Amazon_Products_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep GCN Product Classification.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1a-MNZ9PESfK",
        "colab_type": "text"
      },
      "source": [
        "# Geometric Deep Learning\n",
        "Graph Node Classification using PyTorch Geometric on the Amazon Computers dataset.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r3PDPbYcEm86",
        "colab_type": "text"
      },
      "source": [
        "## Introduction and requirements"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u5JJM0z_l34m",
        "colab_type": "text"
      },
      "source": [
        "PyTorch Geometric is a geometric deep learning (GDN) extension library for PyTorch. In general GDN is used to generalize deep learning for non-Ecludian data, such as graphs.\n",
        "\n",
        "In this notebook we will demonstrate the use of this framework to classify Amazon products to their categories. More specifically, we will use the dataset presented in \"\" where nodes are products and edges are "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UjoTbUQVnCz8",
        "colab_type": "code",
        "outputId": "c9231065-30fa-4877-a477-812578b9dec5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install --upgrade torch-scatter\n",
        "!pip install --upgrade torch-sparse\n",
        "!pip install --upgrade torch-cluster\n",
        "!pip install --upgrade torch-spline-conv \n",
        "!pip install torch-geometric"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch-scatter\n",
            "  Using cached https://files.pythonhosted.org/packages/98/a9/47cd92673b6ba251240d587815c763baac2099b07bb76fecdb3b7ae5cece/torch_scatter-2.0.4.tar.gz\n",
            "Building wheels for collected packages: torch-scatter\n",
            "  Building wheel for torch-scatter (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-scatter: filename=torch_scatter-2.0.4-cp36-cp36m-linux_x86_64.whl size=12144044 sha256=6e8abef8aad383a32ffe53fde00c8353f3bee0dce2183a0decdcaf6364a9a18f\n",
            "  Stored in directory: /root/.cache/pip/wheels/fb/28/28/458ddcee4849d5f8a14dd1be1e957d2e8b2955e8c96b07a12d\n",
            "Successfully built torch-scatter\n",
            "Installing collected packages: torch-scatter\n",
            "Successfully installed torch-scatter-2.0.4\n",
            "Collecting torch-sparse\n",
            "  Using cached https://files.pythonhosted.org/packages/92/d4/7261b5ee3ff529ac3652fda2ad9757c0fb3cde514014a9c26fe71bad38e9/torch_sparse-0.6.4.tar.gz\n",
            "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from torch-sparse) (1.4.1)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy->torch-sparse) (1.18.4)\n",
            "Building wheels for collected packages: torch-sparse\n",
            "  Building wheel for torch-sparse (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-sparse: filename=torch_sparse-0.6.4-cp36-cp36m-linux_x86_64.whl size=23636504 sha256=250417c7bb8e1bb7597aa7587aebb9ee480e4df12009e495c5234affa3ff13c2\n",
            "  Stored in directory: /root/.cache/pip/wheels/70/d3/b3/374d9a7ebf5b1117d651bb20051e387009baa2e37c9339a525\n",
            "Successfully built torch-sparse\n",
            "Installing collected packages: torch-sparse\n",
            "Successfully installed torch-sparse-0.6.4\n",
            "Collecting torch-cluster\n",
            "  Downloading https://files.pythonhosted.org/packages/08/8b/b483508812fe2242ddf645605ee720822943877b11f820ece1dba5c40fc8/torch_cluster-1.5.4.tar.gz\n",
            "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from torch-cluster) (1.4.1)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy->torch-cluster) (1.18.4)\n",
            "Building wheels for collected packages: torch-cluster\n",
            "  Building wheel for torch-cluster (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-cluster: filename=torch_cluster-1.5.4-cp36-cp36m-linux_x86_64.whl size=18151020 sha256=2e4f18a23dd88a3e314bce6a2a3e17cc3c43d1343a0cd99eb34bee98a19943c5\n",
            "  Stored in directory: /root/.cache/pip/wheels/8f/c0/67/f3702763a38ed07752ebfeca0099966ca9149ab9064d83208f\n",
            "Successfully built torch-cluster\n",
            "Installing collected packages: torch-cluster\n",
            "Successfully installed torch-cluster-1.5.4\n",
            "Collecting torch-spline-conv\n",
            "  Downloading https://files.pythonhosted.org/packages/b9/8c/b888f63e797af27a9dc50580f82e2630f32b151e2f70b88e47841f9a0132/torch_spline_conv-1.2.0.tar.gz\n",
            "Building wheels for collected packages: torch-spline-conv\n",
            "  Building wheel for torch-spline-conv (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-spline-conv: filename=torch_spline_conv-1.2.0-cp36-cp36m-linux_x86_64.whl size=6308110 sha256=11c21795883572cf6462c9be86114526f409757ec1cecbb31ab3a057372cc991\n",
            "  Stored in directory: /root/.cache/pip/wheels/fc/18/bc/5106622bc8d198adc7ee928b087fe842da352ada0a43aa5ea2\n",
            "Successfully built torch-spline-conv\n",
            "Installing collected packages: torch-spline-conv\n",
            "Successfully installed torch-spline-conv-1.2.0\n",
            "Collecting torch-geometric\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f5/f2/26359fb7b50d54924ddd23778d4830b2653df9ffe72f85caad2b829dc778/torch_geometric-1.5.0.tar.gz (153kB)\n",
            "\u001b[K     |████████████████████████████████| 153kB 10.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (1.5.0+cu101)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (1.18.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (4.41.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (1.4.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (2.4)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (0.22.2.post1)\n",
            "Requirement already satisfied: numba in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (0.48.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (2.23.0)\n",
            "Collecting plyfile\n",
            "  Downloading https://files.pythonhosted.org/packages/93/c8/cf47848cd4d661850e4a8e7f0fc4f7298515e06d0da7255ed08e5312d4aa/plyfile-0.7.2-py3-none-any.whl\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (1.0.3)\n",
            "Collecting rdflib\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d0/6b/6454aa1db753c0f8bc265a5bd5c10b5721a4bb24160fb4faf758cf6be8a1/rdflib-5.0.0-py3-none-any.whl (231kB)\n",
            "\u001b[K     |████████████████████████████████| 235kB 21.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (2.10.0)\n",
            "Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.6/dist-packages (from torch-geometric) (0.4)\n",
            "Collecting ase\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d0/70/a8b1a7831193aa228defd805891c534d3e4717c8988147522e673458ddce/ase-3.19.1-py3-none-any.whl (2.1MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1MB 42.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch->torch-geometric) (0.16.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->torch-geometric) (4.4.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->torch-geometric) (0.15.1)\n",
            "Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba->torch-geometric) (0.31.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba->torch-geometric) (46.4.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torch-geometric) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torch-geometric) (2.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torch-geometric) (2020.4.5.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torch-geometric) (1.24.3)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->torch-geometric) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->torch-geometric) (2018.9)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from rdflib->torch-geometric) (1.12.0)\n",
            "Collecting isodate\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 6.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing in /usr/local/lib/python3.6/dist-packages (from rdflib->torch-geometric) (2.4.7)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from ase->torch-geometric) (3.2.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ase->torch-geometric) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ase->torch-geometric) (0.10.0)\n",
            "Building wheels for collected packages: torch-geometric\n",
            "  Building wheel for torch-geometric (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torch-geometric: filename=torch_geometric-1.5.0-cp36-none-any.whl size=267918 sha256=7b0781a16eaab42ca8b5f669e7d1722f459d0d601e8044905a78a12e073a0d15\n",
            "  Stored in directory: /root/.cache/pip/wheels/ec/51/31/5786f2ac419ee312f22d4d2877da05f20e7f2d430e22917daf\n",
            "Successfully built torch-geometric\n",
            "Installing collected packages: plyfile, isodate, rdflib, ase, torch-geometric\n",
            "Successfully installed ase-3.19.1 isodate-0.6.0 plyfile-0.7.2 rdflib-5.0.0 torch-geometric-1.5.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xjY9vtO9MgoL",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://raw.githubusercontent.com/rusty1s/pytorch_geometric/master/docs/source/_static/img/pyg_logo_text.svg?sanitize=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8e8e0m5g1Z1",
        "colab_type": "text"
      },
      "source": [
        "First of all, import the necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ghuZ7ABPcG6S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os.path as osp\n",
        "\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "from torch_geometric.datasets import Amazon\n",
        "import torch_geometric.transforms as T\n",
        "from torch_geometric.nn import SplineConv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RfzOoi7dd_7L",
        "colab_type": "text"
      },
      "source": [
        "## Load the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rGDB6F8ChrbE",
        "colab_type": "text"
      },
      "source": [
        "We load the Amazon Computers dataset which is comprised of networks, where nodes represent goods and edges represent that two goods are frequently bought together. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "47SoQUOGcN5N",
        "colab_type": "code",
        "outputId": "14bf004d-2d0d-4eec-b8f3-1cf617c6e661",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "dataset = 'Computers'\n",
        "root='/tmp/Amazon'\n",
        "dataset = Amazon(root, dataset, transform=T.TargetIndegree())\n",
        "data = dataset[0]"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading https://github.com/shchur/gnn-benchmark/raw/master/data/npz/amazon_electronics_computers.npz\n",
            "Processing...\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fe_IZjTS6bvz",
        "colab_type": "text"
      },
      "source": [
        "## Explore the data\n",
        "\n",
        "A graph is used to model pairwise relations (edges) between objects (nodes). A single graph in PyTorch Geometric is described by an instance of torch_geometric.data.Data, which holds the following attributes:\n",
        "\n",
        "*   **data.x:**  Node feature matrix with shape [num_nodes, num_node_features]\n",
        "\n",
        "\n",
        "*   **data.edge_index:**  Graph connectivity in COO format with shape [2, num_edges] and type torch.long\n",
        "\n",
        "*  **data.edge_attr:**  Edge feature matrix with shape [num_edges, num_edge_features]\n",
        "\n",
        "*   **data.y:** The label of each instance of the dataset.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gVP0z4q4qIaJ",
        "colab_type": "code",
        "outputId": "afe2630c-69df-423e-eba3-6fc288a00e69",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        }
      },
      "source": [
        "print(data,end=\"\\n\\n\")\n",
        "\n",
        "print(\"Dataset classes = \" + str(dataset.num_classes))\n",
        "print(\"Labels in [\" + str(int(min(data.y))) + \", \" + str(int(max(data.y))) + \"]\")\n",
        "print(\"data.y = \",end=\"\")\n",
        "print(data.y,end=\"\\n\\n\")\n",
        "\n",
        "print(\"Number of Nodes = \" + str(data.num_nodes))\n",
        "print(\"Features per Node = \" + str(dataset.num_node_features))\n",
        "print(\"data.x = \", end=\"\")\n",
        "print(data.x,end=\"\\n\\n\")\n",
        "\n",
        "print(\"Number of edges = \" + str(data.num_edges))\n",
        "print(\"Features per edge = \" + str(dataset.num_edge_features))\n",
        "print(\"data.edge_index = \", end=\"\")\n",
        "print(data.edge_index)\n",
        "print(\"data.edge_attr = \",end=\"\")\n",
        "print(data.edge_attr)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data(edge_attr=[491722, 1], edge_index=[2, 491722], x=[13752, 767], y=[13752])\n",
            "\n",
            "Dataset classes = 10\n",
            "Labels in [0, 9]\n",
            "data.y = tensor([4, 4, 8,  ..., 8, 4, 0])\n",
            "\n",
            "Number of Nodes = 13752\n",
            "Features per Node = 767\n",
            "data.x = tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
            "        [1., 1., 0.,  ..., 0., 1., 0.],\n",
            "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
            "        ...,\n",
            "        [1., 0., 0.,  ..., 0., 0., 0.],\n",
            "        [1., 0., 0.,  ..., 0., 0., 0.],\n",
            "        [1., 1., 0.,  ..., 1., 1., 0.]])\n",
            "\n",
            "Number of edges = 491722\n",
            "Features per edge = 1\n",
            "data.edge_index = tensor([[    0,     0,     0,  ..., 13751, 13751, 13751],\n",
            "        [  507,  6551,  8210,  ..., 12751, 13019, 13121]])\n",
            "data.edge_attr = tensor([[0.0471],\n",
            "        [0.1029],\n",
            "        [0.8382],\n",
            "        ...,\n",
            "        [0.0431],\n",
            "        [0.0411],\n",
            "        [0.0087]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35qXClA-b_AC",
        "colab_type": "text"
      },
      "source": [
        "Given this dataset, a question that might come to mind is whether the frequency of two products being bought together is related with their category.\n",
        "In the follow in Histogram we will figure out whether two items that are more frequetly bought together, are also more likely to belong in the same class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_w1FiyvVFtq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "import numpy as np\n",
        "\n",
        "productsCount = np.zeros(10)\n",
        "sameClassCount = np.zeros(10)\n",
        "# Iterate through the edges\n",
        "for edge_index in range(data.num_edges):\n",
        "  freqIndex = math.floor(data.edge_attr[edge_index]*10)\n",
        "  if (freqIndex == 10):\n",
        "    freqIndex = 9\n",
        "\n",
        "  productsCount[freqIndex] +=1\n",
        "  # If they belong in the same class \n",
        "  if (data.y[data.edge_index[0][edge_index]] == data.y[data.edge_index[1][edge_index]]):\n",
        "    sameClassCount[freqIndex] +=1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lteoVkiQdDtl",
        "colab_type": "code",
        "outputId": "b53d5a55-01d7-46ef-e3bf-92072bed1581",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Divide same class products with products count of each frequency index\n",
        "np.seterr(divide='ignore', invalid='ignore')\n",
        "y = np.nan_to_num(sameClassCount/productsCount)\n",
        "print(y)\n",
        "print()\n",
        "print(productsCount)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0.79089507 0.7520564  0.73329105 0.70463629 0.85275591 0.82747604\n",
            " 0.13135135 0.         0.52368579 0.11463904]\n",
            "\n",
            "[432491.  28934.  12613.   5004.   1270.   1565.   1850.      0.   5003.\n",
            "   2992.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "55_XYKNCjyQ9",
        "colab_type": "text"
      },
      "source": [
        "We end up with the following graph, where the x axis is the frequency of two products being bought together (divided into 10 bars) and the y axis is the percentage of items that belong in the same category.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YIojlrx7hHUy",
        "colab_type": "code",
        "outputId": "51a608c3-f3d6-4484-9d33-3c964aed3adc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "fig = plt.figure()\n",
        "ax = fig.add_axes([0,0,1,1])\n",
        "ax.set_xlabel('Frequency of two products being bought together')\n",
        "ax.set_ylabel('Percentage of same category products')\n",
        "langs = ['<0.1', '<0.2', '<0.3', '<0.4', '<0.5', '<0.6', '<0.7', '<0.8', '<0.9', '<=1']\n",
        "ax.bar(langs,y)\n",
        "plt.show()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAFNCAYAAAAzYQemAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5glVX3v//eHUS4qqMh4A4YZCaiYKOIIQYyPl+hBPQJRFFBUjJEYBW/RBKNBxBNj1OgvJqDiDeNREFQ8I47BGyp4nQGGy0DQESEMEkEE5KLA4Pf3R1WHTds9XTPT1XvP7PfrefbTVatWVX139eXbq2rttVJVSJKk4dls2AFIkjTuTMaSJA2ZyViSpCEzGUuSNGQmY0mShsxkLEnSkN1j2AGsq+22264WLlw47DAkSVon55xzzi+rav5U2za6ZLxw4UKWL18+7DAkSVonSa6Ybpu3qSVJGjKTsSRJQ2YyliRpyEzGkiQNmclYkqQhMxlLkjRkJmNJkobMZCxJ0pCZjCVJGjKTsSRJQ2YyliRpyEzGkiQN2UY3UYS0KVl41JeHev7L3/XsoZ5fUsOWsSRJQ2YyliRpyEzGkiQNmclYkqQhMxlLkjRkJmNJkobMZCxJ0pD5OWNJ0/Jz0NLcsGUsSdKQmYwlSRoyk7EkSUNmMpYkach6TcZJ9k1yaZJVSY6aYvuCJGcmOS/JBUme1Wc8kiSNot6ScZJ5wHHAM4HdgEOS7Dap2luBU6rqscDBwPF9xSNJ0qjqs2W8J7Cqqi6rqtuBk4H9J9UpYJt2+b7Az3uMR5KkkdRnMt4euHJgfXVbNugY4NAkq4GlwJFTHSjJ4UmWJ1l+7bXX9hGrJElDM+wOXIcAJ1bVDsCzgE8l+b2YquqEqlpcVYvnz58/50FKktSnPpPxVcCOA+s7tGWDXg6cAlBV3we2BLbrMSZJkkZOn8NhLgN2SbKIJgkfDLxwUp3/Ap4GnJjkkTTJeE7vQzvcnyRp2HprGVfVGuAI4AzgEppe0yuTHJtkv7baXwOvSHI+cBJwWFVVXzFJkjSKep0ooqqW0nTMGiw7emD5YmCfPmOQJGnUDbsDlyRJY89kLEnSkJmMJUkaMpOxJElDZjKWJGnITMaSJA2ZyViSpCEzGUuSNGQmY0mShsxkLEnSkPU6HKY2nBNZSNKmz5axJElDZjKWJGnIZkzGSe6dZLN2edck+yW5Z/+hSZI0Hrq0jL8DbJlke+CrwIuBE/sMSpKkcdIlGaeqbgWeCxxfVc8HHtVvWJIkjY8uvamTZG/gRcDL27J5/YWkjYm9vSVpw3VpGb8WeDNwWlWtTPIw4Mx+w5IkaXx0aRk/qKr2m1ipqsuSnNVjTJIkjZUuLeM3dyyTJEnrYdqWcZJnAs8Ctk/ygYFN2wBr+g5MkqRxsbbb1D8HlgP7AecMlN8EvL7PoCRJGifTJuOqOh84P8lpwC1VdSdAknnAFnMUnyRJm7wuz4y/Cmw1sL4V8PUuB0+yb5JLk6xKctQU29+fZEX7+nGSG7qFLUnSpqNLb+otq+rmiZWqujnJvWbaqW1BHwc8HVgNLEuypKouHjjW6wfqHwk8dl2ClyRpU9AlGd+SZI+qOhcgyeOA33TYb09gVVVd1u53MrA/cPE09Q8B3tbhuFJnDkoiaWPQJRm/Djg1yc+BAA8GDuqw3/bAlQPrq4G9pqqYZCdgEfDNDseVJGmTMmMyrqplSR4BPLwturSq7pjlOA4GPjfRSWyyJIcDhwMsWLBglk8tSdJwzZiMk7xkUtEeSaiqf59h16uAHQfWd2jLpnIw8OrpDlRVJwAnACxevLhmOK8kSRuVLrepHz+wvCXwNOBcYKZkvAzYJckimiR8MPDCyZXaVvf9ge93CViSpE1Nl9vURw6uJ7kfcHKH/dYkOQI4g2aWp4+3E00cCyyvqiVt1YOBk6vKFq8kaSx1aRlPdgtNZ6sZVdVSYOmksqMnrR+zHjFIkrTJ6PLM+EvARKt1M2A34JQ+g5IkaZx0aRm/d2B5DXBFVa3uKR5JksZOl2fG356LQCRJGldrm0LxJu66Pf17qmqbXiKSJGnMrG3Wpq0BkrwDuBr4FM0IXC8CHjIn0UmSNAa6zNq0X1UdX1U3VdWvq+qDNGNMS5KkWdAlGd+S5EVJ5iXZLMmLaD7eJEmSZkGXZPxC4AXAL4BrgOczxUhakiRp/XTpTX053paWJKk3M7aMk+yQ5LQk17SvzyfZYS6CkyRpHHS5Tf0JYAnw0Pb1pbZMkiTNgi7JeH5VfaKq1rSvE4H5PcclSdLY6JKMr0tyaNubel6SQ4Hr+g5MkqRx0SUZ/zlNb+r/phn840DgZX0GJUnSOFlrb+ok84B3VtV+cxSPJEljZ60t46q6E9gpyeZzFI8kSWOnyxSKlwHfTbKEgZG3qup9vUUlSdIY6ZKMf9q+NgO27jccSZLGT5cRuN4OkGSbZrVu6j0qSZLGSJcRuBYnuRC4ALgwyflJHtd/aJIkjYcut6k/Dryqqs4CSPJEmhG4Ht1nYJIkjYsunzO+cyIRA1TV2cCa/kKSJGm8dGkZfzvJh4GTgAIOAr6VZA+Aqjq3x/gkSdrkdUnGj2m/vm1S+WNpkvNTp9sxyb7AvwDzgI9W1bumqPMC4Jj2WOdXlXMlS5LGSpfe1E9ZnwO3o3cdBzwdWA0sS7Kkqi4eqLML8GZgn6q6PskD1+dckiRtzLo8M15fewKrquqyqrodOBnYf1KdVwDHVdX1AFV1TY/xSJI0kvpMxtsDVw6sr27LBu0K7Jrku0l+0N7WliRprHR5Ztz3+XcBngzsAHwnyR9V1Q2DlZIcDhwOsGDBgrmOUZKkXnUZ9OOcJK9Ocv91PPZVwI4D6zu0ZYNWA0uq6o6q+hnwY5rkfDdVdUJVLa6qxfPnz1/HMCRJGm1dblMfBDyUpgPWyUn+V5J02G8ZsEuSRe2sTwcDSybV+SJNq5gk29Hctr6sa/CSJG0KZkzGVbWqqt5Ckyg/QzMi1xVJ3p5k27XstwY4AjgDuAQ4papWJjk2ycT8yGcA1yW5GDgTeFNVXbdhb0mSpI1Lp2fGSR4NvAx4FvB54NPAE4FvArtPt19VLQWWTio7emC5gDe0L0mSxtKMyTjJOcANwMeAo6rqtnbTD5Ps02dwkiSNg7Um4ySbAZ+vqndOtb2qnttLVJIkjZG1PjOuqt8BJlxJknrUpTf115O8McmOSbadePUemSRJY6JLB66D2q+vHigr4GGzH44kSeOny0QRi+YiEEmSxlWX3tT3BP4KeFJb9C3gw1V1R49xSZI0Nrrcpv4gcE/g+Hb9xW3ZX/QVlCRJ46RLMn58VT1mYP2bSc7vKyBJksZNl97UdybZeWIlycOAO/sLSZKk8dKlZfwm4MwklwEBdqIZGlOSJM2CLr2pv5FkF+DhbdGlA0NiSpKkDdSlN/XkEbj+IMmNwIVVdU0/YUmSND663KZ+ObA3zRSH0Mw/fA6wKMmxVfWpnmKTJGksdEnG9wAeWVW/AEjyIODfgb2A7wAmY0mSNkCX3tQ7TiTi1jVt2a8AB/6QJGkDdWkZfyvJ6cCp7fqBbdm9aeY5liRJG6BLMn41zTSKT2zXP0kzx3EBT+krMEmSxkWXjzZVkuXAjVX19ST3Au4D3NR7dJIkjYEuH216BXA4sC2wM7A98CHgaf2GJknq08KjvjzU81/+rmcP9fyjpEsHrlcD+wC/BqiqnwAP7DMoSZLGSZdkfFtV3T6xkuQeQPUXkiRJ46VLMv52kr8DtkrydJpe1V/qNyxJksZHl2R8FHAtcCHwl8DSqnpLl4Mn2TfJpUlWJTlqiu2HJbk2yYr25RzJkqSx0+WjTUdW1b8AH5koSPLatmxaSeYBxwFPB1YDy5IsqaqLJ1X9bFUdsY5xS5K0yejSMn7pFGWHddhvT2BVVV3WPnM+Gdh/HWKTJGksTNsyTnII8EKaCSGWDGzaGvhVh2NvD1w5sL6aZjzryZ6X5EnAj4HXV9WVU9SRJGmTtbbb1N8Drga2A/55oPwm4IJZOv+XgJOq6rYkf0kzutdTJ1dKcjjNZ51ZsGDBLJ1akqTRMG0yrqorgCtopk9cH1cBOw6s79CWDZ7juoHVjwLvniaWE4ATABYvXuzHqiRJm5QZnxkn+eMky5LcnOT2JHcm+XWHYy8DdkmyKMnmwMHA4O1ukjxkYHU/4JJ1CV6SpE1Bl97U/0aTSE8FFgMvAXadaaeqWpPkCOAMYB7w8apameRYYHlVLQFek2Q/YA3Nc+jD1utdSJK0EeuSjKmqVUnmVdWdwCeSnAe8ucN+S4Glk8qOHlh+c5fjSJK0KeuSjG9tbzOvSPJumk5dXT4SJUmSOuiSVF/c1jsCuIWmU9bz+gxKkqRx0qVl/Evg9qr6LfD2dmStLfoNS5Kk8dGlZfwN4F4D61sBX+8nHEmSxk+XZLxlVd08sdIu32st9SVJ0jrokoxvSbLHxEqSxwG/6S8kSZLGS5dnxq8DTk3ycyDAg4GDeo1KkqQxMmMyrqplSR4BPLwturSq7ug3LEmSxkfXQT/uAC7qORZJksaSg3dIkjRkJmNJkoasy6xNSXJokqPb9QVJ9uw/NEmSxkOXlvHxNHMaH9Ku3wQc11tEkiSNmS4duPaqqj3amZqoquvbiSMkSdIs6NIyvqMdj7oAkswHftdrVJIkjZEuyfgDwGnAA5P8A3A28M5eo5IkaYx0GfTj00nOAZ5GMwLXAVV1Se+RSZI0JjoN+gH8Ajirrb9Vkj2q6tz+wpIkaXzMmIyTvAM4DPgp7XPj9utT+wtLkqTx0aVl/AJg56q6ve9gJEkaR106cF0E3K/vQCRJGlddWsb/CJyX5CLgtonCqtqvt6gkSRojXZLxJ4F/Ai7EzxdLkjTruiTjW6vqA71HIknSmOryzPisJP+YZO8ke0y8uhw8yb5JLk2yKslRa6n3vCSVZHHnyCVJ2kR0aRk/tv36xwNlM360qR1C8zjg6cBqYFmSJVV18aR6WwOvBX7YNWhJkjYlXUbgesp6HntPYFVVXQaQ5GRgf+DiSfXeQfNM+k3reR5JkjZqnUbgSvJs4FHAlhNlVXXsDLttD1w5sL4a2GvScfcAdqyqLyeZNhknORw4HGDBggVdQpYkaaMx4zPjJB8CDgKOpBmb+vnATht64iSbAe8D/nqmulV1QlUtrqrF8+fP39BTS5I0Urp04HpCVb0EuL6q3g7sDezaYb+rgB0H1ndoyyZsDfwh8K0kl9M8k15iJy5J0rjpkox/0369NclDgTuAh3TYbxmwS5JFSTYHDgaWTGysqhuraruqWlhVC4EfAPtV1fJ1egeSJG3kuiTj05PcD3gPcC5wOXDSTDtV1RrgCOAM4BLglKpameTYJI7eJUlSq0tv6ne0i59PcjqwZVXd2OXgVbUUWDqp7Ohp6j65yzElSdrUdOnA9fz2s8DQfPzoE0keu7Z9JElSd11uU/99Vd2U5InAnwIfAz7Ub1iSJI2PLsn4zvbrs4ETqurLwOb9hSRJ0njpkoyvSvJhms8aL02yRcf9JElSB12S6gtoekT/r6q6AdgWh66UJGnWdOlNfSvwhYH1q4Gr+wxKkqRx4u1mSZKGbNpk3D4bliRJPVtby/j7AEk+NUexSJI0ltb2zHjzJC8EnpDkuZM3VtUXpthHkiSto7Ul41cCLwLuBzxn0rZioFOXJElaf9Mm46o6Gzg7yfKq+tgcxiRJ0liZ8aNNwKeSvAZ4Urv+beBDVXVHf2FJkjQ+uiTj44F7tl8BXgx8EPiLvoKSJGmcdEnGj6+qxwysfzPJ+X0FJEnSuOk0UUSSnSdWkjyMuyaPkCRJG6hLy/hNwJlJLgMC7AS8rNeoJEkaI13Gpv5Gkl2Ah7dFl1bVbf2GJUnS+OjSMqZNvhf0HIskSWPJiSIkSRoyk7EkSUM2YzJO49AkR7frC5Ls2X9okiSNhy4t4+OBvYFD2vWbgON6i0iSpDHTJRnvVVWvBn4LUFXXA5t3OXiSfZNcmmRVkqOm2P7KJBcmWZHk7CS7rVP0kiRtArok4zuSzKOZqYkk84HfzbRTu89xwDOB3YBDpki2n6mqP6qq3YF3A+9bl+AlSdoUdEnGHwBOAx6Y5B+As4F3dthvT2BVVV1WVbcDJwP7D1aoql8PrN6bNuFLkjROugz68ekk5wBPoxmB64CquqTDsbcHrhxYXw3sNblSklcDb6C59f3UqQ6U5HDgcIAFCxZ0OLUkSRuPLr2ptwWuAU4CPgP8Isk9ZyuAqjquqnYG/hZ46zR1TqiqxVW1eP78+bN1akmSRkKX29TnAtcCPwZ+0i5fnuTcJI9by35XATsOrO/Qlk3nZOCADvFIkrRJ6ZKMvwY8q6q2q6oH0HTIOh14FXfNcTyVZcAuSRYl2Rw4GFgyWKEd83rCs2mSvSRJY6VLMv7jqjpjYqWqvgrsXVU/ALaYbqeqWgMcAZwBXAKcUlUrkxybZL+22hFJViZZQfPc+KXr+0YkSdpYdZko4uokf0tzGxngIJrnxvOY4SNOVbUUWDqp7OiB5deuW7iSJG16urSMX0jzvPeL7WtBWzYPeEF/oUmSNB66fLTpl8CR02xeNbvhSJI0fmZMxu2IW38DPArYcqK8qqb8TLAkSVo3XW5Tfxr4T2AR8Hbgcpqe0pIkaRZ0ScYPqKqPAXdU1ber6s+ZZqQsSZK07rr0pr6j/Xp1kmcDPwe27S8kSZLGS5dk/H+S3Bf4a+BfgW2A1/UalSRJY6RLMr6+qm4EbgSeApBkn16jkiRpjHR5ZvyvHcskSdJ6mLZlnGRv4AnA/CRvGNi0Dc2AH5IkaRas7Tb15sB92jpbD5T/Gjiwz6AkSRon0ybjqvo28O0kJ1bVFXMYkyRJY6VLB64tkpwALBys7whckiTNji7J+FTgQ8BHgTv7DUeSpPHTJRmvqaoP9h6JJEljqstHm76U5FVJHpJk24lX75FJkjQmurSMX9p+fdNAWQEPm/1wJEkaP13mM140F4FIkjSuZrxNneReSd7a9qgmyS5J/nf/oUmSNB66PDP+BHA7zWhcAFcB/6e3iCRJGjNdkvHOVfVu2qkUq+pWIL1GJUnSGOmSjG9PshVNpy2S7Azc1mtUkiSNkS69qd8G/AewY5JPA/sAh/UZlCRJ42TGlnFVfQ14Lk0CPglYXFXf6nLwJPsmuTTJqiRHTbH9DUkuTnJBkm8k2WndwpckaePXpTf1n9GMwvXlqjodWJPkgA77zQOOA54J7AYckmS3SdXOo0nujwY+B7x7Xd+AJEkbuy7PjN9WVTdOrFTVDTS3rmeyJ7Cqqi6rqtuBk4H9BytU1ZlthzCAHwA7dAtbkqRNR5dkPFWdLs+atweuHFhf3ZZN5+XAV6bakOTwJMuTLL/22ms7nFqSpI1Hl2S8PMn7kuzcvt4HnDObQSQ5FFgMvGeq7VV1QlUtrqrF8+fPn81TS5I0dF2S8ZE0g358luZW82+BV3fY7ypgx4H1Hdqyu0nyp8BbgP2qyo9MSZLGzlpvN7edsE6vqqesx7GXAbskWUSThA8GXjjp+I8FPgzsW1XXrMc5JEna6K21ZVxVdwK/S3LfdT1wVa0BjgDOAC4BTqmqlUmOTbJfW+09wH2AU5OsSLJkXc8jSdLGrktHrJuBC5N8DbhlorCqXjPTjlW1FFg6qezogeU/7R6qJEmbpi7J+AvtS5Ik9aDLfMafbMemXlBVl85BTJIkjZUuI3A9B1hBMz41SXb32a4kSbOny0ebjqEZTesGgKpaATysx5gkSRorXZLxHYPDYbZ+10cwkiSNoy4duFYmeSEwL8kuwGuA7/UbliRJ46PrCFyPAm4DPgPcCLyuz6AkSRon07aMk2wJvBL4A+BCYO92IA9JkjSL1tYy/iTN5A0X0sxJ/N45iUiSpDGztmfGu1XVHwEk+Rjwo7kJSZKk8bK2lvEdEwvenpYkqT9raxk/Jsmv2+UAW7XrAaqqtuk9OkmSxsC0ybiq5s1lIJIkjasuH22SJEk9MhlLkjRkJmNJkobMZCxJ0pB1GZtakkbSwqO+PNTzX/6uZw/1/Np02DKWJGnITMaSJA2ZyViSpCEzGUuSNGR24JIkjaRx6qDXa8s4yb5JLk2yKslRU2x/UpJzk6xJcmCfsUiSNKp6S8ZJ5gHH0cyFvBtwSJLdJlX7L+Aw4DN9xSFJ0qjr8zb1nsCqqroMIMnJwP7AxRMVqurydtvveoxDkqSR1udt6u2BKwfWV7dlkiRpwEbRmzrJ4UmWJ1l+7bXXDjscSZJmVZ/J+Cpgx4H1HdqydVZVJ1TV4qpaPH/+/FkJTpKkUdFnMl4G7JJkUZLNgYOBJT2eT5KkjVJvybiq1gBHAGcAlwCnVNXKJMcm2Q8gyeOTrAaeD3w4ycq+4pEkaVT1OuhHVS0Flk4qO3pgeRnN7WtJksbWRtGBS5KkTZnJWJKkITMZS5I0ZCZjSZKGzGQsSdKQmYwlSRoyk7EkSUNmMpYkachMxpIkDZnJWJKkITMZS5I0ZCZjSZKGzGQsSdKQmYwlSRoyk7EkSUNmMpYkachMxpIkDZnJWJKkITMZS5I0ZCZjSZKGzGQsSdKQmYwlSRoyk7EkSUPWazJOsm+SS5OsSnLUFNu3SPLZdvsPkyzsMx5JkkZRb8k4yTzgOOCZwG7AIUl2m1Tt5cD1VfUHwPuBf+orHkmSRlWfLeM9gVVVdVlV3Q6cDOw/qc7+wCfb5c8BT0uSHmOSJGnk9JmMtweuHFhf3ZZNWaeq1gA3Ag/oMSZJkkZOqqqfAycHAvtW1V+06y8G9qqqIwbqXNTWWd2u/7St88tJxzocOLxdfThwaS9Br5/tgF/OWGt4jG/DGN+GMb4NY3wbZtTi26mq5k+14R49nvQqYMeB9R3asqnqrE5yD+C+wHWTD1RVJwAn9BTnBkmyvKoWDzuO6RjfhjG+DWN8G8b4Nsyoxzeoz9vUy4BdkixKsjlwMLBkUp0lwEvb5QOBb1ZfTXVJkkZUby3jqlqT5AjgDGAe8PGqWpnkWGB5VS0BPgZ8Kskq4Fc0CVuSpLHS521qqmopsHRS2dEDy78Fnt9nDHNgJG+fDzC+DWN8G8b4NozxbZhRj+9/9NaBS5IkdeNwmJIkDZnJeD20ndJ+2A7j+dm2g9rkOg9IcmaSm5P82wjG9/Qk5yS5sP361BGLb88kK9rX+Un+bJTiG6i7oP0ev3GU4kuyMMlvBq7hh0Ypvrbeo5N8P8nK9udwy1GJL8mLBq7diiS/S7L7CMV3zySfbK/bJUnePBexrUN8myf5RBvf+UmePFfxdZHkEe3P3m1z+bu7Nibjjtofrnu3q/8EvL8dxvN6mmE9J/st8PfAnHyj1yO+XwLPqao/ounR/qkRi+8iYHFV7Q7sC3y4/fjbqMQ34X3AV/qKawPj+2lV7d6+XjlK8bXfy/8LvLKqHgU8GbhjVOKrqk9PXDvgxcDPqmrFqMRH09dmi/b393HAX6bHsf3XI75XALTxPR345yR9z4UwGONMfgW8BnhvjyGtE5PxDJI8Msk/0ww0smuSAE+lGb4TmuE8D5i8X1XdUlVn0yTlUYzvvKr6ebu6EtgqyRYjFN+t7ahsAFsCvXRuWN/42n0PAH5Gc/16sSHxzYUNiO8ZwAVVdT5AVV1XVXeOUHyDDqEZznfWbUB8Bdy7/admK+B24NcjFN9uwDcBquoa4Aagl8/7To6xyz5VdU1VLaPHfwDXlcl4CknuneRlSc4GPgJcDDy6qs6jGa7zhoFEMdUwnxtbfM8Dzq2q20YpviR7JVkJXEjTglozVb1hxJfkPsDfAm+fjZhmO77WoiTnJfl2kj8Zsfh2BSrJGUnOTfI3IxbfoIOAk0Ysvs8BtwBXA/8FvLeqfjVC8Z0P7JfkHkkW0bTed5yiXh8xkuT9uftjhonX780eOCp6/WjTRuxq4ALgL6rqP4cdzBRmLb4kj6K57fSM2QisNSvxVdUPgUcleSTwySRfaT8ONwrxHUNzq+7mzP7cJrMR39XAgqq6LsnjgC8meVRVzUbraTbiuwfwRODxwK3AN5KcU1XfGJH4gOYfQuDWqrpoFuKaMBvx7QncCTwUuD9wVpKvV9VlIxLfx4FHAsuBK4DvtfHOlrXGWFWvn8VzzQlbxlM7kGaozi8kOTrJTgPbrgPul7ueX041zOdGEV+SHYDTgJdU1U9HLb4JVXUJcDPwhyMU317Au5NcDrwO+Ls0g9yMRHxVdVtVXdcunwP8lI638OYiPpoW1Xeq6pdVdSvNeAR7jFB8Ew5mFlvFsxjfC4H/qKo72tvA32X2bgPPxs/fmqp6ffvcfX/gfsCPZym+mWLcKFvGVJWvaV40t2ReC6wAvg4sbMtPBQ5ulz8EvGotxzgM+LdRi4/ml+N84LmjeP2ARcA92uWdgJ8D241KfJOOcwzwxhG7fvOBee3yw2j+cG07QvHdHzgXuBdNK/nrwLNHJb5222btdXvYbH9vZ+H6/S3wiXb53tx1m3ZU4rsXcO92+ek0/3jN2TVch/17+d1dr/cy7AA2lhfNbaEd2+WHAT8CVrU/mFu05fsBxw7sczlNr72baVoCu41KfMBbaZ45rRh4PXCE4nsxTceoFe0f7QNG7fs7sG/vv9Drcf2eN+n6PWeU4mvXD21jvAh49wjG92TgB33GtQHf3/u021bSJOI3jVh8C2k6VF1CkyR3mstr2KHug2n+Jv+apnPZamCbufheT/dyBC5JkobMZ8aSJA2ZyViSpCEzGUuSNGQmY0mShsxkLEnSkJmMNaMkd0764PzCYcfUtyQnJbkgyesnlR+QZLdhxdVFkm8lWecBIJLcL8mr1nGfy5Nstw7195utgRfW933OxjmS7J7kWdPsM+22WYjn7waWFyaZzZHBNEQmY3Xxm7pr9p/dq+ryiQ1pbFI/R0keDDy+qh5dVe+ftPkAmkHw5zqmuRi69n7AOiXjdVVVS6rqXX2eY47sDkyXcNe2bUP93cxVupmjnyl1tEn9EdXcaP8jvzTJv9MM2LBjkjclWda2Jt8+UPctSX6c5Oy2tfnGtvx/WhxJtmuHlSTJvCTvGTjWX7blT273+VyS/0zy6bSDQid5fJLvpZk39UdJtk7ynQzMPxoxTVcAAAfYSURBVNue/zGT3seWuWvO1fOSPKXd9FVg+/YuwJ8M1H8CzcAG72m37ZXknHbbY5JUkgXt+k+T3Ku9Vt9s38s3JrZPiuOYJJ9KM7/qT5K8YuA9n5VkCXDxdPEm2SrJyWnmtT2NZhafiWPfPLB8YJIT2+UHJTmtvWbnt+/tXcDO7Xt7T5KHtNdxRZKLMv1kE3/TxvSjJH/QHn9+ks+338dlSfZpyw9LO793khOTfKD93l2W5MC2fLMkx7ff568lWTqxbQovHohvz3b/bZN8sb3mP0jy6IHr/D9Tmrb7LGyX/779mb7bz2nr+e17+3GSP0kzf++xwEHtuQ8aOObvbVtLPPPb97cyyUeTXJH2LkOSQ9tzrkjy4TS/F++imV1tRZJPt6ecl+Qj7TG+mmSrdv+dk/xHmrnKz0ryiIFr/qEkPwTePc011TAMc8QRXxvHi2aA94lRuk6jGV3nd8Aft9ufAZwAhOYfvNOBJ9HM1HIhzdB429CM2PPGdp9v0cxXDLAdcHm7fDjw1nZ5C5qB5hfRjIZ0I81YuJsB36eZaGBz4DKaliztee5BM0fz/9eW7Qosn+J9/TXw8Xb5ETSz32zZvr+LprkWJwIHDqyvbM95BLAMeBHN8J3fb7d/CXhpu/znwBenOOYxNEOTbtVeiytpJgB4Ms0oaYtmiPcNA+WPBtYMXNubB85zIHBiu/xZ4HXt8jzgvpPfd3u+twzU2XqK2C8fqPMS4PR2+TPAE9vlBcAl7fJhtMPDttfy1Pb7uRuwaiDOpW35g2nmzD1winN/C/hIu/ykidiBfwXe1i4/FVgxcJ3fOLD/Re17fjzNz/aWwNbAT7j7z+k/t8vPAr4++X1MEdfdtq0lnn8D3twu70szLeJ2NBMsfAm4Z7vteJrx4yd/Pxe23+vd2/VTgEPb5W8Au7TLewHfHLjmp9MOleprdF7eplAXv6lmknWgaRkDV1TVD9qiZ7Sv89r1+wC70PxhO62aiQBoW3gzeQbw6IGW0H3bY90O/KiqVrfHWkHzx+hG4Opq5ial2lmJkpwK/H2SN9EkwROnONcTaf5QUlX/meQKmsS9LjMbfQ/YhyYZvJPmj2qAs9rtewPPbZc/xfStkf9XVb8BfpPkTJqh/W5o3/PPZoj3ScAH2vILklzQIe6n0iRPqplH+MYk959UZxnw8ST3pPknYsU0xzpp4OvEbf0/BXbLXTNabZNm2snJvlhVv6Np+T9o4H2e2pb/d3s9pnNS+x6+k2SbJPdr939eW/7NJA9Iss1ajrEPzfX/LfDbJF+atP0L7ddzaH7m1tV08TwR+LO2/D+SXN/WfxrNP7LL2uu3FXDNNMf+2cD35RxgYXudnwCcOnD9B+cqP7V6mDtaG8ZkrPV1y8BygH+sqg8PVkjyurXsv4a7HpNsOelYR1bVGZOO9WRgcL7lO1nLz29V3Zrka8D+wAto/rj14TvAn9C0hv8fzQD+BXx5HY8zeVzaifVbJlfcgONuOW2tqXZsEtyTgGcDJyZ5X1X9+wznmFjejObOyd2mvMzvTzc5+D1dn7kop7tuUxn8mYPu12MixrX+zM2iAJ+sqjd3qDv5d2Irmvd4w+A/0JNs6M+UeuAzY82GM4A/n2j5JNk+yQNpEtUBaZ5pbg08Z2Cfy7krQR446Vh/1bbGSLJrknuv5dyXAg9J8vi2/ta5q2PKR2lajMuq6vop9j2L5rYySXaluZ166Qzv9SaaFv/gMQ4FftK25H5Fczvz7Hb792im4aM911lMbf80z4QfQHN7etk6xPsdmin1SPKHNLeqJ/wiySPTdLL7s4HybwB/1e4zL8l9J7+3NNPS/aKqPkJzLaeb4vCgga/fb5e/Chw5cKzpEsNUvgs8r312/CCa6zGdg9rjPxG4sapu5O7X6cnAL9s7JpdPvIcke9A8/pg433Pa638f4H93iHHyz8Hatk0Xz3dp/lEkyTNoZrKC5ntzYPs7NPEMfKd22x0TvxvTaY/9syTPb/dPJvWX0OgxGWuDVdVXaZ4Rfj/JhcDnaJ4vnkvzbPJ84CvcPcG8lybpnkfznGzCR2lmoTk3zcc2PszaW8C30/xB/tck5wNfo23xVDOP76+BT0yz+/HAZm3MnwUOq6rbpqk74WTgTWk6UO1cTc/y0CREaJLwDQPJ/0jgZe2t4xfTTPc2lQuAM4EfAO+oqp+vQ7wfBO6T5BKazkPnDOxzFM0zwu/RTMg+4bXAU9pjnUMzo9h1wHfTdGx6D00SPL/9Hh0E/Ms0sd+/fX+vBSY+CvYaYHHbaeli4JXT7DuVz9PMonMx8H9pZp26cZq6v23j+xDw8rbsGOBxbUzvouk/MHHcbZOspHnG/2OA9hHHEprvwVdo+jlMd74JZ9Lchr9bB65ptk0Xz9uBZ7Q/588H/hu4qaoupplV7avtPl8DHtLucwJwwUAHrum8CHh5+zuxkuYOkUaYszZpziQ5hqYDynvn6HwPpemA84i21TqS5vq6bAyS3Keqbm7vFPwI2Keq/nsOzncvmn+sDm//mexNki2AO6tqTZK9gQ+u5dayNnE+M9YmKclLgH8A3jDKiVjTOr3tjLU5zZ2C3hJx64Q0g7lsSfO8ttdE3FoAnNI+QrgdeMUcnFMjypaxJElD5jNjSZKGzGQsSdKQmYwlSRoyk7EkSUNmMpYkachMxpIkDdn/D1iljYnA9wJ3AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OugtzxBbkcdc",
        "colab_type": "text"
      },
      "source": [
        "As we see, the items that are frequently bought together, are less likely to belong in the same category. Therefore the edges attribute is not sufficient in order to classify the products, and as a result we certainly need to take advantage of the nodes features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AL0Bn1Oog9UB",
        "colab_type": "text"
      },
      "source": [
        "## Split the training, test and validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5diSPx-zcScF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data.train_mask = torch.zeros(data.num_nodes, dtype=torch.bool)\n",
        "data.train_mask[:data.num_nodes - 1000] = 1\n",
        "\n",
        "data.test_mask = torch.zeros(data.num_nodes, dtype=torch.bool)\n",
        "data.test_mask[data.num_nodes - 500:] = 1\n",
        "\n",
        "data.val_mask = torch.zeros(data.num_nodes, dtype=torch.bool)\n",
        "data.val_mask[data.num_nodes - 1000:data.num_nodes - 500] = 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oh0byX0dhHeZ",
        "colab_type": "text"
      },
      "source": [
        "## Building the Neural Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8u2jdVIkD05",
        "colab_type": "text"
      },
      "source": [
        "The network will consist of two SplineConv layers for convolution. \n",
        "\n",
        "Additionally, we use the exponential ReLU activation function and dropout for regularization."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qXleC9fcko6Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Net(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = SplineConv(dataset.num_features, 16, dim=1, kernel_size=2)\n",
        "        self.conv2 = SplineConv(16, dataset.num_classes, dim=1, kernel_size=2)\n",
        "\n",
        "    def forward(self):\n",
        "        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr\n",
        "        x = F.elu(self.conv1(x, edge_index, edge_attr))\n",
        "        x = self.conv2(x, edge_index, edge_attr)\n",
        "        return F.log_softmax(x, dim=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "resjD8sgkqul",
        "colab_type": "text"
      },
      "source": [
        "Create an instance of our network and use adam optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uND7fsuCd0UL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "model, data = Net().to(device), data.to(device)\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-3)\n",
        "\n",
        "def train():\n",
        "    model.train()\n",
        "    optimizer.zero_grad()\n",
        "    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()\n",
        "    optimizer.step()\n",
        "\n",
        "def test():\n",
        "    model.eval()\n",
        "    logits, accs = model(), []\n",
        "    for _, mask in data('train_mask', 'test_mask'):\n",
        "        pred = logits[mask].max(1)[1]\n",
        "        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()\n",
        "        accs.append(acc)\n",
        "    return accs\n",
        "  \n",
        "def validate():\n",
        "    model.eval()\n",
        "    logits, accs = model(), []\n",
        "    for _, mask in data('val_mask'):\n",
        "        pred = logits[mask].max(1)[1]\n",
        "        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()\n",
        "        accs.append(acc)\n",
        "    return accs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Si5JK3VFhZeT",
        "colab_type": "text"
      },
      "source": [
        "## Training and evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ccb_iVRRR0J",
        "colab_type": "code",
        "outputId": "2fe8c402-58e8-4db4-b47c-65677a5c75b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "for epoch in range(1, 301):\n",
        "    train()\n",
        "    log = 'Epoch: {:03d}, Train: {:.4f}, Test: {:.4f}'\n",
        "    print(log.format(epoch, *test()))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 001, Train: 0.3763, Test: 0.3500\n",
            "Epoch: 002, Train: 0.3903, Test: 0.3520\n",
            "Epoch: 003, Train: 0.2602, Test: 0.2600\n",
            "Epoch: 004, Train: 0.4184, Test: 0.4040\n",
            "Epoch: 005, Train: 0.4330, Test: 0.3980\n",
            "Epoch: 006, Train: 0.3790, Test: 0.3540\n",
            "Epoch: 007, Train: 0.3685, Test: 0.3440\n",
            "Epoch: 008, Train: 0.3749, Test: 0.3420\n",
            "Epoch: 009, Train: 0.4079, Test: 0.3860\n",
            "Epoch: 010, Train: 0.5103, Test: 0.4720\n",
            "Epoch: 011, Train: 0.5710, Test: 0.5240\n",
            "Epoch: 012, Train: 0.6008, Test: 0.5660\n",
            "Epoch: 013, Train: 0.6527, Test: 0.6180\n",
            "Epoch: 014, Train: 0.6444, Test: 0.6400\n",
            "Epoch: 015, Train: 0.6115, Test: 0.5840\n",
            "Epoch: 016, Train: 0.5838, Test: 0.5540\n",
            "Epoch: 017, Train: 0.5804, Test: 0.5500\n",
            "Epoch: 018, Train: 0.5947, Test: 0.5540\n",
            "Epoch: 019, Train: 0.6304, Test: 0.6200\n",
            "Epoch: 020, Train: 0.6830, Test: 0.6660\n",
            "Epoch: 021, Train: 0.7003, Test: 0.6800\n",
            "Epoch: 022, Train: 0.7092, Test: 0.6800\n",
            "Epoch: 023, Train: 0.7123, Test: 0.6940\n",
            "Epoch: 024, Train: 0.7021, Test: 0.6920\n",
            "Epoch: 025, Train: 0.6903, Test: 0.6700\n",
            "Epoch: 026, Train: 0.6900, Test: 0.6820\n",
            "Epoch: 027, Train: 0.7039, Test: 0.6820\n",
            "Epoch: 028, Train: 0.7169, Test: 0.7060\n",
            "Epoch: 029, Train: 0.7287, Test: 0.7140\n",
            "Epoch: 030, Train: 0.7381, Test: 0.7040\n",
            "Epoch: 031, Train: 0.7404, Test: 0.7240\n",
            "Epoch: 032, Train: 0.7397, Test: 0.7200\n",
            "Epoch: 033, Train: 0.7405, Test: 0.7240\n",
            "Epoch: 034, Train: 0.7555, Test: 0.7300\n",
            "Epoch: 035, Train: 0.7726, Test: 0.7460\n",
            "Epoch: 036, Train: 0.7803, Test: 0.7500\n",
            "Epoch: 037, Train: 0.7843, Test: 0.7540\n",
            "Epoch: 038, Train: 0.7860, Test: 0.7600\n",
            "Epoch: 039, Train: 0.7851, Test: 0.7560\n",
            "Epoch: 040, Train: 0.7858, Test: 0.7620\n",
            "Epoch: 041, Train: 0.7903, Test: 0.7720\n",
            "Epoch: 042, Train: 0.7947, Test: 0.7680\n",
            "Epoch: 043, Train: 0.7994, Test: 0.7680\n",
            "Epoch: 044, Train: 0.8038, Test: 0.7800\n",
            "Epoch: 045, Train: 0.8059, Test: 0.7840\n",
            "Epoch: 046, Train: 0.8088, Test: 0.7940\n",
            "Epoch: 047, Train: 0.8127, Test: 0.7900\n",
            "Epoch: 048, Train: 0.8160, Test: 0.7940\n",
            "Epoch: 049, Train: 0.8185, Test: 0.8020\n",
            "Epoch: 050, Train: 0.8214, Test: 0.8060\n",
            "Epoch: 051, Train: 0.8237, Test: 0.8080\n",
            "Epoch: 052, Train: 0.8250, Test: 0.8020\n",
            "Epoch: 053, Train: 0.8269, Test: 0.8080\n",
            "Epoch: 054, Train: 0.8276, Test: 0.8120\n",
            "Epoch: 055, Train: 0.8312, Test: 0.8120\n",
            "Epoch: 056, Train: 0.8343, Test: 0.8160\n",
            "Epoch: 057, Train: 0.8408, Test: 0.8240\n",
            "Epoch: 058, Train: 0.8464, Test: 0.8240\n",
            "Epoch: 059, Train: 0.8500, Test: 0.8280\n",
            "Epoch: 060, Train: 0.8479, Test: 0.8400\n",
            "Epoch: 061, Train: 0.8525, Test: 0.8380\n",
            "Epoch: 062, Train: 0.8460, Test: 0.8200\n",
            "Epoch: 063, Train: 0.8534, Test: 0.8400\n",
            "Epoch: 064, Train: 0.8549, Test: 0.8460\n",
            "Epoch: 065, Train: 0.8605, Test: 0.8460\n",
            "Epoch: 066, Train: 0.8595, Test: 0.8420\n",
            "Epoch: 067, Train: 0.8623, Test: 0.8520\n",
            "Epoch: 068, Train: 0.8614, Test: 0.8540\n",
            "Epoch: 069, Train: 0.8686, Test: 0.8460\n",
            "Epoch: 070, Train: 0.8698, Test: 0.8560\n",
            "Epoch: 071, Train: 0.8730, Test: 0.8560\n",
            "Epoch: 072, Train: 0.8723, Test: 0.8560\n",
            "Epoch: 073, Train: 0.8763, Test: 0.8620\n",
            "Epoch: 074, Train: 0.8770, Test: 0.8580\n",
            "Epoch: 075, Train: 0.8792, Test: 0.8620\n",
            "Epoch: 076, Train: 0.8794, Test: 0.8700\n",
            "Epoch: 077, Train: 0.8817, Test: 0.8640\n",
            "Epoch: 078, Train: 0.8843, Test: 0.8700\n",
            "Epoch: 079, Train: 0.8875, Test: 0.8760\n",
            "Epoch: 080, Train: 0.8861, Test: 0.8760\n",
            "Epoch: 081, Train: 0.8857, Test: 0.8780\n",
            "Epoch: 082, Train: 0.8882, Test: 0.8740\n",
            "Epoch: 083, Train: 0.8897, Test: 0.8740\n",
            "Epoch: 084, Train: 0.8912, Test: 0.8860\n",
            "Epoch: 085, Train: 0.8891, Test: 0.8920\n",
            "Epoch: 086, Train: 0.8934, Test: 0.8860\n",
            "Epoch: 087, Train: 0.8944, Test: 0.8780\n",
            "Epoch: 088, Train: 0.8911, Test: 0.8940\n",
            "Epoch: 089, Train: 0.8954, Test: 0.8900\n",
            "Epoch: 090, Train: 0.8959, Test: 0.8840\n",
            "Epoch: 091, Train: 0.8953, Test: 0.8900\n",
            "Epoch: 092, Train: 0.8955, Test: 0.8980\n",
            "Epoch: 093, Train: 0.8996, Test: 0.8920\n",
            "Epoch: 094, Train: 0.8994, Test: 0.8840\n",
            "Epoch: 095, Train: 0.8985, Test: 0.8960\n",
            "Epoch: 096, Train: 0.9008, Test: 0.8920\n",
            "Epoch: 097, Train: 0.9016, Test: 0.8960\n",
            "Epoch: 098, Train: 0.9017, Test: 0.8880\n",
            "Epoch: 099, Train: 0.9035, Test: 0.8960\n",
            "Epoch: 100, Train: 0.9031, Test: 0.8960\n",
            "Epoch: 101, Train: 0.9043, Test: 0.8920\n",
            "Epoch: 102, Train: 0.9059, Test: 0.8940\n",
            "Epoch: 103, Train: 0.9061, Test: 0.9020\n",
            "Epoch: 104, Train: 0.9064, Test: 0.8980\n",
            "Epoch: 105, Train: 0.9064, Test: 0.8980\n",
            "Epoch: 106, Train: 0.9068, Test: 0.8980\n",
            "Epoch: 107, Train: 0.9088, Test: 0.9020\n",
            "Epoch: 108, Train: 0.9085, Test: 0.9000\n",
            "Epoch: 109, Train: 0.9087, Test: 0.8960\n",
            "Epoch: 110, Train: 0.9094, Test: 0.9000\n",
            "Epoch: 111, Train: 0.9101, Test: 0.8960\n",
            "Epoch: 112, Train: 0.9096, Test: 0.9000\n",
            "Epoch: 113, Train: 0.9104, Test: 0.9060\n",
            "Epoch: 114, Train: 0.9114, Test: 0.9040\n",
            "Epoch: 115, Train: 0.9115, Test: 0.9040\n",
            "Epoch: 116, Train: 0.9116, Test: 0.9080\n",
            "Epoch: 117, Train: 0.9117, Test: 0.9020\n",
            "Epoch: 118, Train: 0.9135, Test: 0.9080\n",
            "Epoch: 119, Train: 0.9129, Test: 0.9040\n",
            "Epoch: 120, Train: 0.9133, Test: 0.9060\n",
            "Epoch: 121, Train: 0.9140, Test: 0.9020\n",
            "Epoch: 122, Train: 0.9140, Test: 0.9020\n",
            "Epoch: 123, Train: 0.9150, Test: 0.9080\n",
            "Epoch: 124, Train: 0.9148, Test: 0.9080\n",
            "Epoch: 125, Train: 0.9155, Test: 0.9100\n",
            "Epoch: 126, Train: 0.9150, Test: 0.9080\n",
            "Epoch: 127, Train: 0.9162, Test: 0.9120\n",
            "Epoch: 128, Train: 0.9155, Test: 0.9060\n",
            "Epoch: 129, Train: 0.9169, Test: 0.9120\n",
            "Epoch: 130, Train: 0.9170, Test: 0.9100\n",
            "Epoch: 131, Train: 0.9181, Test: 0.9140\n",
            "Epoch: 132, Train: 0.9173, Test: 0.9160\n",
            "Epoch: 133, Train: 0.9178, Test: 0.9080\n",
            "Epoch: 134, Train: 0.9144, Test: 0.9140\n",
            "Epoch: 135, Train: 0.9122, Test: 0.8980\n",
            "Epoch: 136, Train: 0.9039, Test: 0.9000\n",
            "Epoch: 137, Train: 0.9023, Test: 0.8800\n",
            "Epoch: 138, Train: 0.9042, Test: 0.9000\n",
            "Epoch: 139, Train: 0.9189, Test: 0.9120\n",
            "Epoch: 140, Train: 0.9132, Test: 0.8980\n",
            "Epoch: 141, Train: 0.9039, Test: 0.9020\n",
            "Epoch: 142, Train: 0.9144, Test: 0.8880\n",
            "Epoch: 143, Train: 0.9195, Test: 0.9080\n",
            "Epoch: 144, Train: 0.9084, Test: 0.9020\n",
            "Epoch: 145, Train: 0.9158, Test: 0.8920\n",
            "Epoch: 146, Train: 0.9196, Test: 0.9060\n",
            "Epoch: 147, Train: 0.9091, Test: 0.9040\n",
            "Epoch: 148, Train: 0.9178, Test: 0.8980\n",
            "Epoch: 149, Train: 0.9199, Test: 0.9040\n",
            "Epoch: 150, Train: 0.9112, Test: 0.9060\n",
            "Epoch: 151, Train: 0.9198, Test: 0.9160\n",
            "Epoch: 152, Train: 0.9193, Test: 0.9020\n",
            "Epoch: 153, Train: 0.9124, Test: 0.9080\n",
            "Epoch: 154, Train: 0.9222, Test: 0.9200\n",
            "Epoch: 155, Train: 0.9185, Test: 0.9080\n",
            "Epoch: 156, Train: 0.9128, Test: 0.9080\n",
            "Epoch: 157, Train: 0.9220, Test: 0.9160\n",
            "Epoch: 158, Train: 0.9180, Test: 0.9040\n",
            "Epoch: 159, Train: 0.9139, Test: 0.9100\n",
            "Epoch: 160, Train: 0.9238, Test: 0.9220\n",
            "Epoch: 161, Train: 0.9175, Test: 0.9080\n",
            "Epoch: 162, Train: 0.9139, Test: 0.9100\n",
            "Epoch: 163, Train: 0.9243, Test: 0.9180\n",
            "Epoch: 164, Train: 0.9197, Test: 0.9040\n",
            "Epoch: 165, Train: 0.9178, Test: 0.9120\n",
            "Epoch: 166, Train: 0.9252, Test: 0.9160\n",
            "Epoch: 167, Train: 0.9219, Test: 0.9080\n",
            "Epoch: 168, Train: 0.9234, Test: 0.9140\n",
            "Epoch: 169, Train: 0.9269, Test: 0.9160\n",
            "Epoch: 170, Train: 0.9231, Test: 0.9100\n",
            "Epoch: 171, Train: 0.9266, Test: 0.9120\n",
            "Epoch: 172, Train: 0.9266, Test: 0.9180\n",
            "Epoch: 173, Train: 0.9230, Test: 0.9180\n",
            "Epoch: 174, Train: 0.9290, Test: 0.9200\n",
            "Epoch: 175, Train: 0.9268, Test: 0.9180\n",
            "Epoch: 176, Train: 0.9246, Test: 0.9240\n",
            "Epoch: 177, Train: 0.9294, Test: 0.9220\n",
            "Epoch: 178, Train: 0.9274, Test: 0.9200\n",
            "Epoch: 179, Train: 0.9267, Test: 0.9200\n",
            "Epoch: 180, Train: 0.9291, Test: 0.9240\n",
            "Epoch: 181, Train: 0.9272, Test: 0.9220\n",
            "Epoch: 182, Train: 0.9282, Test: 0.9200\n",
            "Epoch: 183, Train: 0.9293, Test: 0.9220\n",
            "Epoch: 184, Train: 0.9299, Test: 0.9240\n",
            "Epoch: 185, Train: 0.9291, Test: 0.9240\n",
            "Epoch: 186, Train: 0.9277, Test: 0.9220\n",
            "Epoch: 187, Train: 0.9313, Test: 0.9320\n",
            "Epoch: 188, Train: 0.9299, Test: 0.9320\n",
            "Epoch: 189, Train: 0.9288, Test: 0.9240\n",
            "Epoch: 190, Train: 0.9317, Test: 0.9260\n",
            "Epoch: 191, Train: 0.9306, Test: 0.9280\n",
            "Epoch: 192, Train: 0.9315, Test: 0.9240\n",
            "Epoch: 193, Train: 0.9315, Test: 0.9300\n",
            "Epoch: 194, Train: 0.9317, Test: 0.9260\n",
            "Epoch: 195, Train: 0.9319, Test: 0.9260\n",
            "Epoch: 196, Train: 0.9318, Test: 0.9300\n",
            "Epoch: 197, Train: 0.9322, Test: 0.9240\n",
            "Epoch: 198, Train: 0.9319, Test: 0.9280\n",
            "Epoch: 199, Train: 0.9330, Test: 0.9320\n",
            "Epoch: 200, Train: 0.9325, Test: 0.9280\n",
            "Epoch: 201, Train: 0.9328, Test: 0.9320\n",
            "Epoch: 202, Train: 0.9333, Test: 0.9260\n",
            "Epoch: 203, Train: 0.9329, Test: 0.9280\n",
            "Epoch: 204, Train: 0.9337, Test: 0.9260\n",
            "Epoch: 205, Train: 0.9339, Test: 0.9260\n",
            "Epoch: 206, Train: 0.9339, Test: 0.9320\n",
            "Epoch: 207, Train: 0.9337, Test: 0.9280\n",
            "Epoch: 208, Train: 0.9346, Test: 0.9280\n",
            "Epoch: 209, Train: 0.9334, Test: 0.9300\n",
            "Epoch: 210, Train: 0.9350, Test: 0.9240\n",
            "Epoch: 211, Train: 0.9319, Test: 0.9280\n",
            "Epoch: 212, Train: 0.9359, Test: 0.9200\n",
            "Epoch: 213, Train: 0.9261, Test: 0.9180\n",
            "Epoch: 214, Train: 0.9299, Test: 0.9080\n",
            "Epoch: 215, Train: 0.9093, Test: 0.8860\n",
            "Epoch: 216, Train: 0.9096, Test: 0.8900\n",
            "Epoch: 217, Train: 0.8864, Test: 0.8600\n",
            "Epoch: 218, Train: 0.9157, Test: 0.9020\n",
            "Epoch: 219, Train: 0.9242, Test: 0.9000\n",
            "Epoch: 220, Train: 0.9159, Test: 0.9000\n",
            "Epoch: 221, Train: 0.9142, Test: 0.8980\n",
            "Epoch: 222, Train: 0.9193, Test: 0.8920\n",
            "Epoch: 223, Train: 0.9134, Test: 0.8940\n",
            "Epoch: 224, Train: 0.9126, Test: 0.8840\n",
            "Epoch: 225, Train: 0.9103, Test: 0.8960\n",
            "Epoch: 226, Train: 0.9036, Test: 0.8860\n",
            "Epoch: 227, Train: 0.9079, Test: 0.8800\n",
            "Epoch: 228, Train: 0.8974, Test: 0.8820\n",
            "Epoch: 229, Train: 0.9097, Test: 0.8920\n",
            "Epoch: 230, Train: 0.8854, Test: 0.8540\n",
            "Epoch: 231, Train: 0.9072, Test: 0.8840\n",
            "Epoch: 232, Train: 0.9028, Test: 0.8860\n",
            "Epoch: 233, Train: 0.9263, Test: 0.9080\n",
            "Epoch: 234, Train: 0.9172, Test: 0.8840\n",
            "Epoch: 235, Train: 0.9102, Test: 0.9020\n",
            "Epoch: 236, Train: 0.8981, Test: 0.8860\n",
            "Epoch: 237, Train: 0.9303, Test: 0.9140\n",
            "Epoch: 238, Train: 0.9184, Test: 0.8880\n",
            "Epoch: 239, Train: 0.9211, Test: 0.9120\n",
            "Epoch: 240, Train: 0.9084, Test: 0.8860\n",
            "Epoch: 241, Train: 0.9301, Test: 0.9180\n",
            "Epoch: 242, Train: 0.9264, Test: 0.9040\n",
            "Epoch: 243, Train: 0.9279, Test: 0.9020\n",
            "Epoch: 244, Train: 0.9308, Test: 0.9100\n",
            "Epoch: 245, Train: 0.9263, Test: 0.9020\n",
            "Epoch: 246, Train: 0.9326, Test: 0.9220\n",
            "Epoch: 247, Train: 0.9269, Test: 0.8980\n",
            "Epoch: 248, Train: 0.9376, Test: 0.9280\n",
            "Epoch: 249, Train: 0.9298, Test: 0.9100\n",
            "Epoch: 250, Train: 0.9264, Test: 0.9180\n",
            "Epoch: 251, Train: 0.9346, Test: 0.9140\n",
            "Epoch: 252, Train: 0.9298, Test: 0.9120\n",
            "Epoch: 253, Train: 0.9364, Test: 0.9300\n",
            "Epoch: 254, Train: 0.9240, Test: 0.9180\n",
            "Epoch: 255, Train: 0.9361, Test: 0.9260\n",
            "Epoch: 256, Train: 0.9352, Test: 0.9200\n",
            "Epoch: 257, Train: 0.9380, Test: 0.9140\n",
            "Epoch: 258, Train: 0.9330, Test: 0.9280\n",
            "Epoch: 259, Train: 0.9360, Test: 0.9160\n",
            "Epoch: 260, Train: 0.9400, Test: 0.9360\n",
            "Epoch: 261, Train: 0.9379, Test: 0.9200\n",
            "Epoch: 262, Train: 0.9370, Test: 0.9260\n",
            "Epoch: 263, Train: 0.9393, Test: 0.9240\n",
            "Epoch: 264, Train: 0.9381, Test: 0.9240\n",
            "Epoch: 265, Train: 0.9372, Test: 0.9340\n",
            "Epoch: 266, Train: 0.9395, Test: 0.9240\n",
            "Epoch: 267, Train: 0.9406, Test: 0.9280\n",
            "Epoch: 268, Train: 0.9377, Test: 0.9220\n",
            "Epoch: 269, Train: 0.9382, Test: 0.9340\n",
            "Epoch: 270, Train: 0.9406, Test: 0.9320\n",
            "Epoch: 271, Train: 0.9402, Test: 0.9320\n",
            "Epoch: 272, Train: 0.9394, Test: 0.9300\n",
            "Epoch: 273, Train: 0.9387, Test: 0.9260\n",
            "Epoch: 274, Train: 0.9409, Test: 0.9320\n",
            "Epoch: 275, Train: 0.9410, Test: 0.9280\n",
            "Epoch: 276, Train: 0.9402, Test: 0.9300\n",
            "Epoch: 277, Train: 0.9410, Test: 0.9280\n",
            "Epoch: 278, Train: 0.9413, Test: 0.9260\n",
            "Epoch: 279, Train: 0.9405, Test: 0.9300\n",
            "Epoch: 280, Train: 0.9413, Test: 0.9280\n",
            "Epoch: 281, Train: 0.9417, Test: 0.9280\n",
            "Epoch: 282, Train: 0.9408, Test: 0.9280\n",
            "Epoch: 283, Train: 0.9411, Test: 0.9300\n",
            "Epoch: 284, Train: 0.9419, Test: 0.9300\n",
            "Epoch: 285, Train: 0.9420, Test: 0.9280\n",
            "Epoch: 286, Train: 0.9410, Test: 0.9340\n",
            "Epoch: 287, Train: 0.9420, Test: 0.9300\n",
            "Epoch: 288, Train: 0.9426, Test: 0.9280\n",
            "Epoch: 289, Train: 0.9420, Test: 0.9280\n",
            "Epoch: 290, Train: 0.9419, Test: 0.9300\n",
            "Epoch: 291, Train: 0.9431, Test: 0.9280\n",
            "Epoch: 292, Train: 0.9420, Test: 0.9300\n",
            "Epoch: 293, Train: 0.9428, Test: 0.9280\n",
            "Epoch: 294, Train: 0.9429, Test: 0.9300\n",
            "Epoch: 295, Train: 0.9427, Test: 0.9280\n",
            "Epoch: 296, Train: 0.9430, Test: 0.9280\n",
            "Epoch: 297, Train: 0.9431, Test: 0.9260\n",
            "Epoch: 298, Train: 0.9435, Test: 0.9260\n",
            "Epoch: 299, Train: 0.9424, Test: 0.9320\n",
            "Epoch: 300, Train: 0.9430, Test: 0.9300\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eorD9_oeOtAW",
        "colab_type": "text"
      },
      "source": [
        "Now lets run our network on the validation data, in order to calculate its accuracy on new data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6hh5MnviNsPs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e51b0676-8394-45ca-e8cb-578bac1b3940"
      },
      "source": [
        "print(\"Accuracy: {:.4f}\".format(validate()[0]))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 0.9160\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}