Lab_4_(part1)_PyTorch_Basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "Lab 4 (part1) PyTorch Basics",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CfX2Bxc508wY",
        "colab_type": "text"
      },
      "source": [
        "#Lab 4 (part 1): PyTorch basics\n",
        "In this lab we will use PyTorch to implement linear regression. We will implement Gradient Descent.\n",
        "\n",
        "Let's first import what we need:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FfvFdInZ08wV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "import torch\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sjXceD0_08wa",
        "colab_type": "text"
      },
      "source": [
        "##1. Training set\n",
        "The goal of linear regression is to fit a line to a set of points. We will generate a traing set of n = 100 random samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uAxryP-808wa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Number of samples\n",
        "n=100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6DOzxrYu6QOP",
        "colab_type": "text"
      },
      "source": [
        "PyTorch and other deep learning frameworks use Tensors instead of arrays. Tensors are similar to numpy’s ndarrays (i.e., multi-dimensional arrays), with the addition being that Tensors can also be used on a GPU to accelerate computing. See intro [here](https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py).\n",
        "\n",
        "Let's first define some random x-values in the range -1 to 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1oZLOrIl08wc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = torch.ones(n,2) # Tensor\n",
        "x[:,0].uniform_(-1.,1)\n",
        "print(x[:5])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "grPGHi-o6d-o",
        "colab_type": "text"
      },
      "source": [
        "###Questions 1:\n",
        "- What do you think  the column of 1's is used for?\n",
        "- What does the function `.uniform_()` do?\n",
        "- PyTorch functions with suffix `_` have a special meaning. Can you figure out what they mean?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ugnfKw8baj4M",
        "colab_type": "text"
      },
      "source": [
        "Now, define the ground truth weights of the linear model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jh9xxgjT08wf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w = np.array([3.,2]).astype(np.float32) # Numpy array\n",
        "w = torch.from_numpy(w) # Tensor\n",
        "w"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSqLOWKm6nBm",
        "colab_type": "text"
      },
      "source": [
        "Generate y-values by applying the model and adding random noise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gGXiyC-c08wh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y = x@w + torch.rand(n)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "izhXsmZtCNxE",
        "colab_type": "text"
      },
      "source": [
        "In the above, the @ stands for the dot product operation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nV2_PtID08wj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.scatter(x[:,0], y);"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P-zooDrvCaPq",
        "colab_type": "text"
      },
      "source": [
        "##2. Loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2aUb2pU08wl",
        "colab_type": "text"
      },
      "source": [
        "We want to find weights `w` such that we minimize the *error* between the points and the line `x@w`. Note that here `w` is unknown. For a regression problem the most common *error function* or *loss function* is the mean squared error or **quadratic error**:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mBk3_OGv08wm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def mse(y_hat, y):\n",
        "  return ((y_hat-y)**2).mean()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acUDa8Lu08wo",
        "colab_type": "text"
      },
      "source": [
        "Suppose we believe `w = (-1.0,1.0)` then we can compute `y_hat` which is our *prediction* and then compute our error."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m8u7w7kf08wp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w = np.array([-1.,1]).astype(np.float32)\n",
        "w = torch.as_tensor(w)\n",
        "y_hat = x@w\n",
        "loss = mse(y, y_hat)\n",
        "print(loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WLXwfs7Z-Oey",
        "colab_type": "text"
      },
      "source": [
        "Notice what is printed here: `loss` is a tensor with a scalar value.  \n",
        "\n",
        "Plot the corresponding line:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gvmb4wzK08ws",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.scatter(x[:,0],y)\n",
        "plt.scatter(x[:,0],y_hat);"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jLKoeXvX08wv",
        "colab_type": "text"
      },
      "source": [
        "##Optimization with Autograd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jEtekDz208wu",
        "colab_type": "text"
      },
      "source": [
        "So far we have specified the *model* (linear regression) and the *loss function*. Now we need to handle *optimization*; that is, how do we find the best values for `w`? How do we find the best *fitting* linear regression. We would like to find the values of `w` that minimizes the `mse` loss.\n",
        "\n",
        "**Gradient descent** is an algorithm that minimizes functions. Given a loss function defined by a set of parameters, gradient descent starts with an initial set of parameter values and iteratively moves toward a set of parameter values that minimize the loss. This iterative minimization is achieved by taking steps in the negative direction of the loss gradient w.r.t the parameters.\n",
        "\n",
        "Central to all models implemented in PyTorch is the **Autograd package**. \n",
        "The autograd package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, which means that your backpropagation is defined by how your code is run, and that every single iteration can be different.\n",
        "\n",
        "`torch.Tensor` is a central class. If you set its attribute `.requires_grad = True`, it starts to track all operations on it. When you finish your computation you can call `.backward()` and have all the gradients computed automatically. The gradient for this tensor will be accumulated into `.grad` attribute.\n",
        "\n",
        "First set `w.requires_grad = True` to track computation with it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1xKTl4cC08ww",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w.requires_grad = True\n",
        "w"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g-5gZs9hGsN8",
        "colab_type": "text"
      },
      "source": [
        "There’s one more class which is very important for autograd implementation - a Function.\n",
        "\n",
        "Tensor and Function are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each tensor has a `.grad_fn` attribute that references a Function that has created the Tensor.\n",
        "\n",
        "When you compute the loss again, it is created as a result of an operation, so now it has a `grad_fn`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "102Co5J6E9is",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_hat = x@w\n",
        "loss = mse(y, y_hat)\n",
        "print(loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V-yYneCuHTQK",
        "colab_type": "text"
      },
      "source": [
        "Let’s backprop now to compute the gradient of `w` with respect to the loss:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jk-A8VWiHlbI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "loss.backward()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HLGpXSK6ci7j",
        "colab_type": "text"
      },
      "source": [
        "###Question 2:\n",
        "- Can you figure out how to extract the gradient of the loss w.r.t. to `w`? (Hint: It has just been calculated by calling loss.backward(), but where is it stored?)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-j9mgrNEaNq",
        "colab_type": "text"
      },
      "source": [
        "**Note:** You can stop autograd from tracking history on Tensors by wrapping the code block in\n",
        "\n",
        "```\n",
        "with torch.no_grad()\n",
        "```\n",
        "\n",
        "We usually do this when updating the parameters (weights) during gradient descent, because we do not want these operations to be recorded for our next calculation of the gradient.\n",
        "\n",
        "Putting all the pieces together, here is the update function for gradient descent:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fy4_PzYc08wy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def update():\n",
        "\n",
        "    # predict\n",
        "    y_hat = x@w\n",
        "\n",
        "    # calculate loss\n",
        "    loss = mse(y, y_hat)\n",
        "    \n",
        "    if epoch % 10 == 0: print(loss)\n",
        "    \n",
        "    # calculate gradient of loss\n",
        "    loss.backward()\n",
        "\n",
        "    # update parameters (temporarily stop autograd from tracking history)\n",
        "    with torch.no_grad():\n",
        "        w.sub_(lr * w.grad) # in-place subtraction\n",
        "        w.grad.zero_()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i0sNjTkIdpu-",
        "colab_type": "text"
      },
      "source": [
        "###Question 3:\n",
        "- What does `w.grad.zero_()` do, and why is it necessary to call this function?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kKM7rAZf08w0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 1e-1\n",
        "for epoch in range(100):\n",
        "  update()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FnWOomhS08w2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w.requires_grad = False # Disable gradients, otherwise pyplot refuses to plot x@w\n",
        "plt.scatter(x[:,0],y)\n",
        "plt.scatter(x[:,0],x@w);"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dozosePU08w4",
        "colab_type": "text"
      },
      "source": [
        "## Animate it!\n",
        "This is just for fun :-)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sbFJJrXv08w5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from matplotlib import animation, rc\n",
        "rc('animation', html='jshtml')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nki99Gjm08w7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "w = np.array([-1.,1]).astype(np.float32)\n",
        "w = torch.as_tensor(w)\n",
        "\n",
        "fig = plt.figure()\n",
        "plt.scatter(x[:,0], y, c='orange')\n",
        "line, = plt.plot(x[:,0], x@w)\n",
        "plt.close()\n",
        "\n",
        "def animate(i):\n",
        "    w.requires_grad = True\n",
        "    update()\n",
        "    w.requires_grad = False\n",
        "    line.set_ydata(x@w)\n",
        "    return line,\n",
        "\n",
        "animation.FuncAnimation(fig, animate, np.arange(0, 100), interval=20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YfrAnEku08w9",
        "colab_type": "text"
      },
      "source": [
        "In practice, we usually don't calculate the gradients on the whole training set at once, but we use *mini-batches*. Then gradient descent becomes Stochastic Gradient Descent (SGD)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WsrBg5lu08w-",
        "colab_type": "text"
      },
      "source": [
        "## Vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H65gXtOZ08w_",
        "colab_type": "text"
      },
      "source": [
        "- Learning rate\n",
        "- Epoch\n",
        "- Minibatch\n",
        "- SGD\n",
        "- Model / Architecture\n",
        "- Parameters\n",
        "- Loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d_zrEgpeM7w",
        "colab_type": "text"
      },
      "source": [
        "##Optional task\n",
        "For classification problems, we use *cross entropy loss*, also known as *negative log likelihood loss*. This penalizes incorrect confident predictions, and correct unconfident predictions.\n",
        "\n",
        "Can you implemnt a simple example like above, but for logistic regression? You need to change both the model (recall the output in logistic regression is a sigmoid) and the loss function (cross entropy)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vnOWJtHI08w_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}