Lecture2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2\n",
    "\n",
    "## Last time (Bishop Ch. 1)\n",
    "- What is Machine Learning?\n",
    "- Loss L(guess,actual)\n",
    "- Risk $\\mathbb{E}[L] = \\mathbb{E}_{X,Y}L(h(X),Y)$\n",
    "\n",
    "In general, we don't know the joint probability between x and y.\n",
    "\n",
    "## Today (Bishop Ch 2.1 - 2.4)\n",
    "- Modeling (esp. discrete models) - how do we estimate that probability?\n",
    "- Bayesian inference\n",
    "\n",
    "We were considering data that maybe didn't have features.\n",
    "\n",
    "Suppose we have\n",
    "\n",
    "Data: $y^{(1)}, ..., y^{(n)}$\n",
    "\n",
    "Risk = $\\mathbb{E}_YL(h,Y) = \\int L(h,y)p(y)dy$\n",
    "\n",
    "$p(y)$ unknown\n",
    "\n",
    "Last time, in order to come up with some guess of p(y)\n",
    "- We considered the *empirical distribution*\n",
    "- We also tried assuming a *parametric model*.  We say $p(y)$ takes some particular form, and now the thing we don't know is just $\\theta$, a particular parameter.  $p(y \\ | \\ \\theta)$\n",
    "\n",
    "\n",
    "**Example (continued)**\n",
    "\n",
    "$Y^{(i)} \\in {0,1}$ $Y^{(i)} ~ iid. \\text{Bern}(\\theta)$   $\\ \\ \\ \\theta \\in [0,1]$\n",
    "\n",
    "$p_Y(y) = \\theta^y(1-\\theta)^{1-y}$\n",
    "\n",
    "\n",
    "Nobody actually goes to iTunes and flips a coin to decide whether or not to buy a song.\n",
    "\n",
    "\n",
    "Useful quote from George Box, a statistician: \"All models are wrong, but some are useful.\"\n",
    "\n",
    "Great, so now we have the Bernoulli model, and it's appropraite for a lot of data.  But we don't know $\\theta$.\n",
    "\n",
    "This is a Bayesian idea to have some distribution on what $\\theta$ is.\n",
    "\n",
    "**Bayes Theorem**\n",
    "\n",
    "Random variables $Y, \\Theta$\n",
    "\n",
    "Bayes says:\n",
    "\n",
    "$$P_{\\Theta\\  | \\ Y} (\\theta\\  | \\ y) = \\frac{P_{Y \\ | \\ \\Theta}(y \\ | \\ \\theta) P_{\\Theta}(\\theta)}{P_Y(y)}$$\n",
    "\n",
    "**Bayesian Inference**\n",
    "\n",
    "$P_{Y \\ | \\ \\Theta} (y\\  | \\ \\theta)$: likelihood\n",
    "\n",
    "$P_{\\Theta}(\\theta)$: prior.  This is what really makes it Bayesian.\n",
    "\n",
    "$P_{\\Theta\\  | \\ Y} (\\theta\\  | \\ y)$: posterior.  Your uncertainty after seeing the data.\n",
    "\n",
    "$P_{Y}(y)$: evidence\n",
    "\n",
    "\n",
    "Proposed distribution is a Beta distribution.  We want a continuous distribution that puts mass on 0 and 1.\n",
    "\n",
    "$$\\Theta \\approx Beta(a,b)$$\n",
    "\n",
    "$a,b > 0$ \"hyperparameters\".  Called hyperparameters because they are parameters for our parameter.\n",
    "\n",
    "$p(\\theta) = Beta(\\theta) = \\frac{\\Gamma(a + b)}{\\Gamma(a)\\Gamma(b)} \\theta^{a-1} (1-\\theta)^{b-1}$\n",
    "\n",
    "Gamma function\n",
    "- $m \\in \\mathbb{Z}_+ \\ \\ \\ \\ \\ \\Gamma(m) = (m-1)!$\n",
    "- $t > 0: \\ \\ \\ \\ \\Gamma(t+1) = t\\Gamma(t)$\n",
    "\n",
    "What is Beta distribution is $a = 1$ and $b = 1$?  The uniform distribution.\n",
    "\n",
    "Play with Beta distribution for $(a,b) = (0.5,0.5) ; (a,b) = (2,2) ; (a,b) = (2,5).$\n",
    "\n",
    "What we're really doing is drawing a random distribution.\n",
    "\n",
    "We are drawing a $\\theta$, but we are also drawing a $1 - \\theta$.\n",
    "\n",
    "\n",
    "\n",
    "Calculate on my own (don't just look up on Wikipedia)\n",
    "\n",
    "Mean: $E[\\Theta] = \\frac{a}{a+b}$\n",
    "\n",
    "Var$[\\Theta] = \\frac{ab}{(a+b)^2(a+b+1)}$\n",
    "\n",
    "We have something that can go from very smooth to very sharp.\n",
    "\n",
    "$p(\\theta \\ | y^{(1)},...,y^{(n)}) = p(\\theta \\ | \\ y^{(1:n)})$\n",
    "\n",
    "(Professor is dropping subscripts so that it's easier to write.)\n",
    "\n",
    "Proportional in $\\theta to p_n(y^{(1:n)} \\ | \\ \\theta)p(\\theta)$\n",
    "\n",
    "$= [ \\prod_{i=1}^{n} p(y^{(i)} \\ | \\ \\theta)] p(\\theta)$\n",
    "\n",
    "\n",
    "proprotional in $\\theta$ [\\Theta^{\\sum_{i = 1}^ny^{(i)}(1 - \\theta)^{n - \\sum_{i=1}^ny^{(i)}]\n",
    "\n",
    "* $\\theta^{a-1}(1 - \\theta)^{b - 1}$\n",
    "\n",
    "$p(\\theta \\ | \\ y^{(1:n)}) = Beta(\\sum_{i=1}^ny^{(i)} + a, n - \\sum_{i = 1}^ny^{(i)} + b)$\n",
    "\n",
    "\n",
    "$E[\\Theta \\ | \\ Y^{(1:n)} = y^{(1:n)})$\n",
    "\n",
    "$= \\frac{\\sum y^{(i)} + a}{n + a + b}$ $ \\ \\ \\ c.f. \\theta_ML = \\frac{\\sum_{i=1}^n y{(i)}}{n}$\n",
    "\n",
    "$= (\\frac{n}{n+a+b})\\frac{\\sum y^{(i)}}{n} + (\\frac{a+b}{n+a+b})\\frac{a}{a+b}$\n",
    "\n",
    "\n",
    "\n",
    "How do we choose the prior?  This is a huge question.\n",
    "\n",
    "- 1 option: hey, I've worked at Google before, I have some data\n",
    "- 2 option: we can collect some data\n",
    "\n",
    "We assumed we would have a beta distribution, which lets us have nice different shapes.  But it doesn't, for example, let us have a bimodal distribution.\n",
    "\n",
    "For now beta will let us do some things, but it won't let us do a lot of things.\n",
    "\n",
    "Entire book about choosing priors and related topics: **Robert 2007, The Bayesian Choice**.  Another good book: **Bayesian Data Analysis**.\n",
    "\n",
    "The way we motivated starting this discussion was to say, \"hey we really want the distribution $p(y)$\".  The Bayesian method gives us a way to get at that.\n",
    "\n",
    "Risk = $E_Y L(h,Y)$\n",
    "\n",
    "$= \\int L(h,Y)p(Y)dY$\n",
    "\n",
    "$p(y) \\approx \\frac{1}{n} \\sum_{i=1}^n \\mathbb{1}\\{ y = y^{(i)} \\}$\n",
    "\n",
    "\n",
    "Another option that we pursued was maximum likelihood estimation, in the service of prediction.\n",
    "\n",
    "$p(y) \\approx  p(y \\ | \\ \\theta_{ML})$\n",
    "\n",
    "But now the Bayesian approach. \n",
    "\n",
    "$p(y) \\approx p(y^{n+1} = y \\ | \\ y^{(1)},...,y^{(n)})$\n",
    "\n",
    "posterior predictive distribution.\n",
    "\n",
    "\n",
    "New example\n",
    "\n",
    "$p(y^{(n+1)} \\ | \\ y^{(1:n)} ) = \\int p(y^{(n+1)}, \\theta \\ | \\ y^{(1:n)}) d\\theta$\n",
    "\n",
    "$= \\int p(y^{(n+1)} \\ | \\ \\theta, y^{(1:n)}) p(\\theta \\ | \\ y^{(1:n)}) d\\theta$\n",
    "\n",
    "$= \\int p(y^{(n+1)} \\ | \\ \\theta) p(\\theta \\ | \\ y^{(1:n)}) d\\theta$\n",
    "\n",
    "\n",
    "\n",
    "$p(y^{(n+1)} = 1 \\ | \\ y^{(1:n)}) = \\int \\theta p(\\theta \\ | \\ y^{(1:n)}) d\\theta$\n",
    "\n",
    "$= \\frac{\\sum_{i=1}^n y^{(i)} + a}{n+a+b}$\n",
    "\n",
    "\n",
    "**Note**\n",
    "\n",
    "$Y^{(i)} \\approx iid \\ \\ \\ \\ text{Bern}(\\theta)$\n",
    "\n",
    "$\\Doublearrow$\n",
    "\n",
    "$\\~ Y(1) \\~ \\text{Bin}(n,\\theta) \\ \\ \\ $   Binomial\n",
    "\n",
    "\n",
    "Consider\n",
    "\n",
    "\n",
    "$Y^{(i)} \\approx iid    { 1 .... K   |   w.p. \\theta_1 ... w.p \\theta_k$\n",
    "\n",
    "Discrete categorical multinoulli\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "n draws:   $M_k = \\sum_{i=1}^n \\mathbb{1}{Y^{(i)} = k}$\n",
    "\n",
    "$M_1, ..., M_k) \\approx \\text{Mult}(n, \\theta)$\n",
    "\n",
    "\n",
    "\n",
    "Prior is **conjugate** for a likelihood if the posterior has the same form as the prior.\n",
    "\n",
    "Beta is conjugate for the Bernoulli.  (We proved that in class).\n",
    "\n",
    "\n",
    "\n",
    "What would a conjugate prior for the multinomial might look like?\n",
    "\n",
    "Answer: it's going to be a product of $\\theta$s to the something.\n",
    "\n",
    "\n",
    "\n",
    "Propose the following prior:\n",
    "\n",
    "\n",
    "$p(\\theta_{1:K}) = \\ \\ \\text{...} \\ \\ \\ \\prod_{k=1}^K \\theta_k^{a_k-1}$\n",
    "\n",
    "Will only work if our distribution is over probabilities.\n",
    "\n",
    "$\\theta_k \\in [0,1]$\n",
    "\n",
    "\n",
    "$\\sum_{k=1}^{K} \\theta_l = 1$  The (K-1)-dimensional simplex\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "$p(\\theta_{1:K}) = \\frac{\\Gamma(\\sum_{k=1}^K a_k)}{\\Gamma(a_k)} \\prod_{k=1}^K \\theta_k^{a_k-1}$\n",
    "\n",
    "\n",
    "$\\theta_{1:K} \\approx Dirichilet(a_{1:k}) a_k > 0$\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}