PySpark and Social Media.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spark and Social Media\n",
    "***\n",
    "<a href='https://github.com/pick1'> <img src='sparkjupyter.png' /></a>\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkConf, SparkContext\n",
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "conf = SparkConf().setMaster(\"local\").setAppName(\"FriendsByAge\")\n",
    "sc = SparkContext(conf = conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = sc.textFile(\"fakefriends.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function for slicing the dataset for Ages and Friend Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lineparser(line):\n",
    "    fields = line.split(',')\n",
    "    age = int(fields[2])\n",
    "    numFriends = int(fields[3])\n",
    "    return (age, numFriends)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up new RDD (Spark Dataframe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "rdd = lines.map(lineparser)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Totals by Age\n",
    "**.mapValues leaves the keys untouched. Transforms friends info into tuple with keeping age untouched and adding a '1' for adding in .reduceByKey.**\n",
    "\n",
    "** Next, .reduceByKey adds the total number of friends per age & the total instances of an age.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit \n",
    "totalsByAge = rdd.mapValues(lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Averages by Age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.5 µs ± 149 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
     ]
    }
   ],
   "source": [
    "averagesByAge = totalsByAge.mapValues(lambda x: x[0] / x[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Go Do Something Spark!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(18, 343.375)\n",
      "(19, 213.27272727272728)\n",
      "(20, 165.0)\n",
      "(21, 350.875)\n",
      "(22, 206.42857142857142)\n",
      "(23, 246.3)\n",
      "(24, 233.8)\n",
      "(25, 197.45454545454547)\n",
      "(26, 242.05882352941177)\n",
      "(27, 228.125)\n",
      "(28, 209.1)\n",
      "(29, 215.91666666666666)\n",
      "(30, 235.8181818181818)\n",
      "(31, 267.25)\n",
      "(32, 207.9090909090909)\n",
      "(33, 325.3333333333333)\n",
      "(34, 245.5)\n",
      "(35, 211.625)\n",
      "(36, 246.6)\n",
      "(37, 249.33333333333334)\n",
      "(38, 193.53333333333333)\n",
      "(39, 169.28571428571428)\n",
      "(40, 250.8235294117647)\n",
      "(41, 268.55555555555554)\n",
      "(42, 303.5)\n",
      "(43, 230.57142857142858)\n",
      "(44, 282.1666666666667)\n",
      "(45, 309.53846153846155)\n",
      "(46, 223.69230769230768)\n",
      "(47, 233.22222222222223)\n",
      "(48, 281.4)\n",
      "(49, 184.66666666666666)\n",
      "(50, 254.6)\n",
      "(51, 302.14285714285717)\n",
      "(52, 340.6363636363636)\n",
      "(53, 222.85714285714286)\n",
      "(54, 278.0769230769231)\n",
      "(55, 295.53846153846155)\n",
      "(56, 306.6666666666667)\n",
      "(57, 258.8333333333333)\n",
      "(58, 116.54545454545455)\n",
      "(59, 220.0)\n",
      "(60, 202.71428571428572)\n",
      "(61, 256.22222222222223)\n",
      "(62, 220.76923076923077)\n",
      "(63, 384.0)\n",
      "(64, 281.3333333333333)\n",
      "(65, 298.2)\n",
      "(66, 276.44444444444446)\n",
      "(67, 214.625)\n",
      "(68, 269.6)\n",
      "(69, 235.2)\n"
     ]
    }
   ],
   "source": [
    "results = averagesByAge.collect()\n",
    "for result in results:\n",
    "    print(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}