src/Chapter 1 - Basics/Section 3 - Reading your First Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Library Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql import types as T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above also shows you the \"best practices\" for importing these components into your program.\n",
    "\n",
    "*some of the above imports will be explained later, just know this is how you should import these functions into your Spark application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the essential `imports` that you will need for any `PySpark` program. \n",
    "\n",
    "**`SparkSession`**  \n",
    "The `SparkSession` is how you begin a Spark application. This is where you provide some configuration for your Spark program.\n",
    "\n",
    "**`pyspark.sql.functions`**  \n",
    "You will find that all your data wrangling/analysis will mostly be done by chaining together multiple `functions`. If you find that you get your desired transformations with the base functions, you should:\n",
    "1. Look through the API docs again.\n",
    "2. Ask Google.\n",
    "3. Write a `user defined function` (`udf`).\n",
    "\n",
    "**`pyspark.sql.types`**  \n",
    "When working with spark, you will need to define the type of data for each column you are working with. \n",
    "\n",
    "The possible types that Spark accepts are listed here: [Spark types](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#module-pyspark.sql.types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hello World"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = (\n",
    "    SparkSession.builder\n",
    "    .master(\"local\")\n",
    "    .appName(\"Section 3 - Reading your First Dataset\")\n",
    "    .config(\"spark.some.config.option\", \"some-value\")\n",
    "    .getOrCreate()\n",
    ")\n",
    "\n",
    "sc = spark.sparkContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `SparkSession`. No need to create `SparkContext` as you automatically get it as part of the `SparkSession`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in Data (CSV)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the structure of your data inside the CSV file\n",
    "def get_csv_schema(*args):\n",
    "    return T.StructType([\n",
    "        T.StructField(*arg)\n",
    "        for arg in args\n",
    "    ])\n",
    "\n",
    "# read in your csv file with enforcing a schema\n",
    "def read_csv(fname, schema):\n",
    "    return spark.read.csv(\n",
    "        path=fname,\n",
    "        header=True,\n",
    "        schema=get_csv_schema(*schema)\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "data_path = \"/data\"\n",
    "pets_path = \"/pets.csv\"\n",
    "base_path = os.path.dirname(os.getcwd())\n",
    "\n",
    "path = base_path + data_path + pets_path\n",
    "df = read_csv(\n",
    "    fname=path,\n",
    "    schema=[\n",
    "        (\"id\", T.LongType(), False),\n",
    "        (\"breed_id\", T.LongType(), True),\n",
    "        (\"name\", T.StringType(), True),\n",
    "        (\"birthday\", T.TimestampType(), True),\n",
    "        (\"color\", T.StringType(), True)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>species_id</th>\n",
       "      <th>name</th>\n",
       "      <th>birthday</th>\n",
       "      <th>color</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>King</td>\n",
       "      <td>2014-11-22 12:30:31</td>\n",
       "      <td>brown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Argus</td>\n",
       "      <td>2016-11-22 10:05:10</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  species_id   name            birthday  color\n",
       "0   1           1   King 2014-11-22 12:30:31  brown\n",
       "1   2           3  Argus 2016-11-22 10:05:10   None"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What Happened?\n",
    "Here we read in a `csv` file and put it into a `DataFrame (DF)`: this is one of the three datasets that Spark allows you to use. The other two are `Resilient Distributed Dataset (RDD)` and `Dataset`. `DF`s have replaced `RDD`s as more features have been brought out in version `2.x` of Spark. You should be able to perform anything with `DataFrames` now, if not you will have to work with `RDD`s, which I will not cover.\n",
    "\n",
    "Spark gives you the option to automatically infer the schema and types of columns in your dataset. But you should always specify a `schema` for the data that you're reading in. For each column in the `csv` file we specified:\n",
    "* the `name` of the column\n",
    "* the `data type` of the column\n",
    "* if `null` values can appear in the column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "Congratulations! You've read in your first dataset in Spark. Next we'll look at how you can perform transformations on this dataset :)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}