tutorials/energy-join/01-energy-join-weather-in-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load nyc_energy and enrich it with weather data\n",
    "\n",
    "In this notebook, we try to enrich the NYC Energy data in Jupyter Notebook in a scalable way.\n",
    "We enrich the input data by month, put the monthly enriched data in the temp folder, and save the final result in the current folder every time we have done one month.\n",
    "\n",
    "* Load csv file which is downloaded from:  https://notebooks.azure.com/frlazzeri/projects/automatedml-ms-build/html/nyc_energy.csv\n",
    "* Time range: 1/1/2012  to 8/12/2017\n",
    "* Location: 'PORT AUTH DOWNTN MANHATTAN WALL ST' station at <font color='red'>lat: 40.701, long: -74.009</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install packages if it's not availble.\n",
    "\n",
    "# !pip uninstall -y azureml-opendatasets\n",
    "# !pip install azureml-opendatasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize global variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(datetime.datetime(2012, 1, 1, 0, 0), datetime.datetime(2017, 8, 12, 23, 59))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "\n",
    "start_date = datetime(2012, 1, 1, 0, 0)\n",
    "end_date = datetime(2017, 8, 12, 23, 59)\n",
    "\n",
    "start_date, end_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "68"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import timedelta\n",
    "from dateutil.relativedelta import relativedelta\n",
    "\n",
    "import math\n",
    "\n",
    "\n",
    "r = relativedelta(end_date, start_date)\n",
    "months = r.years * 12 + r.months + math.floor((r.days + 30)/31)\n",
    "months"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40.701, -74.009)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lat, long = 40.701, -74.009\n",
    "lat, long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load ``\"./nyc_energy.csv\"`` (download and save to local) and preview the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>4937.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>4752.1</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>4542.6</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>4357.7</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>4275.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             timeStamp  demand     lat    long\n",
       "0  2012-01-01 00:00:00  4937.5  40.701 -74.009\n",
       "1  2012-01-01 01:00:00  4752.1  40.701 -74.009\n",
       "2  2012-01-01 02:00:00  4542.6  40.701 -74.009\n",
       "3  2012-01-01 03:00:00  4357.7  40.701 -74.009\n",
       "4  2012-01-01 04:00:00  4275.5  40.701 -74.009"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pandas import read_csv\n",
    "\n",
    "\n",
    "df = read_csv('./nyc_energy.csv').drop(columns=['precip', 'temp'], axis=1)\n",
    "df['lat'] = lat\n",
    "df['long'] = long\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extend the timeStamp column so that we can filter it easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>new_datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>4937.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>4752.1</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>4542.6</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>4357.7</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>4275.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             timeStamp  demand     lat    long        new_datetime\n",
       "0  2012-01-01 00:00:00  4937.5  40.701 -74.009 2012-01-01 00:00:00\n",
       "1  2012-01-01 01:00:00  4752.1  40.701 -74.009 2012-01-01 01:00:00\n",
       "2  2012-01-01 02:00:00  4542.6  40.701 -74.009 2012-01-01 02:00:00\n",
       "3  2012-01-01 03:00:00  4357.7  40.701 -74.009 2012-01-01 03:00:00\n",
       "4  2012-01-01 04:00:00  4275.5  40.701 -74.009 2012-01-01 04:00:00"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dateutil import parser\n",
    "\n",
    "\n",
    "df['new_datetime'] = df['timeStamp'].apply(parser.parse)\n",
    "raw_columns = list(df.columns)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the temp folder in which we save the enriched data per month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "!if [ ! -d \"./temp\" ]; then mkdir temp; fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enriching..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2019-04-29 09:00:03.181218] Start enriching...\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.29 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=1/part-00004-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-57.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=84996.24 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=85094.1 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.5 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=2/part-00011-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-64.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43456.5 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43512.0 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=16.26 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=3/part-00001-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-54.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45698.45 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45803.06 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.27 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=4/part-00005-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-58.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=54456.14 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=54511.25 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=19.45 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=5/part-00003-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-56.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44063.01 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44118.81 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.76 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=6/part-00010-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-63.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45127.85 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45182.84 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.56 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=7/part-00006-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-59.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=75718.52 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=75780.16 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.76 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=8/part-00007-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-60.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=129134.28 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=129187.24 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.15 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=9/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=9/part-00008-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-61.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42776.43 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=42830.6 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=20.45 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=10/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=10/part-00002-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-55.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45559.84 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45615.31 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.28 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=11/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=11/part-00009-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-62.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44528.69 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44578.58 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.67 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2012/month=12/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2012/month=12/part-00000-tid-7816671341480880202-0b49e80b-f206-4731-ab5a-61d53f99b595-53.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=88248.81 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=88308.84 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.71 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=1/part-00007-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-73.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44205.78 [ms]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44269.14 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.68 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=2/part-00011-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-77.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=62644.02 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=62696.92 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.12 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=3/part-00008-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-74.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=63616.67 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=63668.2 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.23 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=4/part-00010-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-76.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44978.72 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45061.97 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.39 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=5/part-00003-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-69.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44434.92 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44490.32 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=16.25 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=6/part-00005-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-71.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42086.7 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=42142.35 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.83 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=7/part-00004-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-70.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=41910.92 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=41964.88 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=13.26 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=8/part-00001-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-67.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=46636.82 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=46692.14 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.81 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=9/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=9/part-00006-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-72.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=46989.72 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=47071.96 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=11.18 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=10/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=10/part-00000-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-66.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=162179.34 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=162226.64 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=16.66 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=11/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=11/part-00009-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-75.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43066.49 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43117.76 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.52 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2013/month=12/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2013/month=12/part-00002-tid-236689213593784421-264283c4-dffb-42b8-9bbf-d912ec6814af-68.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45187.0 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45230.84 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.41 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=1/part-00000-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-79.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=49615.68 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=49671.23 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=13.47 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=2/part-00011-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-90.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44652.11 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44703.73 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.69 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=3/part-00001-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-80.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43063.67 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43116.56 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=16.72 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=4/part-00006-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-85.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=57670.61 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=57753.74 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=11.64 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=5/part-00003-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-82.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=47830.11 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=47883.77 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=20.26 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=6/part-00010-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-89.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=41722.47 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=41783.46 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.44 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=7/part-00009-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-88.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42955.4 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43005.24 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.54 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=8/part-00005-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-84.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42296.2 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=42380.61 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=19.23 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=9/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=9/part-00007-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-86.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=48054.2 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=48104.44 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.48 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=10/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=10/part-00002-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-81.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44798.42 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44854.57 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=19.53 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=11/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=11/part-00008-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-87.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=47152.97 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=47238.21 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.0 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2014/month=12/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2014/month=12/part-00004-tid-9219175779481662582-3729dfdb-ab32-4767-b9b6-11d2d644c3ce-83.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=41111.37 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=41203.44 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.96 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=1/part-00006-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-98.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43527.36 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43580.11 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=19.69 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=2/part-00010-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-102.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=39870.61 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=39917.0 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=20.74 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=3/part-00005-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-97.c000.snappy.parquet under container isdweatherdatacontainer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44280.98 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44339.07 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.61 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=4/part-00011-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-103.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43311.91 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43347.73 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.16 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=5/part-00001-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-93.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42948.68 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=42999.33 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=20.54 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=6/part-00008-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-100.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=40811.53 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=40867.21 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.17 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=7/part-00004-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-96.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43527.2 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43582.72 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.76 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=8/part-00007-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-99.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43075.59 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43126.87 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.8 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=9/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=9/part-00009-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-101.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=41673.19 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=41722.83 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.24 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=10/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=10/part-00002-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-94.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=42945.69 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=42996.68 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=18.69 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=11/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=11/part-00003-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-95.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43708.65 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43754.12 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.34 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2015/month=12/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2015/month=12/part-00000-tid-2198075741767757560-e3eb994e-d560-4dfc-941e-0aae74c8d9ed-92.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=60355.98 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=60447.26 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.05 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=1/part-00005-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-110.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=60960.86 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=61040.38 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.98 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=2/part-00011-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-116.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43105.63 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43189.44 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.53 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=3/part-00004-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-109.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=46416.04 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=46471.95 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.41 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=4/part-00008-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-113.c000.snappy.parquet under container isdweatherdatacontainer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43887.97 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43966.98 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.72 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=5/part-00006-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-111.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44077.01 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44163.85 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.48 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=6/part-00010-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-115.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=41326.38 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=41374.81 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.47 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=7/part-00003-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-108.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=52732.89 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=52784.83 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.63 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=8/part-00001-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-106.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44712.9 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44768.49 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=21.75 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=9/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=9/part-00007-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-112.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45721.92 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45792.78 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.39 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=10/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=10/part-00002-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-107.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=48494.7 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=48549.09 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=16.91 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=11/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=11/part-00009-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-114.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=102567.37 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=102611.05 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.52 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2016/month=12/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2016/month=12/part-00000-tid-6700213360605767691-4491b75c-f137-489b-b5df-4204b9326fda-105.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=122545.74 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=122593.63 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.22 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=1/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=1/part-00001-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-119.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=47248.72 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=47308.75 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.4 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=2/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=2/part-00011-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-129.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=44875.39 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=44932.98 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=12.19 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=3/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=3/part-00000-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-118.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=43882.2 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=43949.11 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.48 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=4/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=4/part-00007-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-125.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=53383.3 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=53444.65 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.69 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=5/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=5/part-00002-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-120.c000.snappy.parquet under container isdweatherdatacontainer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=45802.38 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=45874.65 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.22 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=6/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=6/part-00010-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-128.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=47608.01 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=47667.03 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=17.83 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=7/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=7/part-00006-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-124.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=54551.39 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=54612.86 [ms]\n",
      "ActivityStarted, get_enricher\n",
      "ActivityCompleted: Activity=get_enricher, HowEnded=Success, Duration=19.39 [ms]\n",
      "ActivityStarted, enrich_customer_data_no_agg\n",
      "ActivityStarted, enrich\n",
      "Target paths: ['/year=2017/month=8/']\n",
      "Looking for parquet files...\n",
      "Reading them into Pandas dataframe...\n",
      "Reading ISDWeather/year=2017/month=8/part-00003-tid-1321158002197267978-8e3eb092-4b7a-42de-97ee-e23297ed8955-121.c000.snappy.parquet under container isdweatherdatacontainer\n",
      "Done.\n",
      "ActivityCompleted: Activity=enrich, HowEnded=Success, Duration=39574.97 [ms]\n",
      "ActivityCompleted: Activity=enrich_customer_data_no_agg, HowEnded=Success, Duration=39606.91 [ms]\n",
      "[2019-04-29 10:13:28.810135] End enriching...\n"
     ]
    }
   ],
   "source": [
    "import os.path\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from azureml.opendatasets.accessories.location_data import LatLongColumn\n",
    "from azureml.opendatasets.accessories.location_time_customer_data \\\n",
    "    import LocationTimeCustomerData\n",
    "from azureml.opendatasets import NoaaIsdWeather\n",
    "from azureml.opendatasets.environ import PandasEnv\n",
    "\n",
    "\n",
    "if os.path.exists('./nyc_energy_enriched.csv'):\n",
    "    raise RuntimeError('nyc_energy_enriched.csv exists already.')\n",
    "else:\n",
    "    print('[%s] Start enriching...' % datetime.now())\n",
    "    all = pd.DataFrame([])\n",
    "    report_joined = {}\n",
    "    i_date = start_date\n",
    "    for m in range(months):\n",
    "        j_date = i_date + relativedelta(months=1) - timedelta(milliseconds=1)\n",
    "\n",
    "        # This is important to set monotonically increasing index for successful enrichemnt.\n",
    "        df1 = df[(df['new_datetime'] >= i_date) & (df['new_datetime'] <= j_date)].copy()\n",
    "        df1['idx'] = list(range(len(df1.index)))\n",
    "        df1 = df1.set_index('idx')\n",
    "\n",
    "        energy = LocationTimeCustomerData(\n",
    "            df1,\n",
    "            LatLongColumn('lat', 'long'),\n",
    "            'new_datetime')\n",
    "\n",
    "        weather = NoaaIsdWeather(\n",
    "            cols=[\"temperature\", \"precipTime\", \"precipDepth\", \"snowDepth\"],\n",
    "            start_date=i_date,\n",
    "            end_date=j_date)\n",
    "\n",
    "        weather_enricher = weather.get_enricher()\n",
    "        new_energy, processed_weather = weather_enricher.enrich_customer_data_no_agg(\n",
    "            customer_data_object=energy,\n",
    "            location_match_granularity=5, # higher for high join success rate, lower for performance.\n",
    "            time_round_granularity='day')\n",
    "        \n",
    "        # ---=== Begin of cusomtized aggregation ===---\n",
    "        \n",
    "        processed_weather.data['precipDepth'] = processed_weather.data['precipDepth'].apply(\n",
    "            lambda x: np.nan if x == 9999 else x)\n",
    "        processed_weather.data['precipTime'] = processed_weather.data['precipTime'].apply(\n",
    "            lambda x: np.nan if x == 99 else x)\n",
    "\n",
    "        processed_weather.data['precipDepth/precipTime'] = \\\n",
    "        processed_weather.data[['precipDepth', 'precipTime']].apply(\n",
    "            lambda x: np.nan if (\n",
    "                pd.isna(x[0]) or pd.isna(x[1]) or x[1] == 0.0) else (x[0] / x[1]), axis=1)\n",
    "        \n",
    "        aggregations = {\n",
    "            \"temperature\": \"mean\",\n",
    "            \"snowDepth\": \"mean\",\n",
    "            \"precipDepth/precipTime\": \"mean\",\n",
    "            \"precipDepth\": \"max\",\n",
    "            \"precipTime\": \"max\"}\n",
    "        \n",
    "        public_rankgroup = processed_weather.id\n",
    "\n",
    "        public_join_time = [\n",
    "            s for s in list(processed_weather.data.columns)\n",
    "            if s.startswith('ds_join_time')][0]\n",
    "\n",
    "        customer_rankgroup = weather_enricher.location_selector.customer_rankgroup\n",
    "\n",
    "        customer_join_time = [\n",
    "            s for s in list(new_energy.data.columns)\n",
    "            if s.startswith('customer_join_time')][0]\n",
    "\n",
    "        weather_df_grouped = processed_weather.data.groupby(\n",
    "            by=[public_rankgroup, public_join_time]).agg(aggregations)\n",
    "        \n",
    "        joined_dataset = new_energy.data.merge(\n",
    "            weather_df_grouped,\n",
    "            left_on=[customer_rankgroup, customer_join_time],\n",
    "            right_on=[public_rankgroup, public_join_time],\n",
    "            how='left')\n",
    "\n",
    "        final_df = joined_dataset[raw_columns + [\n",
    "            \"temperature\", \"precipTime\", \"precipDepth\", \"snowDepth\", \"precipDepth/precipTime\"]]\n",
    "\n",
    "        report_joined[i_date] = final_df.describe()\n",
    "        \n",
    "        # ---=== End of customized aggregation ===---\n",
    "        \n",
    "        fn = './temp/nyc_energy_enriched_%s.csv' % i_date\n",
    "        final_df.to_csv(fn)\n",
    "\n",
    "        all = pd.concat([all, final_df])\n",
    "        all.to_csv('./nyc_energy_enriched.csv')\n",
    "\n",
    "        i_date += relativedelta(months=1)\n",
    "\n",
    "    print('[%s] End enriching...' % datetime.now())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The final result has been saved to ``\"./nyc_energy_enriched.csv\"``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timeStamp</th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>new_datetime</th>\n",
       "      <th>temperature</th>\n",
       "      <th>precipTime</th>\n",
       "      <th>precipDepth</th>\n",
       "      <th>snowDepth</th>\n",
       "      <th>precipDepth/precipTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>4937.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>7.665934</td>\n",
       "      <td>24.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.046384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>4752.1</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>7.665934</td>\n",
       "      <td>24.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.046384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>4542.6</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>7.665934</td>\n",
       "      <td>24.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.046384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>4357.7</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>7.665934</td>\n",
       "      <td>24.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.046384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>4275.5</td>\n",
       "      <td>40.701</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>7.665934</td>\n",
       "      <td>24.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.046384</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             timeStamp  demand     lat    long        new_datetime  \\\n",
       "0  2012-01-01 00:00:00  4937.5  40.701 -74.009 2012-01-01 00:00:00   \n",
       "1  2012-01-01 01:00:00  4752.1  40.701 -74.009 2012-01-01 01:00:00   \n",
       "2  2012-01-01 02:00:00  4542.6  40.701 -74.009 2012-01-01 02:00:00   \n",
       "3  2012-01-01 03:00:00  4357.7  40.701 -74.009 2012-01-01 03:00:00   \n",
       "4  2012-01-01 04:00:00  4275.5  40.701 -74.009 2012-01-01 04:00:00   \n",
       "\n",
       "   temperature  precipTime  precipDepth  snowDepth  precipDepth/precipTime  \n",
       "0     7.665934        24.0         58.0        0.0                0.046384  \n",
       "1     7.665934        24.0         58.0        0.0                0.046384  \n",
       "2     7.665934        24.0         58.0        0.0                0.046384  \n",
       "3     7.665934        24.0         58.0        0.0                0.046384  \n",
       "4     7.665934        24.0         58.0        0.0                0.046384  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='blue'>The join success rate is 100%</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 49205 entries, 0 to 270\n",
      "Data columns (total 10 columns):\n",
      "timeStamp                 49205 non-null object\n",
      "demand                    49124 non-null float64\n",
      "lat                       49205 non-null float64\n",
      "long                      49205 non-null float64\n",
      "new_datetime              49205 non-null datetime64[ns]\n",
      "temperature               49205 non-null float64\n",
      "precipTime                49205 non-null float64\n",
      "precipDepth               49205 non-null float64\n",
      "snowDepth                 49205 non-null float64\n",
      "precipDepth/precipTime    49205 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(8), object(1)\n",
      "memory usage: 4.1+ MB\n"
     ]
    }
   ],
   "source": [
    "all.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>temperature</th>\n",
       "      <th>precipTime</th>\n",
       "      <th>precipDepth</th>\n",
       "      <th>snowDepth</th>\n",
       "      <th>precipDepth/precipTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>49124.000000</td>\n",
       "      <td>4.920500e+04</td>\n",
       "      <td>49205.000</td>\n",
       "      <td>49205.000000</td>\n",
       "      <td>49205.0</td>\n",
       "      <td>49205.000000</td>\n",
       "      <td>49205.000000</td>\n",
       "      <td>49205.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>6067.447361</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>13.372627</td>\n",
       "      <td>24.0</td>\n",
       "      <td>391.145534</td>\n",
       "      <td>1.072569</td>\n",
       "      <td>19.859086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1285.607657</td>\n",
       "      <td>7.105500e-15</td>\n",
       "      <td>0.000</td>\n",
       "      <td>9.640060</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1042.909500</td>\n",
       "      <td>4.329940</td>\n",
       "      <td>61.989804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2859.600000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>-13.226429</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5133.862250</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>5.637931</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6020.071000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>13.955882</td>\n",
       "      <td>24.0</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.065359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6684.300000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>22.236709</td>\n",
       "      <td>24.0</td>\n",
       "      <td>135.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.711580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>11456.000000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>32.852857</td>\n",
       "      <td>24.0</td>\n",
       "      <td>7630.000000</td>\n",
       "      <td>51.228571</td>\n",
       "      <td>578.402778</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             demand           lat       long   temperature  precipTime  \\\n",
       "count  49124.000000  4.920500e+04  49205.000  49205.000000     49205.0   \n",
       "mean    6067.447361  4.070100e+01    -74.009     13.372627        24.0   \n",
       "std     1285.607657  7.105500e-15      0.000      9.640060         0.0   \n",
       "min     2859.600000  4.070100e+01    -74.009    -13.226429        24.0   \n",
       "25%     5133.862250  4.070100e+01    -74.009      5.637931        24.0   \n",
       "50%     6020.071000  4.070100e+01    -74.009     13.955882        24.0   \n",
       "75%     6684.300000  4.070100e+01    -74.009     22.236709        24.0   \n",
       "max    11456.000000  4.070100e+01    -74.009     32.852857        24.0   \n",
       "\n",
       "        precipDepth     snowDepth  precipDepth/precipTime  \n",
       "count  49205.000000  49205.000000            49205.000000  \n",
       "mean     391.145534      1.072569               19.859086  \n",
       "std     1042.909500      4.329940               61.989804  \n",
       "min        0.000000      0.000000                0.000000  \n",
       "25%        0.000000      0.000000                0.000000  \n",
       "50%       10.000000      0.000000                0.065359  \n",
       "75%      135.000000      0.000000                2.711580  \n",
       "max     7630.000000     51.228571              578.402778  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>temperature</th>\n",
       "      <th>precipTime</th>\n",
       "      <th>precipDepth</th>\n",
       "      <th>snowDepth</th>\n",
       "      <th>precipDepth/precipTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>743.000000</td>\n",
       "      <td>7.440000e+02</td>\n",
       "      <td>744.000</td>\n",
       "      <td>744.000000</td>\n",
       "      <td>744.0</td>\n",
       "      <td>744.000000</td>\n",
       "      <td>744.0</td>\n",
       "      <td>744.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5427.314386</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>15.386461</td>\n",
       "      <td>24.0</td>\n",
       "      <td>81.774194</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.889804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>880.418908</td>\n",
       "      <td>7.110207e-15</td>\n",
       "      <td>0.000</td>\n",
       "      <td>4.159070</td>\n",
       "      <td>0.0</td>\n",
       "      <td>138.531590</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.683908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3946.383000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>7.381250</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4660.270500</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>12.365179</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5433.483000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>15.759633</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6101.083000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>18.396000</td>\n",
       "      <td>24.0</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.198718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7853.300000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>24.327826</td>\n",
       "      <td>24.0</td>\n",
       "      <td>521.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.738443</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            demand           lat     long  temperature  precipTime  \\\n",
       "count   743.000000  7.440000e+02  744.000   744.000000       744.0   \n",
       "mean   5427.314386  4.070100e+01  -74.009    15.386461        24.0   \n",
       "std     880.418908  7.110207e-15    0.000     4.159070         0.0   \n",
       "min    3946.383000  4.070100e+01  -74.009     7.381250        24.0   \n",
       "25%    4660.270500  4.070100e+01  -74.009    12.365179        24.0   \n",
       "50%    5433.483000  4.070100e+01  -74.009    15.759633        24.0   \n",
       "75%    6101.083000  4.070100e+01  -74.009    18.396000        24.0   \n",
       "max    7853.300000  4.070100e+01  -74.009    24.327826        24.0   \n",
       "\n",
       "       precipDepth  snowDepth  precipDepth/precipTime  \n",
       "count   744.000000      744.0              744.000000  \n",
       "mean     81.774194        0.0                1.889804  \n",
       "std     138.531590        0.0                3.683908  \n",
       "min       0.000000        0.0                0.000000  \n",
       "25%       0.000000        0.0                0.000000  \n",
       "50%       0.000000        0.0                0.000000  \n",
       "75%     140.000000        0.0                1.198718  \n",
       "max     521.000000        0.0               13.738443  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "report_joined[datetime(2016, 10, 1, 0, 0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>demand</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>temperature</th>\n",
       "      <th>precipTime</th>\n",
       "      <th>precipDepth</th>\n",
       "      <th>snowDepth</th>\n",
       "      <th>precipDepth/precipTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>720.000000</td>\n",
       "      <td>7.200000e+02</td>\n",
       "      <td>720.000</td>\n",
       "      <td>720.000000</td>\n",
       "      <td>720.0</td>\n",
       "      <td>720.000000</td>\n",
       "      <td>720.0</td>\n",
       "      <td>720.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5371.189944</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>10.440883</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1787.366667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>130.880351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>783.680586</td>\n",
       "      <td>7.110367e-15</td>\n",
       "      <td>0.000</td>\n",
       "      <td>3.318841</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1170.974488</td>\n",
       "      <td>0.0</td>\n",
       "      <td>122.276823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3999.292000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>4.346117</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4622.829500</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>7.377561</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5419.716500</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>11.029123</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2540.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>78.905303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6115.795750</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>12.923214</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2553.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>232.143362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6797.308000</td>\n",
       "      <td>4.070100e+01</td>\n",
       "      <td>-74.009</td>\n",
       "      <td>18.216071</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2646.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>404.605201</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            demand           lat     long  temperature  precipTime  \\\n",
       "count   720.000000  7.200000e+02  720.000   720.000000       720.0   \n",
       "mean   5371.189944  4.070100e+01  -74.009    10.440883        24.0   \n",
       "std     783.680586  7.110367e-15    0.000     3.318841         0.0   \n",
       "min    3999.292000  4.070100e+01  -74.009     4.346117        24.0   \n",
       "25%    4622.829500  4.070100e+01  -74.009     7.377561        24.0   \n",
       "50%    5419.716500  4.070100e+01  -74.009    11.029123        24.0   \n",
       "75%    6115.795750  4.070100e+01  -74.009    12.923214        24.0   \n",
       "max    6797.308000  4.070100e+01  -74.009    18.216071        24.0   \n",
       "\n",
       "       precipDepth  snowDepth  precipDepth/precipTime  \n",
       "count   720.000000      720.0              720.000000  \n",
       "mean   1787.366667        0.0              130.880351  \n",
       "std    1170.974488        0.0              122.276823  \n",
       "min       0.000000        0.0                0.000000  \n",
       "25%       0.000000        0.0                0.000000  \n",
       "50%    2540.000000        0.0               78.905303  \n",
       "75%    2553.000000        0.0              232.143362  \n",
       "max    2646.000000        0.0              404.605201  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "report_joined[datetime(2016, 11, 1, 0, 0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EOF"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}