files for outlier detection for demand data, where slope is used to i…

README.md

@@ -51,7 +51,7 @@ A scenario can be defined by adding an entry to the scenario list ***ScenarioLis
 * **name**: `scenario_name`;
 * **folder_location**: path to folder where MATLAB files are located;
 * **input_data_location**: path to folder where input data are located;
-* **output_data_location**: path to folder where input will be saved;
+* **output_data_location**: path to folder where output data will be saved;
 * **start_index**: start index;
 * **end_index**: end index;
 * **extract**: True/False. Should output data be converted to csv and;
@@ -138,14 +138,30 @@ from prereise.gather.demanddata.eia.test import test_from_excel
 test_from_excel.test_from_excel()
 ```
 
-The notebook [AssembleBAfromExcel_demo.ipynb](https://github.com/intvenlab/PreREISE/blob/sam/prereise/gather/demanddata/EIA/demo/AssembleBAfromExcel_demo.ipynb) illustrates usage.
+The [AssembleBAfromExcel_demo.ipynb](https://github.com/intvenlab/PreREISE/blob/anomaly_detect_1/prereise/gather/demanddata/EIA/demo/AssembleBAfromExcel_demo.ipynb) notebook illustrates usage.
 
 To output the demand profile, cleaning steps were applied to the EIA data:  
 1) missing data imputation - the EIA method was used, i.e., EIA published data was used; beyond this, NA's were converted to float zeros;  
 2) missing hours were added.
 
 The BA counts were then distributed across each region where the BA operates, using the region populations as weights. For example, if a BA operates in both WA and OR, the counts for WA are weighted by the fraction of the total counts in WA relative to the total population of WA and OR.
 
+The next step consist in detecting outliers by looking for large changes in the slope of the demand data. The underlying physical rationale is that demand changes are mostly driven by weather temperature changes (first or higher order), and thermal mass limits the rate at which demand values can change. By looking at the slope of demand data, it is seen that the slope distribution is normally distributed, and outliers can be easily found by imposing a z-score threshold value of 3. These outliers are then replaced by linear interpolation.
+
+To use the outlier detector:
+```python
+from prereise.gather.demanddata.eia import find_fix_outliers
+
+fixed_data = find_fix_outliers.slope_interpolate(BA_file, threshold)
+```
+
+To test
+```python
+from prereise.gather.demanddata.eia.test import test_slope_interpolate
+test_slope_interpolate.test_slope_interpolate()
+```
+The [BA_Anomaly_Detection_demo.ipynb](https://github.com/intvenlab/PreREISE/blob/anomaly_detect_1/prereise/gather/demanddata/EIA/demo//BA_Anomaly_Detection_demo.ipynb) notebook illustrates usage.
+
 
 
 ## 4. Start simulation

prereise/gather/demanddata/EIA/__init__.py

@@ -1 +1 @@
-__all__ = ["get_eia_data"]
+__all__ = ["get_eia_data", "find_fix_outliers"]

prereise/gather/demanddata/EIA/demo/AssembleBAfromExcel_demo.ipynb

@@ -4,7 +4,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Create BA demand counts dataframe from Excel Spreadsheets.\n"
+    "Create BA demand counts dataframe from Excel Spreadsheets.\n",
+    "Tested to work on Mac and Windows.\n"
    ]
   },
   {
@@ -15,10 +16,6 @@
    "source": [
     "import numpy as np\n",
     "import pandas as pd\n",
-    "\n",
-    "from datetime import datetime\n",
-    "from dateutil.parser import parse\n",
-    "\n",
     "import os\n",
     "\n",
     "from prereise.gather.demanddata.eia import get_eia_data"
@@ -33,8 +30,11 @@
     "#Set location of EIA data\n",
     "dir1 = os.path.join('..','test','data')\n",
     "#Set dates\n",
-    "start = pd.to_datetime('2016-01-01 00:00:00')\n",
-    "end = pd.to_datetime('2016-12-31 23:00:00')"
+    "#start = pd.to_datetime('2016-01-01 00:00:00')\n",
+    "#end = pd.to_datetime('2016-12-31 23:00:00')\n",
+    "\n",
+    "start = pd.to_datetime('2015-08-01 00:00:00')\n",
+    "end = pd.to_datetime('2015-10-31 23:00:00')"
    ]
   },
   {
@@ -135,14 +135,47 @@
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>2015-08-01 00:00:00</th>\n",
+       "      <td>7661</td>\n",
+       "      <td>40259.0</td>\n",
+       "      <td>1518</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2015-08-01 01:00:00</th>\n",
+       "      <td>7682</td>\n",
+       "      <td>40616.0</td>\n",
+       "      <td>1487</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2015-08-01 02:00:00</th>\n",
+       "      <td>7560</td>\n",
+       "      <td>40259.0</td>\n",
+       "      <td>1416</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2015-08-01 03:00:00</th>\n",
+       "      <td>7363</td>\n",
+       "      <td>39147.0</td>\n",
+       "      <td>1396</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2015-08-01 04:00:00</th>\n",
+       "      <td>7143</td>\n",
+       "      <td>37567.0</td>\n",
+       "      <td>1344</td>\n",
+       "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "text/plain": [
-       "Empty DataFrame\n",
-       "Columns: [BPAT, CISO, EPE]\n",
-       "Index: []"
+       "                     BPAT     CISO   EPE\n",
+       "2015-08-01 00:00:00  7661  40259.0  1518\n",
+       "2015-08-01 01:00:00  7682  40616.0  1487\n",
+       "2015-08-01 02:00:00  7560  40259.0  1416\n",
+       "2015-08-01 03:00:00  7363  39147.0  1396\n",
+       "2015-08-01 04:00:00  7143  37567.0  1344"
       ]
      },
      "execution_count": 5,