Eastern hydro v3

ATTRIBUTION.md

@@ -2,6 +2,22 @@
 This package contains a variety of materials, including data sets and related materials. The third party data sets and related materials are provided by their respective publishers, and may be subject to separate and additional terms and conditions. The following summarizes the sources of the applicable data, and details regarding the applicable provider. Additional terms and conditions, including certain restrictions on commercial use, redistribution, or other similar restrictions, may apply to the applicable data sets. If you cannot comply with the terms of the applicable data collection, you may not use that data, and your ability to make use of this software package, and/or the results or output you are able to generate through its use may be impacted. Please review the information provided below, and the terms and conditions provided by the publisher at the original source for more information.
 
 
+### Geographical Information
+#### BA to County Mapping
+#### Source
+* Name: Counties in the United States
+* Author: mapchart.net
+* Description: An interactive map contains all the counties in the United
+ States
+* Source: https://mapchart.net
+* Exact source location: https://mapchart.net/usa-counties.html
+
+#### Destination
+* Modifications to source file(s): None
+* Location: ***prereise/gather/data/ba_to_county.txt***
+
+
+---
 ### Generation
 #### Energy Information Administration (EIA)
 ##### Source
@@ -31,15 +47,14 @@ The dataset is used to generate hydro profiles.
 
 ##### Destination
 * Modifications to source file(s): None
-* Location: ***prereise/gather/data/demanddata/eia/demo/eastern_demand_v6_demo/spp/load/****
+* Location: ***prereise/gather/demanddata/eia/demo/eastern_demand_v6_demo/spp/load/****
 
 ##### General Purpose
 The dataset is used to generate demand profiles for the Eastern interconnection.
 
 ##### Note
 Data from this source may be freely copied and distributed except in connection with a commercial publication.
 
-
 ##### Source
 * Name: Legacy BAs geographical coverage
 * Author: SPP
@@ -49,7 +64,7 @@ Data from this source may be freely copied and distributed except in connection
 
 ##### Destination
 * Modifications to source file(s): None
-* Location: ***prereise/gather/data/demanddata/eia/demo/eastern_demand_v6_demo/spp/spp_counties_owners.xlsx***
+* Location: ***prereise/gather/demanddata/eia/demo/eastern_demand_v6_demo/spp/spp_counties_owners.xlsx***
 
 ##### General Purpose
 The dataset is used to generate demand profiles for the Eastern interconnection.
@@ -69,8 +84,8 @@ Legal review pending
 ##### Destination
 * Modifications to source file(s): None
 * Location:
-  * ***prereise/gather/data/demanddata/eia/demo/eastern_demand_v6_demo/miso/20151231_dfal_hist.xls***
-  * ***prereise/gather/data/demanddata/eia/demo/eastern_demand_v6_demo/miso/20161231_dfal_hist.xls***
+  * ***prereise/gather/demanddata/eia/demo/eastern_demand_v6_demo/miso/20151231_dfal_hist.xls***
+  * ***prereise/gather/demanddata/eia/demo/eastern_demand_v6_demo/miso/20161231_dfal_hist.xls***
 
 ##### General Purpose
 The dataset is used to generate demand profiles for the Eastern interconnection.
@@ -91,7 +106,7 @@ When using this dataset, provide a copy of the final product to MISO.
 
 ##### Destination
 * Modifications to source file(s): extract 2015, 2016 and 2017 monthly capacity factors for conventional hydropower
-* Location: ***prereise/gather/data/hydrodata/data/usa_hydro_capacity_factors.csv***
+* Location: ***prereise/gather/hydrodata/data/usa_hydro_capacity_factors.csv***
 
 ##### General Purpose
 The dataset is used to generate hydro profiles.
@@ -107,12 +122,12 @@ The dataset is used to generate hydro profiles.
 
 ##### Destination
 * Modifications to source file(s): monthly files have been concatenated and timestamps have been converted to UTC.
-* Location: ***prereise/gather/data/hydrodata/data/texas_hydro_generation.csv***
+* Location: ***prereise/gather/hydrodata/data/ercot_hydro_2016.csv***
 
 ##### General Purpose
 The dataset is used to generate hydro profiles for the Texas interconnection.
 
-#### Note
+##### Note
 Disclose modifications when redistributing this dataset in modified form.
 
 
@@ -126,12 +141,86 @@ Disclose modifications when redistributing this dataset in modified form.
 
 ##### Destination
 * Modifications to source file(s): power output on 4/26/2019 16:00 UTC for two dams (WAN and PRD) exceeds their respective capacity. These outliers were replaced with an average of the hour before and after the outlier hour, for each of the two dams.
-* Location: ***prereise/gather/data/hydrodata/data/western_hydro_generation.csv***
+* Location: ***prereise/gather/hydrodata/data/usace_hydro_2019.csv***
 
 ##### General Purpose
 The dataset is used to generate hydro profiles for the Western interconnection.
 
 
+#### ISO New England (ISONE)
+##### Source
+* Name: Real-Time Maps and Charts
+* Author: ISONE
+* Description: real-time generation data by fuel type for ISONE in UTC.
+Temporal resolution is adjustable from 2 min to 20 min per sample.
+* Source: https://www.iso-ne.com/
+* Exact source location: https://www.iso-ne.com/isoexpress/
+
+##### Destination
+* Modifications to source file(s): Average of all samples during each hour was
+used. Data from all two-day files was concatenated into one csv file for the
+entire year of 2016. Data for 121 missing hours were imputed.
+* Location: ***prereise/gather/hydrodata/data/neiso_hydro_2016.csv***
+
+##### General Purpose
+The dataset is used to generate hydro profiles for the Eastern interconnection.
+
+#### Southwest Power Pool (SPP)
+##### Source
+* Name: SPP Integrated Marketplace
+* Author: SPP
+* Description: publicly available generation fuel mix data for SPP
+* Source: https://marketplace.spp.org
+* Exact source location: https://marketplace.spp.org/pages/generation-mix-historical
+
+##### Destination
+* Modifications to source file(s): Raw data is provided at 5 min interval and
+was averaged for the hour to create an hourly profile.
+* Location: ***prereise/gather/hydrodata/data/spp_hydro_2016.csv***
+
+##### General Purpose
+The dataset is used to generate hydro profiles for the Eastern interconnection.
+
+##### Note
+Data from this source may be freely copied and distributed except in
+connection with a commercial publication.
+
+#### New York Independent System Operator (NYISO)
+##### Source
+* Name: Open Access Same-Time Information System (OASIS)
+* Author: NYISO
+* Description: historical fuel mix generation data in various temporal
+resolution
+* Source: http://mis.nyiso.com/
+* Exact source location: http://mis.nyiso.com/public/P-63list.htm
+
+##### Destination
+* Modifications to source file(s): Data was reformatted into one excel file
+and averaged among the intervals within one hour. Data for missing hours was
+computed by averaging the non-blank hour before and after the missing hour.  
+* Location: ***prereise/gather/hydrodata/data/nyiso_hydro_2016.csv***
+
+##### General Purpose
+The dataset is used to generate hydro profiles for the Eastern interconnection.
+
+####  PJM Interconnection LLC (PJM)
+##### Source
+* Name: Data Miner 2
+* Author: PJM
+* Description: historical hourly hydro generation data for PJM in 2016
+* Source: http://dataminer2.pjm.com/
+* Exact source location: http://dataminer2.pjm.com/feed/gen_by_fuel
+
+##### Destination
+* Modifications to source file(s): Data for 2015 is not available from the
+tool and the time series data for 2016 is recorded in local time. The last 5
+hours of 2016 EST were used to represent the first 5 hours of the year to
+fill in that missing data.
+* Location: ***prereise/gather/hydrodata/data/pjm_hydro_2016.csv***
+
+##### General Purpose
+The dataset is used to generate hydro profiles for the Eastern interconnection.
+
 ---
 ### Solar
 #### Energy Information Administration (EIA)
@@ -144,7 +233,7 @@ The dataset is used to generate hydro profiles for the Western interconnection.
 
 ##### Destination
 * Modifications to source file(s): convert to `csv`
-* Location: ***prereise/gather/data/solardata/data/3_3_Solar_Y2016.csv***
+* Location: ***prereise/gather/solardata/data/3_3_Solar_Y2016.csv***
 
 ##### General Purpose
 The dataset is used to generate solar profiles.
@@ -162,7 +251,7 @@ The dataset is used to generate solar profiles.
 
 ##### Destination
 * Modifications to source file(s): convert to `csv`
-* Location: ***prereise/gather/data/winddata/data/3_2_Wind_Y2016.csv***
+* Location: ***prereise/gather/winddata/data/3_2_Wind_Y2016.csv***
 
 ##### General Purpose
 The dataset is used to generate wind profiles.

README.md

@@ -216,6 +216,54 @@ data, similarly with the western case, the final hydro profile v2 for Texas is
 constructed by decomposing the total hydro generation profile proportional to
 the generator capacities. Check out the **[texas_hydro_v2_demo.ipynb][hydro_v2_texas_notebook]** notebook for demo.
 
+#### ii. Hydro v3
+This methodology is designed to generate hourly plant level hydro profile
+for Eastern, i.e. eastern hydro v3, which has following features: 
+
+- Pumped storage hydro (HPS) and conventional hydro (HYC) are handled
+separately.
+- Historical hourly hydro profiles of four ISOs (ISONE, NYISO, PJM and SPP)
+, are used directly as regional total hydro profiles to start with.
+- For the rest of areas in Eastern Interconnect, which are not covered by the
+four ISOs, the hydro profile is generated in the same way as western hydro
+profile v2.
+
+Note that usa hydro v3 is the concatenation of eastern hydro v3, western
+hydro v2 and texas hydro v2
+
+##### a. Pumped Storage Hydro
+HPS profile for each plant in [hps_plants_eastern.xlsx][pumped storage hydro
+model], **‘all_plantIDs’** sheet, is generated based on the deterministic
+model presented in **‘profile’** sheet. The HPS profile model is designed based
+on local time. The plant level HPS profiles are generated considering the
+local time and daylight-saving schedule of the locations of the
+corresponding buses. However, during the sanity check afterwards, it turns
+out that this model overestimates the HPS output, the final profile is
+further scaled by 0.65 to be in agreement with values reported in [EIA 923][EIA 923].
+##### b. Conventional Hydro
+The buses of the HYC plants were associated to five regions (ISONE, NYISO,
+PJM, SPP and the area uncovered by the aforementioned regions) using the
+same methodology employed for eastern demand v5. Check Demand Data section
+below for details. For HYC plants locating inside the territories of the
+four ISOs, we decompose the total historical profile of each ISO into plant
+-level profiles proportional to the generator capacities. The time series
+profile data of each ISO is obtained via their official websites: [ISONE
+][ISO New England], [NYISO][NY ISO], [PJM][PJM], [SPP][Southwest Power Pool].
+
+For the rest of HYC plants lying outside the four ISO regions, we apply the
+same methodology that is used to generate hydro profile of California in
+Western hydro v2 (see details in the above subsection). The net demand
+profile is obtained from Eastern base case scenario (Scenario 397). The
+monthly net generations for a state is split proportional to the
+total capacities of the generators if the corresponding state is covered
+partially by the four ISOs mentioned above.   
+
+The final Eastern hydro profile v3 is built by stitching the three parts
+together: pumped storage hydro profiles, conventional hydro profiles within
+the four ISO regions and conventional hydro profiles in the rest areas
+. Check out **[eastern_hydro_v3_demo.ipynb][hydro_v3_eastern_notebook
+]** notebook for demo.
+
 ### D. Demand Data
 
 *Eastern V6*
@@ -388,3 +436,9 @@ usage.
 [hydro_profiles]: https://github.com/Breakthrough-Energy/PreREISE/blob/develop/docs/hydro_profiles.md
 [solar_profiles]: https://github.com/Breakthrough-Energy/PreREISE/blob/develop/docs/solar_profiles.md
 [wind_profiles]: https://github.com/Breakthrough-Energy/PreREISE/blob/develop/docs/wind_profiles.md
+[ISO New England]: https://www.iso-ne.com/isoexpress/
+[NY ISO]: http://mis.nyiso.com/public/P-63list.htm
+[PJM]: http://dataminer2.pjm.com/feed/gen_by_fuel
+[Southwest Power Pool]: https://marketplace.spp.org/pages/generation-mix-historical
+[hydro_v3_eastern_notebook]: https://github.com/Breakthrough-Energy/PreREISE/blob/develop/prereise/gather/hydrodata/eia/demo/eastern_hydro_v3_demo/eastern_hydro_v3_demo.ipynb
+[pumped storage hydro model]: https://github.com/Breakthrough-Energy/PreREISE/blob/develop/prereise/gather/hydrodata/eia/demo/eastern_hydro_v3_demo/hps_plants_eastern.xlsx

prereise/gather/demanddata/eia/demo/eastern_demand_v5_demo/bus_ba_map.ipynb

@@ -16,7 +16,7 @@
     "import json\n",
     "\n",
     "from powersimdata.input.grid import Grid\n",
-    "from prereise.gather.demanddata.eia.map_ba import map_grid_buses_to_county"
+    "from prereise.gather.demanddata.eia.map_ba import map_buses_to_county"
    ]
   },
   {
@@ -51,7 +51,24 @@
    },
    "outputs": [],
    "source": [
-    "bus_ba_map, bus_no_ba_match = map_grid_buses_to_county(grid)"
+    "bus_ba_map = grid.bus[grid.bus[\"Pd\"] > 0][[\"Pd\", \"lat\", \"lon\"]].copy()\n",
+    "bus_ba_map, bus_no_county_match = map_buses_to_county(bus_ba_map)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then check if there are buses where county is null. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "bus_no_county_match"
    ]
   },
   {
@@ -83,7 +100,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "data = json.load(open('ba_to_county.txt'))\n",
+    "data = json.load(open('../../../../data/ba_to_county.txt'))\n",
     "ba_county_list = {}\n",
     "for val in data['groups'].values():\n",
     "    ba_county_list[val['label']] = set(val['paths'])"
@@ -145,23 +162,6 @@
     "bus_no_ba_match"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Then check if there are buses where county is null. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "bus_no_county_match = list(bus_ba_map[bus_ba_map['County'].isna()].index)\n",
-    "bus_no_county_match"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},

prereise/gather/demanddata/eia/demo/eastern_demand_v5_demo/eastern_demand_v5_demo.ipynb

@@ -23,8 +23,8 @@
     "from prereise.gather.demanddata.eia.clean_data import (fix_dataframe_outliers, \n",
     "                                                       replace_with_shifted_demand)\n",
     "from prereise.gather.demanddata.eia.get_eia_data import get_ba_demand\n",
-    "from prereise.gather.demanddata.eia.map_ba import (transform_ba_to_region, \n",
-    "                                                   map_to_loadzone)"
+    "from prereise.gather.demanddata.eia.map_ba import (aggregate_ba_demand,\n",
+    "                                                   get_demand_in_loadzone)"
    ]
   },
   {
@@ -1042,7 +1042,7 @@
     }
    ],
    "source": [
-    "eastern_agg_demand = transform_ba_to_region(eastern_demand_2016, ba_area)\n",
+    "eastern_agg_demand = aggregate_ba_demand(eastern_demand_2016, ba_area)\n",
     "eastern_agg_demand"
    ]
   },
@@ -1606,7 +1606,7 @@
    },
    "outputs": [],
    "source": [
-    "eastern_demand_v5 = map_to_loadzone(eastern_fix_outliers, bus_map)"
+    "eastern_demand_v5 = get_demand_in_loadzone(eastern_fix_outliers, bus_map)"
    ]
   },
   {
@@ -2096,4 +2096,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}