Data Intake: add HIFLD open and related external data

[terrywqf]

Source-1

• Folder structure an ATTRIBUTION file #191
• Name: Electric Power Transmission Lines
• Author: HIFLD Open
• Description: This dataset represents electric power transmission lines. Lines are operated at relatively high voltages varying from 69 kV up to 765 kV, and are capable of transmitting large quantities of electricity over long distances.
• Source: https://hifld-geoplatform.opendata.arcgis.com/
• Exact source location: https://hifld-geoplatform.opendata.arcgis.com/datasets/electric-power-transmission-lines/explore?showTable=true
• Download Date: 11/11/2020
• Terms (if specified): https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::electric-power-transmission-lines/about (Custom License: None (Public Use). Users are advised to read the data set's metadata thoroughly to understand appropriate use and data limitations.)

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/Electric_Power_Transmission_Lines.csv
  /prereise/gather/hiflddata/data/Electric_Power_Transmission_Lines.geojson.zip

General Purpose

The data are used to populate the transmission topology of U.S. power grid that can be used to run OPF in REISE

Source-2

• Folder structure an ATTRIBUTION file #191
• Name: Electric Substations
• Author: HIFLD Open
• Description: This dataset represents electric power substations primarily associated with electric power transmission. In this layer, substations are considered facilities and equipment that switch, transform, or regulate electric power at voltages equal to, or greater than, 69 kilovolts
• Source: https://hifld-geoplatform.opendata.arcgis.com/
• Exact source location: https://hifld-geoplatform.opendata.arcgis.com/datasets/electric-substations/explore?showTable=true
• Download Date: 11/11/2020
• Terms (if specified): https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::electric-substations/about (Custom License: None (Public Use). Users are advised to read the data set's metadata thoroughly to understand appropriate use and data limitations.)

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/Electric_Substations.csv

General Purpose

The data are used to populate the substation topology of U.S. power grid that can be used to run OPF in REISE

Source-3

• Folder structure an ATTRIBUTION file #191
• Name: Power Plants
• Author: HIFLD Open
• Description: This dataset represents electric power plants. Included are the following plant types: hydroelectric dams, fossil fuel (coal, natural gas, or oil), nuclear, solar, wind, geothermal, and biomass.
• Source: https://hifld-geoplatform.opendata.arcgis.com/
• Exact source location: https://hifld-geoplatform.opendata.arcgis.com/datasets/power-plants/explore?showTable=true
• Download Date: 11/11/2020
• Terms (if specified): https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::power-plants/about (Custom License: None (Public Use). Users are advised to read the data set's metadata thoroughly to understand appropriate use and data limitations.)

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/Power_Plants.csv

General Purpose

The data are used to populate the power generation profile of U.S. transmission network that can be used to run OPF in REISE

Source-4

• feat: process plant data to create data frame #197
• Name: General Units
• Author: HIFLD Open
• Description: This dataset represents gererator units in electric power plants, Including boiler or reactor vessel equipment, engines and engine-driven generators, turbo generator units, accessory electric equipment, and miscellaneous power plant equipment are grouped together for each individual facility.
• Source: https://hifld-geoplatform.opendata.arcgis.com/
• Exact source location: https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::generating-units-1/explore
• Download Date: 11/11/2020
• Terms (if specified): https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::generating-units-1/about (Custom License: None (Public Use). Users are advised to read the data set's metadata thoroughly to understand appropriate use and data limitations.)

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/General_Units.csv

General Purpose

The data are used to populate the power generation profile of U.S. transmission network that can be used to run OPF in REISE

Source-5

• feat: add functions to load EIA and EPA data from blob storage #214
• Name: EIA860
• Author: U.S. Energy Information Administration
• Description: This dataset collects generator-level specific information about existing and planned generators and associated environmental equipment at electric power plants with 1 megawatt or greater of combined nameplate capacity. It contains additional details of generator using wind, solar, mutiple fuel, and generators for the energy storage technology.
• Source: https://www.eia.gov/electricity/data/eia860/
• Exact source location: https://www.eia.gov/electricity/data/eia860/archive/xls/eia8602019.zip
• Terms (if specified): https://www.eia.gov/electricity/data/eia860/

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/Generator_Y2019.csv

General Purpose

The data are used to populate the power generation profile of U.S. transmission network that can be used to run OPF in REISE

Source-6

• feat: add functions to load EIA and EPA data from blob storage #214
• Name: National Electric Energy Data System
• Author: United States Environmental Protection Agency
• Description: The dataset contains the generation unit records used to construct the "model" plants that represent existing and planned/committed units in EPA modeling applications of IPM. NEEDS includes basic geographic, operating, air emissions, and other data on these generating units.
• Source: https://www.epa.gov/airmarkets/national-electric-energy-data-system-needs-v6
• Exact source location: https://www.epa.gov/sites/production/files/2021-07/needs_v620_06-30-21.xlsx
• Terms (if specified): https://www.epa.gov/airmarkets/national-electric-energy-data-system-needs-v6

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/needs.csv

General Purpose

The data are used to populate the power generation profile of U.S. transmission network that can be used to run OPF in REISE

Source-7

• feat: add function to assign demand to buses proportional to population #235
• Name: United States Counties Database, Basic
• Author: SimpleMaps.com, Pareto Software, LLC.
• Description: This csv provides the population on U.S. county level - covering 3,233 counties from all 50 states, DC, and the US territories. The dataset is using authoritative sources such as the U.S. Census Bureau and the Bureau of Labor Statistics.
• Source: https://simplemaps.com/data/us-counties
• Exact source location: https://simplemaps.com/data/us-counties
• Terms (if specified): Basic is free - https://simplemaps.com/data/us-counties

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/uscounties.csv

General Purpose

The data are used to compute the load (Pd) based on county and zip population size in U.S.

Source-8

• feat: add function to assign demand to buses proportional to population #235
• Name: US Zip Codes Database, Basic
• Author: SimpleMaps.com, Pareto Software, LLC.
• Description: This csv provides the population on U.S. zip code level - 41,689 unique zip codes including ZCTA, unique, military, and PO box zips. The dataset is using authoritative sources including the U.S. Postal Service™, U.S. Census Bureau, National Weather Service, American Community Survey, and the IRS
• Source: https://simplemaps.com/data/us-zips
• Exact source location: https://simplemaps.com/data/us-zips
• Terms (if specified): Basic is free - https://simplemaps.com/data/us-zips

Destination

• Modifications to source files(s): None
• Location:
  /prereise/gather/hiflddata/data/uszips.csv

General Purpose

The data are used to compute the load (Pd) based on county and zip population size in U.S.

[danielolsen]

For substations and power plants, I see the latest version of the data being from July 2020. For transmission lines, I see a version that was updated two weeks ago. Which version are we going to be working with?

[terrywqf]

For substations and power plants, I see the latest version of the data being from July 2020. For transmission lines, I see a version that was updated two weeks ago. Which version are we going to be working with?

We started this proj and downloaded the datasets from HIFLD at the end of 10/2020 so the substations and power plants data should be their latest in their HIFLD links. We downloaded the tranmission line datasets at the same time when starting the proj. The row counts of the tranmission data remains 89744 - I do not find something like release note or version history of the HIFLD tranmission data in their website. We can add notes when these HIFLD data are downloaded.

[terrywqf]

Note that we will also add more sections for following external data (mostly from EIA):
General_Units.csv, Generator_Y2019.csv, needs.csv
Will ask inputs from Lei and Mingze for this.

I will also add the sections for uscounties.csv and uszips.csv, which are population data sources.

[danielolsen]

All eight data sources in the original post have been added to the attribution file in the hifld branch, so this issue can be closed.