feat: add top-level HIFLD grid orchestration function

[danielolsen]

Pull Request doc

Purpose

Add a single function that kicks off all processing of raw HIFLD data to outputs compatible with PowerSimData. Closes #226.

This is a draft PR for now at least until #235 is merged and #233 is addressed, intended to start conversation on feature design and identify lower-level issues to be addressed in separate PRs. Outstanding issues that are already on my radar to address as part of this PR (design/implementation suggestions welcome!):

• Columns irrelevant to PowerSimData need to be dropped. EDIT: done
• The bus.csv file needs an entry added in const.py for its columns with default values. EDIT: done.
• We need to add missing column information to some data tables: EDIT: done.
  • branch.csv: r, b, ratio, and from_bus_id, to_bus_id for non-transformer branches
  • bus.csv: bus_id (I think we just need to rename the existing index), type, zone_id
  • bus2sub.csv: interconnect
  • dcline.csv: from_bus_id, to_bus_id
  • sub.csv: name (probably just a rename), interconnect_sub_id, lat, lon (renames)
• generators need types, fuel prices, and heat-rate information. EDIT: done.

Ideally, the PowerSimData check_grid function (powersimdata.input.check_grid) will help us identify other similar issues, once Breakthrough-Energy/PowerSimData#551 is complete.

What the code is doing

There's new user-facing function create_csvs within the init.py file. It:

• activates the highest-level functions in each of the relevant data_process sub-modules,
• builds a dictionary of output tables (splitting tables as necessary),
• fills tables with columns of default values as appropriate,
• filters output tables to only the columns expected by PowerSimData, and
• saves all files to the specified output directory.

const.py gets the column default values.

Within transmission.py:

• changes the build_transmission function to use the "line2sub" method by default, (EDIT: done via feat: filter DC lines out of AC transmission network processing within HIFLD construction #237)
• adds the transformer-creation (from feat: add function to create transformers within substations #218), branch-parameter-assignment steps (from feat: add functions to estimate branch parameters (impedance and rating) #219), (EDIT: done via feat: filter DC lines out of AC transmission network processing within HIFLD construction #237)
• adds a function assign_buses_to_lines which translated line-to-substation mapping to line-to-bus, and applies it within build_transmission.
• adds a function add_substation_info_to_buses which adds interconnect and zone IDs to buses, and applies it within build_transmission.
• adds to & from interconnect information to DC lines
• renames columns to their PowerSimData naming conventions, and adds a dummy interconnect_sub_id column to the substations table (I don't remember if we use this for anything, but PowerSimData expects it).

Within generators.py: changing the expected column names for latitude and longitude, to match the renaming in transmission.py.

Testing

Tested manually. Files are created in the right location, although there are a few issues with the outputs, which I think should be addressed in separate PRs to fix the lower-level functions:

• Within branch.csv
  • Currently, lines (either present in the original HIFLD data or added to connect the minimum spanning tree) don't get from_bus_id or to_bus_id assigned in the branch table, only transformers do. Lines do have SUB_1_ID, SUB_2_ID, and VOLTAGE, which should be enough to uniquely identify a bus, since buses are created via the set of unique voltages of lines connected to a given substation. EDIT: done.
  • Dataframe indices aren't unique after the individual frames for the original lines, the minimum-spanning-tree lines, and the transformers are appended together. EDIT: fixed via fix: various issues with HIFLD transmission loading/processing #239.

Tested via the test branch daniel/hifld_top_level_rebased, which combines the code from #236 and #240. Validation uses draft code from the branch for Breakthrough-Energy/PowerSimData#566.

# Building the hifld grid
# This step currently takes 20-30 minutes to download/process all data
from prereise.gather.griddata.hifld import create_csvs
create_csvs("path/to/powersimdata/network/hifld/data")

# Loading and checking the hifld grid
from powersimdata import Grid
from powersimdata.input.check import check_grid
check_grid(Grid("USA", "hifld"))
check_grid(Grid("Eastern", "hifld"))
check_grid(Grid("Western", "hifld"))
check_grid(Grid("ERCOT", "hifld"))
check_grid(Grid(["Eastern", "Western"], "hifld"))
check_grid(Grid(["Eastern", "ERCOT"], "hifld"))
check_grid(Grid(["ERCOT", "Western"], "hifld"))

The whole script takes a while to run (about 20 minutes on my laptop), even with a cached minimum spanning tree, since we don't cache the EPA AMPD data that's downloaded or the heat rate curves that are fitted to these data, along other things. We could improve at least part of this with some KDTree refactors.

Time estimate

30-60 minutes for what's in here currently. Longer if we want to add some caching to speed up performance and/or design pass-throughs to the keyword arguments of build_transmission and build_plant.

[danielolsen]

This code has been tested to successfully make the CSVs necessary to create a grid, once it's rebased onto the code from #240, and has been validated using code from Breakthrough-Energy/PowerSimData#566. The original post has also been updated with what this branch currently does. Therefore, I'm converting this PR out of draft stage.

[danielolsen]

After tweaking the output plant CSV a bit more (adding type, adding heat rate columns, translating to PowerSimData naming expectations), we can also successfully call powersimdata.input.export_data.export_case_mat on the resulting grids.

[kasparm]

Encountered this warning from scipy:

/Users/kmueller/.local/share/virtualenvs/PreREISE-X1G9x7gr/lib/python3.8/site-packages/scipy/stats/_stats_mstats_common.py:170: RuntimeWarning: invalid value encountered in double_scalars
  slope = ssxym / ssxm
/Users/kmueller/.local/share/virtualenvs/PreREISE-X1G9x7gr/lib/python3.8/site-packages/scipy/stats/_stats_mstats_common.py:184: RuntimeWarning: invalid value encountered in sqrt
  t = r * np.sqrt(df / ((1.0 - r + TINY)*(1.0 + r + TINY)))
/Users/kmueller/.local/share/virtualenvs/PreREISE-X1G9x7gr/lib/python3.8/site-packages/scipy/stats/_stats_mstats_common.py:187: RuntimeWarning: invalid value encountered in double_scalars
  slope_stderr = np.sqrt((1 - r**2) * ssym / ssxm / df)

I assume it is not critical.

[kasparm]

First check check_grid(Grid("USA", "hifld")) passes. Second check fails:

>>> check_grid(Grid("Eastern", "hifld"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kmueller/.local/share/virtualenvs/PreREISE-X1G9x7gr/lib/python3.8/site-packages/powersimdata/input/check.py", line 50, in check_grid
    raise ValueError(f"Problem(s) found with grid:\n{collected}")
ValueError: Problem(s) found with grid:
This grid contains 920 connected components, but is specified as having 1 interconnects: ['Eastern'].

Debugging:

gg = nx.from_pandas_edgelist(g_e.branch, "from_bus_id", "to_bus_id")
>>> num_connected_components = len([c for c in nx.connected_components(gg)])
>>> num_connected_components
920

[danielolsen]

@kasparm the first issue is from the generator heat rate curve-fitting, it doesn't seem to be a problem. The second issue was from a commit that I missed when cherry-picking from my testing branch, fixed with the latest push.

[kasparm]

For some reason I'm still getting the same error.

[danielolsen]

For some reason I'm still getting the same error.

Even after running the latest PreREISE code and putting the resulting CSVs into the right folder in PowerSimData? The source of the error is the interconnect column not being filled, so that PowerSimData filters out all NA values when filtering to less than the full USA, which leads to many different islands once the interconnect-less transformers are removed. The most recent commit should solve this problem.

[kasparm]

Looks good now. My installation was off.

[kasparm]

Thank you @danielolsen looks good. The only change I would suggest is to place the create_csv function into it's on file in hifld and then import it in the init.py

[kasparm]

Looks good. Thanks for making the change.