Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: various issues with HIFLD transmission loading/processing #239

Merged
merged 5 commits into from
Nov 9, 2021

Conversation

danielolsen
Copy link
Contributor

Pull Request doc

Purpose

Various fixes for processing of HIFLD transmission data:

  • Correct a lat/lon vs. lon/lat issue (and add a test for said issue) when mapping lines to substations
  • In addition to lines whose status is IN SERVICE, also return lines whose status is NOT AVAILABLE, since this represents most of the lines in the Rocky Mountain states (see Improve handling of DC lines when loading HIFLD data #233 (comment)).
  • Using a rough bounding box of the contiguous U.S., don't return lines for which at least one end is outside of this box (e.g. Alaska, Hawaii, Puerto Rico)
  • Ensure that indices aren't duplicated within the branch dataframe as a results of concatenating individual dataframes for [original lines, MST-connecting lines, transformers]

What the code is doing

  • In const.py, we add data for a rough bounding box for the contiguous U.S. states.
  • In load.py, we use this bounding box to filter lines, and we return both IN SERVICE and NOT AVAILABLE lines.
  • In transmission.py:
    • within map_lines_to_substations_using_coords, we fix a lat/lon vs. lon/lat bug.
    • within filter_islands_and_connect_with_mst, we add a meaningful index to the new lines.
    • within build_transmission, we add a meaningful index to the new transformers.
  • In test_transmission.py, we add a test (test_map_lines_to_substations_using_coords) that catches the lat/lon vs. lon/lat issue as well as tests the combining of substations via coordinate rounding.

Testing

The new unit test initially fails due to the lat/lon issue, but passes once the fix is added.

Usage Example/Visuals

Before:

>>> from prereise.gather.griddata.hifld.data_process.transmission import build_transmission
>>> branch, bus, substations, dc_lines = build_transmission()
>>> len(branch)
74859
>>> branch.index.duplicated().sum()
522

After:

>>> from prereise.gather.griddata.hifld.data_process.transmission import build_transmission
>>> branch, bus, substations, dc_lines = build_transmission()
>>> len(branch)
93908
>>> branch.index.duplicated().sum()
0

After the changes, we have many more branches, and no duplicate indices.

Time estimate

30 minutes.

@danielolsen danielolsen added bug Something isn't working hifld Related to ingestion of the HIFLD data labels Nov 9, 2021
@danielolsen danielolsen requested a review from BainanXia November 9, 2021 02:33
@danielolsen danielolsen self-assigned this Nov 9, 2021
Copy link
Collaborator

@BainanXia BainanXia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@danielolsen danielolsen force-pushed the daniel/hifld_transmission_fixes branch from 818f7e8 to d205919 Compare November 9, 2021 19:12
@danielolsen danielolsen merged commit ca14744 into hifld Nov 9, 2021
@danielolsen danielolsen deleted the daniel/hifld_transmission_fixes branch November 9, 2021 22:08
danielolsen added a commit that referenced this pull request Dec 8, 2021
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Jan 5, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Jan 8, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Jan 31, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Feb 25, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Mar 15, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Apr 1, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
danielolsen added a commit that referenced this pull request Apr 5, 2022
…ssion_fixes

fix: various issues with HIFLD transmission loading/processing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working hifld Related to ingestion of the HIFLD data
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants