Eastern hydro v3

[BainanXia]

Purpose

Clear the technical debt in 2019. Merge eastern hydro profile v3 generation module into the repo.

What the code is doing

A major refactor of the demo notebook eastern_hydro_v3_demo.ipynb is carried out. The new notebook generates a hydro profile for eastern using the same methodology as the one in 2019 with some minor modifications, however, the resultant profiles are not exactly the same:

• All the functions defined in the notebook are either defined in the module or replaced by the existing (refactored) functions in the codebase
• Previously, we used the net demand without curtailment as the shape file to generate the conventional hydro profiles in the rest areas not covered by the four ISOs. Now it is changed to use the net demand of our Eastern base case scenario 397.
• The grid has been changed and hydro plant capacities are projected to 2020, which also includes moving some hydro capacities from Eastern to Western in TX and MT according to EIA 923.
• The code is improved to be more efficient in terms of running time. Now it takes <15 min to run through the demo notebook, whereas, it took more than 40 min before.

Other changes except for the demo notebook:

• A few existing functions used in generating other profiles are refactored to be more general to be used in generating eastern hydro v3, including:
  -- map_grid_buses_to_county-> map_buses_to_county
  -- get_profile -> get_profile_by_state + get_profile_by_dataframe
  -- get_monthly_net_generation -> can be specified to include pumped hydro storage into hydro or not using optional boolean variable hps
• Corresponding notebooks are updated due to the function refactor.
• All the new data files are documented in both README and ATTRIBUTION. (@kasparm I talked with @rouille, we need the legal team to go through this once merged, mainly for the data source of historical hydro profiles of the 4 ISOs, ISONE, NYISO, PJM, SPP)
• The methodology is described in details in README. (@victoriahunt regarding the pumped storage hydro profile generation, I wrote the introduction down based on my understanding of the readme.txt in your exploration folder, feel free to modify this part in README as well as the time series profile data source of the 4 ISOs in ATTRIBUTION upon reviewing)
• This branch is rebased on the latest develop at 5:45 pm on 11/17/20, one conflict is resolved due to the additions in ATTRIBUTION.

Testing

All tests passed. I'm not adding lots of new functions but refactor a decent amount of existing ones to make them more general. Test coverage can be improved upon requests along with this PR.

Where to look

Changes are in following directories:

/prereise/ATTRIBUTION.md
/prereise/README.md
/prereise/gather/data/
/prereise/gather/hydrodata/
/prereise/gather/demanddata/

Time estimate

30 min to 1 hour. A lot of data has been added. New code is mostly within notebooks, which makes it harder to review.

[rouille]

The ATTRIBUTION.md file looks good. That said, @kasparm, @danielolsen and I will probably need to make sure that the data used can be redistributed.

[rouille]

I am not a big fan of mixing the resource parameter, which is not limited to hydro with the optional hps, which is hydro specific. On way to avoid that would be to have a resource2mover dictionary in this module equivalent to the aall_resource variable below that will be passed to this function in place of the resource and hps parameters. Then, you would then remove the HPS value of the hydro entry before calling the function if necessary.

[BainanXia]

@rouille Me neither. I thought about it when I was implementing this. However, the issue is the user does not necessary knows the mapping between resource and mover defined in EIA 923. Looking through the portion of the form which is used in the scope of our concern, hydro will be the only one with one to many mapping (others are all one to one so that we could hard coded in the module), thus, I came up with the current changes with minimum modifications ( but ugly of course...). Not sure whether there are better solutions.

[rouille]

The ATTRIBUTION.md file looks good. That said, @kasparm, @danielolsen and I will probably need to make sure that the data used can be redistributed.

Should we contact the 3 ISOs (PJM, NYISO and NEISO) to make sure we can use their data?

[rouille]

Thanks @BainanXia. Great work. It is nice to have all the methodology in the code base.