feat: extract pg pf from Switch results

[BainanXia]

Pull Request doc

Purpose

Extract time series results, power generation (PG) for each plant (original and expanded), power flow (PF) for each branches (ac and dc), for each investment year, from Switch results. Partially addresses #50

What the code is doing

• switch_to_profiles.py is created with a new class ExtractTimeseries. Creating a class instead of a function due to concerns:
  • Results of raw csvs reading/parsing and further processing can be reused.
  • Better fits later pipeline built to mimic scenario object life cycle.
• Three internal utility functions are added to the new class ExtractTimeseries:
  • _timestamp_to_investment_year: map timestamps to investment year via timepoints
  • _get_parsed_data: feed raw results from pickle file and parameters to function parse_timepoints
  • _calculate_net_pf: calculate net power flow between bus tuples by combining mirror flow of pairs (A, B), (B, A).
• Three user facing functions are added to the new class ExtractTimeseries:
  • get_pg: get time series power generation for both original and expanded plants in every investment year.
  • get_pf: get time series power flow for all ac branches in every investment year.
    • We split power flow among parallel lines based on reactance (x) instead of impedance assuming resistance r << x.
    • We don't sort bus tuples as we did in refactor: distribute branch upgrades among parallel paths #106 assuming there are no parallel lines with reverse directions, i.e. no two lines having (from_bus_A, to_bus_B) and (from_bus_B, to_bus_A) at the same time in the grid we are dealing with. Discussions see this thread. We might add a filter to avoid this happen when the user is adding a new branch, which we never did so far.
  • get_dcline_pf: get time series power flow for all dc branches in every investment year.
    • We split power flow among parallel lines based on capacity (Pmax), although we don't have any parallel dc lines in the system so far.
    • We assume there is no parallel ac and dc lines between any (from_bus_id, to_bus_id) tuple.

Testing

Tested manually.

Where to look

switchwrapper/switch_to_profiles.py

Usage Example/Visuals

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_to_profiles import ExtractTimeseries

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = ExtractTimeseries("results.pickle", "mapping.csv", "./inputs/timepoints_input_v2.csv", grid)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()

pg[2030]

pf[2030]

dcline_pf[2030]

Time estimate

30-40 min

[danielolsen]

Should this have output_processing as the merge target?

[danielolsen]

I like the idea of having a Scenario-like object from which the user can access all inputs and outputs.

If ExtractTimeseries is extracting both the timeseries results and the resulting grid(s), should the name be changed to something more general? Will this be the same user-facing object that retrieves profiles once #107 is in?

[BainanXia]

I like the idea of having a Scenario-like object from which the user can access all inputs and outputs.

If ExtractTimeseries is extracting both the timeseries results and the resulting grid(s), should the name be changed to something more general? Will this be the same user-facing object that retrieves profiles once #107 is in?

I like this idea. What do you think @rouille ? If that's the way to go, shall we do it in this PR or in a separate PR, after we implement the extraction of timeseries of duals (congu, congl, lmp), storage_pg and storage_e that completes #50. Then the follow-up separate PR solely serves for reorganizing grids/input profiles/output timeseries in a user facing object.

[danielolsen]

I guess this idea relates to making the MockScenario as described in #51. Once the output structure of this PR is set (i.e. the name of the new class and the methods that it has), I think we could get started on a feature for #51, and then in parallel we could work on the other aspects of #50, which should be able to get bolted onto the feature for #51 fairly easily.

I think the biggest question in my mind is whether the ExtractTimeseries class (or whatever it is renamed to) will provide all the backend logic for #51, or only part of it.

[BainanXia]

As mentioned in the discussion of #107 , we should be consistent on function/variable names when referring to timestamps_to_timepoints . This is addressed here in 83ec70c

[danielolsen]

The code looks good. We can change some of the higher-level design considerations (the name of the new class, etc.) as necessary when we integrate this into MockScenario creation to address #51.

[BainanXia]

Per @danielolsen 's suggestion, we now make ExtractTimeseries class an universal user facing object that provides all inputs and outputs. The new demo code is:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_to_profiles import ExtractTimeseries

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = ExtractTimeseries(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Now, the last thing we need to do is find a proper name for ExtractTimeseries class. Open to any suggestions @danielolsen @rouille

[rouille]

Per @danielolsen 's suggestion, we now make ExtractTimeseries class an universal user facing object that provides all inputs and outputs. The new demo code is:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_to_profiles import ExtractTimeseries

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = ExtractTimeseries(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Now, the last thing we need to do is find a proper name for ExtractTimeseries class. Open to any suggestions @danielolsen @rouille

Besides its name, I think this class should be moved either to a different module or its own module. The switch_to_profiles module is not appropriate anymore.

[danielolsen]

ExtractOutputs?

[BainanXia]

ExtractOutputs?

There are inputs as well, right?

[danielolsen]

ExtractOutputs?

There are inputs as well, right?

True. SwitchExtract? We extract Switch-formatted things.

[BainanXia]

ExtractOutputs?

There are inputs as well, right?

True. SwitchExtract? We extract Switch-formatted things.

Sounds reasonable. Any other ideas @rouille ?

[rouille]

ExtractOutputs?

There are inputs as well, right?

True. SwitchExtract? We extract Switch-formatted things.

Sounds reasonable. Any other ideas @rouille ?

DataManager, SwitchData, RunData, ExpansionData, CEMData, ... Also, as mentioned above, I would move this class in a new module.

[BainanXia]

ExtractOutputs?

There are inputs as well, right?

True. SwitchExtract? We extract Switch-formatted things.

Sounds reasonable. Any other ideas @rouille ?

DataManager, SwitchData, RunData, ExpansionData, CEMData, ... Also, as mentioned above, I would move this class in a new module.

Currently this class is the only thing in this module, we could simply rename this module then. Btw, This is not SwitchData/RunData, etc, right? We extract CEM outputs generated by Switch and transform it into a PCM compatible format.

[BainanXia]

Per @danielolsen 's suggestion, we now make ExtractTimeseries class an universal user facing object that provides all inputs and outputs. The new demo code is:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_to_profiles import ExtractTimeseries

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = ExtractTimeseries(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Now, the last thing we need to do is find a proper name for ExtractTimeseries class. Open to any suggestions @danielolsen @rouille

Besides its name, I think this class should be moved either to a different module or its own module. The switch_to_profiles module is not appropriate anymore.

The module and the class are renamed into switch_extract.py and SwitchExtract respectively. If there is no objections, I will merge this now @danielolsen @rouille

[BainanXia]

New usage demo after refactor:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_extract import SwitchExtract

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = SwitchExtract(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

[rouille]

New usage demo after refactor:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_extract import SwitchExtract

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = SwitchExtract(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Do we need all these arguments? Are not the name of the files standard? In other words, would it be possible to just provide a path to a directory and load the files in the constructor of the class?

[danielolsen]

Looks good, works on my end as well. This is more than just partially addressing #50, it also gets us most of the way toward #51 and #65.

[danielolsen]

New usage demo after refactor:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_extract import SwitchExtract

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = SwitchExtract(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Do we need all these arguments? Are not the name of the files standard? In other words, would it be possible to just provide a path to a directory and load the files in the constructor of the class?

Not all of these files have a standardized location that they are saved to within the project folder that SwitchWrapper creates on input (see #51 (comment)). Once we do save them, then we could create a static method of the SwitchExtract class, e.g. SwitchExtract.from_folder() which would automatically pass all these paths to the main SwitchExtract init. I think there is enough in this PR though, and we could save that simplified init for a follow-up.

[BainanXia]

New usage demo after refactor:

import sys
from powersimdata import Scenario
sys.path.extend(['/Users/bainanxia/OneDrive - Gates Ventures/Documents/GitHub/SwitchWrapper'])
from switchwrapper.switch_extract import SwitchExtract

grid = Scenario(599).get_grid() # the grid used as input to this Switch run
et = SwitchExtract(
  "results.pickle", 
  "slicing_recovery.csv", 
  "./inputs/timepoints_input_v2.csv", 
  "./inputs/loads.csv", 
  "./inputs/variable_capacity_factors.csv", 
  grid
)
pg = et.get_pg()
pf = et.get_pf()
dcline_pf = et.get_dcline_pf()
demand = et.get_demand()
hydro = et.get_hydro()
wind = et.get_wind()
solar = et.get_solar()

Do we need all these arguments? Are not the name of the files standard? In other words, would it be possible to just provide a path to a directory and load the files in the constructor of the class?

Not all of these files have a standardized location that they are saved to within the project folder that SwitchWrapper creates on input (see #51 (comment)). Once we do save them, then we could create a static method of the SwitchExtract class, e.g. SwitchExtract.from_folder() which would automatically pass all these paths to the main SwitchExtract init. I think there is enough in this PR though, and we could save that simplified init for a follow-up.

I'm thinking about the same thing. We don't have a file directory structure for SwitchWrapper as we did for the PCM yet. Maybe it won't hurt to refactor this after we have one.