Specify demand flexibility in the scenario framework

[lanesmith]

Pull Request doc

Purpose

The goal of this PR is to allow users to specify demand flexibility profiles and parameters in PowerSimData. This PR closes Issue #511.

What the code is doing

Demand flexibility profiles and parameters are specified using the change table (via the add_demand_flexibility function). Demand flexibility profiles are accessed using the same core functionality as the other profile data. Accessing the demand flexibility profiles does differ slightly in that the profiles are not scaled (as you might see with demand, solar, wind, and hydro) and different area granularity can potentially be included (some combination of zone IDs and bus IDs, consistent with the allowed inputs in REISE.jl). This functionality is located in the _get_demand_flexibility_profile function. The final main addition is that the demand flexibility parameters file is created (via the prepare_demand_flexibility_parameters function) using the parameters input by the user. Other changes are less significant, such as including demand flexibility as a recognizable profile type and differentiating between demand profiles and demand flexibility profiles.

Testing

Tests were completed locally using the Native set-up specified here. As has been discussed, there is an error when trying to run the specified scenario in REISE.jl from PowerSimData. Since the scenario creation works as expected (i.e., all input files are created in a single directory) and those files can be used in the standalone version of REISE.jl, I believe this problem lies not with the feature presented in this PR, but with the Native set-up (it seems to be an environment issue). This will be explored in a separate PR.

I have also started to include pytests (located in test_change_table.py), but I ran into an issue with what I was able to implement. Many of the demand-flexibility-related functions rely on checking or including the names of the different versions of the different demand flexibility profiles (e.g., version EFS_x_y_z of the demand flexibility up profile). Since none of these profiles are included in blob storage, the only location they are looked for is locally; unless these profiles are included locally, the functions will fail (e.g., as would be the case when pushing to GitHub). I was trying to implement something like what is seen in mock_input_data.py and test_transform_profile.py, but I couldn't figure out how to overcome the functions looking for the existence of the profiles. Any suggestions are welcome.

Where to look

Most of the main changes are located in:

• powersimdata/input/change_table.py
• powersimdata/input/transform_profile.py
• powersimdata/scenario/execute.py

Smaller ancillary changes were made to:

• powersimdata/data_access/profile_helper.py
• powersimdata/data_access/scenario_list.py
• powersimdata/input/input_data.py

powersimdata/output/output_data.py and powersimdata/scenario/analyze.py include some changes that allow the demand-flexibility-related data to be accessed following the simulation. The test is located in powersimdata/input/tests/test_change_table.py.

Time estimate

This will probably take a bit longer since it is a fairly big new feature.

[danielolsen]

I'm able to use this to create and prepare a scenario...

from powersimdata import Scenario
scenario.set_grid("usa_tamu", "Texas")
scenario.set_time("2016-01-01 00:00", "2016-01-01 23:00", "24H")
scenario.set_base_profile("demand", "vJan2021")
scenario.set_base_profile("solar", "vJan2021")
scenario.set_base_profile("wind", "vJan2021")
scenario.set_base_profile("hydro", "vJan2021")
scenario.set_name("foo", "bar2")
scenario.change_table.add_demand_flexibility(
    {
        "demand_flexibility_up": "EFS_Medium_Moderate_Enhanced_2030_Transportation",
        "demand_flexibility_dn": "EFS_Medium_Moderate_Enhanced_2030_Transportation",
    }
)
scenario.create_scenario()
scenario.prepare_simulation_input()

...but then when I try to run this Scenario manually using REISE.jl, I end up getting an error:

julia> import REISE

julia> import Gurobi

julia> REISE.run_scenario_gurobi(; interval=24, n_interval=1, start_index=1, inputfolder=".")
Compute Server job ID: 5266459b-2415-4adb-93a1-37ef15f46091
Capacity available on 'https://ip-10-0-62-149:61000' - connecting...
Established HTTPS encrypted connection
Reading from folder: .
...loading case.mat
...loading demand.csv
...loading hydro.csv
...loading wind.csv
...loading solar.csv
File case_storage.mat not found in .
...loading demand flexibility parameters
The demand flexibility duration parameter is not defined. Will default to being the size of the interval.
The parameter that indicates if the rolling load balance constraint is enabled is not defined. Will default to being enabled.
...loading demand flexibility up profiles
Demand flexibility up-shift cost profiles not found in .. Will default to no cost for up-shifting demand.
...loading demand flexibility dn profiles
Demand flexibility dn-shift cost profiles not found in .. Will default to no cost for dn-shifting demand.

Compute Server communication statistics:
  Sent: 0.001 MBytes in 4 msgs and 0.23s (0.00 MB/s)
  Received: 0.000 MBytes in 1 msgs and 0.16s (0.00 MB/s)

Connection to Gurobi closed successfully!
ERROR: MethodError: Cannot `convert` an object of type InlineStrings.String7 to an object of type Bool
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Ptr) where T<:Integer at pointer.jl:23
  ...
Stacktrace:
 [1] REISE.DemandFlexibility(::DataFrames.DataFrame, ::DataFrames.DataFrame, ::Nothing, ::Nothing, ::Int64, ::Bool, ::InlineStrings.String7, ::Bool) at C:\Users\DanielOlsen\repos\bes\REISE.jl\src\types.jl:45
 [2] REISE.DemandFlexibility(; flex_amt_up::DataFrames.DataFrame, flex_amt_dn::DataFrames.DataFrame, cost_dn::Nothing, cost_up::Nothing, duration::Int64, enabled::Bool, interval_balance::InlineStrings.String7, rolling_balance::Bool) at .\util.jl:438
 [3] read_demand_flexibility(::String, ::Int64) at C:\Users\DanielOlsen\repos\bes\REISE.jl\src\read.jl:255
 [4] run_scenario(; num_segments::Int64, interval::Int64, n_interval::Int64, start_index::Int64, inputfolder::String, outputfolder::Nothing, threads::Nothing, optimizer_factory::Gurobi.Env, solver_kwargs::Dict{String,Int64}, model_kwargs::Nothing) at C:\Users\DanielOlsen\repos\bes\REISE.jl\src\REISE.jl:69
 [5] run_scenario_gurobi(; solver_kwargs::Nothing, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:interval, :n_interval, :start_index, :inputfolder),Tuple{Int64,Int64,Int64,String}}}) at C:\Users\DanielOlsen\repos\bes\REISE.jl\src\solver_specific\gurobi.jl:13
 [6] top-level scope at REPL[4]:1

Looking a little closer at what's going on under the hood, it seems that demand_flexibility["interval_balance"] is "True" (a string) rather than true (a bool). Is this a bug, or is there something that I'm doing wrong here?

[lanesmith]

Looking a little closer at what's going on under the hood, it seems that demand_flexibility["interval_balance"] is "True" (a string) rather than true (a bool). Is this a bug, or is there something that I'm doing wrong here?

Based on what you describe, I believe this is an issue with the CSV package in Julia. I was encountering a similar issue where "True" was being read as a string instead of as a bool. Based on the available options within CSV.File, this shouldn't be an issue as most reasonable variations of "true" (e.g., True, true, TRUE) are by default supposed to be parsed as bools. In my case, the problem only seemed to occur when a parameter that had "True" or "False" was listed in the last column of the demand flexibility parameters file. The interesting thing is that it recognizes that other "True" values are supposed to be bools, just not values in the last column. I thought I found a workaround by moving the duration parameter to the last column, which was working for my local tests. I guess not having the duration parameter loses that safeguard though.

There doesn't seem to be a better option for loading .csv files into DataFrames within Julia (at least that I'm aware of). I could make a fix in REISE.jl where the value for the boolean demand flexibility parameters are checked and switched to bools if provided as a reasonable string (i.e., any variation of "True" or "False"). I'm also open to other solutions.

[lanesmith]

Another potential solution could be changing how I create the demand flexibility parameters file in execute.py. I could make it so that all the parameters are always created (e.g., rolling_balance and duration are also created even when demand_flexibility_duration isn't provided in the change table), they just would be set to appropriate values (e.g., rolling_balance would be False and duration would be a throwaway value such as 0 (the value of duration would not matter if rolling_balance isn't enabled)). This would probably be the quickest fix and would prevent needing to prop up the CSV package.

[danielolsen]

CSV had been at 0.9.10, but upgrading to 0.10.3 fixed it, so I guess the problem isn't with this PR, but we should probably specify the CSV version within REISE.jl.

For reference, the contents of my demand_flexibility_parameters.csv file were:

,enabled,interval_balance
0,True,True

[danielolsen]

I haven't looked too deeply at the implementation, but I can confirm that it works and makes files that REISE.jl likes. Thanks!

[lanesmith]

CSV had been at 0.9.10, but upgrading to 0.10.3 fixed it, so I guess the problem isn't with this PR, but we should probably specify the CSV version within REISE.jl.

Okay, glad to hear that a later version of CSV fixes this. Unless there is any opposition, I think I'll still make the change to execute.py that I suggested above. This will make it so the demand flexibility parameters are more consistent across scenarios (i.e., all parameters will always be present, even if they're not enabled)

[danielolsen]

@lanesmith I trust you.