data: update ramp_30 to match what is sent to the solver

[danielolsen]

Purpose

Update the values for ramp_30 in the data files to match what is set within REISE and sent to the solver. This will make it easier to keep our Zenodo repository up to date, and will make the data more useful for external researchers.

This should have no impact on how REISE runs, since these values are replaced anyway.

What is the code doing

There are no changes to code. The code that was used to make these changes is:

import numpy as np
import pandas as pd
plant = pd.read_csv('plant.csv', index_col=0, dtype=str)
ramp30_points = {
    'coal': {'xs': (200, 1400), 'ys': (0.4, 0.15)},
    'dfo': {'xs': (200, 1200), 'ys': (0.5, 0.2)},
    'ng': {'xs': (200, 600), 'ys': (0.5, 0.2)},
    }
for fuel, points in ramp30_points.items():
    fuel_idx = np.where(plant.type == fuel)[0]
    slope = (
        (points['ys'][1] - points['ys'][0])
        / (points['xs'][1] - points['xs'][0]))
    intercept = points['ys'][0] - slope * points['xs'][0]
    for idx in fuel_idx:
        pmax = float(plant.Pmax.iloc[idx])
        norm_ramp = pmax * slope + intercept
        if pmax < points['xs'][0]:
            norm_ramp = points['ys'][0]
        if pmax > points['xs'][1]:
            norm_ramp = points['ys'][1]
        plant.ramp_30.iloc[idx] = '{:.2f}'.format(norm_ramp * pmax)
plant.to_csv('plant.csv')

This code mirrors what is done is REISE and REISE.jl (in fact, it was copied and only slightly modified from the pyREISE implementation I wrote).

Time to review

15 minutes or less. You can check out the old version and the new, and compare element-wise to see that the only changes are in the ramp_30 column.

[rouille]

I can see that the only column that is modified is ramp_30 for thermal generators. I will let @BainanXia review the methodology.

[BainanXia]

If I'm understanding correctly, previously we are making these changes in the Matlab script in REISE, i.e. scenario_matlab_script.m and this PR is to modify the input file in PowerSimData, i.e. plant.csv. A quick question, should we remove/comment out those code in REISE so that we don't replace values with the same ones when running scenarios? Also, should we document this somewhere in ReadMe.

[danielolsen]

@BainanXia your interpretation is correct. We can comment these lines out or not, it doesn't matter either way. I see keeping them in the code as documenting the logic behind the values that are run, but I like your suggestion of documenting this logic in the README as well.

[BainanXia]

I can see that the only column that is modified is ramp_30 for thermal generators. I will let @BainanXia review the methodology.

I'll compare the updated plant.csv with grid.plant from scenario 401 (the most up-to-date US basecase).

[BainanXia]

The outputs of following command:

s = Scenario('401')
grid = s.state.get_grid()
grid.plant[grid.plant.type.isin({'coal','dfo','ng'})]['ramp_30']

plant_id
0        88 
1        103
2        103
3        103
4        103
11       1  
19       65 
20       65 
71       8  
72       8  
98       0  
99       0  
100      0  
101      86 
102      86 
103      86 
113      25 
114      25 
115      25 
118      6  
131      0  
140      0  
141      0  
142      0  
143      0  
157      0  
165      0  
199      89 
200      89 
201      89 
         .. 
13834    0  
13835    0  
13836    0  
13837    0  
13838    0  
13839    0  
13840    0  
13841    0  
13842    0  
13843    0  
13844    0  
13845    0  
13846    0  
13847    0  
13848    0  
13849    0  
13850    0  
13851    0  
13852    0  
13853    0  
13854    0  
13855    0  
13856    0  
13857    0  
13858    0  
13859    0  
13860    0  
13861    0  
13862    0  
13863    0  
Name: ramp_30, Length: 8490, dtype: int64

Whereas, what's in the updated plant.csv

plant_id
0        88.60 
1        103.01
2        103.01
3        103.01
4        103.01
11       1.42  
19       65.10 
20       65.10 
71       8.56  
72       8.56  
98       0.98  
99       0.98  
100      0.98  
101      86.98 
102      86.98 
103      86.98 
113      25.85 
114      25.85 
115      25.85 
118      6.11  
131      0.98  
140      0.98  
141      0.98  
142      0.98  
143      0.98  
157      0.98  
165      0.00  
199      89.15 
200      89.15 
201      89.15 
         ...   
13834    113.46
13835    113.46
13836    113.00
13837    113.00
13838    112.22
13839    112.22
13840    111.92
13841    111.92
13842    111.48
13843    111.48
13844    111.48
13845    111.48
13846    111.33
13847    111.33
13848    111.30
13849    111.30
13850    110.94
13851    110.94
13852    110.22
13853    110.13
13854    110.13
13855    110.13
13856    109.83
13857    107.56
13858    107.56
13859    105.08
13860    105.08
13861    102.41
13862    102.41
13863    100.21
Name: ramp_30, Length: 8490, dtype: float64

Apparently, they are different both in terms of type and values. I presume the following two possibilities:

• The calculations in the Matlab and here are different.
• The grid is reconstructed by MatReader in a different way.

[danielolsen]

I checked out the input.mat file for scenario 401 on the server to see what's up, and I found something interesting: the ramp_30 column in this file is floats, not ints, so I don't know why there has been an implicit int conversion somewhere within MATReader.

In input.mat for scenario 401, the ramp_30 values for the final generators is 0, because Pmax is 0, because these are the generators added since 2016 and we turned them off via the change table. So ramp_30 = 0.5 * 0 = 0.

[rouille]

@danielolsen, I guess I messed up. Can you look at the column_type_provider and column_name_provider in powersimdata/input/mat_reader.py and fix the type of the variables? I don't know why I set ramp_30 to int. Thanks

[danielolsen]

Types have been fixed in #148.

[rouille]

@danielolsen, did you rebase onto develop so @BainanXia can pull the new the changes in #148 and test again?

[danielolsen]

@rouille good call, I just did that now.

[BainanXia]

Based on the discussions offline with @danielolsen , due to the grid scaling when creating scenarios, we are expecting to see differences between scenario401.state.get_grid().plant and plant.csv only show up in Western, Texas and new plants in Eastern for 2019 and that's exactly what I observed here (note that plant_id for western starts from 10390):

dif = (grid.plant[grid.plant.type.isin({'coal','dfo','ng'})]['ramp_30'] - plant_to_compare[plant_to_compare.type.isin({'coal','dfo','ng'})]['ramp_30']).to_frame()
dif[abs(dif['ramp_30'])>1]

plant_id ramp_30
10423	3.300767
10424	3.300767
10482	1.846577
10483	1.846577
10505	1.283672
10514	1.823728
10516	1.952511
10517	1.952511
10518	1.952511
10519	1.952511
10534	1.757260
10535	1.757260
10538	2.843278
10539	2.843278
10540	2.843278
10573	2.860623
10574	2.860623
10575	2.860623
10582	2.947653
10583	2.947653
10599	1.971813
10600	1.971813
10601	1.971813
10616	2.786518
10631	3.313037
10632	3.313037
10646	2.374517
10647	2.374517
10648	2.374517
10649	2.374517
...	...
13834	-113.460000
13835	-113.460000
13836	-113.000000
13837	-113.000000
13838	-112.220000
13839	-112.220000
13840	-111.920000
13841	-111.920000
13842	-111.480000
13843	-111.480000
13844	-111.480000
13845	-111.480000
13846	-111.330000
13847	-111.330000
13848	-111.300000
13849	-111.300000
13850	-110.940000
13851	-110.940000
13852	-110.220000
13853	-110.130000
13854	-110.130000
13855	-110.130000
13856	-109.830000
13857	-107.560000
13858	-107.560000
13859	-105.080000
13860	-105.080000
13861	-102.410000
13862	-102.410000
13863	-100.210000

[BainanXia]

I trust you @danielolsen