Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: change unexpected modifications in Western and Texas back #158

Merged
merged 3 commits into from
Apr 27, 2020

Conversation

BainanXia
Copy link
Collaborator

Purpose

Change unexpected modifications in plant.csv and gencost.csv for Western and Texas in #104 back as before.

What is the code doing

No code changes. Only data file modifications.

Validation

checking the plants that I found with zero cost curves as following:

gencost_new.loc[[10423,
10424,
10459,
10460,
10461,
10462,
10482,
10483,
10497,
10498,
10505,
10514,
10515,
10516,
10517,
10518,
10519, 
10534,
10535,
10538,
10539,
10540,
10564,
10573,
10574,
10575,
10582,
10583]]

type	startup	shutdown	n	c2	c1	c0	interconnect
plant_id								
10423	2	0.0	0.0	3	0.001	20.303	838.72	Western
10424	2	0.0	0.0	3	0.001	26.364	838.72	Western
10459	2	0.0	0.0	3	0.001	23.961	662.55	Western
10460	2	0.0	0.0	3	0.001	24.949	662.55	Western
10461	2	0.0	0.0	3	0.001	24.105	662.55	Western
10462	2	0.0	0.0	3	0.001	23.251	662.55	Western
10482	2	0.0	0.0	3	0.001	22.606	733.35	Western
10483	2	0.0	0.0	3	0.001	22.979	733.35	Western
10497	2	0.0	0.0	3	0.002	25.704	1403.47	Western
10498	2	0.0	0.0	3	0.002	23.929	1403.47	Western
10505	2	0.0	0.0	3	0.001	23.622	692.70	Western
10514	2	0.0	0.0	3	0.001	21.827	731.70	Western
10515	2	0.0	0.0	3	0.001	23.801	656.55	Western
10516	2	0.0	0.0	3	0.001	23.264	741.00	Western
10517	2	0.0	0.0	3	0.001	22.918	741.00	Western
10518	2	0.0	0.0	3	0.001	23.396	741.00	Western
10519	2	0.0	0.0	3	0.001	26.180	741.00	Western
10534	2	0.0	0.0	3	0.001	21.129	726.90	Western
10535	2	0.0	0.0	3	0.001	22.330	726.90	Western
10538	2	0.0	0.0	3	0.002	30.184	949.00	Western
10539	2	0.0	0.0	3	0.001	23.682	949.00	Western
10540	2	0.0	0.0	3	0.001	24.834	949.00	Western
10564	2	0.0	0.0	3	0.001	23.375	639.00	Western
10573	2	0.0	0.0	3	0.001	22.462	925.00	Western
10574	2	0.0	0.0	3	0.002	29.087	925.00	Western
10575	2	0.0	0.0	3	0.001	25.033	925.00	Western
10582	2	0.0	0.0	3	0.001	24.823	813.22	Western
10583	2	0.0	0.0	3	0.001	20.736	813.22	Western

Time to review

I modified the csv files using text editor as @danielolsen mentioned in #157 so that git could track the changes in file. It turns out that branch.csv is correct, i.e. no fix is revealed. Also, I've found there are cases in which renewable plants have non-zero cost curves in current files, such as plant_id = 10390, type = hydro. Hence, indeed something is wrong in #104

Half an hour.

Change unexpected modifications in plant.csv and gencost.csv for Western and Texas in #104 back as before.
@BainanXia BainanXia requested a review from danielolsen April 25, 2020 06:07
@BainanXia BainanXia added this to the Spring Board milestone Apr 25, 2020
@danielolsen
Copy link
Contributor

Comparing the gencost CSVs pre- and post-#104, I see 30 changed gencost entries in Western: In addition to the 28 plant ids that Bainan listed, where fossil generators were given cost curves of 0, I see plant 10390 where a hydro generator was given non-zero cost curves, and plant 10470 where a nuclear generator was given a zero cost curve. I see no changes to generators in Texas.

Reviewing the changes to gencost.csv in this PR, I see some six thousand lines of entries changed. However, the overwhelming majority of those changes are to convert values of 0 to values of 0.0, and I can confirm that the only substantive changes are to those same thirty generators in Western whose cost curves were modified in #104.

plant.csv appears to show mostly similar changes (0 to 0.0), but also about 1,500 generators whose ramp_30 values were changed from non-zero values in plant.csv to value 0.0. Since ramp_30 values were updated recently based on the type and Pmax of generators (#144), I believe we want to keep these new values, and therefore we should not make changes to plant.csv, unless there's something I'm missing.

@BainanXia
Copy link
Collaborator Author

BainanXia commented Apr 27, 2020

Moving some bullets from slack discussions with @danielolsen here for record:

  • We still not sure what causes the bug at that time, but by comparing the three files plant.csv, gencost.csv, branch.csv before and after carefully, the only critical unexpected changes are 30 entries mentioned in @danielolsen comment.
  • This PR is created carefully to only fix those 30 entries according to the outputs of @danielolsen's previous notebook and keep everything else unchanged (exact same format) with the input files of current develop branch to make these changes in file trackable in git.
  • The grid retrieval process returns a grid with classical value format, i.e. if an integer appears in a column of value type 'float', it ends up with .0 to indicate the float type.
  • In our current csv inputs in PowerSimData, we have zero decimals when a value is an integer, i.e. 0.0 --> 0, which gives a cleaner look.
  • We should come up with a standard csv value format in the future so that whenever we make changes to those files, we don't fell into the rabbit hole to keep the format exactly the same before and after by trail and error.

Copy link
Contributor

@danielolsen danielolsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest commits, we are changing exactly what we need to, and nothing more. Thank you for leading the cleanup on the data that I messed up originally.

We've also identified a pain point with grid data updates, and have done a little fleshing out of how to mitigate this pain, but we will save that for a future PR.

@rouille
Copy link
Collaborator

rouille commented Apr 27, 2020

This is the command I used to produce the files to_csv('gencost.csv', float_format='%g'). It probably does not help.

@BainanXia BainanXia merged commit effd519 into develop Apr 27, 2020
@BainanXia BainanXia deleted the western_texas_data_fix branch April 27, 2020 20:24
@ahurli ahurli mentioned this pull request Mar 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants