Create grid object from different sources #85

rouille · 2020-01-30T22:17:05Z

Purpose

Refactor Grid oject to allow the use of different models (e.g. TAMU) based on different data formats.

What is the code doing

It builds the grid as we kmow it. The code now allows to build the grid from different models. usa_tamu is the only one implemented so far. Building the grid from the mat files used by MATPOWER/MOST (input.mat) is the next step.

MPC files structure have been modified. Only csv files are used now to build the grid. The information we used in the aux files have been transfered into the appropriate csv files. Also, the csv files have been correctly formatted. Note that the GenMWMin and GenMWMax columns in the plant data frame (attribute of the Grid object) have been dropped. The GenMWMin and GenMWMax values from the aux files is now used inplace of the Pmin and Pmax values in the csv file for renewables generators (hydro, solar and wind) only. Since, there is no more GenMWMin and GenMWMax column some refactoring was necessary. I you look at issue #83, you will see that some refactoring in PreREISE and PostREISE is necessary too. I will create the corresponding PRs soon.

Where to look

grid.py is the interface to build the grid.
usa_tamu_model.py encloses the TAMU object. It is a grid model.
abstract_grid.py is the abstract class defining the attributes of the Grid object.
mpc_reader.py encloses the MPCReader object. It reads MPC files format.
mat_reader encloses an empty MATReader object. Implementation will come in a different PR.
helpers.py has a couple of functions used throughout the construction of the grid.
mock_grid.py has a MockGrid object. It is adapted from the MockGrid object written by @danielolsen in PostREISE. I believe that this mock object sould live in powersimdata. Hence, I have switch the import in test_design.py
test_grid.py tests some function used in the building of the grid.

Time estimate

It does not do anything new. The code looks very different though. it will take some time to review.

kasparm

Nice work.

Tests are running on my system
Design makes sense to me

kasparm · 2020-01-31T01:13:53Z

@danielolsen I have a question regarding naming of file design.py. Could that be part of the helpers.py ? I don't fully understand the name design for this module.

danielolsen · 2020-01-31T01:44:51Z

@danielolsen I have a question regarding naming of file design.py. Could that be part of the helpers.py ? I don't fully understand the name design for this module.

@kasparm my intention was a module that is used to 'design' new scenarios to be run, via the change_tables. Right now the only public function is scale_renewable_stubs(), but I am also intending to add a function that will scale some subset of mesh branches, by looking at congestion shadow prices from a previous scenario. This will automate a lot of the work that had previously been done via copy-and-pasted notebook scripts, and make the process of spinning up advanced scenarios less fragile. The underscored functions could go into scenario/helpers if you think other modules will want to do topological analysis of the network.

danielolsen · 2020-02-03T19:09:01Z

powersimdata/input/design.py

-from postreise.process import const
-from postreise.process.transferdata import download
-from powersimdata.input.profiles import _read_data
-from powersimdata.scenario.helpers import interconnect2name


It looks like these are unnecessary imports that I added when creating this module? I remember I was running into circular dependency issues initially, these might have been cruft left over from trying to circumvent those.

danielolsen · 2020-02-03T19:20:18Z

powersimdata/input/helpers.py

+    return data_frame
+
+
+def add_column_to_data_frame(data_frame, column_dict):


It looks like this function both modifies the input dataframe and returns it. I think it might be more clear if it did one or the other.

Ok. I will change that. Keep you updated.

The opreation is done in place now. Docstring has been updated.

danielolsen · 2020-02-03T19:24:41Z

powersimdata/input/mat_reader.py

@@ -0,0 +1,2 @@
+class MATReader(object):


This is a placeholder for reading from input.mat, yeah?

danielolsen · 2020-02-03T19:27:03Z

powersimdata/input/mpc_reader.py

+from powersimdata.input.helpers import csv_to_data_frame
+
+
+class MPCReader(object):


The naming of the MPCReader and MATReader objects confused me a bit. I think it would help if the docstring for this object made it more clear that it's reading MPC attributes, but from CSVs.

Indeed. We can come up with better names too. Any idea?

What about the MATReader? Do you want to change the name too?

That one I'm more comfortable with, because it does read from a MAT file.

kasparm · 2020-02-05T17:55:38Z

@danielolsen
Could we find a more specific name for the design.py module?
Also, are the private functions useful for other modules and could be moved to the helpers.py?

We don't need to address the changes here.

danielolsen · 2020-02-05T18:12:32Z

@kasparm what about design_transmission.py?

The private functions aren't used by any other modules yet, but I agree that if/once they are, they'd fit better in helpers.py. I am agnostic as to where they should live now.

kasparm · 2020-02-05T18:18:00Z

@danielolsen name sounds good.
Let's leave the private functions where they are for now.

…rid object

danielolsen · 2020-02-06T01:15:09Z

On my local machine, I rebased grid_buildin_tamu onto develop, re-added the changes to design_transmission.py, and added a fix to setup.py that makes sure that the data gets added when you pip install .. From this branch, I created a new branch where I re-applied the gencost changes for v40. When this new branch creates a Scenario, the created case.mat file contains small changes to some fields.

The bus field only differs in the voltage magnitudes and angles (which we don't use), almost definitely due to rounding differences:

> > case_372 = load('C:\Users\dolsen\repos\iv\PowerSimData\compare_case_mat\case_372.mat');
> > case_373 = load('C:\Users\dolsen\repos\iv\PowerSimData\compare_case_mat\case_373.mat');
> > bus_diff = case_373.mpc.bus - case_372.mpc.bus;
sum(abs(bus_diff))
> > sum(abs(bus_diff))
ans =
  Columns 1 through 8
         0         0         0         0         0         0         0    0.1634
  Columns 9 through 16
    1.7518         0         0         0         0         0         0         0
  Column 17
         0

gen has only small differences in Pmin, Pmax, and apf (not used in our simulations):

>> sum(abs(gen_diff))
ans =
  Columns 1 through 8
         0         0         0         0         0    0.0000         0         0
  Columns 9 through 16
    2.0323   11.5991         0         0         0         0         0         0
  Columns 17 through 24
         0         0         0         0    0.1791         0         0         0
  Column 25
         0

branch differences are extremely small:

ans =
  Columns 1 through 8
         0         0    0.0000    0.0017    0.0000    0.0001         0         0
  Columns 9 through 16
         0         0         0         0         0         0         0         0
  Columns 17 through 21
         0         0         0         0         0

gencost differences are smaller yet:

>> sum(abs(gencost_diff))
ans =
   1.0e-11 *
         0         0         0         0    0.0022    0.2328         0

There are no differences in mpc.version, .baseMVA, .genid, .genfuel, .branchid, .dcline, or .dclineid, as verified by MATLAB's isequal.

danielolsen · 2020-02-06T01:17:12Z

My rebased/re-fixed version pushed (with approval from @rouille).

danielolsen

Looks great.

rouille added the grid label Jan 30, 2020

rouille added this to the Paralyzing Snow milestone Jan 30, 2020

rouille requested review from dmuldrew, danielolsen, kasparm and merrielle January 30, 2020 22:17

rouille assigned danielolsen and rouille Jan 30, 2020

kasparm approved these changes Jan 31, 2020

View reviewed changes

danielolsen reviewed Feb 3, 2020

View reviewed changes

rouille added 6 commits February 5, 2020 15:26

chore: Rename directory enclosing TAMU network

b2b42d5

refactor: Add paramater to pass name of dir enclosing network files

d2defa1

refactor: Reorganize MPC files used by TAMU model

8439b14

refactor: Build the grid from different models and sources

eea342d

refactor: Change MPC file creation following the refactoring of the G…

26f5d2d

…rid object

refactor: Use Pmin(max) inplace of GenMWMin(Max)

03bb281

danielolsen force-pushed the grid_buildin_tamu branch from f0efee6 to 69762c1 Compare February 6, 2020 01:16

danielolsen approved these changes Feb 6, 2020

View reviewed changes

dmuldrew approved these changes Feb 6, 2020

View reviewed changes

rouille force-pushed the grid_buildin_tamu branch from f887e30 to 08b68e1 Compare February 7, 2020 18:55

danielolsen force-pushed the grid_buildin_tamu branch from a274fdd to 69762c1 Compare February 7, 2020 19:26

rouille and others added 7 commits February 7, 2020 14:06

feat: Change design of Grid object

bb0e06e

test: Create a MockGrid Object

0b9802a

test: Test function used in the building of the grid

a0f0592

refactor: Add column to data frame inplace

50cd92b

refactor: Rename MPCReader object to CSVReader

00dcfdc

fix: updating setup.py data pointers

fe2b0eb

chore: Update __all__

47568a2

rouille force-pushed the grid_buildin_tamu branch from 8f801b2 to 47568a2 Compare February 7, 2020 22:07

rouille merged commit db440db into develop Feb 7, 2020

rouille deleted the grid_buildin_tamu branch February 7, 2020 22:13

Lab-ITTeam unassigned danielolsen Jul 15, 2020

ahurli mentioned this pull request Mar 11, 2021

Develop into Master #410

Merged

rouille changed the title ~~Grid Building~~ Create grid object from different sources Mar 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create grid object from different sources #85

Create grid object from different sources #85

rouille commented Jan 30, 2020

kasparm left a comment

kasparm commented Jan 31, 2020

danielolsen commented Jan 31, 2020

danielolsen Feb 3, 2020

danielolsen Feb 3, 2020

rouille Feb 3, 2020 •

edited

Loading

rouille Feb 5, 2020

danielolsen Feb 3, 2020

danielolsen Feb 3, 2020

rouille Feb 3, 2020

danielolsen Feb 3, 2020

rouille Feb 3, 2020

danielolsen Feb 3, 2020

kasparm commented Feb 5, 2020

danielolsen commented Feb 5, 2020 •

edited

Loading

kasparm commented Feb 5, 2020

danielolsen commented Feb 6, 2020

danielolsen commented Feb 6, 2020

danielolsen left a comment

		return data_frame


		def add_column_to_data_frame(data_frame, column_dict):

		from powersimdata.input.helpers import csv_to_data_frame


		class MPCReader(object):

Create grid object from different sources #85

Create grid object from different sources #85

Conversation

rouille commented Jan 30, 2020

Purpose

What is the code doing

Where to look

Time estimate

kasparm left a comment

Choose a reason for hiding this comment

kasparm commented Jan 31, 2020

danielolsen commented Jan 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rouille Feb 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasparm commented Feb 5, 2020

danielolsen commented Feb 5, 2020 • edited Loading

kasparm commented Feb 5, 2020

danielolsen commented Feb 6, 2020

danielolsen commented Feb 6, 2020

danielolsen left a comment

Choose a reason for hiding this comment

rouille Feb 3, 2020 •

edited

Loading

danielolsen commented Feb 5, 2020 •

edited

Loading