feat: add functions to estimate branch parameters (impedance and rating)

[danielolsen]

Pull Request doc

Purpose

Add functions to estimate the impedance and rating of each branch (transformers and lines). This PR depends on #218. EDIT: #218 has been merged.

What the code is doing

In const.py, we add assumed line and transformer parameters. @YifanLi86, do we have documentation of where these values come from? Are the assumptions you provided last December still good?

In transmission.py:

• We add a new function get_mst_edges, which either generates these edges from scratch and then caches them locally (the pickle file is about 35 KB), or loads them from the local cache, using a subset of the lines and substations dataframes as a crude hash key in an attempt to only load the solved edges for the same starting network.
• Within build_transmission, we call the get_mst_edges function, and then expand the information returned into a data frame with information required to estimate line impedances and ratings.
• We add new functions calculate_branch_mileage (lines only), estimate_branch_impedance (lines and transformers), and estimate_branch_rating (lines and transformers).
• Within check_for_location_conflicts, we add a bit more printing, so that it's clear to the user than all voltages are assigned by the heuristics, now that the minimum spanning tree edges are added.

Testing

Tested manually.

Usage Example/Visuals

>>> import pandas as pd
>>> from prereise.gather.griddata.hifld.data_process.transmission import (
...     build_transmission,
...     calculate_branch_mileage,
...     create_buses,
...     create_transformers,
...     estimate_branch_impedance,
...     estimate_branch_rating,
... )
>>> # build_transmission has been augmented to include minimum spanning tree
>>> lines, substations = build_transmission()
dropping 6892 substations of 70857 total due to LINES parameter equal to 0
dropping 759 lines with one or more substations listed as 'NOT AVAILABLE' out of a starting total of 71554
dropping 3143 lines with one or more substations not found in substations table out of a starting total of 70795
Evaluating endpoint location mismatches... (this may take several minutes)
dropping 120 lines with one or more substations with non-matching coordinates out of a starting total of 67652
dropping 143 lines with matching SUB_1 and SUB_2 out of a starting total of 67532
Reading cached minimum spanning tree
2140 line voltages can't be found via neighbor consensus
No more missing voltages remain after neighbor minimum
>>> bus, bus2sub = create_buses(lines)
>>> transformers = create_transformers(bus, bus2sub)
>>> # New usage example code starts here
>>> lines["type"] = "Line"
>>> lines["length"] = lines.apply(calculate_branch_mileage, axis=1)
>>> transformers["type"] = "Transformer"
>>> branch = pd.concat([lines, transformers])
>>> branch["x"] = branch.apply(lambda x: estimate_branch_impedance(x, bus), axis=1)
>>> branch["rateA"] = branch.apply(estimate_branch_rating, axis=1)

Time estimate

1 hour.

[YifanLi86]

@danielolsen For reference, I think we can mentioned these are industry generalized/standard values commonly used in studies like MTEP and RIIA (not exactly the same, have been averaged out so not CEII). https://www.misoenergy.org/about/media-center/miso-report-concludes-much-higher-levels-of-renewables-integration-are-achievable/; https://www.misoenergy.org/planning/planning/. They are developed based on various sources like (https://www.hitachiabb-powergrids.com/us/en/offering/product-and-system/high-voltage-switchgear-and-breakers/gas-insulated-transmission-line, https://ieeexplore.ieee.org/abstract/document/4113522) with some engineering judgement and general knowledge.

[danielolsen]

@YifanLi86 could you please come up with a couple of sentences for each of these sets of values? Something that can be included in the repo as documentation. We probably don't need to go through the full data intake procedure for these simple tables, but the data intake template should give a good idea of the sort of information we usually add.

[YifanLi86]

@YifanLi86 could you please come up with a couple of sentences for each of these sets of values? Something that can be included in the repo as documentation. We probably don't need to go through the full data intake procedure for these simple tables, but the data intake template should give a good idea of the sort of information we usually add.

Check this out to see if this is helpful..
Data Intake.docx

[danielolsen]

@YifanLi86 could you please come up with a couple of sentences for each of these sets of values? Something that can be included in the repo as documentation. We probably don't need to go through the full data intake procedure for these simple tables, but the data intake template should give a good idea of the sort of information we usually add.

Check this out to see if this is helpful..
Data Intake.docx

Could you be more specific with which sources are used for which tables?

[YifanLi86]

@YifanLi86 could you please come up with a couple of sentences for each of these sets of values? Something that can be included in the repo as documentation. We probably don't need to go through the full data intake procedure for these simple tables, but the data intake template should give a good idea of the sort of information we usually add.

Check this out to see if this is helpful..
Data Intake.docx

Could you be more specific with which sources are used for which tables?

I splited it into rating and reactance. Not sure if this is adequate, as there are documents that useful for both. Let me know how this looks.

Data Intake.docx

[danielolsen]

This is more helpful, but there's still a lot of ambiguity. Could you please break it down by which specific sources were used in developing:

• Line reactance per mile for lines > 50 miles
• Line ratings for lines <= 50 miles
• Equation structure/values for line ratings as a function of surge impedance loading
• Transformer reactances by voltage
• Transformer ratings

For the links that are general information on MISO's transmission planning, or for the Eastern Interconnection Reliability Assessment Group, could you point to particular document(s)? In general, could you point to where in each document (e.g. table name/number) information comes from?

The static rating for all transformers seems potentially problematic, especially when it's applied to some of the higher voltage transformers for which I would assume higher ratings are often appropriate.

[YifanLi86]

@danielolsen I agree, these data sources are pretty ambiguity. A lot of things are actually just general industry guidelines, e.g., I know 345kV MVA can vary a lot by conductor type, but 1793MVA is just the number everyone is using when TOs developing newprojects without much reference. Same thing for transformer, apprently its rating can vary a lot, however everyone seems just using this value regardless of its kV. I am not fully comfortable with this either, but it might be tricky to track down everything to the exact data source. Let me go over it again and try to get more specific information. I will get back a little bit later.

[danielolsen]

@danielolsen I agree, these data sources are pretty ambiguity. A lot of things are actually just general industry guidelines, e.g., I know 345kV MVA can vary a lot by conductor type, but 1793MVA is just the number everyone is using when TOs developing newprojects without much reference. Same thing for transformer, apprently its rating can vary a lot, however everyone seems just using this value regardless of its kV. I am not fully comfortable with this either, but it might be tricky to track down everything to the exact data source. Let me go over it again and try to get more specific information. I will get back a little bit later.

Thanks! I don't think we need to track down the original source of a given number, but it would be helpful if we can point to at least one specific external source for each of the numbers we're adding (or multiple sources if we've picked a representative value). Then if we find additional or updated sources, it's more straightforward to be able to update our 'consensus' values.

[danielolsen]

If physical transformers are usually modeled in industry as being all 800 MVA, would it be reasonable to equivalently model n transformers in parallel when they're connected to lines at 345 kV and above (since for these lines, a short line would have a rating of >800 MVA)? So the rating would be 800*n MVA, and the impedance would be x/n where x is the per-unit impedance for a single transformer that's looked up via the transformer_reactance table in const.py.

[YifanLi86]

If physical transformers are usually modeled in industry as being all 800 MVA, would it be reasonable to equivalently model n transformers in parallel when they're connected to lines at 345 kV and above (since for these lines, a short line would have a rating of >800 MVA)? So the rating would be 800*n MVA, and the impedance would be x/n where x is the per-unit impedance for a single transformer that's looked up via the transformer_reactance table in const.py.

Yes you are right, that is exactly what people would do in modeling. I think this is partly because transformers most of the time are not limited by thermal effect. I will get you more documents soon.

[danielolsen]

If physical transformers are usually modeled in industry as being all 800 MVA, would it be reasonable to equivalently model n transformers in parallel when they're connected to lines at 345 kV and above (since for these lines, a short line would have a rating of >800 MVA)? So the rating would be 800*n MVA, and the impedance would be x/n where x is the per-unit impedance for a single transformer that's looked up via the transformer_reactance table in const.py.

Yes you are right, that is exactly what people would do in modeling. I think this is partly because transformers most of the time are not limited by thermal effect. I will get you more documents soon.

Great! Do the assumed per-unit reactances already take this sort of parallel arrangement into account? If we keep the default rating as a starting point for all transformers, then for the current values if a transformer has 345 kV as the top end, it'll be modeled as 3 parallel transformers, 500 kV will get 4, and 765 kV will get 7. If we accidentally double-scale the impedance, it could have a pretty large impact for higher voltage transformers.

[YifanLi86]

If physical transformers are usually modeled in industry as being all 800 MVA, would it be reasonable to equivalently model n transformers in parallel when they're connected to lines at 345 kV and above (since for these lines, a short line would have a rating of >800 MVA)? So the rating would be 800*n MVA, and the impedance would be x/n where x is the per-unit impedance for a single transformer that's looked up via the transformer_reactance table in const.py.

Yes you are right, that is exactly what people would do in modeling. I think this is partly because transformers most of the time are not limited by thermal effect. I will get you more documents soon.

Great! Do the assumed per-unit reactances already take this sort of parallel arrangement into account? If we keep the default rating as a starting point for all transformers, then for the current values if a transformer has 345 kV as the top end, it'll be modeled as 3 parallel transformers, 500 kV will get 4, and 765 kV will get 7. If we accidentally double-scale the impedance, it could have a pretty large impact for higher voltage transformers.

Good point...I believe these values should be already taking care of the parallel issue. These transformer data are actually averaged out from ERAG cases (the case itself is CEII but average numbers are not).

[YifanLi86]

Please see here for all the original data source that points to the values that have been used. One thing I think we need to update is to increase the actual route length factor from 1.2 to 1.3, which seems more commonly used now.
data source.docx

[rouille]

General comments. The original line data frame has all its column name in upper case. The new added ones (type, length, start and end) are in lower case. I would try to use the same convention across columns (using upper case for the new ones or lowering all the columns of all the data frames in the load module)

[danielolsen]

Thanks! Just as a heads-up, when we load a Grid object, it automatically adds the from_lat, from_lon, to_lat, and to_lon attributes to the original data in the branch.csv file, so you don't need to do it manually:

>>> from powersimdata import Grid
>>> from powersimdata.utility.distance import haversine
>>> line_length_factor = 1.2
>>> grid = Grid("USA")
>>> lines = grid.branch.query("branch_device_type == 'Line'").copy()
>>> lines["length"] = line_length_factor * lines.apply(
...     lambda x: haversine((x.from_lat, x.from_lon), (x.to_lat, x.to_lon)),
...     axis=1,
... )
>>> lines["x_per_mile"] = lines["x"] / lines["length"]
>>> lines["voltage"] = lines["from_bus_id"].map(grid.bus["baseKV"])
>>> lines.query("length > 10").groupby("voltage")["x_per_mile"].mean()
voltage
69.0     0.009577
100.0    0.006326
115.0    0.003883
138.0    0.002645
161.0    0.002149
230.0    0.001138
345.0    0.000471
500.0    0.000208
765.0    0.000094
Name: x_per_mile, dtype: float64

These values do match with the document, once we round to 4 decimal places, but we should probably specify with a certain number of significant digits instead to avoid distorting the higher-voltage impedances too much (or directly calculate the values in code and use them directly, for maximum transparency).

What about the transformers, what was the process used to generate the values in the document?

No matter which set of values we choose, it is of course not going to match local results perfectly, but we can choose the approach that best matches external benchmarks and justify our decision that way.

[YifanLi86]

Thanks! Just as a heads-up, when we load a Grid object, it automatically adds the from_lat, from_lon, to_lat, and to_lon attributes to the original data in the branch.csv file, so you don't need to do it manually:

>>> from powersimdata import Grid
>>> from powersimdata.utility.distance import haversine
>>> line_length_factor = 1.2
>>> grid = Grid("USA")
>>> lines = grid.branch.query("branch_device_type == 'Line'").copy()
>>> lines["length"] = line_length_factor * lines.apply(
...     lambda x: haversine((x.from_lat, x.from_lon), (x.to_lat, x.to_lon)),
...     axis=1,
... )
>>> lines["x_per_mile"] = lines["x"] / lines["length"]
>>> lines["voltage"] = lines["from_bus_id"].map(grid.bus["baseKV"])
>>> lines.query("length > 10").groupby("voltage")["x_per_mile"].mean()
voltage
69.0     0.009577
100.0    0.006326
115.0    0.003883
138.0    0.002645
161.0    0.002149
230.0    0.001138
345.0    0.000471
500.0    0.000208
765.0    0.000094
Name: x_per_mile, dtype: float64

These values do match with the document, once we round to 4 decimal places, but we should probably specify with a certain number of significant digits instead to avoid distorting the higher-voltage impedances too much (or directly calculate the values in code and use them directly, for maximum transparency).

What about the transformers, what was the process used to generate the values in the document?

No matter which set of values we choose, it is of course not going to match local results perfectly, but we can choose the approach that best matches external benchmarks and justify our decision that way.

Thanks Daniel! Ok just for everyone's information, conversation has been going on in slack, for transformer and phase shifter, the current data is a summary of ERAG data which is not convenient to share or refer to. After discussion, the work going forward looks like: 1. try to find a method to derive these data from our grid data first, and benchmark it using data we have in compliance to all data requirements; 2. try to derive general numbers from PLEXOS WECC data we have now (pending legal check and check with EE for permission to release); 3. wait to get more EI data and reiterate the analysis including EI, since the Gridlab MISO data is not authorized to use outside of the Gridlab MISO project. So the help it can provide is very limited.

[rouille]

Thanks

[BainanXia]

Let's go.