Clean notebooks

[rouille]

Pull Request Etiquette doc

Purpose

Clean notebooks

What the code is doing

No code

Where to look

• The powersimdata/scenario/demo/tutorial.ipynb notebook has been deleted
• powersimdata/scenario/demo/scenario_and_grid_cheatsheet.ipynb: I have rerun the notebooks using a benchmark scenario, add a title, etc.
• powersimdata/input/demo/branch_device_type.ipynb: Same as above. I have also fixed some warnings. This notebook uses a package, networkx, which is not in our requirements.txt/Pipfile. Should we add it to the optional-requirements.txt? What do you think @danielolsen and @jon-hagg

Time estimate

20min.

[danielolsen]

I can see how these notebooks were useful for us, but I'm not sure it's worth maintaining them as notebooks going forward vs. pulling the main conclusions out and recording that information in markdown-format in the main README (for scenario_and_grid_cheatsheet.ipynb) or in a TAMU-specific README (for branch_device_type.ipynb).

[rouille]

I can see how these notebooks were useful for us, but I'm not sure it's worth maintaining them as notebooks going forward vs. pulling the main conclusions out and recording that information in markdown-format in the main README (for scenario_and_grid_cheatsheet.ipynb) or in a TAMU-specific README (for branch_device_type.ipynb).

I agree 100%. I believe that the README already encloses all the information located in scenario_and_grid_cheatsheet.ipynb. What would you add to the README? Regarding branch_device_type.ipynb, should I move it in powersimdata/network/usa_tamu/?

[BainanXia]

I agree with @danielolsen on the point that we might want to maintain these introductions in a lighter way instead of maintaining a notebook in the codebase, which is not convenient to read from general perspective. These notebooks are useful when we are exploring the tamu model ourselves and now we are presenting it others, thus probably we could provide a table that defines the columns of dataframes as Matpower did in the user manual and let the users explore themselves by loading the corresponding object instance into their workspace?

[BainanXia]

@rouille If we are confident that the current README has covered all the info we present in scenario_and_grid_cheatsheet.ipynb, then we could remove it. Regarding the other notebook, it is purely tamu_model dependent info, I think it is fair enough to move it in powersimdata/network/usa_tamu/ if we don't want to extract those info into other forms (such as tables+explanation in README) at this point.

[rouille]

Won't all network models have the same structure so they can be represented by a Grid object? In the affirmative, I think it would make sense to have a README in powersimdata/input/ where the Grid class is located where we list the different fields of the data frames of the attributes composing the Grid class (plant, gencost, etc.).

[danielolsen]

The branch_device_type.ipynb notebook seems to be specifically investigating the branch types of the TAMU network. Our next network may not even have TransformerWinding elements.

The Grid object should ideally have all of the same attributes, even for new network models, but we may not necessarily have all of the same columns in the tables themselves. E.g., should we generate data for the new models that we don't use (ramp_10, apf, etc.) just to ignore it in our simulation engine? How long do we want to support all the MATPOWER columns?

At some point we may also need new information to run Scenarios, such as unit commitment details for more detailed production cost model runs, or investment cost details for capacity expansion model runs (if we will still be preparing/launching those using PowerSimData).

These aren't things we need to consider right now, but definitely things to keep in the back of our minds.

[rouille]

I guess I will move the branch_device_type notebook in powersimdata/network/usa_tamu/ then. I agree that a large number of fields in the data frames of the Grid class are now useless that we move away from MATPOWER. We should think about refactoring the Grid class to stop generating some case.mat and in turn grid.mat with information we do not need. I will have an additional opportunity to modify the grid.mat!

[jenhagg]

I'm wondering if it makes sense to have a single optional-requirements.txt since users might need various subsets of that on a case by case basis. One thing we could do is have the notebook itself install the package. That said, there is only one optional package at the moment, so don't think it will hurt to add one more.

[kasparm]

@rouille What is the status on this?

[rouille]

@rouille What is the status on this?

The notebooks are cleaned/refactored. @BainanXia, @danielolsen and I did not make a decision on where they should go (move in other folder or Dropbox)

[danielolsen]

I vote that we save branch_device_type.ipynb to our dropbox and remove it from our repo. I think scenario_and_grid_cheatsheet.ipynb should stay, but we should get serious about doing automated checks in our CI process to make sure that when we change our codebase, we also detect when this breaks notebooks that are in the repo, so that we can either make sure to update them or decide that the updating process is not worth it and remove them.

[rouille]

I vote that we save branch_device_type.ipynb to our dropbox and remove it from our repo. I think scenario_and_grid_cheatsheet.ipynb should stay, but we should get serious about doing automated checks in our CI process to make sure that when we change our codebase, we also detect when this breaks notebooks that are in the repo, so that we can either make sure to update them or decide that the updating process is not worth it and remove them.

I am not super familiar with the Dropbox structure. Is there a place where branch_device_type.ipynb. would fit well?

[BainanXia]

I vote that we save branch_device_type.ipynb to our dropbox and remove it from our repo. I think scenario_and_grid_cheatsheet.ipynb should stay, but we should get serious about doing automated checks in our CI process to make sure that when we change our codebase, we also detect when this breaks notebooks that are in the repo, so that we can either make sure to update them or decide that the updating process is not worth it and remove them.

I am not super familiar with the Dropbox structure. Is there a place where scenario_and_grid_cheatsheet.ipynb would fit well?

@danielolsen How about Dropbox (Gates Ventures)/BreakthroughEnergySciences/Results/Features? It's empty there and I'm not sure what this folder is designed for originally. It seems to fit at this point.

[danielolsen]

I'm not sure this is results or a feature, it feels more like a reference or an onboarding tool. Maybe BreakthroughEnergySciences/Project/*?

[BainanXia]

I'm not sure this is results or a feature, it feels more like a reference or an onboarding tool. Maybe BreakthroughEnergySciences/Project/*?

My interpretation is the project folder is designed for separate projects other than the synthetic network. If we really want to put it there, it could be BreakthroughEnergySciences/Project/TAMU network model. The reason I choose feature folder is simply because it is a feature of the network model, i.e. how the corresponding information is stored and what does the topology look like, etc. I'm not quite care about where we put those notebooks in the dropbox as long as we keep them for potential future usage, given the current dropbox architecture is not strictly logically organized as the repo does.

[BainanXia]

This is good to go. Thanks.