refactor: reduce memory requirements for mesh branch scaling

[danielolsen]

Purpose

Reduce memory requirements for using the design_transmission module, and as a side-benefit simplify the code.

What is the code doing

Previously, we needed enough memory to store a congestion dataframe five times: we loaded CONGU, we loaded CONGL, we created numpy array versions of both, and we created a new dataframe of the same size to hold the element-wise maximization of these two dataframes. We did a 'clean-up' of CONGU and CONGL afterwards, but that doesn't help us if we don't have enough peak memory.

Instead, we can make use of the fact that element-wise, only one of CONGU and CONGL will be non-zero: we can simply add them instead of performing a type conversion, performing a numpy element-wise maximization, and then storing that result into a new dataframe. I think this new format only requires enough memory to store a congestion dataframe two or three times, depending on the internals of pandas DataFrame addition.

Perhaps most importantly, this function makes use of sparse dataframes as introduced in Breakthrough-Energy/PostREISE#96: we no longer expand the sparse dataframes into non-sparse numpy arrays. Even when starting with sparse dataframes, trying to run the previous code created a MemoryError on our laptops, while the new code runs without a problem.

Time Estimate

Half an hour or less.

[rouille]

I agree that this should use less memory. The operation might also be faster.

[danielolsen]

Tested successfully using sparse dataframe pickles for CONGU and CONGL, as introduced in Breakthrough-Energy/PostREISE#96.