test: make NREL EFS download tests faster #148
Conversation
|
Tests are indeed much faster, thanks for this quick PR. I have some comments on structure but the functional changes look good. |
lanesmith
force-pushed
the
lane/shorter_tests
branch
2 times, most recently
from
5d99dd0 to
1f140ed
Compare
| dummy_demand_df.to_csv("test_demand.csv", index=False) | ||
|
|
||
| # Create a .zip file of the dummy demand data set | ||
| with zipfile.ZipFile("test.zip", "w") as z: | ||
| z.write("test_demand.csv") | ||
| os.remove("test_demand.csv") |
There was a problem hiding this comment.
Please also enclose these file writes within a try/except block as well.
There was a problem hiding this comment.
I see that "test_demand.csv" will get cleaned up either way below, does "test.zip" get covered as well?
There was a problem hiding this comment.
Yeah, test.zip should be cleaned up within _extract_data. I can add an except FileNotFoundError in between the try and finally blocks of test_extract_data in case something goes wrong though.
Can you explain why the above code should be within a try/except block? I don't expect that code to raise an error.
There was a problem hiding this comment.
Ah, I see about the test.zip deletion.
I'm thinking about what happens if e.g. test.zip fails to be written (e.g. in case another program already has it open). Then we get an exception, and "test_demand.csv" is not cleaned up. I guess it's not too big of a problem though, since whoever is running the tests will hopefully fix the problem and re-run and then the file should get cleaned up properly.
lanesmith
force-pushed
the
lane/shorter_tests
branch
from
0f878c8 to
d3afe5b
Compare
Purpose
The goal of this PR is to speed up the NREL EFS data download and extraction tests. Due to the size of the .zip files, the tests take minutes to run (as opposed to the other tests that take seconds). See #145 for reference.
What the code is doing
test_get_efs_data.pywas updated to not testdownload_demand_dataanddownload_flexibility_dataindividually, but rather to test the individual private functions (e.g.,_download_data,_extract_data) that comprise them. While testing the private functions create some more tests, they are much faster because_download_datacan be tested with a much smaller file (the same can't be done withdownload_demand_dataanddownload_flexibility_datasince they are created to only download and extract the large NREL EFS data sets).Where to look
All of the changes are contained within
PreREISE/prereise/gather/demanddata/nrel_efs/tests/test_get_efs_data.py.Time estimate
This should only take a few minutes.