test: Use fake profile data instead of test_usa_tamu #586

c-voegele · 2022-02-14T13:29:41Z

Purpose

Hi Breakthrough Energy Team! Thank you for your work on this interesting and important project!

This PR is an attempt to fix #489.

The test_usa_tamu profiles are no longer needed for test_transform_profile and no Internet connection is required to run the tests.

What the code is doing

To avoid the Azure blob call, I mocked InputData to return random profiles. I just used uniform random data in [0, 1) as suggested here

Testing

I ran tox -e pytest-local -- --cov=powersimdata and tox -e checkformatting and both are passing.

To make sure that test_usa_tamu was not required, I wiped ~/ScenarioData and rebuilt it with LocalConfig().initialize(). After running the tests, test_usa_tamu is no longer loaded.

Where to look

powersimdata/input/tests/test_transform_grid.py

Usage Example/Visuals

n/a

Time estimate

15 minutes

powersimdata/input/tests/test_transform_profile.py

BainanXia · 2022-02-15T16:36:31Z

This is nice! I'm wondering whether it is possible to pull mock_input_data and corresponding raw profile functions into a separate module within powersimdata/tests/ together with other mocks so that it could potentially be used elsewhere, then we could import and patch it in test_transform_profile.py.

BainanXia · 2022-02-15T16:38:52Z

@jon-hagg With the mock_input_data implemented, do we still need truncated test profiles in blob storage under test_usa_tamu folder?

jenhagg · 2022-02-15T21:45:27Z

@jon-hagg With the mock_input_data implemented, do we still need truncated test profiles in blob storage under test_usa_tamu folder?

I would keep them for now - they are used here to test scenario preparation

c-voegele · 2022-02-16T17:31:44Z

This is nice! I'm wondering whether it is possible to pull mock_input_data and corresponding raw profile functions into a separate module within powersimdata/tests/ together with other mocks so that it could potentially be used elsewhere, then we could import and patch it in test_transform_profile.py.

Thanks for your suggestion @BainanXia! I reorganized the code as you suggested. Let me know what you think!

c-voegele · 2022-02-16T17:33:37Z

@jon-hagg With the mock_input_data implemented, do we still need truncated test profiles in blob storage under test_usa_tamu folder?

I would keep them for now - they are used here to test scenario preparation

Perhaps I can remove this dependency on test_usa_tamu in a separate PR?

powersimdata/tests/mock_input_data.py

BainanXia · 2022-02-16T18:30:12Z

This is nice! I'm wondering whether it is possible to pull mock_input_data and corresponding raw profile functions into a separate module within powersimdata/tests/ together with other mocks so that it could potentially be used elsewhere, then we could import and patch it in test_transform_profile.py.

Thanks for your suggestion @BainanXia! I reorganized the code as you suggested. Let me know what you think!

Thanks for the contribution! I think we are good to go except for some minor concerns above.

Fixes: Breakthrough-Energy#489

c-voegele · 2022-02-17T16:23:25Z

This is nice! I'm wondering whether it is possible to pull mock_input_data and corresponding raw profile functions into a separate module within powersimdata/tests/ together with other mocks so that it could potentially be used elsewhere, then we could import and patch it in test_transform_profile.py.

Thanks for your suggestion @BainanXia! I reorganized the code as you suggested. Let me know what you think!

Thanks for the contribution! I think we are good to go except for some minor concerns above.

Thanks! I addressed your comments. Let me know if you find anything else!

BainanXia · 2022-02-17T18:21:34Z

powersimdata/tests/mock_input_data.py

+    MockInputData is a mock of powersimdata.input.input_data.InputData
+    that generates random profiles.
+
+    Exactly 3 of {`start_time`, `end_time`, `periods`, `freq`} must be specified. See <https://pandas.pydata.org/docs/reference/api/pandas.date_range.html>.


BainanXia · 2022-02-17T18:24:20Z

powersimdata/input/tests/test_transform_profile.py

+@pytest.fixture(scope="module")
+def input_data(base_grid):
+    mock_input_data = MockInputData(base_grid)
+    return mock_input_data
+
+
+@pytest.fixture(scope="module", autouse=True)
+def mock_input_data_class(input_data):
+    with patch(
+        "powersimdata.input.transform_profile.InputData"
+    ) as mock_input_data_class:
+        mock_input_data_class.return_value = input_data
+        yield


This is clean.

BainanXia

Great! Thanks for improving test coverage and the patience @c-voegele

c-voegele · 2022-02-18T04:02:56Z

Thanks for the approval!

It doesn't look like I have write permissions so this is as far as I can go.

BainanXia · 2022-02-18T16:05:07Z

Thanks for the approval!

It doesn't look like I have write permissions so this is as far as I can go.

No worries, we will take care of it.

c-voegele force-pushed the c-voegele/test_with_fake_data branch from d1af535 to e14f7ec Compare February 14, 2022 15:53

danielolsen requested review from jenhagg, danielolsen and BainanXia February 14, 2022 16:42

danielolsen reviewed Feb 14, 2022

View reviewed changes

powersimdata/input/tests/test_transform_profile.py Outdated Show resolved Hide resolved

c-voegele force-pushed the c-voegele/test_with_fake_data branch from e14f7ec to 5d306f8 Compare February 15, 2022 04:17

c-voegele requested a review from danielolsen February 15, 2022 04:20

BainanXia requested a review from kasparm February 15, 2022 20:56

c-voegele force-pushed the c-voegele/test_with_fake_data branch from 47e49fd to 1cc34bc Compare February 16, 2022 17:25

BainanXia reviewed Feb 16, 2022

View reviewed changes

powersimdata/tests/mock_input_data.py Show resolved Hide resolved

BainanXia reviewed Feb 16, 2022

View reviewed changes

powersimdata/tests/mock_input_data.py Show resolved Hide resolved

test: Use fake profile data instead of test_usa_tamu

098d9f3

Fixes: Breakthrough-Energy#489

c-voegele force-pushed the c-voegele/test_with_fake_data branch from 1cc34bc to 098d9f3 Compare February 17, 2022 16:21

c-voegele requested a review from BainanXia February 17, 2022 16:23

BainanXia reviewed Feb 17, 2022

View reviewed changes

BainanXia approved these changes Feb 17, 2022

View reviewed changes

BainanXia assigned c-voegele Feb 17, 2022

BainanXia merged commit 733c5b1 into Breakthrough-Energy:develop Feb 18, 2022

danielolsen mentioned this pull request Feb 23, 2022

Enable Export from Scenario/Grid to pypsa.Network #576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Use fake profile data instead of test_usa_tamu #586

test: Use fake profile data instead of test_usa_tamu #586

c-voegele commented Feb 14, 2022

BainanXia commented Feb 15, 2022

BainanXia commented Feb 15, 2022

jenhagg commented Feb 15, 2022

c-voegele commented Feb 16, 2022

c-voegele commented Feb 16, 2022

BainanXia commented Feb 16, 2022

c-voegele commented Feb 17, 2022

BainanXia Feb 17, 2022

BainanXia Feb 17, 2022

BainanXia left a comment

c-voegele commented Feb 18, 2022

BainanXia commented Feb 18, 2022

test: Use fake profile data instead of test_usa_tamu #586

test: Use fake profile data instead of test_usa_tamu #586

Conversation

c-voegele commented Feb 14, 2022

Purpose

What the code is doing

Testing

Where to look

Usage Example/Visuals

Time estimate

BainanXia commented Feb 15, 2022

BainanXia commented Feb 15, 2022

jenhagg commented Feb 15, 2022

c-voegele commented Feb 16, 2022

c-voegele commented Feb 16, 2022

BainanXia commented Feb 16, 2022

c-voegele commented Feb 17, 2022

BainanXia Feb 17, 2022

Choose a reason for hiding this comment

BainanXia Feb 17, 2022

Choose a reason for hiding this comment

BainanXia left a comment

Choose a reason for hiding this comment

c-voegele commented Feb 18, 2022

BainanXia commented Feb 18, 2022