Add coverage report to pytest #471

jenhagg · 2021-04-29T21:33:58Z

Purpose

Whenever we run pytest, include a coverage report in the output. This will help with awareness of test coverage, and enable future improvements such as analysis over time and in different areas of the code, or additional ci checks (e.g. we could fail the build if code coverage decreases).

What the code is doing

New packages:

coverage - the main tool for creating the report
pytest-cov - plugin so we can run coverage via pytest

Add the necessary packages to Pipfile and requirements.txt. Consolidated them under the dev-packages section and removed version pinning for these only. This makes them available whether running pytest directly or via tox.

Also added python 3.9 to the test matrix.

Testing

green checkmark

Usage Example/Visuals

The report looks like this (see the recent github logs for full output)

powersimdata/tests/mock_change_table.py                                  10      1    90%
powersimdata/tests/mock_grid.py                                          57      0   100%
powersimdata/tests/mock_scenario.py                                       9      0   100%
powersimdata/tests/mock_scenario_info.py                                 26      8    69%
powersimdata/tests/mock_ssh.py                                           17      0   100%
powersimdata/tests/test_mocks.py                                         72      0   100%
powersimdata/utility/__init__.py                                          0      0   100%
powersimdata/utility/distance.py                                         38      2    95%
powersimdata/utility/helpers.py                                          63     10    84%
powersimdata/utility/server_setup.py                                     42      8    81%
powersimdata/utility/templates/__init__.py                                0      0   100%
powersimdata/utility/tests/__init__.py                                    0      0   100%
powersimdata/utility/tests/test_distance.py                              23      0   100%
powersimdata/utility/tests/test_helpers.py                               73      0   100%
-----------------------------------------------------------------------------------------
TOTAL                                                                  6572   1908    71%

=============================================================================================== 234 passed, 41 deselected, 1 warning in 33.51s ===============================================================================================

Time estimate

5 mins

rouille · 2021-04-29T21:40:22Z

What does the coverage mean for the module itself, e.g.:

powersimdata/utility/distance.py                                         38      2    95%

w.r.t:

powersimdata/utility/tests/test_distance.py                              23      0   100%

danielolsen · 2021-04-29T21:41:55Z

What does the coverage mean for the module itself, e.g.:

powersimdata/utility/distance.py                                         38      2    95%

w.r.t:

powersimdata/utility/tests/test_distance.py                              23      0   100%

I believe the meaning here is: when the tests run, 100% of the lines of code in test_distance.py are run, while only 95% of the lines of code in distance.py are run.

jenhagg · 2021-04-29T21:43:07Z

Yeah we probably need to exclude test files from the report

danielolsen · 2021-04-29T21:45:03Z

Yeah we probably need to exclude test files from the report

It's interesting that not all of the methods of the mock object are being used though, I think those might be worth keeping an eye on.

jenhagg · 2021-04-29T21:46:22Z

Yeah we probably need to exclude test files from the report

It's interesting that not all of the methods of the mock object are being used though, I think those might be worth keeping an eye on.

Good point - we'd want to exclude test files but not mocks, which are in a tests/ directory

rouille · 2021-04-29T21:54:03Z

Our overall test coverage is not bad I would say. We also have some integration tests (ssh connection required) that are not accounted for in the report.

jenhagg · 2021-04-29T22:19:41Z

Unfortunately our coverage decreases after excluding the test files

Name                                                       Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------------------------------
powersimdata/data_access/context.py                           13      2      4      2    76%
powersimdata/data_access/csv_store.py                         43      1      4      2    94%
powersimdata/data_access/data_access.py                      221     95     34      3    53%
powersimdata/data_access/execute_list.py                      51     18      2      0    62%
powersimdata/data_access/profile_helper.py                    47     19      6      0    57%
powersimdata/data_access/scenario_list.py                     70     18     10      1    71%
powersimdata/data_access/sql_store.py                         62     39     16      0    29%
powersimdata/design/generation/clean_capacity_scaling.py     277    184    161     13    30%
powersimdata/design/generation/cost_curves.py                230    123     95     19    46%
powersimdata/design/generation/curtailment.py                 65      9     41      6    86%
powersimdata/design/investment/const.py                       23      0      0      0   100%
powersimdata/design/investment/create_mapping_files.py        98     86     16      0    11%
powersimdata/design/investment/inflation.py                   12      2      8      3    75%
powersimdata/design/investment/investment_costs.py           206     50     50      9    76%
powersimdata/design/mimic_grid.py                             41     39     18      0     3%
powersimdata/design/scenario_info.py                          80      7     16      6    86%
powersimdata/design/transmission/mwmiles.py                   48     13     25      6    71%
powersimdata/design/transmission/statelines.py                30      8      6      0    78%
powersimdata/design/transmission/upgrade.py                  194     18     95      8    90%
powersimdata/input/abstract_grid.py                           18      0      0      0   100%
powersimdata/input/case_mat.py                                55     51      4      0     7%
powersimdata/input/change_table.py                           395     85    252     38    77%
powersimdata/input/grid.py                                   100     19     32      6    73%
powersimdata/input/helpers.py                                 46     14     12      0    72%
powersimdata/input/input_data.py                              70     13     16      4    78%
powersimdata/input/scenario_grid.py                          164    112     50      0    34%
powersimdata/input/transform_grid.py                         294      0    116      2    99%
powersimdata/input/transform_profile.py                       55     45     20      0    13%
powersimdata/network/csv_reader.py                             8      0      0      0   100%
powersimdata/network/model.py                                 60     22     34      7    56%
powersimdata/network/usa_tamu/constants/plants.py             15      0      2      0   100%
powersimdata/network/usa_tamu/constants/storage.py             4      0      0      0   100%
powersimdata/network/usa_tamu/constants/zones.py              40      0     28      0   100%
powersimdata/network/usa_tamu/model.py                        64      2     32      2    96%
powersimdata/output/output_data.py                            54     41      8      0    21%
powersimdata/scenario/analyze.py                             137    100     20      0    24%
powersimdata/scenario/create.py                              203     91     58     12    49%
powersimdata/scenario/delete.py                               39     39      2      0     0%
powersimdata/scenario/execute.py                             188    148     48      0    17%
powersimdata/scenario/move.py                                 53     53      6      0     0%
powersimdata/scenario/scenario.py                             63     27     22      4    49%
powersimdata/scenario/state.py                                30     14      8      1    45%
powersimdata/tests/mock_analyze.py                            49      3      4      0    94%
powersimdata/tests/mock_builder.py                            10     10      2      0     0%
powersimdata/tests/mock_change_table.py                       10      1      2      0    92%
powersimdata/tests/mock_grid.py                               57      0     22      0   100%
powersimdata/tests/mock_scenario.py                            9      0      0      0   100%
powersimdata/tests/mock_scenario_info.py                      26      8      2      0    71%
powersimdata/tests/mock_ssh.py                                17      0      0      0   100%
powersimdata/utility/distance.py                              38      2      8      0    96%
powersimdata/utility/helpers.py                               63     10     12      0    87%
powersimdata/utility/server_setup.py                          42      8      4      2    78%
--------------------------------------------------------------------------------------------
TOTAL                                                       4287   1649   1433    156    60%

=============================================================================================== 234 passed, 41 deselected, 1 warning in 26.08s ===============================================================================================

danielolsen · 2021-04-29T22:54:06Z

Unfortunately our coverage decreases after excluding the test files

(x + c)/(y + c) > x/y, for all {0 < c}, {0 < x < y}

rouille · 2021-04-29T23:00:08Z

Unfortunately our coverage decreases after excluding the test files

(x + c)/(y + c) > x/y, for all {0 < c}, {0 < x < y}

indeed

pytest.ini

rouille · 2021-04-29T23:04:14Z

Unfortunately our coverage decreases after excluding the test files

Name                                                       Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------------------------------
powersimdata/data_access/context.py                           13      2      4      2    76%
powersimdata/data_access/csv_store.py                         43      1      4      2    94%
powersimdata/data_access/data_access.py                      221     95     34      3    53%
powersimdata/data_access/execute_list.py                      51     18      2      0    62%
powersimdata/data_access/profile_helper.py                    47     19      6      0    57%
powersimdata/data_access/scenario_list.py                     70     18     10      1    71%
powersimdata/data_access/sql_store.py                         62     39     16      0    29%
powersimdata/design/generation/clean_capacity_scaling.py     277    184    161     13    30%
powersimdata/design/generation/cost_curves.py                230    123     95     19    46%
powersimdata/design/generation/curtailment.py                 65      9     41      6    86%
powersimdata/design/investment/const.py                       23      0      0      0   100%
powersimdata/design/investment/create_mapping_files.py        98     86     16      0    11%
powersimdata/design/investment/inflation.py                   12      2      8      3    75%
powersimdata/design/investment/investment_costs.py           206     50     50      9    76%
powersimdata/design/mimic_grid.py                             41     39     18      0     3%
powersimdata/design/scenario_info.py                          80      7     16      6    86%
powersimdata/design/transmission/mwmiles.py                   48     13     25      6    71%
powersimdata/design/transmission/statelines.py                30      8      6      0    78%
powersimdata/design/transmission/upgrade.py                  194     18     95      8    90%
powersimdata/input/abstract_grid.py                           18      0      0      0   100%
powersimdata/input/case_mat.py                                55     51      4      0     7%
powersimdata/input/change_table.py                           395     85    252     38    77%
powersimdata/input/grid.py                                   100     19     32      6    73%
powersimdata/input/helpers.py                                 46     14     12      0    72%
powersimdata/input/input_data.py                              70     13     16      4    78%
powersimdata/input/scenario_grid.py                          164    112     50      0    34%
powersimdata/input/transform_grid.py                         294      0    116      2    99%
powersimdata/input/transform_profile.py                       55     45     20      0    13%
powersimdata/network/csv_reader.py                             8      0      0      0   100%
powersimdata/network/model.py                                 60     22     34      7    56%
powersimdata/network/usa_tamu/constants/plants.py             15      0      2      0   100%
powersimdata/network/usa_tamu/constants/storage.py             4      0      0      0   100%
powersimdata/network/usa_tamu/constants/zones.py              40      0     28      0   100%
powersimdata/network/usa_tamu/model.py                        64      2     32      2    96%
powersimdata/output/output_data.py                            54     41      8      0    21%
powersimdata/scenario/analyze.py                             137    100     20      0    24%
powersimdata/scenario/create.py                              203     91     58     12    49%
powersimdata/scenario/delete.py                               39     39      2      0     0%
powersimdata/scenario/execute.py                             188    148     48      0    17%
powersimdata/scenario/move.py                                 53     53      6      0     0%
powersimdata/scenario/scenario.py                             63     27     22      4    49%
powersimdata/scenario/state.py                                30     14      8      1    45%
powersimdata/tests/mock_analyze.py                            49      3      4      0    94%
powersimdata/tests/mock_builder.py                            10     10      2      0     0%
powersimdata/tests/mock_change_table.py                       10      1      2      0    92%
powersimdata/tests/mock_grid.py                               57      0     22      0   100%
powersimdata/tests/mock_scenario.py                            9      0      0      0   100%
powersimdata/tests/mock_scenario_info.py                      26      8      2      0    71%
powersimdata/tests/mock_ssh.py                                17      0      0      0   100%
powersimdata/utility/distance.py                              38      2      8      0    96%
powersimdata/utility/helpers.py                               63     10     12      0    87%
powersimdata/utility/server_setup.py                          42      8      4      2    78%
--------------------------------------------------------------------------------------------
TOTAL                                                       4287   1649   1433    156    60%

=============================================================================================== 234 passed, 41 deselected, 1 warning in 26.08s ===============================================================================================

A lot of code is tested through integration tests (41 deselected). For example:

powersimdata/input/transform_profile.py                       55     45     20      0    13%

I believe all types of transformation are tested but to do that we need the profiles

danielolsen

I don't understand all the CI stuff, but it is a good idea and it seems to work. Maybe we should also add the code coverage badge to the readme.

danielolsen · 2021-04-29T23:05:34Z

I believe all types of transformation are tested but to do that we need the profiles

Now that we have profiles in blob storage, can we allow these tests from CI?

BainanXia

Learned something new. Thanks!

rouille · 2021-04-29T23:09:50Z

I believe all types of transformation are tested but to do that we need the profiles

Now that we have profiles in blob storage, can we allow these tests from CI?

I thought about it but the profiles are big (>100MB for USA hydro). I don't know if GitHub will be happy about it and/or we want to wait for the checks to finish on each push

rouille · 2021-04-29T23:12:53Z

I don't understand all the CI stuff, but it is a good idea and it seems to work. Maybe we should also add the code coverage badge to the readme.

The test coverage badge would definitely be a huge motivation for writing more tests!

jenhagg · 2021-04-29T23:13:14Z

I don't understand all the CI stuff, but it is a good idea and it seems to work. Maybe we should also add the code coverage badge to the readme.

That would be nice. Looking at pandas, they seem to report coverage to an external service which provides the badge. Will look into this when I can.

jenhagg · 2021-04-29T23:14:56Z

I believe all types of transformation are tested but to do that we need the profiles

Now that we have profiles in blob storage, can we allow these tests from CI?

I thought about it but the profiles are big (>100MB for USA hydro). I don't know if GitHub will be happy about it and/or we want to wait for the checks to finish on each push

We could limit it to the smallest profiles, or use some mock profiles for this purpose, or maybe the github cache action so we only download them once in a while.

danielolsen · 2021-04-29T23:15:08Z

Hopefully we can somehow get at least the profiles tests included.

Jon Hagg added 2 commits April 29, 2021 11:24

ci: add coverage report to pytest

c7521eb

chore: add dev packages to project wide install

32b68fe

jenhagg requested review from rouille, BainanXia and danielolsen April 29, 2021 21:36

jenhagg self-assigned this Apr 29, 2021

jenhagg added this to the Let the Sun Shine milestone Apr 29, 2021

jenhagg added the dependencies Pull requests that update a dependency file label Apr 29, 2021

chore: add .coveragerc and filter report

6431627

danielolsen reviewed Apr 29, 2021

View reviewed changes

pytest.ini Show resolved Hide resolved

danielolsen approved these changes Apr 29, 2021

View reviewed changes

BainanXia approved these changes Apr 29, 2021

View reviewed changes

jenhagg merged commit bedc133 into develop Apr 29, 2021

jenhagg deleted the jon/cov branch April 29, 2021 23:16

This was referenced Apr 30, 2021

ci: add coverage report to pytest Breakthrough-Energy/PostREISE#278

Merged

ci: add coverage report to pytest Breakthrough-Energy/PreREISE#163

Merged

danielolsen mentioned this pull request Jun 11, 2021

chore: merge develop into master for v0.4.2 release #498

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add coverage report to pytest #471

Add coverage report to pytest #471

jenhagg commented Apr 29, 2021 •

edited

Loading

rouille commented Apr 29, 2021

danielolsen commented Apr 29, 2021

jenhagg commented Apr 29, 2021

danielolsen commented Apr 29, 2021

jenhagg commented Apr 29, 2021

rouille commented Apr 29, 2021

jenhagg commented Apr 29, 2021 •

edited

Loading

danielolsen commented Apr 29, 2021

rouille commented Apr 29, 2021

rouille commented Apr 29, 2021

danielolsen left a comment

danielolsen commented Apr 29, 2021

BainanXia left a comment

rouille commented Apr 29, 2021

rouille commented Apr 29, 2021

jenhagg commented Apr 29, 2021

jenhagg commented Apr 29, 2021

danielolsen commented Apr 29, 2021 •

edited

Loading

Add coverage report to pytest #471

Add coverage report to pytest #471

Conversation

jenhagg commented Apr 29, 2021 • edited Loading

Purpose

What the code is doing

Testing

Usage Example/Visuals

Time estimate

rouille commented Apr 29, 2021

danielolsen commented Apr 29, 2021

jenhagg commented Apr 29, 2021

danielolsen commented Apr 29, 2021

jenhagg commented Apr 29, 2021

rouille commented Apr 29, 2021

jenhagg commented Apr 29, 2021 • edited Loading

danielolsen commented Apr 29, 2021

rouille commented Apr 29, 2021

rouille commented Apr 29, 2021

danielolsen left a comment

Choose a reason for hiding this comment

danielolsen commented Apr 29, 2021

BainanXia left a comment

Choose a reason for hiding this comment

rouille commented Apr 29, 2021

rouille commented Apr 29, 2021

jenhagg commented Apr 29, 2021

jenhagg commented Apr 29, 2021

danielolsen commented Apr 29, 2021 • edited Loading

jenhagg commented Apr 29, 2021 •

edited

Loading

jenhagg commented Apr 29, 2021 •

edited

Loading

danielolsen commented Apr 29, 2021 •

edited

Loading