Handle deprecation warnings

[rouille]

Pull Request doc

Purpose

Fix some deprecation warnings. Those are raised in PostREISE.

What the code is doing

Convert sets to lists when querying data frame

Testing

Existing unit tests. I checked the warnings were not raised anymore in PostREISE:

Before

[~/CEM/PostREISE] (develop) brdo$ tox -r
pytest recreate: /Users/brdo/CEM/PostREISE/.tox/pytest
pytest installdeps: pipenv
pytest installed: certifi==2022.6.15,distlib==0.3.5,filelock==3.8.0,pipenv==2022.8.17,platformdirs==2.5.2,virtualenv==20.16.3,virtualenv-clone==0.5.7
pytest run-test-pre: PYTHONHASHSEED='3396604905'
pytest run-test: commands[0] | pipenv sync --dev
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile.lock (e1f1a2)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 57/57 — 00:00:29
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
pytest run-test: commands[1] | pytest
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.9.11, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/pytest/.pytest_cache
rootdir: /Users/brdo/CEM/PostREISE, configfile: pytest.ini
plugins: cov-3.0.0
collected 184 items                                                                                                                                            

postreise/analyze/generation/tests/test_binding.py ....................                                                                                  [ 10%]
postreise/analyze/generation/tests/test_capacity.py ..........................                                                                           [ 25%]
postreise/analyze/generation/tests/test_costs.py ......                                                                                                  [ 28%]
postreise/analyze/generation/tests/test_curtailment.py .....................                                                                             [ 39%]
postreise/analyze/generation/tests/test_emissions.py ............                                                                                        [ 46%]
postreise/analyze/generation/tests/test_summarize.py ............                                                                                        [ 52%]
postreise/analyze/tests/test_demand.py ..                                                                                                                [ 53%]
postreise/analyze/tests/test_time.py ................................                                                                                    [ 71%]
postreise/analyze/transmission/tests/test_congestion_surplus.py ..                                                                                       [ 72%]
postreise/analyze/transmission/tests/test_utilization.py ....                                                                                            [ 74%]
postreise/plot/tests/test_canvas.py .....                                                                                                                [ 77%]
postreise/plot/tests/test_check.py ....                                                                                                                  [ 79%]
postreise/plot/tests/test_plot_bar_generation_max_min_actual.py ...                                                                                      [ 80%]
postreise/plot/tests/test_plot_bar_generation_stack.py ...                                                                                               [ 82%]
postreise/plot/tests/test_plot_bar_generation_vs_capacity.py .........                                                                                   [ 87%]
postreise/plot/tests/test_plot_bar_renewable_max_profile_actual.py ...                                                                                   [ 89%]
postreise/plot/tests/test_plot_bar_shortfall.py ...                                                                                                      [ 90%]
postreise/plot/tests/test_plot_pie_generation_vs_capacity.py .....                                                                                       [ 93%]
postreise/plot/tests/test_plot_shadowprice_map.py .........                                                                                              [ 98%]
postreise/plot/tests/test_plot_sim_vs_hist.py ...                                                                                                        [100%]

======================================================================= warnings summary =======================================================================
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeriesGrouped::test_curtailment_time_series_by_areas
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeriesGrouped::test_curtailment_time_series_by_areas_and_resource
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeriesGrouped::test_curtailment_time_series_by_areas_and_resources
postreise/analyze/generation/tests/test_curtailment.py::TestCalculateCurtailmentTimeSeriesGrouped::test_curtailment_time_series_by_resources_and_areas
  /Users/brdo/CEM/PostREISE/.tox/pytest/lib/python3.9/site-packages/powersimdata/input/helpers.py:182: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
    df_areas[s] = df[get_plant_id_in_states(s, grid) & plant_id]

postreise/analyze/generation/tests/test_curtailment.py: 5 warnings
postreise/plot/tests/test_plot_bar_generation_stack.py: 24 warnings
  /Users/brdo/CEM/PostREISE/.tox/pytest/lib/python3.9/site-packages/powersimdata/input/helpers.py:185: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
    df_areas[l] = df[get_plant_id_in_loadzones(l, grid) & plant_id]

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform darwin, python 3.9.11-final-0 ----------
Name                                                                 Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------------------------------------------------
postreise/__init__.py                                                    0      0      0      0   100%
postreise/analyze/__init__.py                                            0      0      0      0   100%
postreise/analyze/demand.py                                             13      0      2      0   100%
postreise/analyze/generation/__init__.py                                 0      0      0      0   100%
postreise/analyze/generation/binding.py                                 27      0      0      0   100%
postreise/analyze/generation/capacity.py                                48      0      4      0   100%
postreise/analyze/generation/costs.py                                   24      0     10      0   100%
postreise/analyze/generation/curtailment.py                             74      0     20      0   100%
postreise/analyze/generation/emissions.py                               50      1     22      1    97%
postreise/analyze/generation/summarize.py                               71      1     14      1    98%
postreise/analyze/generation/tests/__init__.py                           1      0      0      0   100%
postreise/analyze/generation/tests/test_binding.py                     119      0     10      0   100%
postreise/analyze/generation/tests/test_capacity.py                     85      0      4      0   100%
postreise/analyze/generation/tests/test_costs.py                        40      0      2      0   100%
postreise/analyze/generation/tests/test_curtailment.py                 147      0     44      0   100%
postreise/analyze/generation/tests/test_emissions.py                   104      0     20      0   100%
postreise/analyze/generation/tests/test_summarize.py                    82      0     12      0   100%
postreise/analyze/tests/__init__.py                                      0      0      0      0   100%
postreise/analyze/tests/test_demand.py                                  18      0      0      0   100%
postreise/analyze/tests/test_time.py                                   366      0    112      0   100%
postreise/analyze/time.py                                               71      0     50      1    99%
postreise/analyze/transmission/__init__.py                               0      0      0      0   100%
postreise/analyze/transmission/congestion.py                            16      0      0      0   100%
postreise/analyze/transmission/tests/__init__.py                         1      0      0      0   100%
postreise/analyze/transmission/tests/test_congestion_surplus.py         36      0      4      0   100%
postreise/analyze/transmission/tests/test_utilization.py                27      0      0      0   100%
postreise/analyze/transmission/utilization.py                           48      2      8      2    93%
postreise/data/__init__.py                                               0      0      0      0   100%
postreise/plot/__init__.py                                               0      0      0      0   100%
postreise/plot/canvas.py                                                26      0     24      0   100%
postreise/plot/check.py                                                 19      0     12      0   100%
postreise/plot/colors.py                                                 8      0      0      0   100%
postreise/plot/plot_bar_generation_max_min_actual.py                    50      1     24      1    97%
postreise/plot/plot_bar_generation_stack.py                            106     11     72      6    88%
postreise/plot/plot_bar_generation_vs_capacity.py                      132      7     74     10    92%
postreise/plot/plot_bar_renewable_max_profile_actual.py                 59      1     26      1    98%
postreise/plot/plot_bar_shortfall.py                                    89     16     48      9    80%
postreise/plot/plot_capacity_map.py                                     63     63     16      0     0%
postreise/plot/plot_carbon_bar.py                                       38     38     17      0     0%
postreise/plot/plot_carbon_map.py                                      111    111     24      0     0%
postreise/plot/plot_curtailment_ts.py                                   79     79     26      0     0%
postreise/plot/plot_energy_carbon_stack.py                              81     81     49      0     0%
postreise/plot/plot_generation_ts_stack.py                             124    124     52      0     0%
postreise/plot/plot_heatmap.py                                          43     43     14      0     0%
postreise/plot/plot_interconnection_map.py                              67     67     22      0     0%
postreise/plot/plot_lmp_map.py                                          53     53     16      0     0%
postreise/plot/plot_pie_generation_vs_capacity.py                       95     65     62     10    31%
postreise/plot/plot_powerflow_snapshot.py                               93     93     30      0     0%
postreise/plot/plot_scatter_capacity_vs_capacity_factor.py              42     42     20      0     0%
postreise/plot/plot_scatter_capacity_vs_cost_curve_slope.py             34     34     14      0     0%
postreise/plot/plot_scatter_capacity_vs_curtailment.py                  48     48     20      0     0%
postreise/plot/plot_shadowprice_map.py                                 107     52     22      0    52%
postreise/plot/plot_sim_vs_hist.py                                      59      2     26      2    95%
postreise/plot/plot_states.py                                          119    119     52      0     0%
postreise/plot/plot_tornado.py                                          22     22      4      0     0%
postreise/plot/plot_transmission_upgrades_map.py                        78     78     18      0     0%
postreise/plot/plot_utilization_map.py                                  77     77     18      0     0%
postreise/plot/projection_helpers.py                                    40     33     15      0    13%
postreise/plot/tests/__init__.py                                         0      0      0      0   100%
postreise/plot/tests/test_canvas.py                                     28      0      0      0   100%
postreise/plot/tests/test_check.py                                      22      2     10      4    81%
postreise/plot/tests/test_plot_bar_generation_max_min_actual.py         33      0      4      0   100%
postreise/plot/tests/test_plot_bar_generation_stack.py                  44      0      4      0   100%
postreise/plot/tests/test_plot_bar_generation_vs_capacity.py            64      0      4      0   100%
postreise/plot/tests/test_plot_bar_renewable_max_profile_actual.py      35      0      4      0   100%
postreise/plot/tests/test_plot_bar_shortfall.py                         43      0      4      0   100%
postreise/plot/tests/test_plot_pie_generation_vs_capacity.py            17      0      0      0   100%
postreise/plot/tests/test_plot_shadowprice_map.py                       64      0      8      0   100%
postreise/plot/tests/test_plot_sim_vs_hist.py                           26      0      4      0   100%
------------------------------------------------------------------------------------------------------
TOTAL                                                                 3806   1366   1197     48    62%

============================================================== 184 passed, 33 warnings in 29.39s ===============================================================
format recreate: /Users/brdo/CEM/PostREISE/.tox/format
format installdeps: black, isort
format installed: black==22.6.0,click==8.1.3,isort==5.10.1,mypy-extensions==0.4.3,pathspec==0.9.0,platformdirs==2.5.2,tomli==2.0.1,typing_extensions==4.3.0
format run-test-pre: PYTHONHASHSEED='3396604905'
format run-test: commands[0] | black .
Skipping .ipynb files as Jupyter dependencies are not installed.
You can fix this by running ``pip install black[jupyter]``
All done! ✨ 🍰 ✨
90 files left unchanged.
format run-test: commands[1] | isort .
Skipped 2 files
flake8 recreate: /Users/brdo/CEM/PostREISE/.tox/flake8
flake8 installdeps: flake8, pep8-naming
flake8 installed: flake8==5.0.4,mccabe==0.7.0,pep8-naming==0.13.1,pycodestyle==2.9.1,pyflakes==2.5.0
flake8 run-test-pre: PYTHONHASHSEED='3396604905'
flake8 run-test: commands[0] | flake8 postreise/
flake8 run-test: commands[1] | flake8 --extend-ignore=N816 docs/code
___________________________________________________________________________ summary ____________________________________________________________________________
  pytest: commands succeeded
  format: commands succeeded
  flake8: commands succeeded
  congratulations :)

After:

[~/CEM/PostREISE] (develop) brdo$ tox
pytest recreate: /Users/brdo/CEM/PostREISE/.tox/pytest
pytest installdeps: pipenv, git+https://github.com/Breakthrough-Energy/PowerSimData.git@ben/warnings#egg=PowerSimData
pytest installed: appdirs==1.4.4,azure-core==1.25.0,azure-storage-blob==12.13.1,azure-storage-file-datalake==12.8.0,bcrypt==3.2.2,certifi==2022.6.15,cffi==1.15.1,charset-normalizer==2.1.0,cryptography==37.0.4,distlib==0.3.5,filelock==3.8.0,fs==2.4.14,fs-azureblob==0.2.1,fs.sshfs==1.0.1,idna==3.3,isodate==0.6.1,msrest==0.7.1,networkx==2.8.5,numpy==1.23.2,oauthlib==3.2.0,pandas==1.4.3,paramiko==2.11.0,pipenv==2022.8.17,platformdirs==2.5.2,powersimdata @ git+https://github.com/Breakthrough-Energy/PowerSimData.git@b5e949b905a023213cf0d9e0669c9320491efd3d,property-cached==1.6.4,pycparser==2.21,PyNaCl==1.5.0,python-dateutil==2.8.2,pytz==2022.2.1,requests==2.28.1,requests-oauthlib==1.3.1,scipy==1.9.0,six==1.16.0,tqdm==4.64.0,typing_extensions==4.3.0,urllib3==1.26.11,virtualenv==20.16.3,virtualenv-clone==0.5.7
pytest run-test-pre: PYTHONHASHSEED='1394014111'
pytest run-test: commands[0] | pipenv sync --dev
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile.lock (e1f1a2)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 26/26 — 00:00:10
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
pytest run-test: commands[1] | pytest
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.9.11, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/pytest/.pytest_cache
rootdir: /Users/brdo/CEM/PostREISE, configfile: pytest.ini
plugins: cov-3.0.0
collected 184 items                                                                                                                                            

postreise/analyze/generation/tests/test_binding.py ....................                                                                                  [ 10%]
postreise/analyze/generation/tests/test_capacity.py ..........................                                                                           [ 25%]
postreise/analyze/generation/tests/test_costs.py ......                                                                                                  [ 28%]
postreise/analyze/generation/tests/test_curtailment.py .....................                                                                             [ 39%]
postreise/analyze/generation/tests/test_emissions.py ............                                                                                        [ 46%]
postreise/analyze/generation/tests/test_summarize.py ............                                                                                        [ 52%]
postreise/analyze/tests/test_demand.py ..                                                                                                                [ 53%]
postreise/analyze/tests/test_time.py ................................                                                                                    [ 71%]
postreise/analyze/transmission/tests/test_congestion_surplus.py ..                                                                                       [ 72%]
postreise/analyze/transmission/tests/test_utilization.py ....                                                                                            [ 74%]
postreise/plot/tests/test_canvas.py .....                                                                                                                [ 77%]
postreise/plot/tests/test_check.py ....                                                                                                                  [ 79%]
postreise/plot/tests/test_plot_bar_generation_max_min_actual.py ...                                                                                      [ 80%]
postreise/plot/tests/test_plot_bar_generation_stack.py ...                                                                                               [ 82%]
postreise/plot/tests/test_plot_bar_generation_vs_capacity.py .........                                                                                   [ 87%]
postreise/plot/tests/test_plot_bar_renewable_max_profile_actual.py ...                                                                                   [ 89%]
postreise/plot/tests/test_plot_bar_shortfall.py ...                                                                                                      [ 90%]
postreise/plot/tests/test_plot_pie_generation_vs_capacity.py .....                                                                                       [ 93%]
postreise/plot/tests/test_plot_shadowprice_map.py .........                                                                                              [ 98%]
postreise/plot/tests/test_plot_sim_vs_hist.py ...                                                                                                        [100%]

---------- coverage: platform darwin, python 3.9.11-final-0 ----------
Name                                                                 Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------------------------------------------------
postreise/__init__.py                                                    0      0      0      0   100%
postreise/analyze/__init__.py                                            0      0      0      0   100%
postreise/analyze/demand.py                                             13      0      2      0   100%
postreise/analyze/generation/__init__.py                                 0      0      0      0   100%
postreise/analyze/generation/binding.py                                 27      0      0      0   100%
postreise/analyze/generation/capacity.py                                48      0      4      0   100%
postreise/analyze/generation/costs.py                                   24      0     10      0   100%
postreise/analyze/generation/curtailment.py                             74      0     20      0   100%
postreise/analyze/generation/emissions.py                               50      1     22      1    97%
postreise/analyze/generation/summarize.py                               71      1     14      1    98%
postreise/analyze/generation/tests/__init__.py                           1      0      0      0   100%
postreise/analyze/generation/tests/test_binding.py                     119      0     10      0   100%
postreise/analyze/generation/tests/test_capacity.py                     85      0      4      0   100%
postreise/analyze/generation/tests/test_costs.py                        40      0      2      0   100%
postreise/analyze/generation/tests/test_curtailment.py                 147      0     44      0   100%
postreise/analyze/generation/tests/test_emissions.py                   104      0     20      0   100%
postreise/analyze/generation/tests/test_summarize.py                    82      0     12      0   100%
postreise/analyze/tests/__init__.py                                      0      0      0      0   100%
postreise/analyze/tests/test_demand.py                                  18      0      0      0   100%
postreise/analyze/tests/test_time.py                                   366      0    112      0   100%
postreise/analyze/time.py                                               71      0     50      1    99%
postreise/analyze/transmission/__init__.py                               0      0      0      0   100%
postreise/analyze/transmission/congestion.py                            16      0      0      0   100%
postreise/analyze/transmission/tests/__init__.py                         1      0      0      0   100%
postreise/analyze/transmission/tests/test_congestion_surplus.py         36      0      4      0   100%
postreise/analyze/transmission/tests/test_utilization.py                27      0      0      0   100%
postreise/analyze/transmission/utilization.py                           48      2      8      2    93%
postreise/data/__init__.py                                               0      0      0      0   100%
postreise/plot/__init__.py                                               0      0      0      0   100%
postreise/plot/canvas.py                                                26      0     24      0   100%
postreise/plot/check.py                                                 19      0     12      0   100%
postreise/plot/colors.py                                                 8      0      0      0   100%
postreise/plot/plot_bar_generation_max_min_actual.py                    50      1     24      1    97%
postreise/plot/plot_bar_generation_stack.py                            106     11     72      6    88%
postreise/plot/plot_bar_generation_vs_capacity.py                      132      7     74     10    92%
postreise/plot/plot_bar_renewable_max_profile_actual.py                 59      1     26      1    98%
postreise/plot/plot_bar_shortfall.py                                    89     16     48      9    80%
postreise/plot/plot_capacity_map.py                                     63     63     16      0     0%
postreise/plot/plot_carbon_bar.py                                       38     38     17      0     0%
postreise/plot/plot_carbon_map.py                                      111    111     24      0     0%
postreise/plot/plot_curtailment_ts.py                                   79     79     26      0     0%
postreise/plot/plot_energy_carbon_stack.py                              81     81     49      0     0%
postreise/plot/plot_generation_ts_stack.py                             124    124     52      0     0%
postreise/plot/plot_heatmap.py                                          43     43     14      0     0%
postreise/plot/plot_interconnection_map.py                              67     67     22      0     0%
postreise/plot/plot_lmp_map.py                                          53     53     16      0     0%
postreise/plot/plot_pie_generation_vs_capacity.py                       95     65     62     10    31%
postreise/plot/plot_powerflow_snapshot.py                               93     93     30      0     0%
postreise/plot/plot_scatter_capacity_vs_capacity_factor.py              42     42     20      0     0%
postreise/plot/plot_scatter_capacity_vs_cost_curve_slope.py             34     34     14      0     0%
postreise/plot/plot_scatter_capacity_vs_curtailment.py                  48     48     20      0     0%
postreise/plot/plot_shadowprice_map.py                                 107     52     22      0    52%
postreise/plot/plot_sim_vs_hist.py                                      59      2     26      2    95%
postreise/plot/plot_states.py                                          119    119     52      0     0%
postreise/plot/plot_tornado.py                                          22     22      4      0     0%
postreise/plot/plot_transmission_upgrades_map.py                        78     78     18      0     0%
postreise/plot/plot_utilization_map.py                                  77     77     18      0     0%
postreise/plot/projection_helpers.py                                    40     33     15      0    13%
postreise/plot/tests/__init__.py                                         0      0      0      0   100%
postreise/plot/tests/test_canvas.py                                     28      0      0      0   100%
postreise/plot/tests/test_check.py                                      22      2     10      4    81%
postreise/plot/tests/test_plot_bar_generation_max_min_actual.py         33      0      4      0   100%
postreise/plot/tests/test_plot_bar_generation_stack.py                  44      0      4      0   100%
postreise/plot/tests/test_plot_bar_generation_vs_capacity.py            64      0      4      0   100%
postreise/plot/tests/test_plot_bar_renewable_max_profile_actual.py      35      0      4      0   100%
postreise/plot/tests/test_plot_bar_shortfall.py                         43      0      4      0   100%
postreise/plot/tests/test_plot_pie_generation_vs_capacity.py            17      0      0      0   100%
postreise/plot/tests/test_plot_shadowprice_map.py                       64      0      8      0   100%
postreise/plot/tests/test_plot_sim_vs_hist.py                           26      0      4      0   100%
------------------------------------------------------------------------------------------------------
TOTAL                                                                 3806   1366   1197     48    62%


===================================================================== 184 passed in 26.63s =====================================================================
format installed: black==22.6.0,click==8.1.3,isort==5.10.1,mypy-extensions==0.4.3,pathspec==0.9.0,platformdirs==2.5.2,tomli==2.0.1,typing_extensions==4.3.0
format run-test-pre: PYTHONHASHSEED='1394014111'
format run-test: commands[0] | black .
Skipping .ipynb files as Jupyter dependencies are not installed.
You can fix this by running ``pip install black[jupyter]``
All done! ✨ 🍰 ✨
90 files left unchanged.
format run-test: commands[1] | isort .
Skipped 2 files
flake8 installed: flake8==5.0.4,mccabe==0.7.0,pep8-naming==0.13.1,pycodestyle==2.9.1,pyflakes==2.5.0
flake8 run-test-pre: PYTHONHASHSEED='1394014111'
flake8 run-test: commands[0] | flake8 postreise/
flake8 run-test: commands[1] | flake8 --extend-ignore=N816 docs/code
___________________________________________________________________________ summary ____________________________________________________________________________
  pytest: commands succeeded
  format: commands succeeded
  flake8: commands succeeded
  congratulations :)

Where to look

Changes are located in the decompose_plant_data_frame_into_areas helper function

Usage Example/Visuals

N/A

Time estimate

2min

[BainanXia]

Good to know.