refactor: clean up after capacity scaling tests

[danielolsen]

Purpose

Avoid a diff when running tests. We cannot simply update .gitignore because it will not ignore changes to files that area already tracked in the repo.
Closes #185.

What is the code doing

• Deleting all files in the powersimdata/design/demo/data/save_files folder after tests, by wrapping tests in a try/finally.
• Deleting old (un-needed?) comments

Time to review

5 minutes.

[rouille]

I have the following error when running the tests:

(v1) [~/REM/PowerSimData] (capacity_scaling_test_cleanup) brdo$ pytest powersimdata/design/tests/test_object_persistence.py 
============================================================================================== test session starts ==============================================================================================
platform darwin -- Python 3.6.5, pytest-4.5.0, py-1.8.0, pluggy-0.13.0
rootdir: /Users/brdo/REM/PowerSimData, inifile: pytest.ini
collected 3 items                                                                                                                                                                                               

powersimdata/design/tests/test_object_persistence.py .FF                                                                                                                                                  [100%]

=================================================================================================== FAILURES ====================================================================================================
_____________________________________________________________________________________ test_write_read_json_of_target_object _____________________________________________________________________________________

    def test_write_read_json_of_target_object():
        try:
            # create Pacific
            pacific_solar = Resource('solar', 3)
            pacific_solar.set_capacity(0.25, 3700, 0.215379)
            pacific_solar.set_generation(7000)
            pacific_solar.set_curtailment(0.138483)
            pacific_solar.set_addl_curtailment(0)
    
            pacific_wind = Resource('wind', 3)
            pacific_wind.set_capacity(0.4, 3600, 0.347855)
            pacific_wind.set_generation(11000)
            pacific_wind.set_curtailment(0.130363)
            pacific_wind.set_addl_curtailment(0)
    
            pacific_geo = Resource('geo', 3)
            pacific_geo.set_capacity(1, 4000, 1)
            pacific_geo.set_generation(8000)
            pacific_geo.set_curtailment(0)
            pacific_geo.set_addl_curtailment(0)
    
            pacific_hydro = Resource('hydro', 3)
            pacific_hydro.set_capacity(1, 3900, 1)
            pacific_hydro.set_generation(7000)
            pacific_hydro.set_curtailment(0)
            pacific_hydro.set_addl_curtailment(0)
    
            pacific_nuclear = Resource('nuclear', 3)
            pacific_nuclear.set_capacity(1, 4300, 1)
            pacific_nuclear.set_generation(6500)
            pacific_nuclear.set_curtailment(0)
            pacific_nuclear.set_addl_curtailment(0)
    
            pacific_target = TargetManager('Pacific', 0.25, 'renewables', 200000)
            pacific_target.set_allowed_resources(['solar', 'wind', 'geo'])
            pacific_target.add_resource(pacific_solar)
            pacific_target.add_resource(pacific_wind)
            pacific_target.add_resource(pacific_geo)
            pacific_target.add_resource(pacific_hydro)
            pacific_target.add_resource(pacific_nuclear)
    
>           pacific_target.save_target_as_json()

powersimdata/design/tests/test_object_persistence.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <powersimdata.design.clean_capacity_scaling.TargetManager object at 0x1274f5fd0>

    def save_target_as_json(self):
        """Saves target object as indented JSON file named by region name.
    
        """
        print(os.getcwd())
        json_file = open(os.path.join(
            os.path.dirname(os.path.abspath(__file__)),
            "demo", "data", "save_files",
>           self.region_name+".json"), "w")
E       FileNotFoundError: [Errno 2] No such file or directory: '/Users/brdo/REM/PowerSimData/powersimdata/design/demo/data/save_files/Pacific.json'

powersimdata/design/clean_capacity_scaling.py:801: FileNotFoundError
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
/Users/brdo/REM/PowerSimData
____________________________________________________________________________________ test_write_read_pickle_of_target_object ____________________________________________________________________________________

    def test_write_read_pickle_of_target_object():
        try:
            # create Pacific
            pacific_solar = Resource('solar', 3)
            pacific_solar.set_capacity(0.25, 3700, 0.215379)
            pacific_solar.set_generation(7000)
            pacific_solar.set_curtailment(0.138483)
            pacific_solar.set_addl_curtailment(0)
    
            pacific_wind = Resource('wind', 3)
            pacific_wind.set_capacity(0.4, 3600, 0.347855)
            pacific_wind.set_generation(11000)
            pacific_wind.set_curtailment(0.130363)
            pacific_wind.set_addl_curtailment(0)
    
            pacific_geo = Resource('geo', 3)
            pacific_geo.set_capacity(1, 4000, 1)
            pacific_geo.set_generation(8000)
            pacific_geo.set_curtailment(0)
            pacific_geo.set_addl_curtailment(0)
    
            pacific_hydro = Resource('hydro', 3)
            pacific_hydro.set_capacity(1, 3900, 1)
            pacific_hydro.set_generation(7000)
            pacific_hydro.set_curtailment(0)
            pacific_hydro.set_addl_curtailment(0)
    
            pacific_nuclear = Resource('nuclear', 3)
            pacific_nuclear.set_capacity(1, 4300, 1)
            pacific_nuclear.set_generation(6500)
            pacific_nuclear.set_curtailment(0)
            pacific_nuclear.set_addl_curtailment(0)
    
            pacific_target = TargetManager('Pacific', 0.25, 'renewables', 200000)
            pacific_target.set_allowed_resources(['solar', 'wind', 'geo'])
            pacific_target.add_resource(pacific_solar)
            pacific_target.add_resource(pacific_wind)
            pacific_target.add_resource(pacific_geo)
            pacific_target.add_resource(pacific_hydro)
            pacific_target.add_resource(pacific_nuclear)
    
>           pacific_target.save_target_as_pickle()

powersimdata/design/tests/test_object_persistence.py:157: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <powersimdata.design.clean_capacity_scaling.TargetManager object at 0x127590390>

    def save_target_as_pickle(self):
        """Saves target object as pickle file named by region name.
    
        """
        print(os.getcwd())
        json_file = open(os.path.join(
            os.path.dirname(os.path.abspath(__file__)),
            "demo", "data", "save_files",
>           self.region_name+".pkl"), "wb")
E       FileNotFoundError: [Errno 2] No such file or directory: '/Users/brdo/REM/PowerSimData/powersimdata/design/demo/data/save_files/Pacific.pkl'

powersimdata/design/clean_capacity_scaling.py:815: FileNotFoundError
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
/Users/brdo/REM/PowerSimData
====================================================================================== 2 failed, 1 passed in 0.54 seconds =======================================================================================

I think the problem is that there is no save_files folder. Perhaps, the files could be saved into powersimdata/design/demo/data instead (one level down).

[danielolsen]

The new commits specify the save_files path better (pointing to the right place even if you don't run pytest at package root), create the folder at the start of each test if it does not already exist, and delete the folder after all tests in the module have been completed.

[rouille]

It works now:

(v1) [~/REM/PowerSimData] (capacity_scaling_test_cleanup) brdo$ pytest . -m 'not integration'
============================================================================================== test session starts ==============================================================================================
platform darwin -- Python 3.6.5, pytest-4.5.0, py-1.8.0, pluggy-0.13.0
rootdir: /Users/brdo/REM/PowerSimData, inifile: pytest.ini
collected 147 items / 4 deselected / 143 selected                                                                                                                                                               

powersimdata/design/tests/test_object_persistence.py ...                                                                                                                                                  [  2%]
powersimdata/design/tests/test_resource_target_manager.py ...............                                                                                                                                 [ 12%]
powersimdata/design/tests/test_scenario_info.py ........                                                                                                                                                  [ 18%]
powersimdata/design/tests/test_strategies.py ................                                                                                                                                             [ 29%]
powersimdata/design/tests/test_target_manager_input.py ...                                                                                                                                                [ 31%]
powersimdata/design/tests/test_transmission.py .....................................                                                                                                                      [ 57%]
powersimdata/input/tests/test_change_table.py ..........                                                                                                                                                  [ 64%]
powersimdata/input/tests/test_grid.py ............................                                                                                                                                        [ 83%]
powersimdata/input/tests/test_transform_grid.py ............                                                                                                                                              [ 92%]
powersimdata/tests/test_mocks.py ......                                                                                                                                                                   [ 96%]
powersimdata/utility/tests/test_transfer_data.py .....                                                                                                                                                    [100%]

=================================================================================== 143 passed, 4 deselected in 18.61 seconds ===================================================================================

[danielolsen]

@dmuldrew is this consistent with how the tests are intended to operate? Can the comments be deleted?

[dmuldrew]

I hadn't heard of python package glob yet

[danielolsen]

@dmuldrew I picked that up from @merrielle! I think glob far predates python, and it's just a python implementation of something much older (like regex).

[rouille]

we use glob in the scenario framework. It is great.