Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop py37 support #738

Merged
merged 11 commits into from
Jan 28, 2022
Merged

Drop py37 support #738

merged 11 commits into from
Jan 28, 2022

Conversation

janosh
Copy link
Member

@janosh janosh commented Jan 19, 2022

Closes #735.

Motivation: compatibility with latest pymatgen

@janosh
Copy link
Member Author

janosh commented Jan 19, 2022

I think we can remove the py3devtest job as part of this PR as it's exactly the same as py3test except it doesn't run on the main branch. Not sure what for, seems it just burns compute for no added test coverage.

@janosh
Copy link
Member Author

janosh commented Jan 20, 2022

Note that I removed the add_tags and add_filename powerups from atomate/vasp/powerups.py that were deprecated in favor of identical functions in atomate/common/powerups.py in fe7ba87 almost a year ago. Seems like sufficient time for people to update their code. This makes this a PR a breaking change for people with py38+. People on older Python's are safe since pip will no longer update atomate for them anyway.

@janosh
Copy link
Member Author

janosh commented Jan 21, 2022

CircleCI still fails on Lobsterin._get_potcar_symbol() because fix for TypeError: a bytes-like object is required, not 'str' in latest pymatgen was not released to conda-forge yet. GHA installs from PyPI. I guess we could retry pip install in Circle with latest pymatgen release or wait for update to appear on conda.

@JaGeo
Copy link
Contributor

JaGeo commented Jan 21, 2022

also the MAGMOM fix is not yet in a released version.

@janosh
Copy link
Member Author

janosh commented Jan 25, 2022

The magmom errors are gone with the latest release of pymatgen. There are 3 failing Lobster tests remaining. The error originates in atomate itself this time. It appears that 3 RunFakeVasp calls never complete due to

----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/fireworks/core/rocket.py", line 265, in run
    m_action = t.run_task(my_spec)
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 380, in run_task
    self._verify_inputs()
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 403, in _verify_inputs
    raise ValueError(f"INCAR value of {p} is inconsistent!")
ValueError: INCAR value of LMAXMIX is inconsistent!

@JaGeo Do you have advice on what's the best fix here?

full error logs
=================================== FAILURES ===================================
___________ TestWFLobsterTestBasis.test_single_lobster_db_insertion ____________

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_lobster_db_insertion>

    def test_single_lobster_db_insertion(self):
>       self._single_lobster_db_insertion(fake=True)

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:377: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:363: in _single_lobster_db_insertion
    self._check_run(d, mode="lobsternormal")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_lobster_db_insertion>
d = None, mode = 'lobsternormal'

    def _check_run(self, d, mode):
        if mode not in ["lobsternormal"]:
            raise ValueError("Invalid mode!")
    
>       self.assertEqual(d["formula_pretty"], "CdF2")
E       TypeError: 'NoneType' object is not subscriptable

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:250: TypeError
----------------------------- Captured stdout call -----------------------------
2022-01-25 20:34:42,100 INFO Performing db tune-up
2022-01-25 20:34:42,231 INFO LaunchPad was RESET.
2022-01-25 20:34:42,531 INFO Added a workflow. id_map: {-8470: 1, -8469: 2, -8468: 3}
2022-01-25 20:34:42,640 INFO Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-42-640561
2022-01-25 20:34:42,641 INFO Launching Rocket
2022-01-25 20:34:42,865 INFO RUNNING fw_id: 3 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-42-640561
2022-01-25 20:34:42,876 INFO Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
2022-01-25 20:34:43,186 INFO Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
2022-01-25 20:34:43,189 INFO Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
2022-01-25 20:34:43,317 INFO Rocket finished
2022-01-25 20:34:43,488 INFO Performing db tune-up
2022-01-25 20:34:43,513 INFO LaunchPad was RESET.
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/fireworks/core/rocket.py", line 265, in run
    m_action = t.run_task(my_spec)
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 380, in run_task
    self._verify_inputs()
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 403, in _verify_inputs
    raise ValueError(f"INCAR value of {p} is inconsistent!")
ValueError: INCAR value of LMAXMIX is inconsistent!
------------------------------ Captured log call -------------------------------
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.
INFO     launchpad:launchpad.py:413 Added a workflow. id_map: {-8470: 1, -8469: 2, -8468: 3}
DEBUG    launchpad:launchpad.py:1136 FW with id: 3 is unique!
INFO     rocket.launcher:fw_utilities.py:164 Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-42-640561
INFO     rocket.launcher:fw_utilities.py:86 Launching Rocket
DEBUG    launchpad:launchpad.py:1136 FW with id: 3 is unique!
DEBUG    launchpad:launchpad.py:1466 Created/updated Launch with launch_id: 1
DEBUG    launchpad:launchpad.py:1496 RUNNING FW with id: 3
INFO     rocket.launcher:rocket.py:221 RUNNING fw_id: 3 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-42-640561
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
INFO     rocket.launcher:rocket.py:334 Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
INFO     rocket.launcher:fw_utilities.py:86 Rocket finished
DEBUG    launchpad:launchpad.py:775 Aggregation '[{'$match': {'$or': [{'spec._fworker': {'$exists': False}}, {'spec._fworker': None}, {'spec._fworker': 'Automatically generated Worker'}], 'state': {'$in': ['RUNNING', 'RESERVED']}}}, {'$project': {'fw_id': True, '_id': False}}]'.
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.
_______________ TestWFLobsterTestBasis.test_single_vasp_lobster ________________

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_vasp_lobster>

    def test_single_vasp_lobster(self):
        # will only test lobster_calculation_1
>       self._single_vasp_lobster(fake=True)

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:370: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_vasp_lobster>
fake = True

    def _single_vasp_lobster(self, fake=True):
        # add the workflow
    
        structure = self.struct_mp
        my_wf = get_wf_lobster_test_basis(
            structure=structure,
            c={"vasp_cmd": VASP_CMD, "DB_FILE": None},
            user_kpoints_settings={"grid_density": 100},
            delete_all_wavecars=False,
        )
        if fake:
            my_wf = use_fake_vasp(my_wf, refs_dirs_complex_vasp)
            my_wf = use_fake_lobster(my_wf, ref_dirs=refs_dirs_complex_lobster)
        else:
            my_wf = use_custodian(my_wf)
    
        self.lp.add_wf(my_wf)
    
        # run the workflow
        rapidfire(self.lp)
    
        fw = self.lp.get_fw_by_id(fw_id=1)
    
>       with open(os.path.join(fw.launches[-1].launch_dir, "task_lobster.json")) as f:
E       IndexError: list index out of range

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:303: IndexError
----------------------------- Captured stdout call -----------------------------
2022-01-25 20:34:43,901 INFO Performing db tune-up
2022-01-25 20:34:44,029 INFO LaunchPad was RESET.
2022-01-25 20:34:44,326 INFO Added a workflow. id_map: {-8473: 1, -8472: 2, -8471: 3}
2022-01-25 20:34:44,435 INFO Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-44-435193
2022-01-25 20:34:44,435 INFO Launching Rocket
2022-01-25 20:34:44,660 INFO RUNNING fw_id: 3 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-44-435193
2022-01-25 20:34:44,671 INFO Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
2022-01-25 20:34:44,844 INFO Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
2022-01-25 20:34:44,848 INFO Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
2022-01-25 20:34:44,978 INFO Rocket finished
2022-01-25 20:34:45,143 INFO Performing db tune-up
2022-01-25 20:34:45,173 INFO LaunchPad was RESET.
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/fireworks/core/rocket.py", line 265, in run
    m_action = t.run_task(my_spec)
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 380, in run_task
    self._verify_inputs()
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 403, in _verify_inputs
    raise ValueError(f"INCAR value of {p} is inconsistent!")
ValueError: INCAR value of LMAXMIX is inconsistent!
------------------------------ Captured log call -------------------------------
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.
INFO     launchpad:launchpad.py:413 Added a workflow. id_map: {-8473: 1, -8472: 2, -8471: 3}
DEBUG    launchpad:launchpad.py:1136 FW with id: 3 is unique!
INFO     rocket.launcher:fw_utilities.py:164 Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-44-435193
INFO     rocket.launcher:fw_utilities.py:86 Launching Rocket
DEBUG    launchpad:launchpad.py:1136 FW with id: 3 is unique!
DEBUG    launchpad:launchpad.py:1466 Created/updated Launch with launch_id: 1
DEBUG    launchpad:launchpad.py:1496 RUNNING FW with id: 3
INFO     rocket.launcher:rocket.py:221 RUNNING fw_id: 3 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-44-435193
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
INFO     rocket.launcher:rocket.py:334 Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
INFO     rocket.launcher:fw_utilities.py:86 Rocket finished
DEBUG    launchpad:launchpad.py:775 Aggregation '[{'$match': {'$or': [{'spec._fworker': {'$exists': False}}, {'spec._fworker': None}, {'spec._fworker': 'Automatically generated Worker'}], 'state': {'$in': ['RUNNING', 'RESERVED']}}}, {'$project': {'fw_id': True, '_id': False}}]'.
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.
_____________ TestWFLobsterTestBasis.test_single_vasp_lobster_opt ______________

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_vasp_lobster_opt>

    def test_single_vasp_lobster_opt(self):
        # will only test lobster_calculation_1
>       self._single_vasp_lobster_opt(fake=True)

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:374: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <atomate.vasp.workflows.tests.test_lobster_workflow.TestWFLobsterTestBasis testMethod=test_single_vasp_lobster_opt>
fake = True

    def _single_vasp_lobster_opt(self, fake=True):
        # add the workflow
    
        structure = self.struct_mp
        my_wf = get_wf_lobster_test_basis(
            structure=structure,
            c={"vasp_cmd": VASP_CMD, "DB_FILE": None},
            user_kpoints_settings={"grid_density": 100},
            delete_all_wavecars=False,
            additional_optimization=True,
        )
        if fake:
            my_wf = use_fake_vasp(my_wf, refs_dirs_complex_vasp_opt)
            my_wf = use_fake_lobster(my_wf, ref_dirs=refs_dirs_complex_lobster_opt)
        else:
            my_wf = use_custodian(my_wf)
    
        self.lp.add_wf(my_wf)
    
        # run the workflow
        rapidfire(self.lp)
    
        fw = self.lp.get_fw_by_id(fw_id=1)
    
>       with open(os.path.join(fw.launches[-1].launch_dir, "task_lobster.json")) as f:
E       IndexError: list index out of range

/home/runner/work/atomate/atomate/atomate/vasp/workflows/tests/test_lobster_workflow.py:334: IndexError
----------------------------- Captured stdout call -----------------------------
2022-01-25 20:34:45,228 INFO Performing db tune-up
2022-01-25 20:34:45,375 INFO LaunchPad was RESET.
2022-01-25 20:34:45,682 INFO Added a workflow. id_map: {-8477: 1, -8476: 2, -8475: 3, -8474: 4}
2022-01-25 20:34:45,694 INFO Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-45-694782
2022-01-25 20:34:45,695 INFO Launching Rocket
2022-01-25 20:34:45,728 INFO RUNNING fw_id: 4 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-45-694782
2022-01-25 20:34:45,738 INFO Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
2022-01-25 20:34:46,046 INFO Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
2022-01-25 20:34:46,050 INFO Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
2022-01-25 20:34:46,079 INFO Rocket finished
2022-01-25 20:34:46,245 INFO Performing db tune-up
2022-01-25 20:34:46,269 INFO LaunchPad was RESET.
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/fireworks/core/rocket.py", line 265, in run
    m_action = t.run_task(my_spec)
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 380, in run_task
    self._verify_inputs()
  File "/home/runner/work/atomate/atomate/atomate/vasp/firetasks/run_calc.py", line 403, in _verify_inputs
    raise ValueError(f"INCAR value of {p} is inconsistent!")
ValueError: INCAR value of LMAXMIX is inconsistent!
------------------------------ Captured log call -------------------------------
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.
INFO     launchpad:launchpad.py:413 Added a workflow. id_map: {-8477: 1, -8476: 2, -8475: 3, -8474: 4}
DEBUG    launchpad:launchpad.py:1136 FW with id: 4 is unique!
INFO     rocket.launcher:fw_utilities.py:164 Created new dir /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-45-694782
INFO     rocket.launcher:fw_utilities.py:86 Launching Rocket
DEBUG    launchpad:launchpad.py:1136 FW with id: 4 is unique!
DEBUG    launchpad:launchpad.py:1466 Created/updated Launch with launch_id: 1
DEBUG    launchpad:launchpad.py:1496 RUNNING FW with id: 4
INFO     rocket.launcher:rocket.py:221 RUNNING fw_id: 4 in directory: /home/runner/work/atomate/atomate/atomate/utils/scratch/launcher_2022-01-25-20-34-45-694782
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}}.
INFO     rocket.launcher:rocket.py:334 Task completed: {{atomate.vasp.firetasks.write_inputs.WriteVaspFromIOSet}} 
INFO     rocket.launcher:rocket.py:251 Task started: {{atomate.vasp.firetasks.run_calc.RunVaspFake}}.
INFO     rocket.launcher:fw_utilities.py:86 Rocket finished
DEBUG    launchpad:launchpad.py:775 Aggregation '[{'$match': {'$or': [{'spec._fworker': {'$exists': False}}, {'spec._fworker': None}, {'spec._fworker': 'Automatically generated Worker'}], 'state': {'$in': ['RUNNING', 'RESERVED']}}}, {'$project': {'fw_id': True, '_id': False}}]'.
DEBUG    launchpad:launchpad.py:1122 RESTARTED fw_id, launch_id to (1, 1)
INFO     launchpad:launchpad.py:933 Performing db tune-up
DEBUG    launchpad:launchpad.py:935 Updating indices...
INFO     launchpad:launchpad.py:346 LaunchPad was RESET.

@JaGeo
Copy link
Contributor

JaGeo commented Jan 25, 2022

Thanks for your work. Yeah, it looks like the INCARs produced by the LobsterSet are now different than before?! Would need to inspect further...

@JaGeo
Copy link
Contributor

JaGeo commented Jan 25, 2022

We can fix the error by adapting the test files a bit. I have worked on it. However, it shows me that I might need to work on the LobsterSet in pymatgen further to ensure that such changes do not happen that easily...

I had to change 3 different test files. Now, everything should be working. You can find my changes here: https://github.com/jageo/atomate

@janosh
Copy link
Member Author

janosh commented Jan 25, 2022

@JaGeo Excellent, thanks!

@janosh
Copy link
Member Author

janosh commented Jan 25, 2022

The Lobster INCAR fix worked. All green here, ready for review.

@JaGeo
Copy link
Contributor

JaGeo commented Jan 25, 2022

Great work @janosh
Just to add here: I would be very happy about a new version of atomate including these changes (@itsshawnzhang ?). I would need a compatible version of atomate with one of the latest pymatgen versions for a publication.

@Zhuoying
Copy link
Contributor

@janosh @JaGeo Thanks for your contributions! It looks great (all green), so I am merging it for your convenience to use it.
If there is anything wrt python3.7 drop later when running, please feel free to let us know. Thanks!

@Zhuoying Zhuoying merged commit ac89978 into hackingmaterials:main Jan 28, 2022
@janosh janosh deleted the py38+ branch January 28, 2022 19:50
@janosh
Copy link
Member Author

janosh commented Jan 29, 2022

@JaGeo I believe @computron would have to be the one to release a new version. Since this PR was a breaking change due to dropping 2 deprecated power ups, it would be a major version bump if atomate followed semantic versioning. Looking at passed versions this does not appear to be the case though, hence maintainers may wish to delay a new release instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Drop py37
3 participants