Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance PETSc easyblock to run tests in parallel #2601

Merged
merged 2 commits into from
Oct 27, 2021

Conversation

casparvl
Copy link
Contributor

@casparvl casparvl commented Oct 14, 2021

The PETSc test suite can run in parallel by passing a -j <n> (see https://petsc4py.readthedocs.io/en/stable/manual/tests/) but the previous EasyBlock would just run things serially.

I've tested with 64 tasks (on a 128 core node), which speeds up the test suite from about 3.5 hours to about 30 mins. On another 16-core node, with 16 tasks, it was even down to 17 minutes (probably faster filesystem)

…th 64 cores, where it speeds up the test suite from about 3.5 hours to about 30 mins.
@akesandgren
Copy link
Contributor

This should probably be synced with easybuilders/easybuild-framework#3842 although I'm not sure if it has an impact or not.

@boegel boegel added this to the 4.5.0 milestone Oct 15, 2021
@boegel boegel changed the title Added support for running PETSc tests in parallel. Added support for running PETSc tests in parallel. Oct 15, 2021
@boegel boegel changed the title Added support for running PETSc tests in parallel. add support for running PETSc tests in parallel Oct 15, 2021
@boegel
Copy link
Member

boegel commented Oct 15, 2021

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS PETSc-3.12.4-foss-2019b-Python-3.7.4.eb
  • SUCCESS PETSc-3.12.4-foss-2020a-Python-3.8.2.eb
  • SUCCESS PETSc-3.12.4-intel-2019b-Python-2.7.16.eb
  • SUCCESS PETSc-3.12.4-intel-2019b-Python-3.7.4.eb
  • SUCCESS PETSc-3.12.4-intel-2020a-Python-3.8.2.eb
  • SUCCESS PETSc-3.14.4-foss-2020b.eb
  • SUCCESS PETSc-3.14.4-intel-2020b.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
node2618.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/c4d2263f3cbd928bf849925ddb13058c for a full test report.

@boegel
Copy link
Member

boegel commented Oct 15, 2021

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS PETSc-3.12.4-foss-2019b-Python-3.7.4.eb
  • SUCCESS PETSc-3.14.4-foss-2020b.eb
  • SUCCESS PETSc-3.15.1-foss-2021a.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3553.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/91b0f3e6729fa6103573bd25dc41a601 for a full test report.

@casparvl casparvl requested a review from boegel October 22, 2021 08:14
@casparvl
Copy link
Contributor Author

@akesandgren : I'd probably not make this PR dependend on what is decided in easybuilders/easybuild-framework#3842 , it would delay integration of this block until a decision has been taken in that other PR. Also: the notion in that PR (for now) is not to break EasyBlocks that use ec[parallel] currently, and if we in the end decide to deviate from that, we'll have to find & change all EasyBlocks in which this is used anyway. We might as well update the PETSc EasyBlock then - because what is one more? :)

@boegel
Copy link
Member

boegel commented Oct 27, 2021

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS PETSc-3.9.3-foss-2018a.eb
  • SUCCESS PETSc-3.9.3-intel-2018a.eb
  • SUCCESS PETSc-3.11.1-foss-2019a-Python-3.7.2.eb
  • SUCCESS PETSc-3.11.1-intel-2019a-Python-3.7.2.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
node2602.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/ced64516001c4fc224d22cd494771bce for a full test report.

Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel changed the title add support for running PETSc tests in parallel enhance PETSc easyblock to run tests in parallel Oct 27, 2021
@boegel boegel merged commit 18c81b9 into easybuilders:develop Oct 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants