If microphysics burn fails, retry hydro

[psharda]

Description

If the (VODE) burn fails in microphysics, we should retry hydro with a reduced timestep, just like we retry hydro with a reduced timestep if the CFL is violated.

These changes have been successfully tested on CPUs and GPUs.

Related issues

PopIII test on NVIDIA GPUs crashes due to a burn failure in VODE. This did not occur on AMD GPUs at the same timestep.

Resolves #372 for chemistry

Checklist

Before this pull request can be reviewed, all of these tasks should be completed. Denote completed tasks with an x inside the square brackets [ ] in the Markdown source below:

• I have added a description (see above).
• I have added a link to any related issues see (see above).
• I have read the Contributing Guide.
• I have added tests for any new physics that this PR adds to the code.
• I have tested this PR on my local computer and all tests pass.
• I have manually triggered the GPU tests with the magic comment /azp run.
• I have requested a reviewer for this PR.

[psharda]

@BenWibking I have a conceptual problem with my changes: I can reduce the timestep and retry the burn (for a certain max_retries). Say, the burn is successful on the first or second retry. But I can't just exit chemistry and move on to the next timestep (or module) because the burn was only done for 1/2 or 1/4 of the actual timestep. Don't I need to do the burn twice or four times at the reduced step so that at the end, I have my chemical state for the full timestep?

[github-actions]

clang-tidy made some suggestions

[BenWibking]

@BenWibking I have a conceptual problem with my changes: I can reduce the timestep and retry the burn (for a certain max_retries). Say, the burn is successful on the first or second retry. But I can't just exit chemistry and move on to the next timestep (or module) because the burn was only done for 1/2 or 1/4 of the actual timestep. Don't I need to do the burn twice or four times at the reduced step so that at the end, I have my chemical state for the full timestep?

Yes that's correct. However you should just use the existing code in the hydro update that does this. VODE internally already does what you've done here. The thing that will make it more robust is if you return a boolean success flag to the hydro update so that the whole hydro update is triggered.

[psharda]

@BenWibking done. Tested it on my computer on cpus and it works. But I am wondering if there is a better way to deal with the burns in the first and the second halves of hydro? Currently, I just check if the burn in either half failed, and if it did, I ask for a hydro retry with a reduced timestep.

[github-actions]

clang-tidy made some suggestions

[BenWibking]

@BenWibking done. Tested it on my computer on cpus and it works. But I am wondering if there is a better way to deal with the burns in the first and the second halves of hydro? Currently, I just check if the burn in either half failed, and if it did, I ask for a hydro retry with a reduced timestep.

That's the approach I would take. I don't see what is wrong with this.

[psharda]

Success!

[Level 5 step 732] ADVANCE with time = 4.1376331850e+13 dt = 6.9429380442e+08
WARNNING: Unsuccessful burn. Retrying hydro step.
        >> WARNING: Hydro advance failed on level 5
        >> Re-trying hydro advance at level 5 with reduced timestep (nsubsteps = 2, dt_new = 347146902.2)
[Level 5 step 732] Advanced 1572864 cells

[psharda]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[psharda]

Can be approved now @BenWibking

[BenWibking]

One minor change in the first Strang half-step, otherwise everything looks good.

[psharda]

/azp run

resolved

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).