NOAA issues and testing stats

[eclare108213]

From Bob G:

I'm now largely successful in the base suite on theia with intel -- 99 pass, 6 fail. The bulk of the problem was compiler options -- though I copied a file labeled intel, it was actually gnu options.

One of the failures is the 40x4 roundrobin, which I think is invoking a problem we have long had with CICE here (theia/ NOAA R+D) -- model getting in to a state where it isn't writing output, but also isn't crashing gracefully. I killed it after an hour of no change.

Do we have any way of collecting stats on which tests are most likely to pass or fail? How long they take? It'd be convenient to put a fast, frequently failing, test up front in the queue when we're to a point of requiring all to pass.

[eclare108213]

Which file labeled intel had gnu options in it? We need to fix that.

The test suites now produce timings, which will be flagged in tests that run significantly differently from previous tests. I don't think we've started collecting statistics, but the results are available at
https://github.com/CICE-Consortium/Test-Results/wiki

[rgrumbine]

intel compiler version was the source of 5 of the 6 failures. Default on theia is intel 14. Any version 15 to 18.1.163 clears the problems.

The remaining problem is the 40x4 roundrobin

[eclare108213]

@rgrumbine Did you get the 40x4 roundrobin case to work?

[rgrumbine]

Not yet, I've been off on other things. Just starting back to the benchmark tests.

[rgrumbine]

All regression tests now pass on phase2, but refuses to compile (reprosum, no_I8 issue(?)) on phase3 and theia.

[eclare108213]

@rgrumbine, just checking in on this issue. Did you figure it out?

[rgrumbine]

We've had near zero availability on the phase3 machine, so no. Fingers crossed and all. In the mean time, theia, the R+D machine, is being replaced by Hera at the end of September. I've moved my files over there. But now the baseline tests don't all pass on Hera R+D either. Very likely an issue of intel compiler version vs/with netcdf version.

…

On Fri, Aug 30, 2019 at 3:52 PM Elizabeth Hunke ***@***.***> wrote: @rgrumbine <https://github.com/rgrumbine>, just checking in on this issue. Did you figure it out? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#155?email_source=notifications&email_token=AFQ65HMIVAMGSWJI7KVI4CLQHE635A5CNFSM4FHJJQ22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5SBPEY#issuecomment-526653331>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQ65HJ7A3M33B3SFQ73AKTQHE635ANCNFSM4FHJJQ2Q> .

-- Tel: 301-683-3747 Coupling and Dynamics Group

[rgrumbine]

All regression tests now pass on all three of phase2, phase3, and hera.