Init cooperative wait test for multiple threads

[lcw]

In the cooperative wait test multiple threads are calling MPI functions in a non-serialized way. Thus, MPI needs to be initialized with threadlevel=:multiple.

[vchuravy]

Isn't that our default?

[lcw]

Nope, the default is :serialized, see

MPI.jl/src/environment.jl

Line 114 in a57528f

function Init(;threadlevel=:serialized, finalize_atexit=true, errors_return=true)

[luraess]

Is this always the case here or only if one launches Julia with threads > 1 ?

[lcw]

Good question. I guess it would be only when julia is launched with more than one thread. But Valentin would know better.

[simonbyrne]

We went with :serialized as that is the most common use case, and not all MPI libraries support :multiple by default (or they might incur additional overhead)

[luraess]

So ideally one should always have

(Threads.nthreads() > 1) ? MPI.Init(threadlevel=:multiple) : MPI.Init()

[vchuravy]

This doesn't account for interactive threadpool usage, or foreign threads being dynamically added.

maxthreadid is probably okay, but that get's skewed by GC threads and the IO thread.