Fix a race condition in the tests

[JamesWrigley]

Quick explanation of what the GC tests for RemoteChannels test does:

1. Create RemoteChannels rr and fstore on worker 1 and worker 2 respectively from the master process. At this point only the master process knows about rr and fstore.
2. Master process calls put!(fstore, rr), i.e. we remotecall worker 2 and put rr (which is owned worker 1 but is currently only known about by the master) into fstore.
3. Remotecall into worker 1 and check that it knows about rr.

Step 3 should succeed despite us never previously explicitly communicating with worker 1, because serialize(::ClusterSerializer, ::RemoteChannel) will send a message to the owner of the RemoteChannel to inform them of its existence (see send_add_client()). This happens asynchronously in step 2, and on rare occasions worker 1 would not process that message before step 3, causing the test to fail.

Now we give the check 10s to succeed.

Should fix failures seen in CI: https://github.com/JuliaParallel/DistributedNext.jl/actions/runs/11880658178/job/33104808773

Test Summary:        | Pass  Total  Time
GC tests for Futures |   42     42  4.0s
GC tests for RemoteChannels: Test Failed at /home/runner/work/DistributedNext.jl/DistributedNext.jl/test/distributed_exec.jl:280
  Expression: remotecall_fetch((k->begin
                #= /home/runner/work/DistributedNext.jl/DistributedNext.jl/test/distributed_exec.jl:280 =#
                haskey(DistributedNext.PGRP.refs, k)
            end), wid1, rrid) == true
   Evaluated: false == true

Stacktrace:
 [1] top-level scope
   @ ~/work/DistributedNext.jl/DistributedNext.jl/test/distributed_exec.jl:256
 [2] macro expansion
   @ /opt/hostedtoolcache/julia/nightly/x86/share/julia/stdlib/v1.12/Test/src/Test.jl:1700 [inlined]
 [3] macro expansion
   @ ~/work/DistributedNext.jl/DistributedNext.jl/test/distributed_exec.jl:280 [inlined]
 [4] macro expansion
   @ /opt/hostedtoolcache/julia/nightly/x86/share/julia/stdlib/v1.12/Test/src/Test.jl:679 [inlined]
Test Summary:               | Pass  Fail  Total  Time
GC tests for RemoteChannels |   10     1     11  2.8s
ERROR: LoadError: Some tests did not pass: 10 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/runner/work/DistributedNext.jl/DistributedNext.jl/test/distributed_exec.jl:255
in expression starting at /home/runner/work/DistributedNext.jl/DistributedNext.jl/test/runtests.jl:18
Package DistributedNext errored during testing
Error: Process completed with exit code 1.

I was able to reproduce the failure roughly every 30 test executions. Checked this fix by running the tests locally 100 times.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.37%. Comparing base (e990442) to head (4c00a27).
Report is 26 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master       #8   +/-   ##
=======================================
  Coverage   86.37%   86.37%           
=======================================
  Files          10       10           
  Lines        1982     1982           
=======================================
  Hits         1712     1712           
  Misses        270      270           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jpsamaroo]

Great find! Gotta love how much of a pain it is to test async behavior...