[jobs] backoff cluster teardown

[cg505]

Update the managed jobs terminate_cluster function to retry more, and back off exponentially. This should help to mitigate issues we see in high concurrency environments like botocore.exceptions.NoCredentialsError: Unable to locate credentials, which we suspect are due to high cloud API load/throttling.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • Normal sky jobs launch echo hi
  • Cancelling a job
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • /quicktest-core
  • /smoke-test managed_jobs
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[cg505]

/quicktest-core

[cg505]

/smoke-test managed_jobs

[cg505]

/quicktest-core

[cg505]

/smoke-test managed_jobs

[cg505]

will fix the tests tomorrow

[Michaelvll]

Thanks @cg505! LGTM.

[Michaelvll]

Just wondering if we really need to sleep longer than a normal attempt, it feels fine to me to sleep much shorter to ensure better performance.

[cg505]

The key value for throttling is the interval between requests, but only the backoff amount (sleep time) is increased. E.g. with initial_backoff=1, assuming each attempt takes 10s:

• Attempt 1 begins at t=0
• backoff 1s at t=10
• Attempt 2 begins at t=11 (interval: 11)
• backoff 1.6s at t=21
• Attempt 3 begins at t=22.6 (interval: 11.6s)
• backoff 2.56 at t=32.6
• Attempt 4 begins at t=35.2 (interval 12.56s)
• backoff 4.1s at t=45.2
• Attempt 5 begins at t=49.3 (interval 14.1s)

... you get the idea. The backoff doesn't really work as expected.

[Michaelvll]

I thought the main issue with the in parallel teardown is that multiple cloud requests are triggered at the exact same time, and the exponential backoff with random jitter is to have those requests to be triggered at a different timestamp? Why a smaller initial backoff not effective in the example above?

[cg505]

No, I think the issue is just load/throttling. We naturally get a bit of jitter because the controller processes will pick up the cancellation/success/failure at slightly different times anyway.

[cg505]

smoke tests look flaky