[Nix] adjust pytest retrys

[dougch]

Resolved issues:

part of #3841

Description of changes:

Several tests were failing under nix on arm (e.g. happy_path), but upon investigation, the pytest arguments for nix were less forgiving than our other ci jobs.

This PR aligns the default pytest retry count, from 0 to 2 globally.

These changes address the following test failures:

The following tests FAILED:
        271 - integrationv2_happy_path (Failed)
        275 - integrationv2_ocsp (Failed)
        277 - integrationv2_record_padding (Failed)
        278 - integrationv2_renegotiate (Failed)
        279 - integrationv2_session_resumption (Failed)
        280 - integrationv2_signature_algorithms (Failed)
        282 - integrationv2_version_negotiation (Failed)

An ad-hoc job running just the above on both architectures is here

Call-outs:

Unfortunately, a few tests need more retries than just 2 or a pause between retries. Iterating through these failures, I've bumped up the retry count/delay using pytest rerunfailures flaky decorator, but only on specific tests while running on arm.

For renegotiate, I also observed many of the retries were from us hitting the sub-process timeout of 5 seconds. Experimenting with huge timeouts led to landing on 8 seconds as a more reliable timeout.

Testing:

How is this change tested (unit tests, fuzz tests, etc.)? local/ci

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[goatgoose]

What are the errors when these tests fail due to flakiness? It seems like the tests could be broken if they require 5 or 7 retries in order to succeed?

[dougch]

Current working theory is that this is the test framework/interactions with subprocesses. I bumped the timeout values from 5 seconds to 60, and then noticed that the retried tests count exactly matches the number of tests running for 60 seconds. So these are being timed out by pytest, unclear why.

[goatgoose]

Interesting ok. I guess the concern is that if we make a change that causes one of these tests to become flaky, we likely won't notice it since they're retrying so many times. But I guess the extra retries only apply to arm, so as long as we continue running them in an environment with retries disabled we should notice a flaky regression.

[dougch]

To be fair, we're running these now on x86 with 2 retires, so there is no CI running these specific integrationv2 tests with 0 retires. I don't believe we're collecting retry metrics, which would be an interesting datapoint...