Pants flaky-test policy

[jsirois]

This issue just serves to document the current Pants flaky test policy. If that policy changes over time, this issue can be updated.

Flaky Test Policy

(As of 3/24/2020 as discussed here: https://groups.google.com/d/msg/pants-committers/D-9zkIa9zbQ/KO0pCwf4AAAJ):

When a test flakes in CI or the test is already associated with a flaky-test issue, feel free to:

• If the test is a python test, decorate it with:

  @pytest.mark.skip(reason="[pointer]")

  Or:

  @unittest.skip("[pointer]")

• If the test is a rust test rust add:

  #[ignore] // [pointer]

The [pointer] can just be flaky, but it could also be flaky: [url] where the url points to a github issue labelled flaky-test if you intend to work on the flaky test or would like to leave extra info about how the test flakes for someone who will pick up the work. If the skipped test has a flaky-test issue associated, please leave the issue open but further label it test-skipped. The issue can be closed when the test is deleted or fixed and un-skipped.

If a test target is flaky due to timeout it's harder to place blame on any individual test in the suite. You may not be in a good position to skip the test, but use your judgement. At the least, please also label any associated issue with test-timeout.