Rework time dilution in F3 tests and refine assertions (#862)

* Rework time dilution in F3 tests and refine assertions

Progress the clock during tests independently of condition check. Rework
the assertions to use built-in eventual assertion. Use consistent names
for the asserting functions.

Fixes #818
Fixes #847

* Reduce filality time and fix bugs in isntantiation/assertions

* Repeat flaky tests 10 times on CI with/without race detector

* Disable fail fast on CI and reduce clock advance by to 500ms

* Fix typo in adcance clock step

* Every 30 m advance by 1 s

* Adjust base manifest

* Increase timeouts to accommodate slow race detector tests

* Increase timeout further to find what works on CI

Author: masih

.github/workflows/repeat-flaky.yml

@@ -0,0 +1,34 @@
+name: Repeat Flaky Tests
+
+on:
+  pull_request:
+
+jobs:
+  test:
+    name: ${{ matrix.test }} ${{ matrix.go }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        go:
+          - 1.22
+          - 1.23
+        test:
+          - TestF3Simple
+          - TestF3WithLookback
+          - TestF3PauseResumeCatchup
+          - TestF3FailRecover
+          - TestF3DynamicManifest_WithoutChanges
+          - TestF3DynamicManifest_WithRebootstrap
+          - TestF3DynamicManifest_WithPauseAndRebootstrap
+          - TestF3DynamicManifest_RebootstrapWithCompression
+          - TestF3LateBootstrap
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
+      - name: Repeat ${{ matrix.test }}
+        run: go test -timeout 10m -count 10 -v --run='^${{ matrix.test }}' --failfast .
+      - name: Repeat ${{ matrix.test }} with race detector
+        run: go test -timeout 10m -count 10 -v --race --run='^${{ matrix.test }}' --failfast .