tests: failures in test_timeline_archival_chaos

[jcsp]

Multiple failure modes:

• A) Assertion on timeline states: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-10379/12768980436/index.html#testresult/1897a41a54810ddd/retries
• B) Didn't make it through enough iterations: https://neon-github-public-dev.s3.amazonaws.com/reports/main/12748358128/index.html#testresult/327cbe4cb33ec42/retries
• C) Log error for failure uploading manifest in compaction: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-10244/12748633778/index.html#testresult/550fccea6ef552ad/retries
• D) Failed to clean up uninitialized timeline directory https://neon-github-public-dev.s3.amazonaws.com/reports/main/12966756686/index.html#testresult/5523f7d15f46f7f7/retries
• E) failed to freeze and flush: cannot flush frozen layers when flush_loop is not running, state is Exited https://neon-github-public-dev.s3.amazonaws.com/reports/pr-10558/13030683056/index.html#/testresult/916489f58539180
• F) leftover tasks warning WARN shutdown_pageserver{exit_code=0}: stopping left-over name="timeline_delete" tenant_shard_id=4fb531bf3b088b282c8a950ec97545e7 timeline_id=Some(cf17d70625d82ca45684b9bd7d2ce30f) kind=TimelineDeletionWorker\n')https://neon-github-public-dev.s3.amazonaws.com/reports/pr-10608/13072224498/index.html#/testresult/ffcc90cce37c63d0

[jcsp]

Failure mode C: #10524

[jcsp]

Failure mode D: #10532

[jcsp]

Failed mode E: #10594

[jcsp]

Failure mode B: #10595

[jcsp]

Failure mode F: #10702

[jcsp]

Case A: the test is detecting that something has gone offloaded while it is really offload_ing_ -- presumably the pageserver shows it in the list of offloaded things before that manifest is persistent.

In the general case, this is a test bug: the pageserver is allowed to un-offload things if it wants to, it's not illegal. However, what the test is really trying to verify is that things don't un-offload across restarts, so we don't want to drop the check entirely.

[jcsp]

There's a TODO in the code relevant to case A:

    // Last step: mark timeline as offloaded in S3
    // TODO: maybe move this step above, right above deletion of the local timeline directory,
    // then there is no potential race condition where we partially offload a timeline, and
    // at the next restart attach it again.
    // For that to happen, we'd need to make the manifest reflect our *intended* state,
    // not our actual state of offloaded timelines.
    tenant.store_tenant_manifest().await?;