try: startup speedup

[koivunej]

Startup can take a long time. We suspect it's the initial logical size calculations. Long term solution is to not block the tokio executors but do most of I/O in spawn_blocking.

See: #4025, #4183

Short-term solution to above:

• Delay global background tasks until initial tenant loads complete
  • This works off the assumption that tenant-specific or timeline-specific already have randomized starts
• Do not consider safekeepers without additional WAL worthy of connecting to
  • This will delay the initial logical size calculation so it will most likely be started by metrics collection first
  • Reverted, see try: startup speedup #4366 (comment)
• Just limit how many init logical size calculations can we have at the same time to cores / 2

This PR is to try. I think the additions to initial tenant loading are useful regardless. We can however test this out on staging, to see if there is any positive effect.

[koivunej]

Woops, late drafting; let's see about the tests before reviewing.

[koivunej]

Regress tests passed, ready for review.

[LizardWizzard]

Looks good to try on staging

[koivunej]

Received a NAK from @arssher for:

Second part is not good, as safekeepers will never know that remote_consistent_lsn is up to date, and will push timeline to the broker forever (timeline will stay active at safekeepers).

Actions taken: a80bf2f

[github-actions]

753 tests run: 722 passed, 0 failed, 31 skipped (full report)

---

_{The comment gets automatically updated with the latest test results
793536a at 2023-05-29T17:54:53.553Z :recycle:}

[koivunej]

Very difficult to say if this helped or not. I forgot that we have quite the error margins.

ps-0.eu-west: 132s or 127s before, after 206s, 118s, 108s, 117s.
ps-1.eu-west: 91s or 93s before, after 84s, 64s, 84s.
ps-99.us-east: no change around 2s but difficult to tell, actively used for testing.

I think this might be more about warming up the s3 on repeat attempts. I'll revert the limiting part for proper measurements.

Noted:

• 3s "time difference" comparing since_creation_millis to elapsed_millis below
• Not an issue; they are not entirely comparable
• More important is that tese two were logged quite close to each other in time (0.01s)

2023-05-30T06:50:09.748448Z  INFO load{tenant_id=498d4021c9c525280ba808ac5d1c0022}: activation attempt finished since_creation_millis=81878 tenant_id=498d4021c9c525280ba808ac5d1c0022 activated_timelines=1 total_timelines=1 post_state="Active"
2023-05-30T06:50:09.753052Z  INFO Initial load completed. elapsed_millis=84217

[koivunej]

After #4368 it seems the results are the same, confirming my suspicion that the absence of spawn_blocking on the load path is to blame.