[WIP] [IGNORE] proxy: conrad is messing around

[conradludgate]

Problem

Error handling has always been a pain in the ass (#5599). I think much of that is due to errors being so deep in the stack.

Additionally, the full handle_client future is 8KiB, which seems wasteful once we get to proxy_pass.

This has been an idea I wanted to do for ages now, this weekend I got the courage and motivation to try it out.

Summary of changes

Refactor proxy to use a state machine paradigm. Proxy processes requests in a very linear way, so these state machines are straightforward.

The main benefit here is that each stage in the state machine is only 1 step away from the error handling for that session. There are no complicated error chains to worry about. All errors are now ReportableErrors. These have a direct blame associated with them (as introduced in my RequestMonitoring PR).

Current state machine flows:

---
title: Standard Proxy
---
stateDiagram-v2
    [*] --> NeedsHandshake
    NeedsHandshake --> NeedsAuthentication

    NeedsAuthentication --> NeedsPasswordHack: if no SNI or endpoint option
    NeedsPasswordHack --> NeedsAuthSecret

    NeedsAuthentication --> NeedsAuthSecret
    NeedsAuthSecret --> NeedsWakeCompute
    NeedsWakeCompute --> NeedsCompute
    NeedsCompute --> ProxyPass
    ProxyPass --> [*]

Loading

---
title: Passwordless Proxy
---
stateDiagram-v2
    [*] --> NeedsHandshake
    NeedsHandshake --> NeedsAuthentication
    NeedsAuthentication --> NeedsLinkAuthentication
    NeedsLinkAuthentication --> NeedsLinkAuthenticationResponse
    NeedsLinkAuthenticationResponse --> NeedsCompute
    NeedsCompute --> ProxyPass
    ProxyPass --> [*]

Loading

The core of this setup is in state_machine.rs with the Stage trait. Stage essentially describes Stage -> Option<Result<Stage>> as a step in the state machine.

In this setup, the largest future (NeedsCompute) is 904 bytes, whereas ProxyPass, the final stage, is only 224 bytes. This means that 10000 active database connections go down from 85.4MiB to 2.13MiB (this is not counting additional allocations each connection might hold)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2280 tests run: 2189 passed, 0 failed, 91 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_statvfs_pressure_usage: debug

Code coverage (full report)

• functions: 54.5% (10663 of 19562 functions)
• lines: 81.5% (61118 of 74983 lines)

---

_{The comment gets automatically updated with the latest test results
7fed0ba at 2024-01-23T14:09:38.196Z :recycle:}