App access requests periodically fail with 500 error

[r0mant]

Application access uses 5-minute chunks to store the session events. The audit log writer is attached to the session chunk and is closed after each 5 minute interval when the session chunk "expires":

https://github.com/gravitational/teleport/blob/v9.2.4/lib/srv/app/session.go#L134

If the session expires and the audit writer is closed while the application request is being processed, app server transport will fail to emit the audit event here:

https://github.com/gravitational/teleport/blob/v9.2.4/lib/srv/app/transport.go#L157

resulting in the error being returned and the client receiving 500 response, even though the request actually made it to the target application.

This can be reproduced very easily locally by reducing the session TTL to something like 5 seconds (in the code) and with a simple "header dumper" application:

➜  apps git:(master) ✗ curl \
  --cacert /Users/r0mant/.tsh/keys/root.gravitational.io/cas/root.pem \
  --cert /Users/r0mant/.tsh/keys/root.gravitational.io/alice-app/root/dumper-x509.pem \
  --key /Users/r0mant/.tsh/keys/root.gravitational.io/alice \
  https://dumper.root.gravitational.io:3080
Internal Server Error%

2022-05-27T14:08:30-07:00             ERRO "Error forwarding to /, err: failed to emit audit event\n\taudit writer is closed" forward/fwd.go:181
2022-05-27T14:08:30-07:00 [APP:WEB]   INFO Round trip: GET /, code: 500, duration: 2.005467s tls:version: 304, tls:resume:false, tls:csuite:1303, tls:server:dumper.root.gravitational.io forward/fwd.go:187

The solution that comes to mind is that the session writer should not close while there are requests in flight.

This issue is the likely cause of https://github.com/gravitational/cloud/issues/1273.

[justinas]

This has been fixed in master by #13145, and has also been backported to v9 by #13469 and made it to the 9.3.6 release.

The changes did not backport cleanly to v8/v7, as the app proxy code seems to have undergone some changes since those branched-off. If this bug is deemed important enough, we could spend some time to manually backport to older versions.