Playing a leaf SSH session recorded at the proxy fails

[capnspacehook]

Expected behavior:
Playing a leaf SSH session recorded at the proxy succeeds

Current behavior:
Playing a leaf SSH session recorded at the proxy fails

Bug details:

• Teleport version
  Teleport v14.0.0-beta.1

• Recreation steps
  Using tsh or the WebUI, connect to a node and try to play back the recorded session. The leaf auth server must be set to record sessions on the proxy and the root auth server must not be set to record sessions on the proxy, otherwise playing back the session succeeds.

• Debug logs

2023-09-12T18:24:58-04:00 [WEB]       DEBU Unable to find events for session 923f620d-507b-4796-8153-86c0fb64b9ec. error:[
ERROR REPORT:                                                                                                                     
Original Error: *trace.NotFoundError open /var/lib/teleport2-new/log/records/923f620d-507b-4796-8153-86c0fb64b9ec.tar: no such fil
e or directory 
Stack Trace:                                                                                                             [0/37604]
Caught:
        github.com/gravitational/teleport/lib/httplib/httplib.go:206 github.com/gravitational/teleport/lib/httplib.ConvertResponse
        github.com/gravitational/teleport/lib/auth/http_client.go:314 github.com/gravitational/teleport/lib/auth.(*HTTPClient).Get
        github.com/gravitational/teleport/lib/auth/http_client.go:918 github.com/gravitational/teleport/lib/auth.(*HTTPClient).Get
SessionEvents
        github.com/gravitational/teleport/lib/web/apiserver.go:3480 github.com/gravitational/teleport/lib/web.(*Handler).siteSessi
onEventsGet
        github.com/gravitational/teleport/lib/web/apiserver.go:3656 github.com/gravitational/teleport/lib/web.(*Handler).bindDefau
ltEndpoints.(*Handler).WithClusterAuth.func34
        github.com/gravitational/teleport/lib/httplib/httplib.go:117 github.com/gravitational/teleport/lib/web.(*Handler).bindDefa
ultEndpoints.(*Handler).WithClusterAuth.MakeHandler.MakeHandlerWithErrorWriter.func157
        github.com/julienschmidt/[email protected]/router.go:399 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
        net/http/server.go:2179 github.com/gravitational/teleport/lib/web.NewHandler.func1.StripPrefix.func1
        net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
        github.com/gravitational/teleport/lib/web/apiserver.go:463 github.com/gravitational/teleport/lib/web.NewHandler.func1
        net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
        github.com/julienschmidt/[email protected]/router.go:460 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
        github.com/gravitational/teleport/lib/web/apiserver.go:317 github.com/gravitational/teleport/lib/web.(*APIHandler).ServeHT
TP
        github.com/gravitational/[email protected]/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy
/ratelimit.(*TokenLimiter).ServeHTTP
        github.com/gravitational/[email protected]/connlimit/connlimit.go:75 github.com/gravitational/oxy/con
nlimit.(*ConnLimiter).ServeHTTP
        github.com/gravitational/teleport/lib/httplib/httplib.go:94 github.com/gravitational/teleport/lib/httplib.MakeTracingHandl
er.func1
        net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
        go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:212 go.opentelemetry.io/contrib/instrumen
tation/net/http/otelhttp.(*Handler).ServeHTTP
        net/http/server.go:2938 net/http.serverHandler.ServeHTTP
        net/http/server.go:2009 net/http.(*conn).serve
        runtime/asm_amd64.s:1650 runtime.goexit
User Message: open /var/lib/teleport2-new/log/records/923f620d-507b-4796-8153-86c0fb64b9ec.tar: no such file or directory
] web/apiserver.go:3482

---

The recorded session is recorded and uploaded to the root cluster, but when playback is requested it is searched for in the leaf cluster.

[zmb3]

@capnspacehook have we confirmed that v13 does not exhibit the same behavior?

[capnspacehook]

Yes, v13 does not have this bug, only v14.

[capnspacehook]

It seems I came to the wrong conclusions earlier: This only happens when the root cluster is not recording at the proxy, and the leaf cluster is recording at the proxy. V13 is affected actually as I was testing wrong earlier.

[russjones]

@capnspacehook Looks like this feature is working as expected? If so, can you close this ticket?

[capnspacehook]

I'm not sure how this is working as expected, to my knowledge playback of a recorded session should always succeed. I have an open PR to fix the issue just waiting on another review.