"kubectl logs -f" does not work

[gclawes]

Description

What happened:
kubectl logs -f hangs when accessing a cluster through the new kubernetes service. Omitting -f works.

After cancelling the command with ^C, this appears in the logs of the teleport kubernetes service:

ERRO [KUBERNETE] Error copying upstream response Body: context canceled forward/fwd.go:203
2020/11/29 23:31:57 http: superfluous response.WriteHeader call from github.com/gravitational/teleport/lib/kube/proxy.(*responseStatusRecorder).WriteHeader (forwarder.go:1619)

What you expected to happen:
kubectl logs -f to work as intended

How to reproduce it (as minimally and precisely as possible):

1. Configure teleport cluster (separate proxy and auth components).
2. Install teleport in kubernetes cluster with teleport-kube-agent connecting to proxy.
3. Try kubectl logs -f on any pod in cluster.
4. After command hangs, press Ctrl-C

Environment

• Teleport version (use teleport version): 5.0.0

• Tsh version (use tsh version): 5.0.0

• OS (e.g. from /etc/os-release): client: macOS, proxy: debian, auth: ubuntu, kubernetes: k3s

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): Dedicated hardware*

*NOTE: running on RPi4 cluster using a container derived from the build file modifications here: #3384 (comment)

Browser environment

• Browser Version (for UI-related issues): N/A
• Install tools: N/A
• Others: N/A

Relevant Debug Logs If Applicable

• tsh --debug
• teleport --debug

[awly]

@gclawes tested this and it looks like our k8s code will batch output on kubectl logs -f.
It should deliver logs to kubectl once there are ~32kb of fresh log data.
Can you try kubectl logs -f with a chatty pod, and wait a few minutes to see if new logs show up?

The code should also be flushing every 100ms, but it's not happening for some reason. I'll look into that.

[gclawes]

Confirmed that logs from a chatty pod stream in ~32kb increments.

[awly]

Fix will be available in 5.0.1 (likely some time this month)