Support k8s clusters behind firewall/NAT using a single teleport cluster

[awly]

Feature Request

Similar to IoT mode, but for k8s protocol:

• start a proxy (or a set of proxies) in k8s cluster as pods
• start an externally-reachable proxy (or a set of proxies) outside of the cluster
• have the in-cluster proxy tunnel to the external proxy
• users can talk to the external proxy, which will forward requests to the appropriate in-cluster proxy over the tunnel

This should support multiple k8s clusters per a single teleport cluster.
The in-cluster proxies should advertise which k8s cluster they are representing.

Motivation

Currently, the only way to achieve something similar is with trusted clusters: one leaf cluster per k8s cluster and a shared root cluster.
This is harder to configure and manage, and auth servers in leaf clusters are unnecessary, wasting compute resources.

Who's it for?

OSS User, Pro, Enterprise

[awly]

@benarent WDYT?

[klizhentas]

I like this strategy because it fits very well with our existing flow of nodes joining the cluster, as far as I understand, your suggestion is to let teleport proxy to join the cluster.

What would help if you wrote down imaginary documentation - quickstart for this feature consisting of folks setting up teleport using new flow, so we can see what use cases will work and how folks would set this up, for example:

• User has cluster A and wants to join cluster B, the would

  • Get a join token (example command)

command here

• Start proxy and point to server

teleport start -- command here

This will allow us to elaborate on the use cases we would be able to cover and may be think through some edge cases.

[webvictim]

I like the idea overall - numerous people have expressed a desire to have something that looks like "one auth server with many Teleport proxies connected" where each proxy represents a different k8s cluster.

I do have some concerns around how we document this feature in a way people will be able to understand plus what the tsh experience will be like for bigger setups, but I think we'd be able to work through them.

[awly]

Thanks for the feedback!
Right now our biggest limitation is the implicit relationship between proxies and k8s clusters. It's 1:1 mapping that is hidden from most teleport components and API.

In terms of UX example:

• a company has 2 k8s clusters: K1 and K2
• a company has an existing Teleport setup:
  • auth server at auth.example.com (non-public)
  • proxy at proxy.example.com (public, potentially LB in front of multiple proxies)
• admin wants to setup k8s access to K1 and K2 through this existing cluster
• admin generates an invite token for a proxy:

$ tctl nodes add --roles=proxy

• admin creates a proxy k8s pod in K1, with:

$ teleport start \
    --token ${JOIN_TOKEN} \
    --ca-pin ${CA_PIN} \
    --k8s-cluster K1 \
    --auth-server 'proxy.example.com' # note: address of the public proxy, not auth server

(figuratively, it's really a yaml blob)

• admin repeats the process for K2
• a user logs into teleport with:

$ tsh login --proxy=proxy.example.com

• a user checks the available k8s clusters:

$ tsh k8s clusters
Name  Status
K1    online
K2    online

• a user chooses cluster K2:

$ tsh k8s login K2

• a user accesses K2 using kubectl
• proxy.example.com sees K2 in the client cert and routes the request to the proxy pod in K2

We can reuse the existing tunneling logic used by trusted clusters to get proxies connected. The probing logic for --auth-server from node IoT mode can be used to let the proxies "guess" what they're connecting to.

The biggest piece of new logic is teaching tsh and k8s proxies about different k8s clusters:

• proxies should announce which cluster they are for
• this information is aggregated by either auth server or other proxies
• tsh explicitly exposes switching between k8s clusters

[awly]

As a side note, it would be great to abstract all the tunneling/connectivity features away from IoT, trusted clusters and k8s clusters. I don't know how yet, but those connectivity details should be hidden from the user.

[awly]

This was implemented in #4611
Docs are still pending.