Teleport 13 Test Plan #24576

r0mant · 2023-04-13T22:11:46Z

Manual Testing Plan

Below are the items that should be manually tested with each release of Teleport.
These tests should be run on both a fresh installation of the version to be released
as well as an upgrade of the previous version of Teleport.

User accounting @atburke

Verify that active interactive sessions are tracked in /var/run/utmp on Linux.
Verify that interactive sessions are logged in /var/log/wtmp on Linux.

Combinations @strideynet

For some manual testing, many combinations need to be tested. For example, for
interactive sessions the 12 combinations are below.

Teleport with EKS/GKE @tigrato

Deploy Teleport on a single EKS cluster
Deploy Teleport on two EKS clusters and connect them via trusted cluster feature
Deploy Teleport Proxy outside GKE cluster fronting connections to it (use this script to generate a kubeconfig)
Deploy Teleport Proxy outside EKS cluster fronting connections to it (use this script to generate a kubeconfig)

Teleport with multiple Kubernetes clusters @AntonAM

Note: you can use GKE or EKS or minikube to run Kubernetes clusters.
Minikube is the only caveat - it's not reachable publicly so don't run a proxy there.

Kubernetes auto-discovery @tigrato

Kubernetes Secret Storage @AntonAM

Kubernetes Secret storage for Agent's Identity
- Install Teleport agent with a short-lived token
  - Validate if the Teleport is installed as a Kubernetes Statefulset
  - Restart the agent after token TTL expires to see if it reuses the same identity.
- Force cluster CA rotation

Kubernetes Pod RBAC @AntonAM

Kubernetes credentials forwarding @tigrato

Components running v13
- Validate the user details are shared using user impersonation
  - Proxy -> kube service
  - Proxy -> leaf proxy -> kube service
Different versions
- Proxy v13, kube service v12
- Proxy v13, leaf proxy v13, leaf kube service v12
- Proxy v13, leaf proxy v12, leaf kube service v12

Teleport with FIPS mode @atburke

Perform trusted clusters, Web and SSH sanity check with all teleport components deployed in FIPS mode.

ACME @marcoandredinis

Teleport can fetch TLS certificate automatically using ACME protocol.

Migrations @r0mant @zmb3

Migrate trusted clusters from 12.x to 13.0.0-alpha.1
- Migrate auth server on main cluster, then rest of the servers on main cluster
  SSH should work for both main and old clusters
- Migrate auth server on remote cluster, then rest of the remote cluster
  SSH should work

Command Templates

When interacting with a cluster, the following command templates are useful:

OpenSSH

# when connecting to the recording proxy, `-o 'ForwardAgent yes'` is required.
ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %[email protected] -s proxy:%h:%p" \
  node.example.com

# the above command only forwards the agent to the proxy, to forward the agent
# to the target node, `-o 'ForwardAgent yes'` needs to be passed twice.
ssh -o "ForwardAgent yes" \
  -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %[email protected] -s proxy:%h:%p" \
  node.example.com

# when connecting to a remote cluster using OpenSSH, the subsystem request is
# updated with the name of the remote cluster.
ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %[email protected] -s proxy:%h:%[email protected]" \
  node.foo.com

Teleport

# when connecting to a OpenSSH node, remember `-p 22` needs to be passed.
tsh --proxy=proxy.example.com --user=<username> --insecure ssh -p 22 node.example.com

# an agent can be forwarded to the target node with `-A`
tsh --proxy=proxy.example.com --user=<username> --insecure ssh -A -p 22 node.example.com

# the --cluster flag is used to connect to a node in a remote cluster.
tsh --proxy=proxy.example.com --user=<username> --insecure ssh --cluster=foo.com -p 22 node.foo.com

Teleport with SSO Providers

GitHub External SSO @Tener

Teleport OSS
- GitHub organization without external SSO succeeds
- GitHub organization with external SSO fails
Teleport Enterprise
- GitHub organization without external SSO succeeds
- GitHub organization with external SSO succeeds

`tctl sso` family of commands @Tener

For help with setting up sso connectors, check out the Quick GitHub/SAML/OIDC Setup Tips

tctl sso configure helps to construct a valid connector definition:

tctl sso configure github ... creates valid connector definitions
tctl sso configure oidc ... creates valid connector definitions
tctl sso configure saml ... creates valid connector definitions

tctl sso test test a provided connector definition, which can be loaded from
file or piped in with tctl sso configure or tctl get --with-secrets. Valid
connectors are accepted, invalid are rejected with sensible error messages.

Teleport Plugins @EdwardDowling

Test receiving a message via Teleport Slackbot
Test receiving a new Jira Ticket via Teleport Jira

AWS Node Joining @nklaassen

Docs

On EC2 instance with ec2:DescribeInstances permissions for local account:
TELEPORT_TEST_EC2=1 go test ./integration -run TestEC2NodeJoin
On EC2 instance with any attached role:
TELEPORT_TEST_EC2=1 go test ./integration -run TestIAMNodeJoin
EC2 Join method in IoT mode with node and auth in different AWS accounts
IAM Join method in IoT mode with node and auth in different AWS accounts

Kubernetes Node Joining @hugoShaka

Join a Teleport node running in the same Kubernetes cluster via a Kubernetes ProvisionToken

Azure Node Joining @atburke

Docs

Join a Teleport node running in an Azure VM

Cloud Labels @atburke

Create an EC2 instance with tags in instance metadata enabled
and with tag foo: bar. Verify that a node running on the instance has label
aws/foo=bar.
Create an Azure VM with tag foo: bar. Verify that a node running on the
instance has label azure/foo=bar.

Passwordless @codingllama

This feature has additional build requirements, so it should be tested with a pre-release build from Drone (eg: https://get.gravitational.com/teleport-v10.0.0-alpha.2-linux-amd64-bin.tar.gz).

This sections complements "Users -> Managing MFA devices". tsh binaries for
each operating system (Linux, macOS and Windows) must be tested separately for
FIDO2 items.

Device Trust @sshahcodes

Device Trust requires Teleport Enterprise.

This feature has additional build requirements, so it should be tested with a
pre-release build from Drone (eg:
https://get.gravitational.com/teleport-v10.0.0-alpha.2-linux-amd64-bin.tar.gz).

Client-side enrollment requires a signed tsh for macOS, make sure to use the
tsh binary from tsh.app.

A simple formula for testing device authorization is:

# Before enrollment.
# Replace with other kinds of access, as appropriate (db, kube, etc)
tsh ssh node-that-requires-device-trust
> ERROR: ssh: rejected: administratively prohibited (unauthorized device)

# Register the device.
# Get the serial number from "Apple -> About This Mac".
tctl devices add --os=macos --asset-tag=<SERIAL_NUMBER> --enroll

# Enroll the device.
tsh device enroll --token=<TOKEN_FROM_COMMAND_ABOVE>
tsh logout; tsh login

# After enrollment
tsh ssh node-that-requires-device-trust
> $

Hardware Key Support @Joerger

Hardware Key Support is an Enterprise feature and is not available for OSS.

You will need a YubiKey 4.3+ to test this feature.

This feature has additional build requirements, so it should be tested with a pre-release build from Drone (eg: https://get.gravitational.com/teleport-ent-v11.0.0-alpha.2-linux-amd64-bin.tar.gz).

Server Access

These tests should be carried out sequentially. tsh tests should be carried out on Linux, MacOS, and Windows.

tsh login as user with Webauthn login and no hardware key requirement.
Request a role with role.role_options.require_session_mfa: hardware_key - tsh login --request-roles=hardware_key_required

Assuming the role should force automatic re-login with yubikey
tsh ssh
- Requires yubikey to be connected for re-login
- Prompts for per-session MFA

Request a role with role.role_options.require_session_mfa: hardware_key_touch - tsh login --request-roles=hardware_key_touch_required

Assuming the role should force automatic re-login with yubikey
- Prompts for touch if not cached (last touch within 15 seconds)
tsh ssh
- Requires yubikey to be connected for re-login
- Prompts for touch if not cached

tsh logout and tsh login as the user with no hardware key requirement.
Upgrade auth settings to auth_service.authentication.require_session_mfa: hardware_key

Using the existing login session (tsh ls) should force automatic re-login with yubikey
tsh ssh
- Requires yubikey to be connected for re-login
- Prompts for per-session MFA

Upgrade auth settings to auth_service.authentication.require_session_mfa: hardware_key_touch

Using the existing login session (tsh ls) should force automatic re-login with yubikey
- Prompts for touch if not cached
tsh ssh
- Requires yubikey to be connected for re-login
- Prompts for touch if not cached

Other

Set auth_service.authentication.require_session_mfa: hardware_key_touch in your cluster auth settings.

Database Access: tsh proxy db --tunnel

HSM Support @nklaassen

Docs

Moderated session @marcoandredinis

Using tsh join an SSH session as two moderators (two separate terminals, role requires one moderator).

Ctrl+C in the Implement a prototype for a proxying SSH server that implements concepts expressed in readme #1 terminal should disconnect the moderator.
Ctrl+C in the Implement a functional prototype #2 terminal should disconnect the moderator and terminate the session as session has no moderator.

Using tsh join an SSH session as two moderators (two separate terminals, role requires one moderator).

t in any terminal should terminate the session for all participants.

Performance @rosstimothy @fspmarshall @espadolini

Scaling Test

Scale up the number of nodes/clusters a few times for each configuration below.

Verify that there are no memory/goroutine/file descriptor leaks
Compare the baseline metrics with the previous release to determine if resource usage has increased
Restart all Auth instances and verify that all nodes/clusters reconnect

Perform reverse tunnel node scaling tests for all backend configurations:

etcd - 10k
DynamoDB - 10k
Firestore - 10k

Perform the following additional scaling tests on DynamoDB:

10k direct dial nodes.
500 trusted clusters.

Soak Test

Run 30 minute soak test directly against direct and tunnel nodes
and via label based matching. Tests should be run against a Cloud
tenant.

tsh bench ssh --duration=30m user@direct-dial-node ls
tsh bench ssh --duration=30m user@reverse-tunnel-node ls
tsh bench ssh --duration=30m user@foo=bar ls
tsh bench ssh --duration=30m --random user@foo ls

Concurrent Session Test

Cluster with 1k reverse tunnel nodes

Run a concurrent session test that will spawn 5 interactive sessions per node in the cluster:

tsh bench web sessions --max=5000 user ls
tsh bench web sessions --max=5000 --web user ls

Verify that all 5000 sessions are able to be established.
Verify that tsh and the web UI are still functional.

Robustness

Connectivity Issues:

Verify that a lack of connectivity to Auth does not prevent access to
resources which do not require a moderated session and in async recording
mode from an already issued certificate.
Verify that a lack of connectivity to Auth prevents access to resources
which require a moderated session and in async recording mode from an already
issued certificate.
Verify that an open session is not terminated when all Auth instances
are restarted.

Teleport with Cloud Providers

AWS @tcsc

Deploy Teleport to AWS. Using DynamoDB & S3
Deploy Teleport Enterprise to AWS. Using HA Setup https://gravitational.com/teleport/docs/aws-terraform-guide/

GCP @tcsc

Deploy Teleport to GCP. Using Cloud Firestore & Cloud Storage
Deploy Teleport to GKE. Google Kubernetes engine.
Deploy Teleport Enterprise to GCP.

IBM @hugoShaka

Deploy Teleport to IBM Cloud. Using IBM Database for etcd & IBM Object Store
Deploy Teleport to IBM Cloud Kubernetes.
Deploy Teleport Enterprise to IBM Cloud.

Application Access @mdwn

Database Access @smallinsky

TLS Routing @smallinsky

Verify that teleport proxy v2 configuration starts only a single listener for proxy service, in contrast with v1 configuration. @GavinFrazar
Given configuration:

version: v2
proxy_service:
  enabled: "yes"
  public_addr: ['root.example.com']
  web_listen_addr: 0.0.0.0:3080

There should be total of three listeners, with only *:3080 for proxy service. Given the configuration above, 3022 and 3025 will be opened for other services.

lsof -i -P | grep teleport | grep LISTEN
  teleport  ...  TCP *:3022 (LISTEN)
  teleport  ...  TCP *:3025 (LISTEN)
  teleport  ...  TCP *:3080 (LISTEN) # <-- proxy service

In contrast for the same configuration with version v1, there should be additional ports 3023 and 3024.

lsof -i -P | grep teleport | grep LISTEN
  teleport  ...  TCP *:3022 (LISTEN)
  teleport  ...  TCP *:3025 (LISTEN)
  teleport  ...  TCP *:3023 (LISTEN) # <-- extra proxy service port
  teleport  ...  TCP *:3024 (LISTEN) # <-- extra proxy service port
  teleport  ...  TCP *:3080 (LISTEN) # <-- proxy service

Run Teleport Proxy in multiplex mode auth_service.proxy_listener_mode: "multiplex" @GavinFrazar
- Trusted cluster
  - Setup trusted clusters using single port setup web_proxy_addr == tunnel_addr
```
kind: trusted_cluster
spec:
  ...
  web_proxy_addr: root.example.com:443
  tunnel_addr: root.example.com:443
  ...
```
Database Access
- Verify that tsh db connect works through proxy running in multiplex mode
  - Postgres @smallinsky
  - MySQL @GavinFrazar
  - MariaDB @GavinFrazar
  - MongoDB @Tener
  - CockroachDB @Tener
  - Redis @greedy52
  - MSSQL @gabrielcorado
  - Snowflake @smallinsky
  - Elasticsearch. @Tener
  - Cassandra/ScyllaDB. @gabrielcorado
  - Oracle @smallinsky
- Verify connecting to a database through TLS ALPN SNI local proxy tsh db proxy with a GUI client. @smallinsky @GavinFrazar @Tener @greedy52
- Verify tsh proxy db with teleport proxy behind ALB. @smallinsky
Application Access @smallinsky
- Verify app access through proxy running in multiplex mode
SSH Access @GavinFrazar
- Connect to a OpenSSH server through a local ssh proxy ssh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh" [email protected]
- Connect to a OpenSSH server on leaf-cluster through a local ssh proxyssh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh --user=%r --cluster=leaf-cluster %h:%p" [email protected]
- Verify tsh ssh access through proxy running in multiplex mode
Kubernetes access: @gabrielcorado
- Verify kubernetes access through proxy running in multiplex mode
Teleport Proxy single port multiplex mode behind L7 load balancer
- Agent can join through Proxy and maintain reverse tunnel @smallinsky
- tsh login and tctl @smallinsky
- SSH Access: tsh ssh and tsh config @gabrielcorado
- Database Access: tsh proxy db and tsh db connect @gabrielcorado
- Application Access: tsh proxy app and tsh aws @gabrielcorado
- Kubernetes Access: tsh proxy kube @smallinsky

Desktop Access @ibeckermayer

Binaries compatibility @fheinecke

Verify tsh runs on:
- Windows 10
- MacOS

Machine ID

SSH @strideynet

With a default Teleport instance configured with a SSH node:

Verify you are able to create a new bot user with tctl bots add robot --roles=access. Follow the instructions provided in the output to start tbot
Verify you are able to connect to the SSH node using openssh with the generated ssh_config in the destination directory
Verify that after the renewal period (default 20m, but this can be reduced via configuration), that newly generated certificates are placed in the destination directory
Verify that sending both SIGUSR1 and SIGHUP to a running tbot process causes a renewal and new certificates to be generated
Verify that you are able to make a connection to the SSH node using the ssh_config provided by tbot after each phase of a manual CA rotation.

Ensure the above tests are completed for both:

Directly connecting to the auth server
Connecting to the auth server via the proxy reverse tunnel

DB Access @timothyb89

With a default Postgres DB instance, a Teleport instance configured with DB access and a bot user configured:

Verify you are able to connect to and interact with a database using tbot db while tbot start is running

Host users creation @lxea

Host users creation docs
Host users creation RFD

Verify host users creation functionality
- non-existing users are created automatically
- users are added to groups
  - non-existing configured groups are created
  - created users are added to the teleport-system group
- users are cleaned up after their session ends
  - cleanup occurs if a program was left running after session ends
- sudoers file creation is successful
  - Invalid sudoers files are not created
- existing host users are not modified
- setting disable_create_host_user: true stops user creation from occurring

CA rotations @espadolini

Verify the CA rotation functionality itself (by checking in the backend or with tctl get cert_authority)
- standby phase: only active_keys, no additional_trusted_keys
- init phase: active_keys and additional_trusted_keys
- update_clients and update_servers phases: the certs from the init phase are swapped
- standby phase: only the new certs remain in active_keys, nothing in additional_trusted_keys
- rollback phase (second pass, after completing a regular rotation): same content as in the init phase
- standby phase after rollback: same content as in the previous standby phase
Verify functionality in all phases (clients might have to log in again in lieu of waiting for credentials to expire between phases)
- SSH session in tsh from a previous phase
- SSH session in web UI from a previous phase
- New SSH session with tsh
- New SSH session with web UI
- New SSH session in a child cluster on the same major version
- New SSH session in a child cluster on the previous major version
- New SSH session from a parent cluster
- Application access through a browser
- Application access through curl with tsh apps login
- kubectl get po after tsh kube login
- Database access (no configuration change should be necessary if the database CA isn't rotated, other Teleport functionality should not be affected if only the database CA is rotated)

EC2 Discovery @lxea

EC2 Discovery docs

Verify EC2 instance discovery
- Only EC2 instances matching given AWS tags have the installer executed on them
- Only the IAM permissions mentioned in the discovery docs are required for operation
- Custom scripts specified in different matchers are executed
- Custom SSM documents specified in different matchers are executed
- New EC2 instances with matching AWS tags are discovered and added to the teleport cluster
  - Large numbers of EC2 instances (51+) are all successfully added to the cluster
- Nodes that have been discovered do not have the install script run on the node multiple times

IP Pinning

Add a role with pin_source_ip: true (requires Enterprise) to test IP pinning.
Testing will require changing your IP (that Teleport Proxy sees).
Docs: IP Pinning

Verify that it works for SSH Access @AntonAM
- You can access tunnel node with tsh ssh on root cluster
- You can access direct access node with tsh ssh on root cluster
- You can access tunnel node from Web UI on root cluster
- You can access direct access node from Web UI on root cluster
- You can access tunnel node with tsh ssh on leaf cluster
- You can access direct access node with tsh ssh on leaf cluster
- You can access tunnel node from Web UI on leaf cluster
- You can access direct access node from Web UI on leaf cluster
- You can download files from nodes in Web UI (small arrows at top left corner)
- If you change your IP you no longer can access nodes.
Verify that it works for Kube Access @AntonAM
- You can access Kubernetes cluster through standalone Kube service on root cluster
- You can access Kubernetes cluster through agent inside Kubernetes on root cluster
- You can access Kubernetes cluster through standalone Kube service on leaf cluster
- You can access Kubernetes cluster through agent inside Kubernetes on leaf cluster
- If you change your IP you no longer can access Kube clusters.
Verify that it works for DB Access @smallinsky
- You can access DB servers on root cluster
- You can access DB servers on leaf cluster
- If you change your IP you no longer can access DB servers.
Verify that it works for App Access @mdwn
- You can access App service on root cluster
- You can access App service on leaf cluster
- If you change your IP you no longer can access App services.
Verify that it works for Desktop Access @ibeckermayer
- You can access Desktop service on root cluster
- You can access Desktop service on leaf cluster
- If you change your IP you no longer can access Desktop services.

Documentation @ptgott @alexfornuto

Checks should be performed on the version of documentation corresponding to the
major release we're testing for. For example, for Teleport 12 release use
branch/v12 branch and make sure to select "Version 12.0" in the documentation
version switcher.

Resources

Quick GitHub/SAML/OIDC Setup Tips

The text was updated successfully, but these errors were encountered:

hugoShaka · 2023-04-18T16:03:58Z

Creating a Kubernetes join token returns no message: #24733

nklaassen · 2023-04-18T22:14:53Z

~~Web UI has no favicon: #24773~~

Fixed

nklaassen · 2023-04-18T23:29:14Z

Agentless/OpenSSH guide doesn't work: #24778

Tener · 2023-04-19T10:41:21Z

~~public_addr no longer accepts https://: #24796~~

Fixed

Joerger · 2023-04-20T00:19:59Z

~~Hardware Key support bug: #24866~~

And the fix: #24867

smallinsky · 2023-04-20T10:16:52Z

Minor UX log entry issue for tsh db connect #24879 cc: @GavinFrazar

nklaassen · 2023-04-20T18:13:01Z

~~Can't SSH to agentless nodes from Web UI: #24922~~

Fixed

Joerger · 2023-04-20T18:16:02Z

Enhanced session recording does not capture disk events - looks like a known issue so I'm checking it as complete.

nklaassen · 2023-04-20T18:56:09Z

tsh proxy ssh tries to prompt for password on invalid login when stdin is not a terminal: #24925

capnspacehook · 2023-04-20T19:54:51Z

~~Forwarding SSH agent with OpenSSH to agentless node hangs on exit: #24936~~

Fixed

rosstimothy · 2023-04-20T21:30:05Z

~~Proxy is unable to join the cluster when using the default Kube join mechanism in the Helm chart: #24941~~

Fixed

nklaassen · 2023-04-20T21:37:00Z

~~tsh attempts relogin for "ambiguous host" errors: #24943~~

Fixed

nklaassen · 2023-04-20T23:25:35Z

~~scp to agentless nodes allowed in spite of RBAC denial: #24949~~

Fixed

nklaassen · 2023-04-21T23:22:24Z

YubiHSM2 SDK version 2023.01 not supported: #25017

GavinFrazar · 2023-04-21T23:49:43Z

~~Trusted cluster OpenSSH tsh config incorrect config generation: #25018~~

This is working as expected.

Tener · 2023-04-24T12:58:53Z

DynamoDB db access requires additional configuration which isn't mentioned in docs or handled by tsh: #25063

strideynet · 2023-04-24T14:17:18Z

~~Can't openssh or Web SSH from root cluster to Agentless in leaf cluster (this request can be only executed by a proxy): #25068~~

Fixed

strideynet · 2023-04-24T14:26:29Z

~~Web SSH connections to an Agentless node do not show the node name in the session recordings list: #25072~~

Fixed

strideynet · 2023-04-24T15:48:29Z

~~Role impersonated certificates do not work with Agentless SSH proxy re-issuing #25083~~

Fixed

GavinFrazar · 2023-04-24T22:05:48Z

agentless OpenSSH guide does not explain required permissions to create node resources: #25129

AntonAM · 2023-04-25T15:45:21Z

~~Proxy can't connect to the Auth when installing Teleport with helm chart #25149~~

Fixed

ibeckermayer · 2023-04-25T17:45:57Z

Several UI bugs in Discover for Desktop Access

Joerger · 2023-04-26T02:21:52Z

~~tsh ssh -J leaf.proxy.example.com leaf-node only works when root auth/proxy is shut down - #25178~~

Fixed

ptgott · 2023-04-26T17:27:19Z

Differences in docs pages between master and v13, including the git commits on master that aren't present in v13 for each page (I can't just use git log here because backport refs aren't identical to their source refs):

$ git diff --name-only origin/master origin/branch/v13 -- docs/pages | xargs -I{} bash -c '
git log --oneline origin/branch/v13..origin/master -- {}
'
59ebccb538 docs: Login Rule k8s operator docs (#23888)
59ebccb538 docs: Login Rule k8s operator docs (#23888)
bb1f9899c1 Alphabetize the GUI Client page (#25013)
3d17be5a1d docs: add information on viewing status and logs for systemd service (#25139)
59ebccb538 docs: Login Rule k8s operator docs (#23888)

Looks like these all have outstanding backports, so I think all is good.

ptgott · 2023-04-26T19:03:38Z

@alexfornuto @avatus What do you think is the best way right now to ensure that the "Upcoming Releases" page only exists for the default docs version?

We could add a redirect from /preview/upcoming-releases/ to https://goteleport.com/docs/preview/upcoming-releases in the non-default branches, but that will quickly become difficult to maintain (unless we change the test plan to be really specific about what we need to change with each release).

Another option is deleting this page for non-default versions and not adding a redirect, but that might lead to 404s.

rosstimothy · 2023-04-26T20:47:49Z

etcd 10k Test

etcd Metrics

Teleport Metrics

Network Metrics

Soak Test

> tsh bench --duration=30m ssh root@node-agents-6dcccfd8df-22rfr-01 ls

* Requests originated: 17998
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         195 ms
50         227 ms
75         284 ms
90         354 ms
95         403 ms
99         548 ms
100        1353 ms

> tsh bench --duration=30m ssh --random root@all ls

* Requests originated: 17998
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         229 ms
50         258 ms
75         301 ms
90         354 ms
95         397 ms
99         533 ms
100        2475 ms

> tsh bench --duration=30m ssh root@foo=bar ls

* Requests originated: 17982
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         719 ms
50         1294 ms
75         4591 ms
90         18735 ms
95         24047 ms
99         28415 ms
100        35935 ms

Firestore 10k Test

Teleport Metrics

Network Metrics

Issues

Joining several thousand instances at once takes longer for all instances to successfully join. Mass joining instances results in high resource usage #25154
Cycling auth servers while all 10k nodes were joined revealed that there may be connections leaked from joining. After all instances reconnected to Auth the Proxy had far fewer connections, goroutines, and consumed less memory. Auth was stopped ~16:20 and started ~16:50 in the etcd graphs. Auth was stopped ~22:08 and started ~22:18 in the Firestore graphs. Reduced resource consumption after Auth restart #25235
Benchmarks were performed on Cloud but due to High memory usage on Auth with label based tsh bench ssh #25165 the results were very inconsistent.

ptgott · 2023-04-27T20:27:05Z

I've suggested some edits to the teleport-cluster Helm guide in the course of testing it: #25287

ptgott · 2023-04-27T20:42:42Z

In the Teleport Enterprise Cloud Getting Started guide, v13 has some UI differences from v12 for adding servers (including light mode), but I'm not going to update the screenshots this week since I have some higher-priority items to take care of. The overall server registration flow shown in the guide still works as intended.

hugoShaka · 2023-04-27T20:48:28Z

Outdated OneLogin screenshot: #25290

hugoShaka · 2023-04-27T22:30:08Z

Bucket ACL issues with terraform: #25113

flyinghermit · 2023-04-28T10:36:13Z

~~tsh ssh returns ambiguous EOF error when devices are locked : https://github.com/gravitational/teleport.e/issues/1240~~

Fixed

rosstimothy · 2023-04-28T22:06:25Z

~~tsh ssh and tsh ls not working when cluster is upgraded to alpha.2: #25365~~

Fixed

capnspacehook · 2023-05-02T20:57:15Z

Using OpenSSH ssh to connect to leaf agentless nodes results in hostkey warning: #25511

capnspacehook · 2023-05-02T22:46:27Z

~~joining agentless moderated sessions doesn't work: #25522~~

Working as expected

capnspacehook · 2023-05-03T14:57:12Z

creating moderated sessions for a leaf node is not enforced: #25557

fspmarshall · 2023-05-04T17:22:17Z

Dynamo Loadtesting

10k Tunnel Node Scaling

10k Direct Dial Node Scaling

500 Trusted Cluster Scaling

note: Elevated CPU is presumed to be due to a cache bug that was causing frequent recents of the "remote proxy" cache, and will be fixed in the final v13 release.

Benchmarks (1k Nodes)

Tunnel bench:

tsh bench ssh --duration=30m root@node-agents-77ff5cb7c7-zspkf-19 ls
                 
* Requests originated: 17999
* Requests failed: 0
   
Histogram
            
Percentile Response Duration
---------- -----------------
25         118 ms                                                                                                                 
50         127 ms
75         136 ms           
90         140 ms   
95         143 ms
99         155 ms
100        6839 ms

Tunnel Random:

tsh bench ssh --duration=30m --random root@all ls

* Requests originated: 17999
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         123 ms
50         130 ms
75         140 ms
90         150 ms
95         159 ms
99         187 ms
100        6823 ms

Label-Based:

tsh bench ssh --duration=30m root@fullname=node-agents-77ff5cb7c7-zxxg4-19 ls
                            
                 
* Requests originated: 17999
* Requests failed: 0
                                                 
Histogram
                            
Percentile Response Duration
---------- -----------------
25         164 ms
50         172 ms
75         179 ms           
90         187 ms           
95         192 ms
99         209 ms
100        6859 ms

Direct Dial:

tsh bench ssh --duration=30m root@node-agents-77ff5cb7c7-zsw4p-19 ls
            
* Requests originated: 17999
* Requests failed: 0
                                                                             
Histogram

Percentile Response Duration
---------- -----------------
25         108 ms
50         113 ms
75         116 ms
90         126 ms           
95         133 ms           
99         145 ms
100        6803 ms

r0mant added the test-plan A list of tasks required to ship a successful product release. label Apr 13, 2023

r0mant closed this as completed May 9, 2023

Teleport 13 Test Plan #24576

Teleport 13 Test Plan #24576

Comments

r0mant commented Apr 13, 2023 • edited by AntonAM Loading

Manual Testing Plan

User accounting @atburke

Combinations @strideynet

Teleport with EKS/GKE @tigrato

Teleport with multiple Kubernetes clusters @AntonAM

Kubernetes auto-discovery @tigrato

Kubernetes Secret Storage @AntonAM

Kubernetes Pod RBAC @AntonAM

Kubernetes credentials forwarding @tigrato

Teleport with FIPS mode @atburke

ACME @marcoandredinis

Migrations @r0mant @zmb3

Command Templates

OpenSSH

Teleport

Teleport with SSO Providers

GitHub External SSO @Tener

tctl sso family of commands @Tener

Teleport Plugins @EdwardDowling

AWS Node Joining @nklaassen

Kubernetes Node Joining @hugoShaka

Azure Node Joining @atburke

Cloud Labels @atburke

Passwordless @codingllama

Device Trust @sshahcodes

Hardware Key Support @Joerger

Server Access

Other

HSM Support @nklaassen

Moderated session @marcoandredinis

Performance @rosstimothy @fspmarshall @espadolini

Scaling Test

Soak Test

Concurrent Session Test

Robustness

Teleport with Cloud Providers

AWS @tcsc

GCP @tcsc

IBM @hugoShaka

Application Access @mdwn

Database Access @smallinsky

TLS Routing @smallinsky

Desktop Access @ibeckermayer

Binaries compatibility @fheinecke

Machine ID

SSH @strideynet

DB Access @timothyb89

Host users creation @lxea

CA rotations @espadolini

EC2 Discovery @lxea

IP Pinning

Documentation @ptgott @alexfornuto

Resources

hugoShaka commented Apr 18, 2023

nklaassen commented Apr 18, 2023 • edited by zmb3 Loading

nklaassen commented Apr 18, 2023

Tener commented Apr 19, 2023 • edited by zmb3 Loading

Joerger commented Apr 20, 2023 • edited by zmb3 Loading

smallinsky commented Apr 20, 2023

nklaassen commented Apr 20, 2023 • edited by zmb3 Loading

Joerger commented Apr 20, 2023

nklaassen commented Apr 20, 2023

capnspacehook commented Apr 20, 2023 • edited by zmb3 Loading

rosstimothy commented Apr 20, 2023 • edited by zmb3 Loading

nklaassen commented Apr 20, 2023 • edited by zmb3 Loading

nklaassen commented Apr 20, 2023 • edited by capnspacehook Loading

nklaassen commented Apr 21, 2023

GavinFrazar commented Apr 21, 2023 • edited by r0mant Loading

Tener commented Apr 24, 2023

strideynet commented Apr 24, 2023 • edited by capnspacehook Loading

strideynet commented Apr 24, 2023 • edited by zmb3 Loading

strideynet commented Apr 24, 2023 • edited by capnspacehook Loading

GavinFrazar commented Apr 24, 2023

AntonAM commented Apr 25, 2023 • edited by zmb3 Loading

ibeckermayer commented Apr 25, 2023

Joerger commented Apr 26, 2023 • edited by rosstimothy Loading

r0mant commented Apr 13, 2023 •

edited by AntonAM

Loading

`tctl sso` family of commands @Tener

nklaassen commented Apr 18, 2023 •

edited by zmb3

Loading

Tener commented Apr 19, 2023 •

edited by zmb3

Loading

Joerger commented Apr 20, 2023 •

edited by zmb3

Loading

nklaassen commented Apr 20, 2023 •

edited by zmb3

Loading

capnspacehook commented Apr 20, 2023 •

edited by zmb3

Loading

rosstimothy commented Apr 20, 2023 •

edited by zmb3

Loading

nklaassen commented Apr 20, 2023 •

edited by zmb3

Loading

nklaassen commented Apr 20, 2023 •

edited by capnspacehook

Loading

GavinFrazar commented Apr 21, 2023 •

edited by r0mant

Loading

strideynet commented Apr 24, 2023 •

edited by capnspacehook

Loading

strideynet commented Apr 24, 2023 •

edited by zmb3

Loading

strideynet commented Apr 24, 2023 •

edited by capnspacehook

Loading

AntonAM commented Apr 25, 2023 •

edited by zmb3

Loading

Joerger commented Apr 26, 2023 •

edited by rosstimothy

Loading

flyinghermit commented Apr 28, 2023 •

edited by rosstimothy

Loading

rosstimothy commented Apr 28, 2023 •

edited

Loading

capnspacehook commented May 2, 2023 •

edited

Loading