Teleport 15 Test Plan #36663
Labels
test-plan
A list of tasks required to ship a successful product release.
Comments
r0mant
added
the
test-plan
Desktop Access @probakowski @ibeckermayer
Binaries / OS compatibility @fheineckeVerify that our software runs on the minimum supported OS versions as per Windows @ravicious
macOS @camscale
Linux @camscale
Machine ID @strideynet
With an SSH node registered to the Teleport cluster:
With a Postgres DB registered to the Teleport cluster:
With a Kubernetes cluster registered to the Teleport cluster:
With a HTTP application registered to the Teleport cluster:
Host users creation @lxeaHost users creation docs
CA rotations @fspmarshall
Proxy Peering
EC2 Discovery @marcoandredinis
Azure Discovery @marcoandredinis
GCP Discovery @atburke
IP Pinning @AntonAMAdd a role with
Assist @jakule @ryanclark @tigrato @xacrimon @justinasAssist is not supported by
Resources |
|
Certificate presented by root proxy for leaf agentless nodes is not trusted by client #36801 |
|
No audit log entries when SCP denied: #36820 |
|
Desktop session clipboard and directory sharing icon state is unclear: #36825 |
|
Login failure events aren't emitted if MFA is enabled: #36837 |
|
#31410, which I reported for v14, is still not fixed |
|
tsh does not work on Windows 10 rev. 1607. This is a regression introduced somewhere between 13.0.0 and 13.4.15. [Alan]: Fixed by #36859. |
|
agent locking is broken: https://github.com/gravitational/teleport-private/issues/1340 |
|
A minor issue with tsh in FIPS mode (not sure if related to FIPS itself, though): #36922 |
|
Unable to use RDP in FIPS mode: #36928 |
|
Nit: |
|
Helm chart |
|
Jump host fails with unknown certificate authority: #36964 |
|
default |
Performance Test ResultsCloudLoad Tests30k Resources
5K Concurrent Sessions
Soak TestsOrigin: us-east-1 Target: us-east-1/usr/local/bin/tsh bench ssh --duration=30m root@node-agents-67588c8d58-26f2m-00 ls
* Requests originated: 17998
* Requests failed: 0
Histogram
Percentile Response Duration
---------- -----------------
25 219 ms
50 224 ms
75 229 ms
90 234 ms
95 236 ms
99 245 ms
100 1228 ms
/usr/local/bin/tsh bench ssh --duration=30m root@fullname=node-agents-67588c8d58-26f2m-00 ls
* Requests originated: 17996
* Requests failed: 0
Histogram
Percentile Response Duration
---------- -----------------
25 481 ms
50 488 ms
75 495 ms
90 501 ms
95 505 ms
99 514 ms
100 1522 ms
/usr/local/bin/tsh bench ssh --duration=30m --random root@all ls
* Requests originated: 17998
* Requests failed: 0
Histogram
Percentile Response Duration
---------- -----------------
25 218 ms
50 224 ms
75 232 ms
90 242 ms
95 251 ms
99 289 ms
100 784 ms
Origin: us-east-1 Target: us-west-2/usr/local/bin/tsh bench ssh --duration=30m root@ip-172-31-35-106 ls
* Requests originated: 17993
* Requests failed: 0
Histogram
Percentile Response Duration
---------- -----------------
25 783 ms
50 801 ms
75 827 ms
90 850 ms
95 858 ms
99 878 ms
100 1102 ms
Origin: us-west-2 Target: us-east-1/usr/local/bin/tsh bench ssh --duration=30m root@node-agents-67588c8d58-26f2m-00 ls
* Requests originated: 17992
* Requests failed: 0
Histogram
Percentile Response Duration
---------- -----------------
25 823 ms
50 836 ms
75 845 ms
90 851 ms
95 858 ms
99 870 ms
100 1980 ms
etcd130k Resources
500 Trusted Clusters
Postgres130k Resources
Note The postgres backend exhibited some odd memory usage behaviors that were not observed when testing other backends. Firestore130k Resources
Footnotes |
|
IBM changed its admin etcd login process, docs are not working anymore: #37059 |
|
Cancelling running query doesn't work for CockroachDB #37074 |
|
Found a seemingly Cloud-specific issue with inviting new users to a cluster. #37159 |
|
Okta integration installer doesn't create SSO connector: #37160 |
|
Goroutine leak on PostgreSQL database access: #37219 |
Database Access load test (PostgreSQL and MySQL)Setup (same as previous test)EKS with a single node group:
Teleport cluster (all deployed on the EKS cluster):
Databases:
Note: Databases were configured using discovery running inside the database agent.
MySQL10 connections/second
50 connections/second
PostgreSQL10 connections/second
50 connections/second
|
SSH Connection ResumptionVerify that SSH works, and that resumable SSH is not interrupted across a Teleport Cloud tenant upgrade.
(resumed connections to peered nodes work with a local tsh after #37352) Verify that SSH works, and that resumable SSH is not interrupted across a control plane restart (of either the root or the leaf cluster).
|
|
The "SSH gRPC" transport client code doesn't unblock the connection on |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Manual Testing Plan
Below are the items that should be manually tested with each release of Teleport.
These tests should be run on both a fresh installation of the version to be released
as well as an upgrade of the previous version of Teleport.
Adding nodes to a cluster @lxea
Labels @lxea
Trusted Clusters @bl-nero
RBAC @bl-nero
Make sure that invalid and valid attempts are reflected in audit log. Do this with both Teleport and Agentless nodes.
Verify that custom PAM environment variables are available as expected. @atburke
Users @codingllama
With every user combination, try to login and signup with invalid second
factor, invalid password to see how the system reacts.
WebAuthn in the release
tshbinary is implemented using libfido2 forlinux/macOS. Ask for a statically built pre-release binary for realistic
tests. (
tsh fido2 diagshould work in our binary.) Webauthn in Windowsbuild is implemented using
webauthn.dll. (tsh webauthn diagwithsecurity key selected in dialog should work.)
Touch ID requires a signed
tsh, ask for a signed pre-release binary so youmay run the tests.
Windows Webauthn requires Windows 10 19H1 and device capable of Windows
Hello.
Adding Users Password Only
Adding Users OTP
Adding Users WebAuthn
Adding Users via platform authenticator
Managing MFA devices
tsh mfa addtsh mfa addtsh mfa addtsh mfa lstsh mfa rmtsh mfa rmsecond_factor: oninauth_service, should failsecond_factor: optionalinauth_service, should succeedLogin Password Only
Login with MFA
tsh mfa addLogin OIDC
Login SAML
Login GitHub
Deleting Users
Backends @rosstimothy
Session Recording @atburke
Enhanced Session Recording @jakule
disk,commandandnetworkevents are being logged.enhanced_recordingrole option.Auditd @jakule
teleport/lib/auditd/common.go
Lines 25 to 34 in 7744f72
Audit Log @rosstimothy
Audit log with dynamodb
Audit log with Firestore
Failed login attempts are recorded
Interactive sessions have the correct Server ID
server_idis the ID of the node in "session_recording: node" modeserver_idis the ID of the node in "session_recording: proxy" modeforwarded_byis the ID of the proxy in "session_recording: proxy" modeNode/Proxy ID may be found at
/var/lib/teleport/host_uuidin thecorresponding machine.
Node IDs may also be queried via
tctl nodes ls.Exec commands are recorded
scpcommands are recordedSubsystem results are recorded
Subsystem testing may be achieved using both
Recording Proxy mode
and
OpenSSH integration.
Assuming the proxy is
proxy.example.com:3023andnode1is a node runningOpenSSH/sshd, you may use the following command to trigger a subsystem audit
log:
sftp -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %[email protected] -s proxy:%h:%p" root@node1External Audit Storage @nklaassen
External Audit Storage must be tested on an Enterprise Cloud tenant.
Instructions for deploying a custom release to a cloud staging tenant: https://github.com/gravitational/teleport.e/blob/master/dev-deploy.md
tsh play <session-id>worksInteract with a cluster using
tsh@capnspacehookThese commands should ideally be tested for recording and non-recording modes as they are implemented in a different ways.
Interact with a cluster using
ssh@strideynetMake sure to test both recording and regular proxy modes.
Verify proxy jump functionality @atburke Jump host fails with unknown certificate authority #36964
Log into leaf cluster via root, shut down the root proxy and verify proxy jump works.
Interact with a cluster using the Web UI @capnspacehook
X11 Forwarding @marcoandredinis
xeyesandxclip:apt install x11-apps xclipxeyes. Thenbrew install xclip.ssh_service.x11.enabled = yestsh ssh -X user@node xeyestsh ssh -X root@node xeyestsh ssh -Y server01 "echo Hello World | xclip -sel c && xclip -sel c -o"should print "Hello World"tsh ssh -X server01 "echo Hello World | xclip -sel c && xclip -sel c -o"should fail with "BadAccess" X errorUser accounting @atburke
/var/run/utmpon Linux./var/log/wtmpon Linux.Combinations @capnspacehook
For some manual testing, many combinations need to be tested. For example, for
interactive sessions the 12 combinations are below.
Add an agentless Node in a local cluster.
Add a Teleport Node in a local cluster.
Add an agentless Node in a remote (leaf) cluster.
Add a Teleport Node in a remote (leaf) cluster.
Teleport with EKS/GKE @AntonAM
Teleport with multiple Kubernetes clusters @tigrato
Note: you can use GKE or EKS or minikube to run Kubernetes clusters.
Minikube is the only caveat - it's not reachable publicly so don't run a proxy there.
tsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas both clusterstsh kube loginkubectl get nodes,kubectl exec -it $SOME_POD -- shon the new clustertsh login, check thattsh kube lshas all clustersnameandlabelsStep 2login value matching the rowsnamecolumnnameorlabelsin the search bar worksnamecolumKubernetes auto-discovery @AntonAM
tctl create.tctl create -f.tctl rm.Kubernetes Secret Storage @AntonAM
StatefulsetKubernetes Pod RBAC @AntonAM
kubernetes_resources:{"kind":"pod","name":"*","namespace":"*"}- must allow access to every pod.{"kind":"pod","name":"<somename>","namespace":"*"}- must allow access to pod<somename>in every namespace.{"kind":"pod","name":"*","namespace":"<somenamespace>"}- must allow access to any pod in<somenamespace>namespace.*wildcards -<some-name>-*and regex fornameandnamespacefields.go-client.kubernetes_resources:kubernetes_groupsthat denies exec into a podsearch_as_rolesis not allowed.Teleport with FIPS mode @bl-nero
ACME @bl-nero
Migrations @tigrato
SSH should work for both main and old clusters
SSH should work
Command Templates
When interacting with a cluster, the following command templates are useful:
OpenSSH
Teleport
Teleport with SSO Providers
GitHub External SSO @capnspacehook
tctl ssofamily of commands @flyinghermitFor help with setting up sso connectors, check out the [Quick GitHub/SAML/OIDC Setup Tips]
tctl sso configurehelps to construct a valid connector definition:tctl sso configure github ...creates valid connector definitionstctl sso configure oidc ...creates valid connector definitionstctl sso configure saml ...creates valid connector definitionstctl sso testtest a provided connector definition, which can be loaded fromfile or piped in with
tctl sso configureortctl get --with-secrets. Validconnectors are accepted, invalid are rejected with sensible error messages.
tctl sso test.SSO login on remote host @atburke
tshshould be running on a remote host (e.g. over an SSH session) and use thelocal browser to complete and SSO login. Run
tsh login --callback <remote.host>:<port> --bind-addr localhost:<port> --auth <auth>on the remote host. Note that the
--callbackURL must be able to resolve to the--bind-addrover HTTPS.Teleport Plugins @EdwardDowling
Teleport Operator @hugoShaka
teleport-clusterHelm chart and the operator enabledAWS Node Joining @hugoShaka
Docs
ec2:DescribeInstancespermissions for local account:TELEPORT_TEST_EC2=1 go test ./integration -run TestEC2NodeJoinTELEPORT_TEST_EC2=1 go test ./integration -run TestIAMNodeJoinKubernetes Node Joining @hugoShaka
Azure Node Joining @marcoandredinis
Docs
GCP Node Joining @hugoShaka
Docs
Cloud Labels @hugoShaka @marcoandredinis
and with tag
foo:bar. Verify that a node running on the instance has labelaws/foo=bar.foo:bar. Verify that a node running on theinstance has label
azure/foo=bar.Passwordless @codingllama
This feature has additional build requirements, so it should be tested with a
pre-release build from Drone (eg:
https://get.gravitational.com/tsh-v10.0.0-alpha.2.pkg).This sections complements "Users -> Managing MFA devices".
tshbinaries foreach operating system (Linux, macOS and Windows) must be tested separately for
FIDO2 items.
Diagnostics
Commands should pass all tests.
tsh fido2 diag(macOS/Linux)tsh touchid diag(macOS only)tsh webauthnwin diag(Windows only)Registration
tsh mfa add, choose WEBAUTHN andpasswordless)
tsh mfa add, choose TOUCHID)tsh mfa add, choose WEBAUTHN andpasswordless)
Login
tsh login --auth=passwordless)tsh login --auth=passwordless)tsh login --auth=passwordless --mfa-mode=cross-platformuses FIDO2tsh login --auth=passwordless --mfa-mode=platformuses platform authenticatortsh login --auth=passwordless --mfa-mode=autoprefers platform authenticatorthe same device)
(
auth_service.authentication.passwordless = false)(
auth_service.authentication.connector_name = passwordless)(
tsh login --auth=local)Touch ID support commands
tsh touchid lsworkstsh touchid rmworks (careful, may lock you out!)Device Trust @codingllama
Device Trust requires Teleport Enterprise.
This feature has additional build requirements, so it should be tested with a
pre-release build from Drone (eg:
https://get.gravitational.com/teleport-ent-v10.0.0-alpha.2-linux-amd64-bin.tar.gz).Client-side enrollment requires a signed
tshfor macOS, make sure to use thetshbinary fromtsh.app.A simple formula for testing device authorization is:
Inventory management
tctl devices add)tctl devices add --enroll)tctl devices ls)tctl devices rm)tctl devices rm)tctl devices enroll)tctl devices enroll)Device enrollment
Enroll/authn device on macOS (
tsh device enroll)Enroll/authn device on Windows (
tsh device enroll)Enroll/authn device on Linux (
tsh device enroll)Linux users need read/write permissions to /dev/tpmrm0. The simplest way is
to assign yourself to the
tssgroup. Seehttps://goteleport.com/docs/access-controls/device-trust/device-management/#troubleshooting.
Verify device extensions on TLS certificate
Note that different accesses have different certificates (Database, Kube,
etc).
Verify device extensions on SSH certificate
Device authorization
device_trust.mode other than "off" or "" not allowed (OSS)
device_trust.mode="off" doesn't impede access (Enterprise and OSS)
device_trust.mode="optional" doesn't impede access, but issues device
extensions on login
device_trust.mode="required" enforces enrolled devices
device_trust.mode="required" is enforced by processes, and not only by
Auth APIs
Testing this requires issuing a certificate without device extensions
(mode="off"), then changing the cluster configuration to mode="required" and
attempting to access a process directly, without a login attempt.
Role-based authz enforces enrolled devices
(device_trust.mode="off" or "optional",
role.spec.options.device_trust_mode="required")
Device authorization works correctly for both require_session_mfa=false
and require_session_mfa=true
Device authorization applies to SSH access (all items above)
Device authorization applies to Trusted Clusters (root with
mode="optional" and leaf with mode="required")
Device authorization applies to Database access (all items above)
Device authorization applies to Kubernetes access (all items above)
Cluster-wide device authorization does not apply to App access
Role-based device authorization applies to App access
Device authorization does not apply to Windows Desktop access
(both cluster-wide and role) @ibeckermayer
Device audit (see lib/events/codes.go)
data (for certificates with device extensions)
Binary support
tshfor macOS gives a sane errormessage for
tsh device enrollattempts.Device support commands
tsh device collect(macOS)tsh device asset-tag(macOS)tsh device collect(Windows)tsh device asset-tag(Windows)tsh device collect(Linux)tsh device asset-tag(Linux)Hardware Key Support @jakule
Hardware Key Support is an Enterprise feature and is not available for OSS.
You will need a YubiKey 4.3+ to test this feature.
This feature has additional build requirements, so it should be tested with a pre-release build from Drone (eg:
https://get.gravitational.com/teleport-ent-v11.0.0-alpha.2-linux-amd64-bin.tar.gz).Server Access
These tests should be carried out sequentially.
tshtests should be carried out on Linux, MacOS, and Windows.tsh loginas user with Webauthn login and no hardware key requirement.role.role_options.require_session_mfa: hardware_key-tsh login --request-roles=hardware_key_requiredtsh sshrole.role_options.require_session_mfa: hardware_key_touch-tsh login --request-roles=hardware_key_touch_requiredtsh sshtsh logoutandtsh loginas the user with no hardware key requirement.auth_service.authentication.require_session_mfa: hardware_keytsh ls) should force automatic re-login with yubikeytsh sshauth_service.authentication.require_session_mfa: hardware_key_touchtsh ls) should force automatic re-login with yubikeytsh sshOther
Set
auth_service.authentication.require_session_mfa: hardware_key_touchin your cluster auth settings.tsh proxy db --tunnelHSM Support @nklaassen
Docs
Moderated session @strideynet
Using
tshjoin an SSH session as two moderators (two separate terminals, role requires one moderator).Ctrl+Cin the Implement a prototype for a proxying SSH server that implements concepts expressed in readme #1 terminal should disconnect the moderator.Ctrl+Cin the Implement a functional prototype #2 terminal should disconnect the moderator and terminate the session as session has no moderator.Using
tshjoin an SSH session as two moderators (two separate terminals, role requires one moderator).tin any terminal should terminate the session for all participants.Performance @rosstimothy @fspmarshall
Scaling Test
Scale up the number of nodes/clusters a few times for each configuration below.
Perform reverse tunnel node scaling tests with actual nodes for Cloud:
Perform simulated node scaling tests with actual nodes via
tctl loadtest node-heartbeats --count=15000 --ttl=2m --interval=1m --labels=2 --concurrency=32for:Perform the following additional scaling tests on etcd:
Soak Test
and via label based matching. Tests should only be run against a Cloud
tenant.
Concurrent Session Test
Run a concurrent session test that will spawn 5 interactive sessions per node in the cluster. Tests should only be run against a Cloud tenant:
Robustness
resources which do not require a moderated session and in async recording
mode from an already issued certificate.
which require a moderated session and in async recording mode from an already
issued certificate.
are restarted.
Teleport with Cloud Providers
AWS @camscale
GCP @tigrato
IBM @hugoShaka
Application Access @mdwn
debug_app: trueworks.name.rootProxyPublicAddrand well aspublicAddr.name.rootProxyPublicAddr.app.session.startandapp.session.chunkevents are created in the Audit Log.app.session.chunkpoints to a 5 minute session archive with multipleapp.session.requestevents inside.tsh play <chunk-id>can fetch and print a session chunk archive.tsh apps login.tshcommands.tsh awstsh aws --endpoint-url(this is a hidden flag)tsh apps login.tsh azcommands.tsh proxy azandazcommands.tsh apps login.tsh gcloudcommands.tsh gsutilcommands.tsh proxy gcloudandgcloud/gsutilcommands.tctl create.tctl create -f.tctl rm.Add Applicationlinks to documentation.Database Access @greedy52 + team
IMPORTANT: for self-hosted databases, please verify before and after rotating
dbanddb_clientCAs. Note that databases need to be reconfigured withthe new certs and CAs. And some databases require slightly different setup:
update self-hosted database guides #36260
select pg_sleep(10)followed by ctrl-c is a good query to test.)assume_role_arn: ""andexternal_id: "<id>"assume_role_arn: ""andexternal_id: "<id>"assume_role_arn: ""andexternal_id: "<id>"Verify all supported modes:
keep,best_effort_dropdb.session.startis emitted when you connect.db.session.endis emitted when you disconnect.db.session.queryis emitted when you execute a SQL query.tsh db lsshows only databases matching role'sdb_labels.db_users.db_names.db.session.startis emitted when connection attempt is denied.db_names.db.session.queryis emitted when command fails due to permissions.tsh db connect.tctl create.tctl create -f.tctl rm.Please configure discovery in Discovery Service instead of Database Service.
assume_role_arnandexternal_idis set.name,description,type, andlabelsStep 2login value matching the rowsnamecolumnlabelsTLS Routing @smallinsky
v2configuration starts only a single listener for proxy service, in contrast withv1configuration. @smallinskyGiven configuration:
*:3080for proxy service. Given the configuration above, 3022 and 3025 will be opened for other services.v1, there should be additional ports 3023 and 3024.multiplexmodeauth_service.proxy_listener_mode: "multiplex"@smallinskyweb_proxy_addr == tunnel_addrtsh db connectworks through proxy running inmultiplexmodetsh proxy dbwith a GUI client. @greedy52multiplexmodessh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh" [email protected]ssh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh --user=%r --cluster=leaf-cluster %h:%p" [email protected]tsh sshaccess through proxy running in multiplex modemultiplexmode, usingtshmultiplexmode behind L7 load balancertsh loginandtctl@greedy52tsh sshandtsh config@smallinskytsh proxy dbandtsh db connect@greedy52tsh proxy appandtsh aws@smallinskytsh proxy kube@smallinskyIGS:
Access Monitoring @smallinsky
Access List @mdwn
Verify Okta Sync Service @tcsc
okta_import_rulerule configuration.The text was updated successfully, but these errors were encountered: