Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Access agentless nodes using hostname with ssh as client #31281

Closed
marcoandredinis opened this issue Aug 31, 2023 · 16 comments
Closed

Access agentless nodes using hostname with ssh as client #31281

marcoandredinis opened this issue Aug 31, 2023 · 16 comments
Assignees
@lxea @capnspacehook
Labels
bug test-plan-problem Issues which have been surfaced by running the manual release test plan

Comments

@marcoandredinis
Copy link
Contributor

marcoandredinis commented Aug 31, 2023 •
edited
Loading

Expected behavior:
Doing tsh ssh <user>@<host> should be the same as doing ssh <user>@<host>.<cluster> after correctly configuring ssh client.

Current behavior:
✅ tsh ssh root@ip-172-31-24-131

❌ ssh [email protected]

> ssh [email protected]
ERROR: ssh: subsystem request failed

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

After doing tctl nodes ls we get the node's name and try ssh client again:
✅ ssh [email protected]

Bug details:

2023-08-31T14:43:22Z DEBU [PROXY]     conn(92.250.90.215:34966->172.31.39.120:443, user=root) auth attempt fingerprint:[email protected] SHA256:uJpFHj00KQcyjKTFiV45+iozG0FLJXgba8gBHpiDjlU local:172.31.39.120:443 remote:92.250.90.215:34966 user:root srv/authhandlers.go:304
2023-08-31T14:43:22Z DEBU [PROXY]     conn(92.250.90.215:34966->172.31.39.120:443, user=root) auth attempt with key [email protected] SHA256:uJpFHj00KQcyjKTFiV45+iozG0FLJXgba8gBHpiDjlU, &ssh.Certificate{Nonce:[]uint8{0x11, 0x98, 0x20, 0x22, 0xb8, 0xc3, 0x38, 0xa4, 0xe0, 0x9f, 0x32, 0xf5, 0x74, 0x1d, 0x67, 0xfb, 0xe, 0xd5, 0x85, 0xa2, 0x66, 0xec, 0x98, 0x1, 0x26, 0xee, 0xc0, 0xff, 0x65, 0x0, 0x9c, 0x2c}, Key:(*ssh.rsaPublicKey)(0xc00261bb60), Serial:0x0, CertType:0x1, KeyId:"marco", ValidPrincipals:[]string{"dinis", "root", "ubuntu", "-teleport-internal-join"}, ValidAfter:0x64f09d58, ValidBefore:0x64f14654, Permissions:ssh.Permissions{CriticalOptions:map[string]string{}, Extensions:map[string]string{"login-ip":"92.250.90.215", "permit-agent-forwarding":"", "permit-port-forwarding":"", "permit-pty":"", "private-key-policy":"none", "teleport-roles":"{\"version\":\"v1\",\"roles\":[\"editor\",\"access\"]}", "teleport-route-to-cluster":"lenix", "teleport-traits":"{\"logins\":[\"dinis\",\"root\",\"ubuntu\"]}"}}, Reserved:[]uint8{}, SignatureKey:(*ssh.rsaPublicKey)(0xc001b74710), Signature:(*ssh.Signature)(0xc0026052c0)} fingerprint:[email protected] SHA256:uJpFHj00KQcyjKTFiV45+iozG0FLJXgba8gBHpiDjlU local:172.31.39.120:443 remote:92.250.90.215:34966 user:root srv/authhandlers.go:307
2023-08-31T14:43:22Z DEBU [PROXY]     Successfully authenticated fingerprint:[email protected] SHA256:uJpFHj00KQcyjKTFiV45+iozG0FLJXgba8gBHpiDjlU local:172.31.39.120:443 remote:92.250.90.215:34966 user:root srv/authhandlers.go:394
2023-08-31T14:43:22Z DEBU [SSH:PROXY] Incoming connection 92.250.90.215:34966 -> 172.31.39.120:443 version: SSH-2.0-Go, certtype: "user" sshutils/server.go:500
2023-08-31T14:43:22Z DEBU [KEEPALIVE] Starting keep-alive loop with interval 5m0s and max count 3. srv/keepalive.go:64
2023-08-31T14:43:22Z DEBU [PROXY]     Handling request [email protected], want reply true. id:18 local:172.31.39.120:443 login:root remote:92.250.90.215:34966 teleportUser:marco regular/sshserver.go:1576
2023-08-31T14:43:22Z WARN             agent forwarding to proxy only supported in recording mode regular/sshserver.go:1595
2023-08-31T14:43:22Z DEBU [PROXY]     Handling request subsystem, want reply true. id:18 local:172.31.39.120:443 login:root remote:92.250.90.215:34966 teleportUser:marco regular/sshserver.go:1576
2023-08-31T14:43:22Z DEBU [NODE]      parse_proxy_subsys("proxy:ip-172-31-24-131:3022@lenix") regular/proxy.go:70
2023-08-31T14:43:22Z DEBU [NODE]      newProxySubsys({default ip-172-31-24-131 3022 lenix}). regular/proxy.go:182
2023-08-31T14:43:22Z DEBU [PROXY]     Subsystem request: proxySubsys(cluster=default/lenix, host=ip-172-31-24-131, port=3022). id:18 local:172.31.39.120:443 login:root remote:92.250.90.215:34966 teleportUser:marco regular/sshserver.go:1837
2023-08-31T14:43:22Z DEBU [SUBSYSTEM] Starting subsystem trace.fields:map[dst:172.31.39.120:443 src:92.250.90.215:34966 subsystem:proxySubsys(cluster=default/lenix, host=ip-172-31-24-131, port=3022)] regular/proxy.go:214
2023-08-31T14:43:22Z DEBU [SUBSYSTEM] proxy connecting to host=ip-172-31-24-131 port=3022, exact port=true trace.fields:map[dst:172.31.39.120:443 src:92.250.90.215:34966 subsystem:proxySubsys(cluster=default/lenix, host=ip-172-31-24-131, port=3022)] regular/proxy.go:247
2023-08-31T14:43:22Z WARN [PROXY]     Subsystem request proxySubsys(cluster=default/lenix, host=ip-172-31-24-131, port=3022) failed: direct dialing to nodes not found in inventory is not supported. id:18 local:172.31.39.120:443 login:root remote:92.250.90.215:34966 teleportUser:marco regular/sshserver.go:1841
2023-08-31T14:43:22Z ERRO             failure handling SSH "subsystem" request error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError direct dialing to nodes not found in inventory is not supported
Stack Trace:
	github.com/gravitational/teleport/lib/proxy/router.go:288 github.com/gravitational/teleport/lib/proxy.(*Router).DialHost
	github.com/gravitational/teleport/lib/srv/regular/proxy.go:257 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).proxyToHost
	github.com/gravitational/teleport/lib/srv/regular/proxy.go:224 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).Start
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1840 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSubsystem
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1585 github.com/gravitational/teleport/lib/srv/regular.(*Server).dispatch
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1544 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSessionRequests
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: direct dialing to nodes not found in inventory is not supported] regular/sshserver.go:2051
2023-08-31T14:43:22Z DEBU [PROXY]     Releasing associated resources - context has been closed. id:18 local:172.31.39.120:443 login:root remote:92.250.90.215:34966 teleportUser:marco srv/monitor.go:397
2023-08-31T14:43:22Z DEBU [SSH:PROXY] Closed connection 92.250.90.215:34966. sshutils/server.go:505

This is also valid for remote clusters
Trying to access a Node in a Agentless Node using ssh fails
@marcoandredinis marcoandredinis added bug test-plan-problem Issues which have been surfaced by running the manual release test plan labels Aug 31, 2023
@marcoandredinis marcoandredinis mentioned this issue Aug 31, 2023
Closed
@marcoandredinis marcoandredinis changed the title Access agentless nodes using hostname Access agentless nodes using hostname using ssh Aug 31, 2023
@marcoandredinis marcoandredinis changed the title Access agentless nodes using hostname using ssh Access agentless nodes using hostname with ssh as client Aug 31, 2023
@r0mant
Copy link
Collaborator

r0mant commented Aug 31, 2023

@lxea @capnspacehook Shouldn't agentless mode support dialing by hostname? Is there a bug?

@r0mant
Copy link
Collaborator

r0mant commented Aug 31, 2023

@marcoandredinis What happens if you try to connect using ssh root@ip-172-31-24-131?

@marcoandredinis
Copy link
Contributor Author

marcoandredinis commented Aug 31, 2023 •
edited
Loading

@marcoandredinis What happens if you try to connect using ssh root@ip-172-31-24-131?

I'll try that as well
But my expectation is that it won't match the regex* we define in ssh config and will try to access the ip-172-31-24-131 node.

Host *.{{ .ClusterName }} !{{ .ProxyHost }}
    ... tsh proxy ssh ...

@marcoandredinis
Copy link
Contributor Author

New set up (just to ensure I didn't messed up previously, or I did it twice 😅 )

Cluster
lenixleaf - SSH node with Teleport Agent
mynode - SSH Node in Agentless Mode (added using teleport join openssh)

dinis@lenix ~/p/ssh-combinations (master)> tsh ssh dinis@lenixleaf exit
dinis@lenix ~/p/ssh-combinations (master)> ssh [email protected] exit
dinis@lenix ~/p/ssh-combinations (master)> tsh ssh root@mynode exit
dinis@lenix ~/p/ssh-combinations (master)> ssh [email protected] exit
ERROR: ssh: subsystem request failed

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
dinis@lenix ~/p/ssh-combinations (master) [255]> ssh root@mynode exit
ssh: Could not resolve hostname mynode: Name or service not known

Logs when I try to connect to connect using ssh [email protected] exit

2023-09-01T11:03:43+01:00 DEBU [PROXY]     conn(127.0.0.1:36156->127.0.0.2:13080, user=root) auth attempt fingerprint:[email protected] SHA256:iBWTODZvOQ1c8mz1FcN281TlNP1DT9q+A/sa0mLisLM local:127.0.0.2:13080 remote:127.0.0.1:36156 user:root srv/authhandlers.go:304
2023-09-01T11:03:43+01:00 DEBU [PROXY]     conn(127.0.0.1:36156->127.0.0.2:13080, user=root) auth attempt with key [email protected] SHA256:iBWTODZvOQ1c8mz1FcN281TlNP1DT9q+A/sa0mLisLM, &ssh.Certificate{Nonce:[]uint8{0xe1, 0x73, 0x24, 0xa7, 0xa7, 0xbe, 0xe2, 0xee, 0xb0, 0x81, 0x47, 0x77, 0x24, 0x6d, 0x18, 0xf6, 0xfc, 0x7, 0x7, 0x6f, 0x75, 0xfe, 0xa7, 0xc5, 0x91, 0x78, 0xed, 0xb1, 0xbc, 0xe5, 0x2e, 0x95}, Key:(*ssh.rsaPublicKey)(0xc003d889d0), Serial:0x0, CertType:0x1, KeyId:"marco", ValidPrincipals:[]string{"root", "dinis", "-teleport-internal-join"}, ValidAfter:0x64f1b518, ValidBefore:0x64f25e14, Permissions:ssh.Permissions{CriticalOptions:map[string]string{}, Extensions:map[string]string{"login-ip":"127.0.0.1", "permit-agent-forwarding":"", "permit-port-forwarding":"", "permit-pty":"", "private-key-policy":"none", "teleport-roles":"{\"version\":\"v1\",\"roles\":[\"editor\",\"access\"]}", "teleport-route-to-cluster":"lenixleaf", "teleport-traits":"{\"logins\":[\"root\",\"dinis\"]}"}}, Reserved:[]uint8{}, SignatureKey:(*ssh.rsaPublicKey)(0xc003d88a10), Signature:(*ssh.Signature)(0xc004ae3180)} fingerprint:[email protected] SHA256:iBWTODZvOQ1c8mz1FcN281TlNP1DT9q+A/sa0mLisLM local:127.0.0.2:13080 remote:127.0.0.1:36156 user:root srv/authhandlers.go:307
2023-09-01T11:03:43+01:00 DEBU [PROXY]     Successfully authenticated fingerprint:[email protected] SHA256:iBWTODZvOQ1c8mz1FcN281TlNP1DT9q+A/sa0mLisLM local:127.0.0.2:13080 remote:127.0.0.1:36156 user:root srv/authhandlers.go:394
2023-09-01T11:03:43+01:00 DEBU [SSH:PROXY] Incoming connection 127.0.0.1:36156 -> 127.0.0.2:13080 version: SSH-2.0-Go, certtype: "user" sshutils/server.go:500
2023-09-01T11:03:43+01:00 DEBU [PROXY]     Handling request [email protected], want reply true. id:21 local:127.0.0.2:13080 login:root remote:127.0.0.1:36156 teleportUser:marco regular/sshserver.go:1576
2023-09-01T11:03:43+01:00 DEBU [KEEPALIVE] Starting keep-alive loop with interval 5m0s and max count 3. srv/keepalive.go:64
2023-09-01T11:03:43+01:00 WARN             agent forwarding to proxy only supported in recording mode regular/sshserver.go:1595
2023-09-01T11:03:43+01:00 DEBU [PROXY]     Handling request subsystem, want reply true. id:21 local:127.0.0.2:13080 login:root remote:127.0.0.1:36156 teleportUser:marco regular/sshserver.go:1576
2023-09-01T11:03:43+01:00 DEBU [NODE]      parse_proxy_subsys("proxy:mynode:3022@lenixleaf") regular/proxy.go:70
2023-09-01T11:03:43+01:00 DEBU [NODE]      newProxySubsys({default mynode 3022 lenixleaf}). regular/proxy.go:182
2023-09-01T11:03:43+01:00 DEBU [PROXY]     Subsystem request: proxySubsys(cluster=default/lenixleaf, host=mynode, port=3022). id:21 local:127.0.0.2:13080 login:root remote:127.0.0.1:36156 teleportUser:marco regular/sshserver.go:1837
2023-09-01T11:03:43+01:00 DEBU [SUBSYSTEM] Starting subsystem trace.fields:map[dst:127.0.0.2:13080 src:127.0.0.1:36156 subsystem:proxySubsys(cluster=default/lenixleaf, host=mynode, port=3022)] regular/proxy.go:214
2023-09-01T11:03:43+01:00 DEBU [SUBSYSTEM] proxy connecting to host=mynode port=3022, exact port=true trace.fields:map[dst:127.0.0.2:13080 src:127.0.0.1:36156 subsystem:proxySubsys(cluster=default/lenixleaf, host=mynode, port=3022)] regular/proxy.go:247
2023-09-01T11:03:43+01:00 WARN [PROXY]     Subsystem request proxySubsys(cluster=default/lenixleaf, host=mynode, port=3022) failed: direct dialing to nodes not found in inventory is not supported. id:21 local:127.0.0.2:13080 login:root remote:127.0.0.1:36156 teleportUser:marco regular/sshserver.go:1841
2023-09-01T11:03:43+01:00 ERRO             failure handling SSH "subsystem" request error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError direct dialing to nodes not found in inventory is not supported
Stack Trace:
	github.com/gravitational/teleport/lib/proxy/router.go:288 github.com/gravitational/teleport/lib/proxy.(*Router).DialHost
	github.com/gravitational/teleport/lib/srv/regular/proxy.go:257 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).proxyToHost
	github.com/gravitational/teleport/lib/srv/regular/proxy.go:224 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).Start
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1840 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSubsystem
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1585 github.com/gravitational/teleport/lib/srv/regular.(*Server).dispatch
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1544 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSessionRequests
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: direct dialing to nodes not found in inventory is not supported] regular/sshserver.go:2051
2023-09-01T11:03:43+01:00 DEBU [PROXY]     Releasing associated resources - context has been closed. id:21 local:127.0.0.2:13080 login:root remote:127.0.0.1:36156 teleportUser:marco srv/monitor.go:397
2023-09-01T11:03:43+01:00 DEBU [SSH:PROXY] Closed connection 127.0.0.1:36156. sshutils/server.go:505

I edited the ~/.ssh/config to add --debug flag to tsh proxy ssh and got the following logs

2023-09-01T11:05:05+01:00 INFO [CLIENT]    ALPN connection upgrade required for "127.0.0.2.nip.io:13080": false. client/api.go:723
2023-09-01T11:05:05+01:00 ERRO [CLIENT]    [KEY AGENT] Unable to connect to SSH agent on socket: "". error:[
ERROR REPORT:
Original Error: *net.OpError dial unix: missing address
Stack Trace:
	github.com/gravitational/teleport/lib/utils/agentconn/agent_unix.go:32 github.com/gravitational/teleport/lib/utils/agentconn.Dial
	github.com/gravitational/teleport/lib/client/api.go:4552 github.com/gravitational/teleport/lib/client.connectToSSHAgent
	github.com/gravitational/teleport/lib/client/keyagent.go:135 github.com/gravitational/teleport/lib/client.NewLocalAgent
	github.com/gravitational/teleport/lib/client/api.go:1115 github.com/gravitational/teleport/lib/client.NewClient
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:3396 github.com/gravitational/teleport/tool/tsh/common.makeClientForProxy
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:3381 github.com/gravitational/teleport/tool/tsh/common.makeClient
	github.com/gravitational/teleport/tool/tsh/common/proxy.go:63 github.com/gravitational/teleport/tool/tsh/common.onProxyCommandSSH
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:1360 github.com/gravitational/teleport/tool/tsh/common.Run
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:540 github.com/gravitational/teleport/tool/tsh/common.Main
	github.com/gravitational/teleport/tool/tsh/main.go:24 main.main
	runtime/proc.go:267 runtime.main
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: dial unix: missing address] client/api.go:4554
2023-09-01T11:05:05+01:00 DEBU [KEYSTORE]  Reading certificates from path "/home/dinis/.tsh/keys/127.0.0.2.nip.io/marco-ssh/lenixleaf-cert.pub". client/keystore.go:354
2023-09-01T11:05:05+01:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-01 21:56:36 +0000 UTC". client/client_store.go:106
2023-09-01T11:05:05+01:00 INFO [KEYAGENT]  Loading SSH key for user "marco" and cluster "lenixleaf". client/keyagent.go:196
2023-09-01T11:05:05+01:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-01 21:56:36 +0000 UTC". client/client_store.go:106
2023-09-01T11:05:05+01:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-01 21:56:36 +0000 UTC". client/client_store.go:106
2023-09-01T11:05:05+01:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-01 21:56:36 +0000 UTC". client/client_store.go:106
2023-09-01T11:05:05+01:00 DEBU [KEYAGENT]  "Checking key: [email protected] AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAg0UtEXmvYvs2NnB66ZyU6R10PgtohjgMYsR5adyk+dgYAAAADAQABAAABAQDtMIWU2UNgE/Zwb7f2GViAXLWk+23ptAojLC/aGRd4KxHmdKBlZOW1Li4qrMza0EFTpbx9N1LG1nffAtcLMDhv/zsiMpzi+9ZrKNnpsx4VUYPCdK+fgCoW6cHcgeggmKPMz/t/bE5uTBfHmOY2Kl/9AAje4RFis/l0KSsjxyBhhl4PvC2zU3osv9v2xCj1ImcZTKVBdWzMhzjq0C9JkMvHdc2M91Ny8bEyBy8NhpaRvkhtTzfcFYp7aas2sXyP75Ux8FuUMpykglpop9+y+/CNeIxy+/Quduedqkh3qFDlobp13jlmErkwsPvHvuYwvsNyi+Ln3yUJAAEvHq+RxhvXAAAAAAAAAAAAAAACAAAAAAAAAN8AAAAuODdlZDUyNzEtNGEwNy00MDMwLWE5MjUtNjgyZTk1YzAyNTQ2Lmxlbml4bGVhZgAAACQ4N2VkNTI3MS00YTA3LTQwMzAtYTkyNS02ODJlOTVjMDI1NDYAAAATbGVuaXhsZWFmLmxlbml4bGVhZgAAAAlsZW5peGxlYWYAAAAJbG9jYWxob3N0AAAACTEyNy4wLjAuMQAAAAM6OjEAAAAQMTI3LjAuMC4yLm5pcC5pbwAAAChyZW1vdGUua3ViZS5wcm94eS50ZWxlcG9ydC5jbHVzdGVyLmxvY2FsAAAAAGTxs4T//////////wAAAAAAAABJAAAAFHgtdGVsZXBvcnQtYXV0aG9yaXR5AAAADQAAAAlsZW5peGxlYWYAAAAPeC10ZWxlcG9ydC1yb2xlAAAACQAAAAVQcm94eQAAAAAAAAEXAAAAB3NzaC1yc2EAAAADAQABAAABAQCvHgRzW8LYczdnEvJUNyDK8eN0ojMfNxeB04piR11FfiMPuOXgaM6fWUTxgDpiv1kgWBRUlYPdG/tM/uWuo556VYCStT1MIejCF+dAKflwsvMRlPQVhebOIZplZkwHNbb2edtjrd24lcCWy2N9nxdef2IUjkz4aebzKNkLZT6ToR/KtgTlZgTR6A1Ovz5NaAAQZ7VG1sZKhy8EjDsLNnJ47pYERq2Wv4wkC7+Z6fbHsFnf8e+Y2hm/qyd4aMIKCGmic/V0O/ovYbRJi5gju+dukaUnlA28HNCL/ADvcM2JZlS5EJZR6qPjvcFzn+TvK0JXWwh7MiZ54R+xZ+aKbpmtAAABFAAAAAxyc2Etc2hhMi01MTIAAAEAG6R6DHqF8dD3es1DXF6rTUQ5zu1W6z6zYAfHK3+IR9jc6lSHDYAsdKj3ohPOxz8zc7yNBORAg2ej1JurXIv5GbKXFB8gvYsFlWQnp91Qad0Jn3NzU0xMYuY36DcoCSJaAu/JlQkUpFyMPCZpZu8pu/t+XI1462DHH1nwQWNs26mVCq55WZ5yUyLWt0YgTagETID0GhtKmSfWIQ/mTbJGS0vQ25G04lyelCgnUPDqi2DDZlIgse4tUL8H5h8dUs0gtYrSyDyGO7gRR91Sn7cSEhVrLLnYqDHbrCtPuTKu3K/G3gdv4jIz3L7+iAG748+JAv5LSdZa0lG/JHPyfq4YvA==\n." client/keyagent.go:368
2023-09-01T11:05:05+01:00 DEBU [KEYAGENT]  Validated host 127.0.0.2.nip.io:13080. client/keyagent.go:374

ERROR REPORT:
Original Error: *errors.errorString ssh: subsystem request failed
Stack Trace:
	github.com/gravitational/teleport/[email protected]/observability/tracing/ssh/session.go:183 github.com/gravitational/teleport/api/observability/tracing/ssh.(*Session).RequestSubsystem
	github.com/gravitational/teleport/tool/tsh/common/proxy.go:228 github.com/gravitational/teleport/tool/tsh/common.sshProxy
	github.com/gravitational/teleport/tool/tsh/common/proxy.go:80 github.com/gravitational/teleport/tool/tsh/common.onProxyCommandSSH.func1
	github.com/gravitational/teleport/lib/client/api.go:570 github.com/gravitational/teleport/lib/client.RetryWithRelogin
	github.com/gravitational/teleport/tool/tsh/common/proxy.go:68 github.com/gravitational/teleport/tool/tsh/common.onProxyCommandSSH
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:1360 github.com/gravitational/teleport/tool/tsh/common.Run
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:540 github.com/gravitational/teleport/tool/tsh/common.Main
	github.com/gravitational/teleport/tool/tsh/main.go:24 main.main
	runtime/proc.go:267 runtime.main
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: ssh: subsystem request failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

@marcoandredinis
Copy link
Contributor Author

The same happens when accessing the Nodes in a leaf cluster, using a login in root cluster
For Teleport Nodes I can connect using WebUI, tsh and ssh
For Agentless Nodes I can only connect using WebUI, tsh but not ssh

@lxea
Copy link
Contributor

lxea commented Sep 1, 2023

I was able to connect to an agentless leaf node by passing an explicit port to tsh --

$  ssh -F ~/ssh_config_teleport -p 22 agentless_node.lan.leafcluster

However subsequent re-connections says the remote host key has changed for some reason

$ ssh -F ~/ssh_config_teleport -p 22 [email protected]
The authenticity of host 'agentless_node.lan.leafcluster (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:Qy+aE9WQa63W7GomQv1f3nE78dgkcNJm3/i7PH36RqE.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'agentless_node.lan.leafcluster' (RSA) to the list of known hosts.
Last login: Fri Sep  1 11:44:19 IST 2023 from 192.168.122.24 on ssh
Have a lot of fun...
alex@suse:~> logout
Connection to agentless_node.lan.leafcluster closed.
alex@corn ~
$ ssh -F ~/ssh_config_teleport -p 22 agentless_node.lan.leafcluster
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:dmDmF9gfrPXdTuH7hILRYujJ9VEqPIqtYpalCrhuCjc.
Please contact your system administrator.
Add correct host key in /home/alex/.tsh/known_hosts to get rid of this message.
Offending RSA key in /home/alex/.tsh/known_hosts:3
Host key for agentless_node.lan.leafcluster has changed and you have requested strict checking.
Host key verification failed.

@marcoandredinis
Copy link
Contributor Author

I can confirm the exact same behaviour 👍
Passing the host's port works, but the 2nd access fails (same error you got)

@lxea
Copy link
Contributor

lxea commented Sep 4, 2023

I can connect okay when using an agentless node on a local cluster like ssh -F config_file -p 22 [email protected]

The port being required to be passed seems to be expected acording to the guide https://goteleport.com/docs/server-access/guides/openssh/#step-33-connect-to-your-sshd-host however when connecting to a node on a leaf cluster you seem to be expected to include .leafcluster.rootcluster however this seems to give the same error, and pass -p 22 also doesnt work with it

@r0mant
Copy link
Collaborator

r0mant commented Sep 5, 2023

@marcoandredinis @lxea @capnspacehook Is this a regression in behavior compared to v13 and v12?

@marcoandredinis
Copy link
Contributor Author

I'm getting the same thing using v13:

dinis@lenix ~/p/ssh-combinations (master)> tsh version
Teleport v13.3.7 git:api/v13.3.7-37-gdb1dc70117 go1.21.0
Proxy version: 13.3.7
Proxy: 127.0.0.2.nip.io:13080
dinis@lenix ~/p/ssh-combinations (master)> ssh -F ssh_config_teleport [email protected]
ERROR: ssh: subsystem request failed

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
dinis@lenix ~/p/ssh-combinations (master) [255]> ssh -F ssh_config_teleport [email protected] -p 4444
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.1 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.
Last login: Tue Sep  5 17:25:39 2023 from 127.0.0.1
root@mynode:~#

V12 seems to have a different onboarding flow
Should I test there as well?

@r0mant
Copy link
Collaborator

r0mant commented Sep 5, 2023

@marcoandredinis Yeah can you please check v12 as well, the client experience shouldn't have really changed with agentless to my knowledge, just how you register the server with the cluster.

@marcoandredinis
Copy link
Contributor Author

I might be missing something but can you actually register a node in v12? In the sense that it actually appears as a Teleport resource.

I followed this guide https://goteleport.com/docs/ver/12.x/server-access/guides/openssh/ but you don't end up with a node registration (as in, you don't see it when list servers in UI/cli).
So, while you get access to it using either ssh or tsh ssh, you always need to pass the port.

As a summary, I think the issue is that, given that you can do tsh ssh <user>@<host> then you should also be able do ssh -F ssh_config_teleport <user>@<host>. That doesn't work, and for ssh you must pass the port.
Not a regression from v13, tho.
The docs do have a reference for using the port, which kind of makes all of this ok.

Using a single cluster everything works if you pass the port in the ssh command.

For trusted clusters, if you try to access a node in a leaf cluster using ssh it works but fails if you do it a 2nd time:
This was also observed by Alex

dinis@lenix ~/p/ssh-combinations (master)> ssh -F ssh_config_teleport_root [email protected] -p 4444 exit
The authenticity of host '[mynode.lenixleaf]:4444 (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:yJsOdyS81xN0rMgypI5AG7DFmAOcxaPvyPErYB7UArk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[mynode.lenixleaf]:4444' (RSA) to the list of known hosts.
dinis@lenix ~/p/ssh-combinations (master)> ssh -F ssh_config_teleport_root [email protected] -p 4444 exit
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:iJ7GUeuN5B5hVnh8nI+oiWFp0qC1uxpx+6yCd6p/zJw.
Please contact your system administrator.
Add correct host key in /home/dinis/.tsh/known_hosts to get rid of this message.
Offending RSA key in /home/dinis/.tsh/known_hosts:3
Host key for [mynode.lenixleaf]:4444 has changed and you have requested strict checking.
Host key verification failed.

@capnspacehook
Copy link
Contributor

OpenSSH nodes can't be registered pre v13, that's right. I'm assuming @r0mant just meant connecting to OpenSSH nodes with tsh or ssh shouldn't have changed in behavior from v12 to v13+.

I can repro the host key changing bug on a remote cluster though.

@Joerger
Copy link
Contributor

Joerger commented Sep 9, 2023

I got curious about this Issue after reading @capnspacehook's status so I took a look as well.

It looks like the agentless node public key exchange is passing the root cluster's Host CA instead of the leaf cluster's. I suspect this is because we are connecting to the leaf cluster through the root Proxy.

In the example below, you can see agentless nodes from both leaf and root pass the root cluster's ssh public key - ssh-rsa SHA256:3mJp7uQQBpToHZ2FN3Fu2rqgo1hwI98bGwK9l0iDeJY

> ssh -v -p 22 agentless-node.leaf.example.com
...
debug1: Server host certificate: [email protected] SHA256:8r4gvTFHy8Eb9AlGXku7juX5TGdOmXandxy/9LV4qZ4, serial 0 ID "" CA ssh-rsa SHA256:3mJp7uQQBpToHZ2FN3Fu2rqgo1hwI98bGwK9l0iDeJY valid after 2023-09-08T19:19:17
debug1: No matching CA found. Retry with plain key
...

> ssh -v -p 22 agentless-node.root.example.com
...
debug1: Server host certificate: [email protected] SHA256:N1t5HtAUN8dchHRjHta4X8Obp4CqknpUAKujP9S3QvY, serial 0 ID "" CA ssh-rsa SHA256:3mJp7uQQBpToHZ2FN3Fu2rqgo1hwI98bGwK9l0iDeJY valid after 2023-09-08T19:22:07
debug1: Host 'agentless-node.root.example.com' is known and matches the RSA-CERT host certificate.
...

If you update the Root CA entry in ~/.tsh/known_hosts to include the hostname *.leaf.example.com, then you'll get this error instead:

debug1: Found CA key in /home/bjoerger/.tsh/known_hosts:1
Certificate invalid: name is not a listed principal

A better workaround is to jumphost through the leaf cluster proxy. Here's a scuffed example, it'd be cleaner using Proxy Templates:

> cat ~/.ssh/config
...
Host *.leaf.example.com !root.example.com agentless-node
    Port 5022
    ProxyCommand "/home/bjoerger/gravitational/teleport/build/tsh" proxy ssh -J leaf.example.com:5080 %r@agentless-node:%p

> ssh -v -p 22 agentless-node.leaf.example.com
// Works first time without prompting for you to accept unkown host key

@strideynet
Copy link
Contributor

strideynet commented Jan 16, 2024 •
edited
Loading

The issue of the proxy connections presenting a certificate signed by the root rather than the leaf for an agentless node is still occurring as of test plan v15.

In addition, the wrong cluster name is used for the principals - leading to the Certificate invalid: name is not a listed principal error if you patch the known_hosts file:

debug1: Server host certificate: [email protected] SHA256:28AeviUZNXceNp6sSwAMI65utTJXkH+ngGdLDico5t4, serial 0 ID "" CA ssh-rsa SHA256:Vcx3pHuh1dhowW4ssS7hXLx7Lqc6sfg9NjmaL8wFo2Y valid after 2024-01-16T17:03:50
debug2: Server host certificate hostname: noah-v15-agentless.root.tele.ottr.sh
debug2: Server host certificate hostname: noah-v15-agentless
debug2: Server host certificate hostname: localhost
debug2: Server host certificate hostname: 127.0.0.1
debug2: Server host certificate hostname: ::1
debug2: Server host certificate hostname: f1a3daa0-3582-48fb-b30a-485cddfe45b2.leaf.tele.ottr.sh
debug2: Server host certificate hostname: 34.171.84.37
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'noah-v15-agentless.leaf.tele.ottr.sh' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in /Users/noah/.tsh/known_hosts:2
Certificate invalid: name is not a listed principal

This was referenced Jan 16, 2024
Closed
Open
@strideynet
Copy link
Contributor

Moving discussion of the cert issues to #36801

Closing this issue as the original reported problem is resolved (it was an issue with the port number)

@strideynet strideynet closed this as not planned Won't fix, can't repro, duplicate, stale Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lxea lxea

@capnspacehook capnspacehook

Labels
bug test-plan-problem Issues which have been surfaced by running the manual release test plan
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@r0mant @marcoandredinis @lxea @strideynet @Joerger @capnspacehook