Azure VM Discovery and joining does not support VMs belonging to a VMSS

[hugoShaka]

I'm not sure if this is a bug or a feature request.

Expected behaviour:

Azure VM Discovery service discovers VMs belonging to a VM Scale Set (VMs replicated).

Current behaviour:

Azure VM Discovert service does not see VMs in a VMSS, even with the appropriate permissions.

Bug details:

• Teleport version: v14.0.0-beta.1
• Recreation steps:
  • create a VMSS through the UI (here the VMSS is called mininode)
  • validate the discovery VM can see the VMSS (az vmss list)
  • run the discovery service with --debug
  • observe that it does not discover VMs from the VMSS (here it should see 2 VMs mininode_1 and mininode_2)
• Debug logs:

  2023-09-12T15:35:22Z DEBU [PROC:1]    Service has started. pid:29262.1 service:discovery.stop service/supervisor.go:280
  2023-09-12T15:35:22Z INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. pid:29262.1 service/connect.go:683
  2023-09-12T15:35:22Z DEBU [DISCOVERY] Azure instances discovered (SubscriptionID: 060a97ea-3a57-4218-9be5-dba3f19ff2b5, Instances: [discovery-agent, node3]), starting installation pid:29262.1 discovery/discovery.go:764
  2023-09-12T15:35:22Z DEBU [DISCOVERY] Running Teleport installation on these virtual machines: SubscriptionID: 060a97ea-3a57-4218-9be5-dba3f19ff2b5, VMs: [discovery-agent, node3] pid:29262.1 discovery/discovery.go:732
  

Unlike most cloud providers, VMs belonging to a VMSS are not exposed through the regular VM list endpoint on Azure. You must hit the VMSS endpoint to get a list of VMSSes and their VMs.

[strideynet]

It looks like this potentially affects more than just discovery and may also be preventing joining for nodes within a VMSS:

Instance failed to establish connection to cluster: rpc error: code = PermissionDenied desc = no VM found with matching VM ID

[hugoShaka]

Yup, in the current azure join flow, a VM from a VMSS cannot join as it is not listed by the VM endpoint.
Implementing https://github.com/gravitational/teleport.e/issues/2164 would allow VMSS-joining but not VM discovery.
VM discovery would also require the discovery service to consume the VMSS endpoint.