Metric "server_interactive_sessions_total" missing IoT sessions

[travelton]

Description

What happened:
The metrics server_interactive_sessions_total, exported via --diag-addr, is not counting interactive sessions for IoT sessions.

What you expected to happen:
IoT sessions should be counted in server_interactive_sessions_total metric.

How to reproduce it (as minimally and precisely as possible):

1. Enable --diag-addr=127.0.0.1:3000 on Teleport Cluster.
2. Ensure an IoT node is joined.
3. curl http://127.0.0.1:3000/metrics | grep server_interactive_sessions_total.
4. Expect sessions to be 0.
5. Connect to IoT node.
6. Expect sessions to be 1. (It's not, it stays zero)

Environment

• Teleport version (use teleport version): 4.2.9

[webvictim]

In addition (while validating this report), I noticed that the server_interactive_sessions_total counter on my personal Teleport installation (1 directly connected node, ~6 IoT nodes) behaves weirdly.

No sessions connected:

$ curl -s http://localhost:3434/metrics | grep sessions
# HELP server_interactive_sessions_total Number of active sessions
# TYPE server_interactive_sessions_total gauge
server_interactive_sessions_total -1

Start a session (to the non-IoT node), then check again:

$ curl -s http://localhost:3434/metrics | grep sessions
# HELP server_interactive_sessions_total Number of active sessions
# TYPE server_interactive_sessions_total gauge
server_interactive_sessions_total 0

Disconnect session and check again:

$ curl -s http://localhost:3434/metrics | grep sessions
# HELP server_interactive_sessions_total Number of active sessions
# TYPE server_interactive_sessions_total gauge
server_interactive_sessions_total -1

The value for sessions shouldn't be able to go below zero.

[webvictim]

:(

[russjones]

Best: 2
Worst: 3

[awly]

Figured out the issue @webvictim was having: #4228

As for IoT nodes - the sessions counter works as expected on the node itself. @travelton, which teleport process were you watching those metrics on? Node or proxy?
We only count the sessions on the node (doesn't matter whether IoT and regular).
Maybe we should be keeping an aggregate counter on the proxy?

[zmb3]

Pretty sure @travelton filed this on my behalf. We were watching metrics on the proxy. Would love to see an aggregate counter maintained by the proxy, as the nodes themselves are often deployed in hard-to-reach network segments, making scraping them directly more difficult than scraping the proxy.

[awly]

@zmb3 added proxy_ssh_sessions_total, a new metric on the proxy for an aggregate counter.
It will appear in 4.4.