Rate limits for unauthenticated endpoints can cause denial of service in environments using a single NATed IP

[webvictim]

Expected behavior

Teleport rate limits should be higher for login attempts in NATed office environments, or otherwise configurable through config file/dynamic cluster resource config.

Current behavior

Lots of users logging in at once (for example at the beginning of a day) can cause Teleport to emit errors on login:

Current unauthenticated rate limits appear to be an average of 20 requests per minute with an allowed burst of 40:

This was a Teleport Cloud tenant, but the rate limits are the same for all clusters.

Bug details:

• Teleport version: 13.3.5 11.3.20

[zmb3]

I raised this in #24623 (comment) and some of the limits applied where were reverted in #27747.

The max rate reached error is from oxy's ratelimiter, so it is definitely us and not a rate limit from an upstream API.

[jentfoo]

It's worth calling out that the PR's above exist only in master and wont be released till Teleport 14. So issues in T13 are not from the PR's linked above.

That said, those changes actually may improve the situation in a couple ways:

• Default rates are doubled compared to T13: https://github.com/gravitational/teleport/pull/24623/files#diff-cc34e8b8bef47626c4518b46eb5a56458acedd961d290d5a79b5fe5ff63fa832R370
• In a couple cases the limiter was switched to only limit unauthenticated requests: https://github.com/gravitational/teleport/pull/24623/files#diff-5a12dd7747d28af8fe5043638cc11f31a3314b2a23f3e23fe0bcdc952a805fa6L569-R585 / https://github.com/gravitational/teleport/pull/24623/files#diff-5a12dd7747d28af8fe5043638cc11f31a3314b2a23f3e23fe0bcdc952a805fa6L725-R741
• It adds a concept of a high rate endpoint that we can easily switch endpoints to if needed

So there is some potential that the customer may have an improved experience with T14. But a feature improvement to make these values configurable would be best for everyone.

[zmb3]

Good point. It would be great to know which endpoint is being rate limited above so we can determine if the upcoming changes will help the situation.

[webvictim]

The customer was unable to get further details on the particular endpoint as when they went to run tsh -d login, the rate limit was no longer in force. I suspect it was either webapi/ping or webapi/find though.

[jentfoo]

With 13 neither ping or find have any rate limits:

teleport/lib/web/apiserver.go

Lines 557 to 571 in 8268e8d

	h.GET("/webapi/find", httplib.MakeHandler(h.find))
	// Issue host credentials.
	h.POST("/webapi/host/credentials", httplib.MakeHandler(h.hostCredentials))
	}
	// bindDefaultEndpoints binds the default endpoints for the web API.
	func (h *Handler) bindDefaultEndpoints() {
	h.bindMinimalEndpoints()
	// ping endpoint is used to check if the server is up. the /webapi/ping
	// endpoint returns the default authentication method and configuration that
	// the server supports. the /webapi/ping/:connector endpoint can be used to
	// query the authentication configuration for a specific connector.
	h.GET("/webapi/ping", httplib.MakeHandler(h.ping))
	h.GET("/webapi/ping/:connector", httplib.MakeHandler(h.pingWithConnector))

[webvictim]

My apologies, I made an error - the tenant is on v11.3.20. It seems the limiter code is quite different there.

[jentfoo]

@webvictim is further action needed on this issue?

[webvictim]

Given that the customer will need to upgrade to v12 and beyond soon to remain in support, I think it's OK to close this.