Investigate centralised rate-limiting system

[gouthamve]

I am now tweaking the limits in our system, and it's confusing. The limits right now are per distributor/ingester. Which means the limits for each user keeps changing as we scale distributors and ingesters. Further if we want to know how much to set it, it should be ActualLimit / # distributors for the distributor limits and for ingester limits it is ActualLimit * 3 / # ingesters where is 3 is replication.

This is super confusing and having global limits makes configuration simpler. Looking at tools available to do that, they rely on consistent stores like etcd (https://github.com/youtube/doorman). Not sure if we want that complexity yet.

Having said that, this tool sounds interesting: https://github.com/tsenart/patrol

[bboreham]

This was discussed at #844 (comment)

[gouthamve]

Yes, that is per-ingester user limits, which ideally should scale with the number of ingesters. This is put in to make sure one user doesn't overload an ingester.

But given we hash on ALL the metric labels (including name) now, I think it is fair to assume that the distribution will be even. What we need is global series limits for each user and an ingester limit for the number of series for all users. i.e, max number of series per ingester.

[tomwilkie]

Also see #255

[pracucci]

I've drafted a design document with a proposal to introduce a global rate limiter:
https://docs.google.com/document/d/1QjPWSobORQ_14_Geaxve1ee5rlfb_nk2GQYl6OZCglc/edit