ruler: cap the number of remote eval retries

[dimitarvdimitrov]

Problem

The retries happen more aggressively than actual evaluations. With the current setup an error spike results in 3x the query rate - initial query, and two retries fairly quickly 100ms & 200ms after that.

What this PR does

Introduce a soft rate limit for the retries. The default is 170 retries/sec, which is half the average rate of rule evaluations in clusters at GL. If a retry is above the rate limit, we'd wait (time.Sleep) until it is within the limit, while also not exceeding a 2s backoff. The idea is not to overextend into the next evaluation.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[github-actions]

💻 Deploy preview deleted.

[tacole02]

Looks great! Thanks, @dimitarvdimitrov !

[bboreham]

The PR title and description seem to apply to a circuit-breaker version.

[bboreham]

Reviewing note: I checked up on this:

// As of Go 1.23, the garbage collector can recover unreferenced,
// unstopped timers. 

[grafanabot]

The backport to r316 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-10375-to-r316 origin/r316
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x ffee57de406dd651dccf104db25ee93ec46a3c0e
# Push it to GitHub
git push --set-upstream origin backport-10375-to-r316
git switch main
# Remove the local backport branch
git branch -D backport-10375-to-r316

Then, create a pull request where the base branch is r316 and the compare/head branch is backport-10375-to-r316.

[grafanabot]

The backport to r320 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-10375-to-r320 origin/r320
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x ffee57de406dd651dccf104db25ee93ec46a3c0e
# Push it to GitHub
git push --set-upstream origin backport-10375-to-r320
git switch main
# Remove the local backport branch
git branch -D backport-10375-to-r320

Then, create a pull request where the base branch is r320 and the compare/head branch is backport-10375-to-r320.

[grafanabot]

The backport to r321 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-10375-to-r321 origin/r321
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x ffee57de406dd651dccf104db25ee93ec46a3c0e
# Push it to GitHub
git push --set-upstream origin backport-10375-to-r321
git switch main
# Remove the local backport branch
git branch -D backport-10375-to-r321

Then, create a pull request where the base branch is r321 and the compare/head branch is backport-10375-to-r321.

[grafanabot]

The backport to r319 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-10375-to-r319 origin/r319
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x ffee57de406dd651dccf104db25ee93ec46a3c0e
# Push it to GitHub
git push --set-upstream origin backport-10375-to-r319
git switch main
# Remove the local backport branch
git branch -D backport-10375-to-r319

Then, create a pull request where the base branch is r319 and the compare/head branch is backport-10375-to-r319.