FEATURE: add cli argument to modify controller workqueue ratelimiter

[ImpSy]

Purpose of this PR

Add the capacity to customize the controller workqueue ratelimiter and remove a potential bug on high usage cluster

Proposed changes:

• cli argument to modify the default controller-runtime workqueue ratelimiter
• Keep the default value (from controller-runtime) in the code but set them in the chart

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

1- controler-runtime uses workqueue.DefaultControllerRateLimiter by default which has a rather low BucketRatelimiter

With only 10 QPS and 100 Bucket max size we easily reach those number, before migrating to controller-runtime the value was actually 50/500 (0dc641b#diff-915090b1cf1857dc448319dbbbc11699ba40e674744723a5772a4624a1c79282L59)

2- BucketRatelimiter does not have a max delay

We encounter a bug on a high usage cluster a few years ago where we discover that BucketRatelimiter rely on golang.org/x/time/rate Limiter.Reserve() function that is not upperBound:

• If QPS > Bucket max size then the rateLimiter return rate.InfDuration

Being able to set a MaxDelay fix this

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

This changed has been implemented on our fork for almost 2 years -> spotinst#5

[vara-bonthu]

Thank you for the contribution! The PR looks good to me, except pending resolution of the comment.

We’re planning to run some load tests in v2 with this change to evaluate if it improves performance and whether we actually need a queue-per-app solution.

Thanks, @ImpSy!

[ImpSy]

@vara-bonthu @ChenYi015 Added the documentation to the charts like you wanted
And thanks for the review 🙏

[vara-bonthu]

/approve

wait for another approval

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015, vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015,vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

/lgtm