SNAT port exhaustion alerting #343
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RMcVelia
reviewed
Jan 10, 2025
4 tasks
neillturner
force-pushed
the
port-exhaustion
branch
from
7afe812 to
418e577
Compare
shaheislamdfe
approved these changes
Jan 10, 2025
|
tested this successfully before merging by setting port usage down from 900 to 20 and saw it get triggered and email sent. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
We want to be alerted if SNAT port exhaustion happens on the kubernetes load balancer as well as when port usage is high on the load balancer for a node.
Changes proposed in this pull request
Two Azure alerts created with actions to send to slack.
I don't believe we can use prometheus for this as the load balancer is outside of kubernetes so does report info to prometheus AFAIK.
Guidance to review
this has been applied to the platform test environment via
make platform-test terraform-aks-cluster-apply CONFIRM_PLATFORM_TEST=yes
so can be seen in the azure portal.
Not user which slack webhook to use.
To test the slack integration we could set the ConnectionState to Success instead of Failed to generate an alert
and reduce the port threshold from 900 to a small value to test the only alert.
Checklist
Devopslabel