DEV-46209 - Fix evaluation headers concurrency issue #37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also added the following changes: * small refactor on how headers are built based on similar existing code * added / changed logs to contain rule and org information for easier investigations * removed in eval.go some redundant logz changed (reverted to original code)
|
run-test |
ohadza
approved these changes
Aug 20, 2024
ohadza
approved these changes
Aug 20, 2024
yasmin-tr
deleted the
DEV-46209-fix-evaluation-headers-concurrency-issue
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this feature?
This is a fix to bug introduced on the code adding logzio headers to the alert evaluation.
We got the exception of:
fatal error: concurrent map iteration and map writethat was crashing the service.This happened more often the more alert rules were evaluated at the same time, and the exception was originating at the following place in eval.go : 321
The error means that the map object (the headers) had access and write operations at the same time.
Root Cause
The root cause for this is that the headers were added from the api request context and since the evaluate API is done in bulk for multiple alert rules at the same time, they were sharing the same headers object - which is passed as reference into the evaluation.
What Was the Fix?
To fix the concurrency issue of accessing the same object we needed to clone it.
So instead of passing same headers reference we just cloned it to be a separate reference for each alert evaluation that is sent.
** Also added the following changes:**
Why do we need this feature?
The obvious reason - server should not crash because of error when running multiple alert evaluations.
We need to support big scale and evaluation of a lot of alerts a the same time.