[ResponseOps][flapping] change action behavior when flapping

[doakalexi]

Resolves #143445

Summary

This pr changes the behavior of alerting actions respecting flapping.
To recover a flapping alert the alert must be reported as recovered for 4 (this will be configurable eventually) consecutive runs before the framework makes the alert as recovered.

Link to a spreadsheet that helps vizualize the changes: https://docs.google.com/spreadsheets/d/1amoUDFx-Sdxirnabd_E4OSnyur57sLhY6Ce79oDYgJA/edit#gid=0

Also Resolves #147205 to recover those alerts

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

To verify

• Create an alert and let it run until it gets into a flapping state (changes from active/recovered at least 4 times)
• Once flapping, verify the following cases:
• Verify that a recovered alert will be returned as an active notification 3 times before finally recovering
• Verify that an alert that does not recover 3 times before an active alert is reported never reports as a recovered alert

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[weltenwort]

infra plugin code changes LGTM

[CoenWarmer]

AO changes LGTM

[ersin-erdal]

LGTM
Tested locally, works as expected.

[pmuellr]

code LGTM

Only thing I'm curious about is the new pendingRecoveredCount property on the alert meta. Seems like this would be highly correlated with flappingHistory. Like, the number of true entries is probably close to the pendingRecoveredCount. But since we don't know which state was flipped to in the flapping history, just that it changed between active and recovered, we can't actually calculate the precise number that pendingRecoveredCount is tracking. Is that right?

[doakalexi]

But since we don't know which state was flipped to in the flapping history, just that it changed between active and recovered, we can't actually calculate the precise number that pendingRecoveredCount is tracking. Is that right?

Yes that is correct! We also need to reset the value once it's reported active or finally recovered, and I don't think we can use the flapping history to do that.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: d0490ec
• Storybooks Preview

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
alerting	40.7KB	40.7KB	+31.0B

History

• 💔 Build #101759 failed efa8186
• 💚 Build #101048 succeeded afcf9e0
• 💚 Build #100804 succeeded 39e025f
• 💔 Build #100721 failed 30c1094
• 💚 Build #99947 succeeded 6874ace

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream