[ResponseOps] move alert UUID generation from rule registry to alerting framework

[pmuellr]

Currently the Rule Registry generates a unique UUID for every alert, which will be reused when the same alert with the same id is active during the next execution. When the alert recovers, and becomes active again, with the same alert id, a new UUID will be generated.

This functionality needs to be moved into the alerting framework itself, so it can manage the UUIDs. Instead of the Rule Registry generating the UUID, we'll make it available via the alert functions from the alert factory.

This is a blocker to implement flapping alert detection in alerting, as per current design, alerts that are flapping will reuse the same UUID even after they recover. Only after the alert is no long flapping will the UUID stop being used. Because the alerting framework determines flapping state, and because the rule registry operates as a "wrapper" over the existing alert executors it's used with, the easiest way to deal with this is to have the alerting framework generate the UUID.

Additional advantage - assuming we write these UUIDs into the event log, we'll be able to "join" over this field in searches over the event log and AAD indices combined.

Another advantage is that we'll save some space. The rule registry currently "wraps" a rule's task state to maintain the alert id / uuid relationship. Below is an example of what the task state looks like for a rule with one alert.

The Rule Registry takes the task state returned by a rule, set's it to be the value of the wrapped property, and then adds trackedAlerts which contains the id / uuid mapping. The other top-level properties are maintained by the alerting framework. And you can see that alertInstances contains everything in trackedAlerts but the alert UUID. So, we should be able to remove this wrapped wrapping completely, once we add the alert UUID to ... state? meta? not sure where.

That implies either a migration, or having code that can read both the old wrapped style or new unwrapped style (and presumably always write the new style). But we typically handle these with migrations.

{
  "alertTypeState": {
    "wrapped": {
      "groups": [ "some-thing, metricbeat" ],
      "groupBy": [ "fields.eks_cluster_name", "agent.type" ],
      "filterQuery": ""
    },
    "trackedAlerts": {
      "some-thing, metricbeat": {
        "alertId": "some-thing, metricbeat",
        "alertUuid": "5cee8f86-5d0b-4d4a-92f7-a3c731b7ecc3",
        "started": "2022-03-11T20:46:24.170Z"
      },
    }
  },
  "alertInstances": {
    "some-thing, metricbeat": {
      "state": {
        "start": "2022-03-11T20:46:24.383Z",
        "duration": 16913558599000000
      },
      "meta": {
        "lastScheduledActions": {
          "group": "metrics.threshold.fired",
          "date": "2022-03-11T20:46:24.390Z"
        }
      }
    },
  },
  "previousStartedAt": "2022-09-23T14:59:02.346Z"
}

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)