[Bug] Rules being duplicated when updated

[zkink]

Describe the bug
A clear and concise description of what the bug is.
When updating rules, instances appear to be duplicating some rules as part of the update. Within our cluster, we can see the same rule exist multiple times. Both rules appear to be prebuilt rules that have been created by the author "Elastic". This is a few example rules where this can be seen to have occurred.

Example Rules:
Enumeration of Kernel Modules
Rule ID 1: aeffacb0-199f-11ee-be89-1b8cf30adee4
Rule ID 2: 2d8aaff0-4027-11ee-bc86-b1f97eb72d5d

Abnormal Process ID or Lock File Created
Rule ID 1: 1b6df8c0-2599-11ee-8571-99b83d216ac8
Rule ID 2: d76ede10-402c-11ee-bc86-b1f97eb72d5d

For each of these rules (there are a few more), these rules have one show up in "Elastic Rules" and one show up in "Custom Rules". It appears that when rule updates get performed, they sometimes are being put into a custom rule and the elastic rule has stayed unmodified.

To Reproduce
Steps to reproduce the behavior:

1. Go to Detection Rules (SIEM)
2. Click on Rule Updates
3. Click on Update All
4. Once updates have been performed, sometimes (this appears to be an interment issue), rules can be found in Custom rather than prebuilt.

Expected behavior
A clear and concise description of what you expected to happen.
What would be expected is for rules to update rather than creating a new record/ID.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Mac
• Version: Sonoma 14.2.1

Additional context
Add any other context about the problem here.
This appears to be a long-term issue that has been intermittent. It appears only sometimes when rules are updated, and looking in a different instance, a different rule could be duplicated. When looking, it appears to be randomly occurring rather than occurring for the same rule each time.

[approksiu]

@zkink, thanks for opening this bug! Do you mind opening it in https://github.com/elastic/kibana/issues as it relates to the rules management in kibana. This way you will be able to easier track the resolution. Thank you!

[jpdjere]

Hi @zkink. Thanks for reporting this.

++ to recreating this as a Kibana ticket. It's related to Rule Management and not to the Detection Rules themselves, I can follow though better there. Feel free to ping me directly in a new ticket there.

Having said that, could you please let us know the following:

• which version of Kibana are you using?
• which version of the Prebuilt Rules package are/were you on? Before and after the rule updates appeared.
• in cases in which you are seeing duplicated rules, are the rule_ids for the two rules identical? I understand that you posted ids in the ticket body, but I want to make sure that we make a distinction between the rule_id, which identifies a prebuilt rule asset (and should be constant for all versions of a prebuilt rule), vs simply id which is the id of the instance of the rule (and will change when the rule gets updated).

If you can reproduce the issue again, you can get the data for both rules by navigating to the Dev Tools and making the request:

GET .kibana_alerting_cases/_search
{
  "_source": [
    "alert.params.ruleId"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "type": "alert"
          }
        },
        {
          "match_phrase": {
            "alert.name": "Abnormal Process ID or Lock File Created"
          }
        }
      ]
    }
  }
}

and the response should look like:

{
  # ... other fields
  "hits": {
    # ... other fields
    "max_score": 5.1785355,
    "hits": [
      {
        "_index": ".kibana_alerting_cases_8.13.0_001",
        "_id": "alert:b1137f12-4b54-4984-b09b-0188bba052b4",  // <---- instance id
        "_score": 5.1785355,
        "_source": {
          "alert": {
            "params": {
              "ruleId": "c138e3cd-4cae-45a3-bdba-dc45b4ebf700"  // <---- rule_id
            }
          }
        }
      }
    ]
  }
}

[zkink]

Moved to Kibana Repo per request