[Security Solution] Initial rule upgrade/install endpoints implementation

[xcrzx]

Resolves: #148186
Resolves (partially): #148184

Summary

This PR adds two new endpoints:

• POST /internal/detection_engine/prebuilt_rules/installation/_perform (see this issue for more detail on the endpoint interface)
• POST /internal/detection_engine/prebuilt_rules/upgrade/_perform (see this issue for more detail on the endpoint interface)

For both endpoints, I've implemented two modes: ALL_RULES and SPECIFIC_RULES. So from the rules management page, all rules could be installed or upgraded in bulk or one by one if needed.

Things not covered by this PR

• Unit and integrational test for both endpoints
• Rule upgrade: Missing the ability to pick the MERGED version for rule upgrades, but it is not needed so long as we do not allow rule modification
• Rule upgrade: Missing the ability to specify rule field versions to pick

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 3b37309

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	400	404	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	480	484	+4
total			+6

History

• 💚 Build #130136 succeeded 48307d3918bbc356aad570e43931a128585cf8b4
• 💚 Build #129940 succeeded 7a2e0be0bd54ce9ea43718d02c740f1457da38ba
• 💚 Build #129874 succeeded b69be26236f05f6c1340917ac3f79eb679868b2a
• 💔 Build #129803 failed 90dc44906fa6d95b7cde116598629cecf44b0aea
• 💔 Build #129674 failed 2ec1f8585718b4a251ecbcb16e64046774e24269
• 💔 Build #129609 failed 9b944c602007f2273b76e6e9376bcdb1e1854003

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

[jpdjere]

Is the endpoint open to returning other kind of information other than stats in the future; do we have any future use cases for that?

Just wandering why we would hold all data within the single stats property instead of making the stats themselves first-class properties.

[xcrzx]

Currently, I do not anticipate extending the endpoint to return information other than stats. However, it's always good practice to keep future use cases in mind and design with flexibility for potential changes. If a new use case does arise in the future, it would be pretty straightforward to adapt this endpoint.

[jpdjere]

I think it's not necessary to create a Set to de-duplicate the ruleIds, since the Map will override any repeated value with the last key-pair created anyways.
I guess both ways achieve the same thing, not sure if this could make a difference time-complexity-wise, since the number of rules we'll be dealing is not sufficiently large.

[xcrzx]

The idea here is to gather all available rule IDs to iterate over, as these three structures only partially overlap. There could be missing base rules or rules that are no longer available in the package, meaning no latest version, and so on. Ultimately, we aim to create a structure similar to the following:

const map = {
  rule_id_1: {
    current: undefined,
    base: undefined,
    target: {},
  },
  rule_id_2: {
    current: {},
    base: {},
    target: undefined,
  },
};

But a similar result could probably be achieved by iterating over the three structures one by one.

[jpdjere]

So, if I understand correctly, for the current Milestone we won't need to pass the pick_version param, since we will always want to update to the target version of the rule.
Is that correct?

[xcrzx]

Yes, that's right. The version to pick will always be TARGET.

[jpdjere]

Reviewed, great changes, thanks for this! Some of the refactoring you did here on the diff logic made it even easier to understand.

I'm approving now but I haven't tested the endpoints; maybe we can sync tomorrow on how you've been testing? I guess that's where the generate_assets_route comes in handy :)

[xcrzx]

I'm going to merge the PR so @jpdjere can start integrating it as soon as possible. If any issues arise, I can address them in follow-ups. However, since the functionality is under a feature flag, the risk of introducing any bugs is quite low.