Modify Unit Test to Support Alert Suppression for EQL Sequences

[shashank-elastic]

Pull Request

Issue link(s): #4326

Summary - What I changed

• For stack versions 8.18 and above the Unit Test test_eql_non_sequence_support_only can be safely ignored.
• We don't have to test if alert suppression is used in sequences only , With [Security Solution][Detection Engine] Adds support for suppressing EQL sequence alerts kibana#189725 starting in 8.18 alert suppression for eql sequences is available.
• Code change is made to skip the test accordingly

How To Test

8.18 Testing

detection-rules on  issue-4326 [$!?] is 📦 v0.4.10 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] 
❯ pytest  -v tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.12.8, pytest-8.1.1, pluggy-1.4.0 -- /Users/shashankks/elastic_workspace/detection-rules/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                          

tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only SKIPPED (Test only applicable to 8.14 to 8.17 stacks for eql ...) [100%]

============================================================== 1 skipped in 62.44s (0:01:02) ==============================================================

8.17 Testing

 pytest  -v tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.12.8, pytest-8.1.1, pluggy-1.4.0 -- /Users/shashankks/elastic_workspace/detection-rules/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                                          

tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only PASSED                                                            [100%]

=================================================================== 1 passed in 54.16s ====================================================================
(.venv) 

• Remote Unit Tests are also skipping the test appropriately

• Unit test to pass

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Code changes do not introduce new warnings or errors.
• Variables and functions are well-named and descriptive.
• Any unnecessary / commented-out code is removed.
• Ensure that the code is modular and reusable where applicable.
• Check for proper exception handling and messaging.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.

Additional Checks

• Ensure that the enhancement does not break existing functionality.
• Review the enhancement with a peer or team member for additional insights.
• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that all dependencies are up-to-date and compatible with the changes.
• Confirm that the proper version label is applied to the PR patch, minor, major.

[Mikaayenson]

For testing theres more we can do.

• Can you build the rule in kibana then try exporting and importing the ndjson back and forth into kibana
• Can you share the ndjson and toml file
• Can you share the output of the make test-cli and make test-cli-remote commands

[shashank-elastic]

Additional Testing

• Create a EQL Rule with Alert Supression on 8.18 Kibana
  

Successful Rule Import to Repo

❯ python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/rules_export_eql_supression.ndjson --required-only  
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/custom_rules/rules/eql_sequence_alert_supression.toml
1 results exported
1 rules converted
0 exceptions exported
0 actions connectors exported
(.venv) 
detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] 
❯ 

Successful Rule Export from Kibana

❯  python -m detection_rules kibana export-rules -r "4a06ba5b-09f1-4522-a91a-a40d014e2a37" -d /Users/shashankks/elastic_workspace/detection-rules/custom_rules
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 results exported
1 rules converted
0 exceptions exported
0 action connectors exported
1 rules saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules
0 exception lists saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/exceptions
0 action connectors saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/action_connectors
(.venv) 
detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 2s 

Successful Rule Import from Kibana

Expected to Fail as rule exists

❯ python -m detection_rules kibana import-rules -id "4a06ba5b-09f1-4522-a91a-a40d014e2a37"                                      
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 rule(s) failed to import!
 - 4a06ba5b-09f1-4522-a91a-a40d014e2a37: (409) rule_id: "4a06ba5b-09f1-4522-a91a-a40d014e2a37" already exists
(.venv) 
detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 3s 
❯ 

Changed rule id and name just to test

❯ python -m detection_rules kibana import-rules -id "4a06ba5b-09f1-4522-a91a-a40d014e2a38" 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

WARNING: Rule path does not match required path: eql_sequence_alert_supression.toml != eql_sequence_alert_supression_1.toml
1 rule(s) successfully imported
 - 4a06ba5b-09f1-4522-a91a-a40d014e2a38
(.venv) 
detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 5s 

• Files for reference

rules_export_eql_supression.ndjson.txt
eql_sequence_alert_supression.toml.txt

make test-cli

Exporting rule by ID: 0a97b20f-4144-49ea-be32-b540ecc445de
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Importing rule by ID: 0a97b20f-4144-49ea-be32-b540ecc445de
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

make test-remote-cli

Running detection-rules remote CLI tests...
Performing a quick rule alerts search...
Requires .detection-rules-cfg.json credentials file set.
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

No alerts detected
Performing a rule export...
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
- skipping Potential Abuse of Resources by High Token Count and Large Response Sizes - ValidationError
6 results exported
5 rules converted
0 exceptions exported
0 action connectors exported
5 rules saved to tmp-export
0 exception lists saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/exceptions
0 action connectors saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/action_connectors
1 errors saved to tmp-export/_errors.txt

Custom Rule has an error local to my branch, It does not have the latest KEEP command for ES|QL

[eric-forte-elastic]

For your additional testing, can you also run the unit tests between import and export? Remember the import and export only puts the rules through schema validation and not through the unit testing (which the two test scripts also do not check).

Also as a note for the python -m detection_rules kibana import-rules -id "4a06ba5b-09f1-4522-a91a-a40d014e2a37" you can also use the -o overwrite flag to import the same rule to check that way if preferred.

[shashank-elastic]

@eric-forte-elastic

On main/pvt branch after importing the rule. The test is skipped

detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] 
❯ python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/rules_export_eql_supression.ndjson --required-only  
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/rules/eql_sequence_alert_supression.toml
1 results exported
1 rules converted
0 exceptions exported
0 actions connectors exported
(.venv) 

detection-rules on  issue-4326 [$?] is 📦 v0.4.11 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] 
❯ pytest  -v tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only                                   
========================================================== test session starts ===========================================================
platform darwin -- Python 3.12.8, pytest-8.1.1, pluggy-1.4.0 -- /Users/shashankks/elastic_workspace/detection-rules/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                         

tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only SKIPPED (Test is applicable to 8.14 --> 8.17...) [100%]

===================================================== 1 skipped in 62.47s (0:01:02) ======================================================

On say a older protected branch 8.17. The test runs and errors as expected

❯ pytest  -v tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only
========================================================== test session starts ===========================================================
platform darwin -- Python 3.12.8, pytest-8.1.1, pluggy-1.4.0 -- /Users/shashankks/elastic_workspace/detection-rules/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/shashankks/elastic_workspace/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 1 item                                                                                                                         

tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only FAILED                                           [100%]

================================================================ FAILURES ================================================================
________________________________________ TestAlertSuppression.test_eql_non_sequence_support_only _________________________________________

self = <tests.test_all_rules.TestAlertSuppression testMethod=test_eql_non_sequence_support_only>

    @unittest.skipIf(PACKAGE_STACK_VERSION < Version.parse("8.14.0"),
                     "Test only applicable to 8.14+ stacks for eql non-sequence rule alert suppression feature.")
    def test_eql_non_sequence_support_only(self):
        for rule in self.all_rules:
            if (
                isinstance(rule.contents.data, EQLRuleData) and rule.contents.data.get("alert_suppression")
                and rule.contents.data.is_sequence  # noqa: W503
            ):
                # is_sequence method not yet available during schema validation
                # so we have to check in a unit test
>               self.fail(
                    f"{self.rule_str(rule)} Sequence rules cannot have alert suppression"
                )
E               AssertionError: 4a06ba5b-09f1-4522-a91a-a40d014e2a37 - EQL Sequence Alert Supression -> Sequence rules cannot have alert suppression

tests/test_all_rules.py:1462: AssertionError
======================================================== short test summary info =========================================================
FAILED tests/test_all_rules.py::TestAlertSuppression::test_eql_non_sequence_support_only - AssertionError: 4a06ba5b-09f1-4522-a91a-a40d014e2a37 - EQL Sequence Alert Supression -> Sequence rules cannot have alert suppression
=========================================================== 1 failed in 53.61s ===========================================================
(.venv) 
detection-rules on  8.17 [$!+?⇣] is 📦 v0.4.8 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 54s 
❯