[Security Content] Introduce Investigate Plugin in Investigation Guides

[w0rk3r]

Issues

https://github.com/elastic/security-team/issues/5456
https://github.com/elastic/ia-trade-team/issues/152

Summary

BLOCKED until we work on the unit tests from https://github.com/elastic/security-team/issues/5456

Introduces the usage of the investigate plugin in the investigation guides.

I followed the format we used in the osquery transform, where label and providers are required fields and relativeTo and relativeFrom are optional.

Here is what it looks like in the investigation guides:

Here are the raw syntax for them:

!{investigate{"label":"Alerts associated with the host in the last 48h","providers":[[{"field":"event.kind","excluded":false,"queryType":"phrase","value":"signal","valueType":"string"},{"field":"host.name","excluded":false,"queryType":"phrase","value":"{{host.name}}","valueType":"string"}]],"relativeFrom":"now-48h/h","relativeTo":"now"}}

!{investigate{"label":"Alerts associated with the user in the last 48h","providers":[[{"field":"event.kind","excluded":false,"queryType":"phrase","value":"signal","valueType":"string"},{"field":"user.id","excluded":false,"queryType":"phrase","value":"{{user.id}}","valueType":"string"}]],"relativeFrom":"now-48h/h","relativeTo":"now"}}

!{investigate{"label":"Investigate the Subject Process Network Events","providers":[[{"field":"process.entity_id","excluded":false,"queryType":"phrase","value":"{{process.entity_id}}","valueType":"string"},{"field":"event.category","excluded":false,"queryType":"phrase","value":"network","valueType":"string"}]]}}

[w0rk3r]

@Mikaayenson I've added a compatibility note as this syntax will be only compatible with 8.8+ stacks only

[brokensound77]

We will need to add support for this in the RuleTransforms dataclass. I will try to knock it out real fast

[brokensound77]

I added initial support for investigate to the Transform schema in 5c28b5a. It is failing, so we will need to check if the existing manually converted transforms in these rules are correct, or if the schema needs tweaking (possibly lists).

The values of the new types in definitions may also need expanding

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[brokensound77]

Docs do not show description as optional, but @w0rk3r testing shows that it does - updating schema

[brokensound77]

(venv38) ➜  detection-rules-fork git:(initial_inv_queries) ✗ python -m detection_rules view-rule detection-rules-fork/rules/windows/command_and_control_common_webservices.toml --rule-format

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[metadata]
creation_date = "2020/11/04"
integration = ["endpoint"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.3.0"
updated_date = "2023/12/07"

[rule]
author = ["Elastic"]
description = """
Adversaries may implement command and control (C2) communications that use common web services to hide their activity.
This attack technique is typically targeted at an organization and uses web services common to the victim network, which
allows the adversary to blend into legitimate traffic activity. These popular services are typically targeted since they
have most likely been used before compromise, which helps malicious traffic blend in.
"""
from = "now-9m"
index = ["logs-endpoint.events.*"]
language = "eql"
license = "Elastic License v2"
name = "Connection to Commonly Abused Web Services"
note = """## Triage and analysis

### Investigating Connection to Commonly Abused Web Services

Adversaries may use an existing, legitimate external Web service as a means for relaying data to/from a compromised system. Popular websites and social media acting as a mechanism for C2 may give a significant amount of cover due to the likelihood that hosts within a network are already communicating with them prior to a compromise.

This rule looks for processes outside known legitimate program locations communicating with a list of services that can be abused for exfiltration or command and control.

> **Note**:
> This investigation guide uses the [Osquery Markdown Plugin](https://www.elastic.co/guide/en/security/master/invest-guide-run-osquery.html) introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display unrendered Markdown in this guide.
> This investigation guide uses the [Investigate Markdown Plugin](https://www.elastic.co/guide/en/security/master/interactive-investigation-guides.html) introduced in Elastic Stack version 8.8.0. Older Elastic Stack versions will display unrendered Markdown in this guide.

#### Possible investigation steps

- Investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.
- Investigate other alerts associated with the user/host during the past 48 hours.
  - !{investigate{"label":"Alerts associated with the user in the last 48h","providers":[[{"excluded":false,"field":"event.kind","queryType":"phrase","value":"signal","valueType":"string"},{"excluded":false,"field":"user.id","queryType":"phrase","value":"{{user.id}}","valueType":"string"}]],"relativeFrom":"now-48h/h","relativeTo":"now"}}
  - !{investigate{"label":"Alerts associated with the host in the last 48h","providers":[[{"excluded":false,"field":"event.kind","queryType":"phrase","value":"signal","valueType":"string"},{"excluded":false,"field":"host.name","queryType":"phrase","value":"{{host.name}}","valueType":"string"}]],"relativeFrom":"now-48h/h","relativeTo":"now"}}
- Verify whether the digital signature exists in the executable.
- Identify the operation type (upload, download, tunneling, etc.).
- Examine the host for derived artifacts that indicate suspicious activities:
  - Analyze the process executable using a private sandboxed analysis system.
  - Observe and collect information about the following activities in both the sandbox and the alert subject host:
    - Attempts to contact external domains and addresses.
      - Use the Elastic Defend network events to determine domains and addresses contacted by the subject process by filtering by the process' `process.entity_id`.
        - !{investigate{"label":"Investigate the Subject Process Network Events","providers":[[{"excluded":false,"field":"process.entity_id","queryType":"phrase","value":"{{process.entity_id}}","valueType":"string"},{"excluded":false,"field":"event.category","queryType":"phrase","value":"network","valueType":"string"}]]}}
      - Examine the DNS cache for suspicious or anomalous entries.
        - !{osquery{"label":"Osquery - Retrieve DNS Cache","query":"SELECT * FROM dns_cache"}}
    - Use the Elastic Defend registry events to examine registry keys accessed, modified, or created by the related processes in the process tree.
    - Examine the host services for suspicious or anomalous entries.
      - !{osquery{"label":"Osquery - Retrieve All Services","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services"}}
      - !{osquery{"label":"Osquery - Retrieve Services Running on User Accounts","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services WHERE\nNOT (user_account LIKE '%LocalSystem' OR user_account LIKE '%LocalService' OR user_account LIKE '%NetworkService' OR\nuser_account == null)\n"}}
      - !{osquery{"label":"Osquery - Retrieve Service Unsigned Executables with Virustotal Link","query":"SELECT concat('https://www.virustotal.com/gui/file/', sha1) AS VtLink, name, description, start_type, status, pid,\nservices.path FROM services JOIN authenticode ON services.path = authenticode.path OR services.module_path =\nauthenticode.path JOIN hash ON services.path = hash.path WHERE authenticode.result != 'trusted'\n"}}
  - Retrieve the files' SHA-256 hash values using the PowerShell `Get-FileHash` cmdlet and search for the existence and reputation of the hashes in resources like VirusTotal, Hybrid-Analysis, CISCO Talos, Any.run, etc.

### False positive analysis

- This rule has a high chance to produce false positives because it detects communication with legitimate services. Noisy false positives can be added as exceptions.

### Response and remediation

- Initiate the incident response process based on the outcome of the triage.
- Isolate the involved host to prevent further post-compromise behavior.
- If the triage identified malware, search the environment for additional compromised hosts.
  - Implement temporary network rules, procedures, and segmentation to contain the malware.
  - Stop suspicious processes.
  - Immediately block the identified indicators of compromise (IoCs).
  - Inspect the affected systems for additional malware backdoors like reverse shells, reverse proxies, or droppers that attackers could use to reinfect the system.
- Remove and block malicious artifacts identified during triage.
- Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.
- Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.
- Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).
"""
risk_score = 21
rule_id = "66883649-f908-4a5b-a1e0-54090a1d3a32"
severity = "low"
tags = [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Threat Detection",
    "Tactic: Command and Control",
    "Resources: Investigation Guide",
    "Data Source: Elastic Defend",
]
timestamp_override = "event.ingested"
type = "eql"

query = '''
network where host.os.type == "windows" and network.protocol == "dns" and
    process.name != null and user.id not in ("S-1-5-18", "S-1-5-19", "S-1-5-20") and
    /* Add new WebSvc domains here */
    dns.question.name :
    (
        "raw.githubusercontent.*",
        "*.pastebin.*",
        "*drive.google.*",
        "*docs.live.*",
        "*api.dropboxapi.*",
        "*dropboxusercontent.*",
        "*onedrive.*",
        "*4shared.*",
        "*.file.io",
        "*filebin.net",
        "*slack-files.com",
        "*ghostbin.*",
        "*ngrok.*",
        "*portmap.*",
        "*serveo.net",
        "*localtunnel.me",
        "*pagekite.me",
        "*localxpose.io",
        "*notabug.org",
        "rawcdn.githack.*",
        "paste.nrecom.net",
        "zerobin.net",
        "controlc.com",
        "requestbin.net",
        "cdn.discordapp.com",
        "discordapp.com",
        "discord.com",
        "script.google.com",
        "script.googleusercontent.com"
    ) and
    /* Insert noisy false positives here */
    not (
      (
        process.executable : (
          "?:\\Program Files\\*.exe",
          "?:\\Program Files (x86)\\*.exe",
          "?:\\Windows\\System32\\WWAHost.exe",
          "?:\\Windows\\System32\\smartscreen.exe",
          "?:\\Windows\\System32\\MicrosoftEdgeCP.exe",
          "?:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\*\\MsMpEng.exe",
          "?:\\Users\\*\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe",
          "?:\\Users\\*\\AppData\\Local\\BraveSoftware\\*\\Application\\brave.exe",
          "?:\\Users\\*\\AppData\\Local\\Vivaldi\\Application\\vivaldi.exe",
          "?:\\Users\\*\\AppData\\Local\\Programs\\Opera*\\opera.exe",
          "?:\\Users\\*\\AppData\\Local\\Programs\\Fiddler\\Fiddler.exe",
          "?:\\Users\\*\\AppData\\Local\\Programs\\Microsoft VS Code\\Code.exe",
          "?:\\Users\\*\\AppData\\Local\\Microsoft\\OneDrive\\OneDrive.exe",
          "?:\\Windows\\system32\\mobsync.exe",
          "?:\\Windows\\SysWOW64\\mobsync.exe"
        ) and process.code_signature.trusted == true
      ) or
    
      /* Discord App */
      (process.name : "Discord.exe" and (process.code_signature.subject_name : "Discord Inc." and
       process.code_signature.trusted == true) and dns.question.name : ("discord.com", "cdn.discordapp.com", "discordapp.com")
      ) or 

      /* MS Sharepoint */
      (process.name : "Microsoft.SharePoint.exe" and (process.code_signature.subject_name : "Microsoft Corporation" and
       process.code_signature.trusted == true) and dns.question.name : "onedrive.live.com"
      ) or 

      /* Firefox */
      (process.name : "firefox.exe" and (process.code_signature.subject_name : "Mozilla Corporation" and
       process.code_signature.trusted == true)
      ) or 

      /* Dropbox */
      (process.name : "Dropbox.exe" and (process.code_signature.subject_name : "Dropbox, Inc" and
       process.code_signature.trusted == true) and dns.question.name : ("api.dropboxapi.com", "*.dropboxusercontent.com")
      ) or 

      /* Obsidian - Plugins are stored on raw.githubusercontent.com */
      (process.name : "Obsidian.exe" and (process.code_signature.subject_name : "Dynalist Inc" and
       process.code_signature.trusted == true) and dns.question.name : "raw.githubusercontent.com"
      ) or 

      /* WebExperienceHostApp */
      (process.name : "WebExperienceHostApp.exe" and (process.code_signature.subject_name : "Microsoft Windows" and
       process.code_signature.trusted == true) and dns.question.name : ("onedrive.live.com", "skyapi.onedrive.live.com")
      )
    )
'''


[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1102"
name = "Web Service"
reference = "https://attack.mitre.org/techniques/T1102/"

[[rule.threat.technique]]
id = "T1568"
name = "Dynamic Resolution"
reference = "https://attack.mitre.org/techniques/T1568/"
[[rule.threat.technique.subtechnique]]
id = "T1568.002"
name = "Domain Generation Algorithms"
reference = "https://attack.mitre.org/techniques/T1568/002/"



[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"
[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1567"
name = "Exfiltration Over Web Service"
reference = "https://attack.mitre.org/techniques/T1567/"
[[rule.threat.technique.subtechnique]]
id = "T1567.001"
name = "Exfiltration to Code Repository"
reference = "https://attack.mitre.org/techniques/T1567/001/"

[[rule.threat.technique.subtechnique]]
id = "T1567.002"
name = "Exfiltration to Cloud Storage"
reference = "https://attack.mitre.org/techniques/T1567/002/"



[rule.threat.tactic]
id = "TA0010"
name = "Exfiltration"
reference = "https://attack.mitre.org/tactics/TA0010/"

[brokensound77]

[w0rk3r]

Tested with: py.exe -m detection_rules export-rules --rule-id 66883649-f908-4a5b-a1e0-54090a1d3a32

It compiles the markdown just fine:

[Mikaayenson]

@brokensound77 we still need to update test_if_plugins_explicitly_defined

[brokensound77]

@brokensound77 we still need to update test_if_plugins_explicitly_defined

Thanks, updated!

[Mikaayenson]

LGTM, before merging it might be good to update the CLI.md with the example:

(detection_dev) ➜  detection-rules git:(initial_inv_queries) python -m detection_rules dev transforms guide-plugin-convert
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Enter plugin contents []: !{investigate{"label":"Alerts associated with the host in the last 48h","providers":[[{"field":"event.kind","excluded":false,"queryType":"phrase","value":"signal","valueType":"string"},{"field":"host.name","excluded":false,"queryType":"phrase","value":"{{host.name}}","valueType":"string"}]],"relativeFrom":"now-48h/h","relativeTo":"now"}}
[transform]

[[transform.investigate]]
label = "Alerts associated with the host in the last 48h"
providers = [[{field = "event.kind", excluded = false, queryType = "phrase", value = "signal", valueType = "string"}, {field = "host.name", excluded = false, queryType = "phrase", value = "{{host.name}}", valueType = "string"}]]
relativeFrom = "now-48h/h"
relativeTo = "now"