[RFC] Support id in threat.indicator for STIX 2.1

[brett-fitz]

Summary

This STIX 2.1 domain object: indicator, is currently represented in the threat.indicator namespace.
Requesting support for field id in the threat field set under the indicator namespace: threat.indicator.id.
This field is a required STIX 2.1 property for indicators.

Motivation:

When storing STIX 2.1 indicators in elasticsearch using ECS schema, the properties id, spec_version, type, created, and
modified must be stored. Currently, the threat field set only maps the following (to my understanding).

• type --> threat.indicator.type
• spec_version --> threat.indicator.marking.tlp_version (loose)
• created --> event.created == threat.indicator.first_seen
• modified --> threat.indicator.modified_at

Detailed Design:

Provide additional details around the design of the proposed changes.

Field names

• threat.indicator.id OR event.id --> Caveat: This doesn't work nicely when multiple indicators are in the same event.

Example values for the fields

• domain-name--ecb120bf-2694-4902-a737-62b74539a41b
• ipv4-address--efcd5e80-570d-4131-b213-62cb18eaa6a8

Suggested appropriate datatypes

• Array[keyword]

Any example events that map to the proposed use case(s)

{
    // Metadata about the indicator event
    "@timestamp": "2019-08-10T11:09:23.000Z",
    "event": {
        "kind": "enrichment",
        "category": "threat",
        "type": "indicator",
        "reference": "https://urlhaus.abuse.ch",
        "severity": 7,
        "risk_score": 10,
        "original": "2020-10-29 19:16:38"
    },

    // Metadata about the indicator data
    "threat.indicator": {
        "first_seen": "2020-11-05T17:25:47.000Z",
        "last_seen": "2020-11-05T17:25:47.000Z",
        "modified_at": "2020-11-05T17:25:47.000Z"
        "sightings": "10",
        "type": [
            "ipv4-addr",
            "port",
            "domain-name",
            "email-addr"
        ],
        "description": "Email address, domain, port, and IP address observed using an Angler EK campaign.",
        "provider": "Abuse.ch",
        "reference": "https://urlhaus.abuse.ch/url/1292596/",
        "confidence": "High",
        "ip": "1.2.3.4",
        "id": [
            "ipv4-address--efcd5e80-570d-4131-b213-62cb18eaa6a8",
            "domain-name--ecb120bf-2694-4902-a737-62b74539a41b",
            "email-addr--2d77a846-6264-5d51-b586-e43822ea1ea3"
        ]
        "domain": "malicious.evil",
        "port": 443,
        "email.address": "[email protected]",
        "marking.tlp": "WHITE",
        "scanner_stats": 4
    },

    // Any indicators should also be copied to relevant related.* field
    "related": {
        "hash": [
            "1eee2bf3f56d8abed72da2bc523e7431",
            "0c415dd718e3b3728707d579cf8214f54c2942e964975a5f925e0b82fea644b4"
        ],
        "hosts": [
            "nefarious.evil"
        ],
        "ip": [
            "1.2.3.4"
        ]
    },

    // Tags for context
    "tags": [
        "threatintel",
        "forwarded"
    ]
}

[P1llus]

Hello @brett-fitz ! Thanks for creating the issue and wanting to expand on the threat.* ECS schema.

From what I can see, there is nothing against having an ID field, which would could include an array of relevant ID's, but there are a few things we might want to clear up first:

1. I assume that the examle data you provided is not from the AbuseCH, as their format returned from the API is not STIX, and their ID format is a bit different? If that's the case then this is not an issue.
2. You mention that event.id could be difficult because one might have multiple indicators in a single document, I do agree because it also can affect the enrichments added when an Indicator Match rule matches, as it usually uses the threat.* fields, however its good to note that it is only meant to store max one of each indicator type, so not an array of IP's etc.
3. Are the ID fields in the format already returned by STIX? Or is it a generated ID format which merges the indicator type and its ID?
4. We might want to allow TI sources that do not use STIX to still populate the ID field, which means the format of the ID's might not always be the same as the one you provided, do you think that would be okay?

[brett-fitz]

Hello @P1llus!

1,4. Yup, any ID value, STIX or custom, is fine. An array here would make sense as well to allow for identification of the same observable across systems that use different ID formats.
2. Appreciate the context there, we were curious whether it was expected to have one or more indicators in an event and how we would connect indicators/observables together. We are using this documentation hub for examples: https://docs.elastic.co/integrations/threat-intelligence-intro
3. Thats one format, you will also see: indicator--9299f726-ce06-492e-8472-2b52ccb53191
Format: [construct type]-[GUID]
Documentation:

• https://stixproject.github.io/documentation/suggested-practices/
• https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_mlbmudhl16lr
• https://oasis-open.github.io/cti-documentation/examples/sighting-of-an-indicator

[brett-fitz]

@P1llus Hey! Curious what the next steps are in the RFC process? Let me know if you need me to submit a PR. 😄

[P1llus]

@ebeahan what is the current process here?

[brett-fitz]

@P1llus @ebeahan any update here? Happy to submit the required schema updates in a PR.

[ebeahan]

@brett-fitz Yeah I think a PR is the next step. We'll find the right folks to weigh-in on the PR and to confirm the proposed addition fits into how we've been using the threat.* fields today.