merging in ATT&CK v10.0 release

Author: isaisabel

CHANGELOG.md

@@ -1,16 +1,44 @@
-### Changes to STIX for April 2021 ATT&CK Content Release (ATT&CK-v9.0)
+
+# Changes to ATT&CK in STIX 2.0
+## 21 October 2021 - ATT&CK Spec v2.1.0
+Changes to ATT&CK in STIX for October 2021 ATT&CK Content Release (ATT&CK-v10.0)
+
+| Feature | [Available in STIX 2.0](https://github.com/mitre/cti) | [Available in STIX 2.1](https://github.com/mitre-attack/attack-stix-data) |
+|:--------|:-----------------------------------------------------:|:-------------------------------------------------------------------------:|
+| Added full objects for data sources and data components. See [the data sources section of the USAGE document](https://github.com/mitre-attack/attack-stix-data/blob/master/USAGE.md#data-sources-and-data-components) for more information about data sources, data components, and their relationships with techniques. | :white_check_mark: | :white_check_mark: |
+| Added `x_mitre_attack_spec_version` field to all object types. This field tracks the version of the ATT&CK Spec used by the object. Consuming software can use this field to determine if the data format is supported; if the field is absent the object will be assumed to use ATT&CK Spec version `2.0.0`. | :x: | :white_check_mark: |
+
+## 21 June 2021 - ATT&CK Spec v2.0.0
+Release of ATT&CK in STIX 2.1.
+
+The contents of this repository is not affected, but you can find ATT&CK in STIX 2.1 (ATT&CK spec v2.0.0+) on our new [attack-stix-data](https://github.com/mitre-attack/attack-stix-data) GitHub repository. Both MITRE/CTI (this repository) and attack-stix-data will be maintained and updated with new ATT&CK releases for the foreseeable future, but the data model of attack-stix-data includes quality-of-life improvements not found on MITRE/CTI. 
+
+| Feature | [Available in STIX 2.0](https://github.com/mitre/cti) | [Available in STIX 2.1](https://github.com/mitre-attack/attack-stix-data) |
+|:--------|:-----------------------------------------------------:|:-------------------------------------------------------------------------:|
+| Added `x_mitre_modified_by_ref` field to all object types. This field tracks the identity of the individual or organization which created the current _version_ of the object. | :x: | :white_check_mark: | 
+| Added `x_mitre_domains` field to all non-relationship objects. This field tracks the domains the object is found in. | :x: | :white_check_mark: |
+| Added [collection](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md) objects to track information about specific releases of the dataset and to allow the dataset to be imported into [ATT&CK Workbench](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/). | :x: | :white_check_mark: |
+| Added a [collection index](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/blob/master/docs/collections.md) to list the contents of this repository and to allow the data to be imported into [ATT&CK Workbench](https://github.com/center-for-threat-informed-defense/attack-workbench-frontend/). | :x: | :white_check_mark: |
+
+## 29 April 2021
+Changes to ATT&CK in STIX for April 2021 ATT&CK Content Release (ATT&CK-v9.0)
+
 1. Replaced `GCP`, `AWS` and `Azure` platforms under the enterprise domain with `IaaS` (Infrastructure as a Service).
 2. Added `Containers` and `Google Workspace` to the platforms of the enterprise domain.
 3. Revised the data sources of the enterprise domain. Data sources are still represented as a string array, but the elements within that array are now formatted `"data source: data component"` to reflect the new data source representation. More information on the new data sources can be found on our [attack-datasources](https://github.com/mitre-attack/attack-datasources) GitHub repository. Note that the data sources in the ICS domain was not affected by this change.
 
 With the release of ATT&CK version 9 we are also hosting an excel representation of the knowledge base on our website. You can find that representation and more about ATT&CK tools on the updated [Working with ATT&CK](https://attack.mitre.org/resources/working-with-attack/) page.
 
-### Changes to STIX for October 2020 ATT&CK Content Release (ATT&CK-v8.0)
+## 27 October 2020
+Changes to ATT&CK in STIX for October 2020 ATT&CK Content Release (ATT&CK-v8.0)
+
 1. Added new platforms under the enterprise domain: `Network` and `PRE`.
 2. Deprecated the pre-ATT&CK domain. Pre-ATT&CK has been migrated to two new tactics in the Enterprise domain tagged with the `PRE` platform. Please see the new [PRE matrix](https://attack.mitre.org/matrices/enterprise/PRE/) for the replacing Enterprise tactics and techniques. All objects within the pre-ATT&CK domain have been marked as deprecated, along with a new description pointing to their new home in Enterprise.
 3. Added the [ATT&CK for ICS domain](ics-attack).
 
-### Changes to STIX for July 2020 ATT&CK Content Release (ATT&CK-v7.0)
+## 8 July 2020 - ATT&CK Spec v1.3.0
+Changes to ATT&CK in STIX for July 2020 ATT&CK Content Release (ATT&CK-v7.0)
+
 1. Added sub-techniques:
     - A sub-technique is an attack-pattern where `x_mitre_is_subtechnique` is `true`. 
     - Relationships of type `subtechnique-of` between sub-techniques and techniques convey their hierarchy.
@@ -20,17 +48,21 @@ With the release of ATT&CK version 9 we are also hosting an excel representation
 
 We've also rewritten the [USAGE](USAGE.md) document with additional information about the ATT&CK data model and more examples of how to access and use ATT&CK in Python.
 
-### Changes to STIX for October 2019 ATT&CK Content Release (ATT&CK-v6.0)
+## 24 October 2019
+Changes to ATT&CK in STIX for October 2019 ATT&CK Content Release (ATT&CK-v6.0)
 1. Added cloud platforms under the enterprise domain: `AWS`, `GCP`, `Azure`, `Office 365`, `Azure AD`, and `SaaS`.
 
-### Changes to STIX for July 2019 ATT&CK Content Release (ATT&CK-v5.0)
+## 31 July 2019
+Changes to ATT&CK in STIX for July 2019 ATT&CK Content Release (ATT&CK-v5.0)
 1. Descriptions added to relationships of type `mitigates` under the enterprise domain 
 
-### Changes to STIX for April 2019 ATT&CK Content Release (ATT&CK-v4.0)
+## 30 April 2019 - ATT&CK Spec v1.2.0
+Changes to ATT&CK in STIX for April 2019 ATT&CK Content Release (ATT&CK-v4.0)
 1. `x_mitre_impact_type` added for enterprise techniques within the `Impact` tactic
 2. Descriptions added to relationships between software/groups
 
-### Changes to STIX for October 2018 ATT&CK Content Release (ATT&CK-v3.0)
+## 23 October 2018 - ATT&CK Spec v1.1.0
+Changes to ATT&CK in STIX for October 2018 ATT&CK Content Release (ATT&CK-v3.0)
 
 1. `x_mitre_platforms` added for enterprise malware/tools
 2. `x_mitre_detection` added to attack-patterns
@@ -44,4 +76,4 @@ We've also rewritten the [USAGE](USAGE.md) document with additional information
 10. Changed ===Windows=== subheadings to ### Windows subheadings (Windows is just one example)
 11. Added space between asterisks (ex. *Content to * Content) to populate markdown correctly
 12. Changed "true" to True in `x_mitre_deprecated`
-13. Added old ATT&CK IDs to Mobile/PRE-ATT&CK objects whose IDs have changed as `x-mitre-old-attack-id`
+13. Added old ATT&CK IDs to Mobile/PRE-ATT&CK objects whose IDs have changed as `x-mitre-old-attack-id`

USAGE.md

@@ -31,6 +31,7 @@ If you are looking for ATT&CK data represented in STIX 2.1, please see our [atta
       - [Collisions with technique ATT&CK IDs](#collisions-with-technique-attck-ids)
     + [Groups](#groups)
     + [Software](#software)
+    + [Data Sources and Data Components](#data-sources-and-data-components)
     + [Relationships](#relationships)
 - [Accessing ATT&CK data in python](#accessing-attck-data-in-python)
   * [Requirements and imports](#requirements-and-imports)
@@ -82,6 +83,7 @@ ATT&CK uses a mix of predefined and custom STIX objects to implement ATT&CK conc
 | [Mitigation](#mitigations)       | [course-of-action](https://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230929) | no |
 | [Group](#groups)                 | [intrusion-set](https://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230941)  | no |
 | [Software](#software)            | [malware](http://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230945) or [tool](http://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230961) | no |
+| [Data Source](#data-source)      | `x-mitre-data-source` | yes |
 
 Two additional object types are found in the ATT&CK catalog:
 
@@ -125,6 +127,7 @@ The most commonly used ID format is what is referred to as the ATT&CK ID or simp
 | [Mitigation](#mitigations)       | `Mxxxx` |
 | [Group](#groups)                 | `Gxxxx`  |
 | [Software](#software)            | `Sxxxx` |
+| [Data Source](#data-source)      | `DSxxxx` |
 
 ATT&CK IDs are typically, but not always, unique. See [Collisions with Technique ATT&CK IDs](#collisions-with-technique-attck-ids) for an edge case involving ID collisions between mitigations and techniques.
 
@@ -178,7 +181,7 @@ Techniques depart from the attack-pattern format with the following fields. Doma
 |:------|:-----|:--------|:------------|
 | `x_mitre_detection` | string | All techniques | Strategies for identifying if a technique has been used by an adversary. |
 | `x_mitre_platforms` | string[] | All techniques | List of platforms that apply to the technique. |
-| `x_mitre_data_sources` | string[] | Enterprise and ICS domains | Sources of information that may be used to identify the action or result of the action being performed. |
+| `x_mitre_data_sources` | string[] | Enterprise* & ICS domains | Sources of information that may be used to identify the action or result of the action being performed. |
 | `x_mitre_is_subtechnique` | boolean | Enterprise domain | If true, this `attack-pattern` is a sub-technique. See [sub-techniques](#sub-techniques). |
 | `x_mitre_system_requirements` | string[] | Enterprise domain | Additional information on requirements the adversary needs to meet or about the state of the system (software, patch level, etc.) that may be required for the technique to work. |
 | `x_mitre_tactic_type` | string[] | Mobile domain |  "Post-Adversary Device Access", "Pre-Adversary Device Access", or "Without Adversary Device Access". |
@@ -189,6 +192,8 @@ Techniques depart from the attack-pattern format with the following fields. Doma
 | `x_mitre_impact_type` | string[] | Enterprise domain in the _Impact_ tactic | Denotes if the technique can be used for integrity or availability attacks. |
 
 
+\* In the Enterprise domain data sources are represented via [x-mitre-data-source](#data-sources) and [x-mitre-data-component](#data-components) objects, and their relationship with techniques through relationships of type `detects`. The `x_mitre_data_sources` field will still be maintained on enterprise techniques for backwards-compatibility purposes but we advise against its use as it does not include the full context of the data model.
+
 See [mapping matrices, tactics and techniques](#mapping-matrices-tactics-and-techniques) for more information about how techniques map into tactics and matrices.
 
 #### Sub-Techniques
@@ -227,8 +232,59 @@ Both `malware` and `tool` type software depart from the STIX format with the fol
 | Field | Type | Description |
 |:------|:-----|-------------|
 | `x_mitre_platforms` | string[] | List of platforms that apply to the software. |
-| `x_mitre_aliases` | string[] | List of aliases for the given software |
+| `x_mitre_aliases` | string[] | List of aliases for the given software. |
+
+
+### Data Sources and Data Components
+
+Data Sources and Data Components represent data which can be used to detect techniques. Data components are nested within a data source but have their own STIX object.
+
+- A Data Component can only have one parent Data Source.
+- A Data Source can have any number of Data Components.
+- Data Components can map to any number of techniques.
+
+The general structure of data sources and data components is as follows:
+
+<!-- diagram generated with https://asciiflow.com/ -->
+```
+           "detects"       x_mitre_data_source_ref
+          relationship      embedded relationship
+               │                      │
+┌───────────┐  ▼  ┌────────────────┐  │  ┌───────────┐
+│Technique 1│◄────┤                │  │  │           │
+└───────────┘     │                │  ▼  │           │
+                  │Data Component 1├────►│           │
+┌───────────┐     │                │     │           │
+│Technique 2│◄────┤                │     │Data Source│
+└───────────┘     └────────────────┘     │           │
+                                         │           │
+┌───────────┐     ┌────────────────┐     │           │
+│Technique 3│◄────┤Data Component 2├────►│           │
+└───────────┘     └────────────────┘     └───────────┘
+```
+
+Prior to ATT&CK v10 data sources were stored in a `x_mitre_data_sources` field on techniques. This representation is still available for backwards-compatibility purposes, and does properly reflect the current set of data sources. However, because information is lost in that representation we advise against using it except in legacy applications. The ATT&CK for ICS domain still uses only the `x_mitre_data_sources` field.
+
+#### Data Sources
+
+A Data Source in ATT&CK is defined by an `x-mitre-data-source` object. As a custom STIX type they follow only the generic [STIX Domain Object pattern](https://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230920). 
 
+Data Sources extend the generic SDO format with the following fields:
+
+| Field | Type | Description |
+|:------|:-----|-------------|
+| `x_mitre_platforms` | string[] | List of platforms that apply to the data source. |
+| `x_mitre_collection_layers` | string[] | List of places the data can be collected from. |
+
+#### Data Components
+
+A Data Component in ATT&CK is represented as an `x-mitre-data-component` object. As a custom STIX type they follow only the generic [STIX Domain Object pattern](https://docs.oasis-open.org/cti/stix/v2.0/csprd01/part2-stix-objects/stix-v2.0-csprd01-part2-stix-objects.html#_Toc476230920). 
+
+Data Components extend the generic SDO format with the following field:
+
+| Field | Type | Description |
+|:------|:-----|-------------|
+| `x_mitre_data_source_ref` | embedded relationship (string) | STIX ID of the data source this component is a part of. |
 
 ### Relationships
 
@@ -245,8 +301,9 @@ Relationships oftentimes have descriptions which contextualize the relationship
 | `intrusion-set` | `uses`        | `malware` or `tool` | No | Group using a software. |
 | `intrusion-set` | `uses`        | `attack-pattern`    | No | Group using a technique, which is also considered a procedure example. |
 | `malware` or `tool` | `uses`    | `attack-pattern`    | No | Software using a technique, which is also considered a procedure example. |
-| `course-of-action`  | `mitigates` | `attack-pattern`  | No | Mitigation mitigating technique. |
+| `course-of-action`  | `mitigates` | `attack-pattern`  | No | Mitigation mitigating a technique. |
 | `attack-pattern`    | `subtechnique-of` | `attack-pattern` | Yes | Sub-technique of a technique, where the `source_ref` is the sub-technique and the `target_ref` is the parent technique. |
+| `x-mitre-data-component` | `detects` | `attack-pattern` | Yes | Data component detecting a technique. | 
 | any type    | `revoked-by`      | any type | Yes | The target object is a replacement for the source object. Only occurs where the objects are of the same type, and the source object will have the property `revoked = true`. See [Working with deprecated and revoked objects](#Working-with-deprecated-and-revoked-objects) for more information on revoked objects. |
 
 Note that because groups use software and software uses techniques, groups can be considered indirect users of techniques used by their software. See [Getting techniques used by a group's software](#Getting-techniques-used-by-a-groups-software).
@@ -815,14 +872,23 @@ def technique_mitigated_by_mitigations(thesrc):
     """return technique_id => {mitigation, relationship} for each mitigation of the technique."""
     return get_related(thesrc, "course-of-action", "mitigates", "attack-pattern", reverse=True)
 
-# technique:subtechnique
+# technique:sub-technique
 def subtechniques_of(thesrc):
     """return technique_id => {subtechnique, relationship} for each subtechnique of the technique."""
     return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern", reverse=True)
 
 def parent_technique_of(thesrc):
     """return subtechnique_id => {technique, relationship} describing the parent technique of the subtechnique"""
     return get_related(thesrc, "attack-pattern", "subtechnique-of", "attack-pattern")[0]
+
+# technique:data-component
+def datacomponent_detects_techniques(thesrc):
+    """return datacomponent_id => {technique, relationship} describing the detections of each data component"""
+    return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern")
+
+def technique_detected_by_datacomponents(thesrc):
+    """return technique_id => {datacomponent, relationship} describing the data components that can detect the technique"""
+    return get_related(thesrc, "x-mitre-data-component", "detects", "attack-pattern", reverse=True)
 ```
 
 Example usage: 

enterprise-attack/attack-pattern/attack-pattern--0042a9f5-f053-4769-b3ef-9ad018dfa298.json

@@ -1,6 +1,6 @@
 {
     "type": "bundle",
-    "id": "bundle--2c20675c-0c44-43ec-a61a-3321d8ff23a0",
+    "id": "bundle--0e9e6e4b-a446-46db-ae72-8caecc3b1af7",
     "spec_version": "2.0",
     "objects": [
         {

enterprise-attack/attack-pattern/attack-pattern--005a06c6-14bf-4118-afa0-ebcd8aebb0c9.json

@@ -1,6 +1,6 @@
 {
     "type": "bundle",
-    "id": "bundle--4a43e057-f553-406a-b518-ea56ca37fcff",
+    "id": "bundle--1f92ee4e-b3ca-4a15-b0bb-85e9a4c53c92",
     "spec_version": "2.0",
     "objects": [
         {

enterprise-attack/attack-pattern/attack-pattern--00d0b012-8a03-410e-95de-5826bf542de6.json

@@ -1,6 +1,6 @@
 {
     "type": "bundle",
-    "id": "bundle--bd843ccd-82d0-4daf-afaa-fb28f02c7d16",
+    "id": "bundle--8b12b94e-dd57-4a2d-bd0a-b2df92b3c3bc",
     "spec_version": "2.0",
     "objects": [
         {

enterprise-attack/attack-pattern/attack-pattern--00f90846-cbd1-4fc5-9233-df5c2bf2a662.json

@@ -1,6 +1,6 @@
 {
     "type": "bundle",
-    "id": "bundle--953493eb-be43-4c58-a00c-76d2e82bffc0",
+    "id": "bundle--4e218ffe-7b7d-49f1-8f21-88bfbb2f4b77",
     "spec_version": "2.0",
     "objects": [
         {