Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cisco Umbrella] Fix Proxy Log CSV fields #4085

Merged
merged 7 commits into from
Sep 7, 2022
Merged

[Cisco Umbrella] Fix Proxy Log CSV fields #4085

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6711563
[Cisco Umbrella] Fix Proxy Log CSV fields
legoguy1000 Aug 28, 2022
22b5bb4
fix comments
legoguy1000 Aug 29, 2022
ef7eaef
Split Itentity Types
legoguy1000 Aug 29, 2022
5c77032
Revert back to identity field
legoguy1000 Aug 29, 2022
5c5311f
update docs
legoguy1000 Sep 2, 2022
66c462b
revert custom pipeline changes
legoguy1000 Sep 7, 2022
c55e479
formatted
legoguy1000 Sep 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/cisco_umbrella/_dev/build/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
dependencies:
ecs:
reference: [email protected]-rc1
reference: [email protected]
2 changes: 2 additions & 0 deletions packages/cisco_umbrella/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ datasets for receiving logs from an AWS S3 bucket using an SQS notification queu

When using Cisco Managed S3 buckets that does not use SQS there is no load balancing possibilities for multiple agents, a single agent should be configured to poll the S3 bucket for new and updated files, and the number of workers can be configured to scale vertically.

The field `cisco.umbrella.identity` is described by the documentation as `An identity can be a high-level entity within your system (e.g a network) or very granular (e.g a single user). It is important to define how granular the identities will be.`. This will depend on the customer environment and maybe configurable. Due to this variability, this field isn't normalized into ECS fields by default. A custom ingest pipeline can be used to perform this normalization. This pipeline can be added to the integration config in the `identities_pipeline` option which defaults to `cisco-umbrella-identities-customization`. This option does not need to be used and will not error if it is not set or doesn't exist.

The `log` dataset collects Cisco Umbrella logs.

{{event "log"}}
Expand Down
5 changes: 5 additions & 0 deletions packages/cisco_umbrella/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.3.2"
changes:
- description: Fix proxy log CSV fields
type: bugfix
link: https://github.com/elastic/integrations/pull/4085
- version: "1.3.1"
changes:
- description: Set default endpoint to empty string
Expand Down
8 changes: 6 additions & 2 deletions ...ella/data_stream/log/_dev/test/pipeline/test-umbrella-cloudfirewalllogs.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
"cisco": {
"umbrella": {
"datacenter": "ams1.edc",
"identity_types": "CDFW Tunnel Device",
"identity_types": [
"CDFW Tunnel Device"
],
"origin_id": "[211039844]"
}
},
Expand Down Expand Up @@ -76,7 +78,9 @@
"cisco": {
"umbrella": {
"datacenter": "ams1.edc",
"identity_types": "CDFW Tunnel Device",
"identity_types": [
"CDFW Tunnel Device"
],
"origin_id": "[211039844]"
}
},
Expand Down
13 changes: 10 additions & 3 deletions ...cisco_umbrella/data_stream/log/_dev/test/pipeline/test-umbrella-dnslogs.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
"elasticuser2",
"some other identity"
],
"identity_types": "SomeIdentityType",
"identity_types": [
"SomeIdentityType"
],
"policy_identity_type": "Test Policy Name"
}
},
Expand Down Expand Up @@ -90,7 +92,9 @@
"elasticuser2",
"some other identity"
],
"identity_types": "SomeIdentityType",
"identity_types": [
"SomeIdentityType"
],
"policy_identity_type": "Test Policy Name"
}
},
Expand Down Expand Up @@ -160,7 +164,10 @@
"elastic_machine",
"Elastic User ([email protected])"
],
"identity_types": "Roaming Computers,AD Users",
"identity_types": [
"Roaming Computers",
"AD Users"
],
"policy_identity_type": "Roaming Computers"
}
},
Expand Down
7 changes: 4 additions & 3 deletions packages/cisco_umbrella/data_stream/log/_dev/test/pipeline/test-umbrella-proxylogs.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
"2020-07-23 23:48:56","elasticuser","someotheruser","192.168.1.1","67.43.156.12","81.2.69.144","","ALLOWED","https://elastic.co/blog/ext_id=Anyclip","https://google.com/elastic","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","200","850","","","","Business Services","AVDetectionName","Malicious","MalwareName","","","Roaming Computers",""
"2020-07-23 23:48:56","elasticuser","someotheruser","192.168.1.1","67.43.156.12","81.2.69.144","","BLOCKED","https://elastic.co/blog/ext_id=Anyclip","https://google.com/elastic","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","200","850","","","","Business Services","AVDetectionName","Malicious","MalwareName","","","Roaming Computers",""
"2017-10-02 23:52:53","elasticuser","ActiveDirectoryUserName,ADSite,Network","192.168.192.135","67.43.156.12","","","ALLOWED","http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","","","","","","","","Networks"
"2020-07-23 23:48:56","elasticuser","192.168.1.1","67.43.156.12","81.2.69.144","","ALLOWED","https://elastic.co/blog/ext_id=Anyclip","https://google.com/elastic","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","200","850","","","","Business Services","AVDetectionName","Malicious","MalwareName","","","Roaming Computers",""
"2020-07-23 23:48:56","elasticuser","192.168.1.1","67.43.156.12","81.2.69.144","","BLOCKED","https://elastic.co/blog/ext_id=Anyclip","https://google.com/elastic","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","200","850","","","","Business Services","AVDetectionName","Malicious","MalwareName","","","Roaming Computers",""
"2017-10-02 23:52:53","elasticuser","192.168.192.135","67.43.156.12","","","ALLOWED","http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","","","","","","","","Networks"
"2017-10-02 23:52:53","TheComputerName","192.168.192.135","89.160.20.129","89.160.20.130","","ALLOWED", "http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","Search Engines","","","","","","Roaming Computer","","TheComputerName, ADSite,Network","Roaming Computer, Site, Network","GET","","","the.js","","",""
123 changes: 112 additions & 11 deletions ...sco_umbrella/data_stream/log/_dev/test/pipeline/test-umbrella-proxylogs.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@
"amp_disposition": "MalwareName",
"av_detections": "AVDetectionName",
"categories": "Business Services",
"computer_name": "elasticuser",
"identities": "someotheruser",
"identity_types": "Roaming Computers",
"identity": "elasticuser",
"identity_types": [
"Roaming Computers"
],
"puas": "Malicious"
}
},
Expand All @@ -34,7 +35,7 @@
},
"event": {
"category": "network",
"original": "\"2020-07-23 23:48:56\",\"elasticuser\",\"someotheruser\",\"192.168.1.1\",\"67.43.156.12\",\"81.2.69.144\",\"\",\"ALLOWED\",\"https://elastic.co/blog/ext_id=Anyclip\",\"https://google.com/elastic\",\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\",\"200\",\"850\",\"\",\"\",\"\",\"Business Services\",\"AVDetectionName\",\"Malicious\",\"MalwareName\",\"\",\"\",\"Roaming Computers\",\"\"",
"original": "\"2020-07-23 23:48:56\",\"elasticuser\",\"192.168.1.1\",\"67.43.156.12\",\"81.2.69.144\",\"\",\"ALLOWED\",\"https://elastic.co/blog/ext_id=Anyclip\",\"https://google.com/elastic\",\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\",\"200\",\"850\",\"\",\"\",\"\",\"Business Services\",\"AVDetectionName\",\"Malicious\",\"MalwareName\",\"\",\"\",\"Roaming Computers\",\"\"",
"type": [
"allowed"
]
Expand Down Expand Up @@ -92,9 +93,10 @@
"amp_disposition": "MalwareName",
"av_detections": "AVDetectionName",
"categories": "Business Services",
"computer_name": "elasticuser",
"identities": "someotheruser",
"identity_types": "Roaming Computers",
"identity": "elasticuser",
"identity_types": [
"Roaming Computers"
],
"puas": "Malicious"
}
},
Expand All @@ -119,7 +121,7 @@
},
"event": {
"category": "network",
"original": "\"2020-07-23 23:48:56\",\"elasticuser\",\"someotheruser\",\"192.168.1.1\",\"67.43.156.12\",\"81.2.69.144\",\"\",\"BLOCKED\",\"https://elastic.co/blog/ext_id=Anyclip\",\"https://google.com/elastic\",\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\",\"200\",\"850\",\"\",\"\",\"\",\"Business Services\",\"AVDetectionName\",\"Malicious\",\"MalwareName\",\"\",\"\",\"Roaming Computers\",\"\"",
"original": "\"2020-07-23 23:48:56\",\"elasticuser\",\"192.168.1.1\",\"67.43.156.12\",\"81.2.69.144\",\"\",\"BLOCKED\",\"https://elastic.co/blog/ext_id=Anyclip\",\"https://google.com/elastic\",\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\",\"200\",\"850\",\"\",\"\",\"\",\"Business Services\",\"AVDetectionName\",\"Malicious\",\"MalwareName\",\"\",\"\",\"Roaming Computers\",\"\"",
"type": [
"denied"
]
Expand Down Expand Up @@ -175,16 +177,15 @@
"cisco": {
"umbrella": {
"blocked_categories": "Networks",
"computer_name": "elasticuser",
"identities": "ActiveDirectoryUserName,ADSite,Network"
"identity": "elasticuser"
}
},
"ecs": {
"version": "8.4.0"
},
"event": {
"category": "network",
"original": "\"2017-10-02 23:52:53\",\"elasticuser\",\"ActiveDirectoryUserName,ADSite,Network\",\"192.168.192.135\",\"67.43.156.12\",\"\",\"\",\"ALLOWED\",\"http://google.com/the.js\",\"www.google.com\",\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\",\"200\",\"562\",\"1489\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"Networks\"",
"original": "\"2017-10-02 23:52:53\",\"elasticuser\",\"192.168.192.135\",\"67.43.156.12\",\"\",\"\",\"ALLOWED\",\"http://google.com/the.js\",\"www.google.com\",\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\",\"200\",\"562\",\"1489\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"Networks\"",
"type": [
"allowed"
]
Expand Down Expand Up @@ -235,6 +236,106 @@
"user_agent": {
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
}
},
{
"@timestamp": "2017-10-02T23:52:53.000Z",
"cisco": {
"umbrella": {
"categories": "Search Engines",
"file_name": "the.js",
"identities": [
"TheComputerName",
"ADSite",
"Network"
],
"identity": "TheComputerName",
"identity_types": [
"Roaming Computer",
"Site",
"Network"
],
"request_method": "GET"
}
},
"destination": {
"address": "89.160.20.130",
"as": {
"number": 29518,
"organization": {
"name": "Bredband2 AB"
}
},
"geo": {
"city_name": "Linköping",
"continent_name": "Europe",
"country_iso_code": "SE",
"country_name": "Sweden",
"location": {
"lat": 58.4167,
"lon": 15.6167
},
"region_iso_code": "SE-E",
"region_name": "Östergötland County"
},
"ip": "89.160.20.130"
},
"ecs": {
"version": "8.4.0"
},
"event": {
"category": "network",
"original": "\"2017-10-02 23:52:53\",\"TheComputerName\",\"192.168.192.135\",\"89.160.20.129\",\"89.160.20.130\",\"\",\"ALLOWED\", \"http://google.com/the.js\",\"www.google.com\",\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\",\"200\",\"562\",\"1489\",\"\",\"\",\"Search Engines\",\"\",\"\",\"\",\"\",\"\",\"Roaming Computer\",\"\",\"TheComputerName, ADSite,Network\",\"Roaming Computer, Site, Network\",\"GET\",\"\",\"\",\"the.js\",\"\",\"\",\"\"",
"type": [
"allowed"
]
},
"http": {
"request": {
"bytes": 562,
"referrer": "www.google.com"
},
"response": {
"bytes": 1489,
"status_code": 200
}
},
"log": {
"file": {
"path": "/test/path/proxylogs"
}
},
"observer": {
"product": "Umbrella",
"type": "proxy",
"vendor": "Cisco"
},
"related": {
"ip": [
"192.168.192.135",
"89.160.20.129",
"89.160.20.130"
]
},
"source": {
"address": "192.168.192.135",
"ip": "192.168.192.135",
"nat": {
"ip": "89.160.20.129"
}
},
"tags": [
"preserve_original_event"
],
"url": {
"domain": "google.com",
"extension": "js",
"original": "http://google.com/the.js",
"path": "/the.js",
"scheme": "http"
},
"user_agent": {
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
}
}
]
}
8 changes: 6 additions & 2 deletions packages/cisco_umbrella/data_stream/log/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,11 @@ tags:
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if processors}}
processors:
{{#if processors}}
{{processors}}
{{/if}}
{{/if}}
- add_fields:
target: _config
fields:
identity_pipeline: {{identities_pipeline}}
Loading