Update Deployment and Devices integrations to ECS 8.17.0 (part 3) #12572

taylor-swanson · 2025-02-03T17:00:08Z

Proposed commit message

Updates the following integrations to ECS 8.17.0:

juniper_srx
modsecurity
netflow
osquery
panw
pfsense
proxysg

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
~~[ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices~~

Related issues

Relates [meta] Upgrade integrations to ECS 8.17 #11952

elasticmachine · 2025-02-03T18:13:45Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

dwhyrock

Couple minor questions, otherwise LGTM.

dwhyrock · 2025-02-03T21:55:37Z

packages/pfsense/data_stream/log/_dev/test/pipeline/test-pfsense-haproxy.log-expected.json

@@ -406,14 +406,14 @@
            ]
        },
        {
-            "@timestamp": "2024-06-13T20:53:49.000-04:00",
+            "@timestamp": "2025-06-13T20:53:49.000-04:00",


Curious why the date year is being updated here, but not in the file below (packages/pfsense/data_stream/log/_dev/test/pipeline/test-pfsense-ipsec.log-expected.json), even though they are both part of the pfsense package?

@timestamp is a weird field when it comes to pipeline tests.

The main issue is that (most) of these logs don't contain a year, so when the expected file generates, it gets whatever year the current year is. This works fine until the new year comes, and then the test will fail, since the pipeline will generate a document with the new year, but this no longer matches what is in the expected file.

The "solution" is to mark the @timestamp as a dynamic field (see test-common-config.yml). That config takes a regex pattern. You could do .*, but to make it slightly more robust, some integrations do some form of regex against the RFC3339 format, although it tends to miss things like the time zone offset at the end. What this does it it will validate the value of @timestamp against the pattern, but not the value in the expected file. Why it updates some of the entries and not all of them, I'm not sure. Technically, it's not supposed to update any of the values if the field is marked as dynamic.

dwhyrock · 2025-02-03T21:56:50Z

packages/proxysg/data_stream/log/fields/fields.yml

-        type: keyword
-      - name: access_security_policy_reason
-        type: keyword
+        - name: appliance_name


Did these have negative effects being at the same level as fields: ?

Functionally, they are the same thing, just a formatting change.

dwhyrock · 2025-02-03T21:57:13Z

packages/proxysg/data_stream/log/fields/fields.yml

@@ -204,7 +204,7 @@
          type: keyword
        - name: icap_status
          type: keyword
-        - name: certificate_hostname 


ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/juniper_srx

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/modsecurity

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/netflow

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/osquery

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/panw

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/pfsense

ECS version in build manifest changed from [email protected] to [email protected]. The set ecs.version processor in pipelines was changed 8.17.0. Previously the pipeline was setting version 8.11.0. The ecs.version in sample_event.json files was changed to 8.17.0. Previously sample_event.json files contained 8.11.0. [git-generate] go run github.com/andrewkroh/go-examples/ecs-update@latest -ecs-version=8.17.0 [email protected] -pr=1 packages/proxysg

elasticmachine · 2025-02-04T21:00:00Z

💚 Build Succeeded

Buildkite Build
Commit: 916f610

History

💚 Build #21610 succeeded f4a06652c6be41dc2b85cc575b097b3a93905090
💚 Build #21485 succeeded 7e3e78e4b17a0c988e22e85d09a3bafcfed2cea6

cc @taylor-swanson

elastic-sonarqube · 2025-02-04T21:00:14Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-02-04T21:21:13Z

Package juniper_srx - 1.22.0 containing this change is available at https://epr.elastic.co/package/juniper_srx/1.22.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:14Z

Package modsecurity - 1.19.0 containing this change is available at https://epr.elastic.co/package/modsecurity/1.19.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:15Z

Package netflow - 2.21.0 containing this change is available at https://epr.elastic.co/package/netflow/2.21.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:16Z

Package osquery - 1.20.0 containing this change is available at https://epr.elastic.co/package/osquery/1.20.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:16Z

Package panw - 5.1.0 containing this change is available at https://epr.elastic.co/package/panw/5.1.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:17Z

Package pfsense - 1.21.0 containing this change is available at https://epr.elastic.co/package/pfsense/1.21.0/

elastic-vault-github-plugin-prod · 2025-02-04T21:21:18Z

Package proxysg - 0.5.0 containing this change is available at https://epr.elastic.co/package/proxysg/0.5.0/

taylor-swanson self-assigned this Feb 3, 2025

taylor-swanson mentioned this pull request Jan 8, 2025

[meta] Upgrade integrations to ECS 8.17 #11952

Open

taylor-swanson marked this pull request as ready for review February 3, 2025 18:13

taylor-swanson requested a review from a team as a code owner February 3, 2025 18:13

dwhyrock approved these changes Feb 3, 2025

View reviewed changes

qcorporation force-pushed the main branch from 410d8ec to 703eb18 Compare February 4, 2025 03:56

qcorporation requested review from a team as code owners February 4, 2025 03:56

qcorporation requested review from AndersonQ and rdner February 4, 2025 03:56

andrewkroh removed Integration:panw Palo Alto Next-Gen Firewall Integration:juniper_srx Juniper SRX Integration:netflow NetFlow Records Integration:modsecurity ModSecurity Audit labels Feb 4, 2025

qcorporation force-pushed the main branch from 7801e00 to ed31586 Compare February 4, 2025 18:34

taylor-swanson added 8 commits February 4, 2025 14:29

changelogs

916f610

taylor-swanson force-pushed the ecs/8.17-part3 branch from f4a0665 to 916f610 Compare February 4, 2025 20:29

taylor-swanson merged commit 60807db into elastic:main Feb 4, 2025
5 checks passed

taylor-swanson deleted the ecs/8.17-part3 branch February 4, 2025 21:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Deployment and Devices integrations to ECS 8.17.0 (part 3) #12572

Update Deployment and Devices integrations to ECS 8.17.0 (part 3) #12572

taylor-swanson commented Feb 3, 2025 •

edited

Loading

elasticmachine commented Feb 3, 2025

dwhyrock left a comment

dwhyrock Feb 3, 2025

taylor-swanson Feb 4, 2025

dwhyrock Feb 3, 2025

taylor-swanson Feb 4, 2025

dwhyrock Feb 3, 2025

elasticmachine commented Feb 4, 2025

elastic-sonarqube bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

Update Deployment and Devices integrations to ECS 8.17.0 (part 3) #12572

Update Deployment and Devices integrations to ECS 8.17.0 (part 3) #12572

Conversation

taylor-swanson commented Feb 3, 2025 • edited Loading

Proposed commit message

Checklist

Related issues

elasticmachine commented Feb 3, 2025

dwhyrock left a comment

Choose a reason for hiding this comment

dwhyrock Feb 3, 2025

Choose a reason for hiding this comment

taylor-swanson Feb 4, 2025

Choose a reason for hiding this comment

dwhyrock Feb 3, 2025

Choose a reason for hiding this comment

taylor-swanson Feb 4, 2025

Choose a reason for hiding this comment

dwhyrock Feb 3, 2025

Choose a reason for hiding this comment

elasticmachine commented Feb 4, 2025

💚 Build Succeeded

History

elastic-sonarqube bot commented Feb 4, 2025

Quality Gate passed

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

elastic-vault-github-plugin-prod bot commented Feb 4, 2025

taylor-swanson commented Feb 3, 2025 •

edited

Loading