Skip to content

Commit

Permalink
New fileset for googlecloud firewall logs (elastic#14553)
Browse files Browse the repository at this point in the history 
This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps
parsing firewall logs generated by rules under VPC Network -> Firewall Rules.

Note that GCP only logs firewall events under the following conditions:
- Logging needs to be enabled for each individual rule in order to log.
- Only TCP and UDP rules can be logged.
  • Loading branch information
@adriansr
adriansr authored Nov 19, 2019
1 parent eff54c3 commit 4a66f0b
Show file tree
Hide file tree
Showing 21 changed files with 3,123 additions and 1,090 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Remove beta flag for some filebeat modules. {pull}14374[14374]
- Add attack_pattern_kql field to MISP threat indicators. {pull}14470[14470]
- Add fileset to the Zeek module for the intel.log. {pull}14404[14404]
- New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. {pull}14553[14553]

*Heartbeat*
- Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]
Expand Down
186 changes: 152 additions & 34 deletions filebeat/docs/fields.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8700,40 +8700,87 @@ Fields from Google Cloud logs.
[float]
=== vpcflow
=== destination.instance
Fields for Google Cloud VPC flow logs.
If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
*`googlecloud.vpcflow.reporter`*::
*`googlecloud.destination.instance.project_id`*::
+
--
The side which reported the flow. Can be either 'SRC' or 'DEST'.
ID of the project containing the VM.
type: keyword
--
*`googlecloud.vpcflow.rtt.ms`*::
*`googlecloud.destination.instance.region`*::
+
--
Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
Region of the VM.
type: long
type: keyword
--
*`googlecloud.destination.instance.zone`*::
+
--
Zone of the VM.
type: keyword
--
[float]
=== destination.instance
=== destination.vpc
If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
*`googlecloud.destination.vpc.project_id`*::
+
--
ID of the project containing the VM.
type: keyword
--
*`googlecloud.destination.vpc.vpc_name`*::
+
--
VPC on which the VM is operating.
type: keyword
--
*`googlecloud.destination.vpc.subnetwork_name`*::
+
--
Subnetwork on which the VM is operating.
type: keyword
--
*`googlecloud.vpcflow.destination.instance.project_id`*::
[float]
=== source.instance
If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
*`googlecloud.source.instance.project_id`*::
+
--
ID of the project containing the VM.
Expand All @@ -8743,7 +8790,7 @@ type: keyword
--
*`googlecloud.vpcflow.destination.instance.region`*::
*`googlecloud.source.instance.region`*::
+
--
Region of the VM.
Expand All @@ -8753,7 +8800,7 @@ type: keyword
--
*`googlecloud.vpcflow.destination.instance.zone`*::
*`googlecloud.source.instance.zone`*::
+
--
Zone of the VM.
Expand All @@ -8764,13 +8811,13 @@ type: keyword
--
[float]
=== destination.vpc
=== source.vpc
If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
*`googlecloud.vpcflow.destination.vpc.project_id`*::
*`googlecloud.source.vpc.project_id`*::
+
--
ID of the project containing the VM.
Expand All @@ -8780,7 +8827,7 @@ type: keyword
--
*`googlecloud.vpcflow.destination.vpc.vpc_name`*::
*`googlecloud.source.vpc.vpc_name`*::
+
--
VPC on which the VM is operating.
Expand All @@ -8790,7 +8837,7 @@ type: keyword
--
*`googlecloud.vpcflow.destination.vpc.subnetwork_name`*::
*`googlecloud.source.vpc.subnetwork_name`*::
+
--
Subnetwork on which the VM is operating.
Expand All @@ -8801,79 +8848,150 @@ type: keyword
--
[float]
=== source.instance
=== firewall
If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
Fields for Google Cloud Firewall logs.
[float]
=== rule_details
Description of the firewall rule that matched this connection.
*`googlecloud.vpcflow.source.instance.project_id`*::
*`googlecloud.firewall.rule_details.priority`*::
+
--
ID of the project containing the VM.
The priority for the firewall rule.
type: long
--
*`googlecloud.firewall.rule_details.action`*::
+
--
Action that the rule performs on match.
type: keyword
--
*`googlecloud.vpcflow.source.instance.region`*::
*`googlecloud.firewall.rule_details.direction`*::
+
--
Region of the VM.
Direction of traffic that matches this rule.
type: keyword
--
*`googlecloud.firewall.rule_details.reference`*::
+
--
Reference to the firewall rule.
type: keyword
--
*`googlecloud.vpcflow.source.instance.zone`*::
*`googlecloud.firewall.rule_details.source_range`*::
+
--
Zone of the VM.
List of source ranges that the firewall rule applies to.
type: keyword
--
*`googlecloud.firewall.rule_details.destination_range`*::
+
--
List of destination ranges that the firewall applies to.
type: keyword
--
[float]
=== source.vpc
*`googlecloud.firewall.rule_details.source_tag`*::
+
--
List of all the source tags that the firewall rule applies to.
If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
type: keyword
--
*`googlecloud.vpcflow.source.vpc.project_id`*::
*`googlecloud.firewall.rule_details.target_tag`*::
+
--
ID of the project containing the VM.
List of all the target tags that the firewall rule applies to.
type: keyword
--
*`googlecloud.vpcflow.source.vpc.vpc_name`*::
*`googlecloud.firewall.rule_details.ip_port_info`*::
+
--
VPC on which the VM is operating.
List of ip protocols and applicable port ranges for rules.
type: array
--
*`googlecloud.firewall.rule_details.source_service_account`*::
+
--
List of all the source service accounts that the firewall rule applies to.
type: keyword
--
*`googlecloud.vpcflow.source.vpc.subnetwork_name`*::
*`googlecloud.firewall.rule_details.target_service_account`*::
+
--
Subnetwork on which the VM is operating.
List of all the target service accounts that the firewall rule applies to.
type: keyword
--
[float]
=== vpcflow
Fields for Google Cloud VPC flow logs.
*`googlecloud.vpcflow.reporter`*::
+
--
The side which reported the flow. Can be either 'SRC' or 'DEST'.
type: keyword
--
*`googlecloud.vpcflow.rtt.ms`*::
+
--
Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
type: long
--
[[exported-fields-haproxy]]
== haproxy fields
Expand Down
50 changes: 48 additions & 2 deletions filebeat/docs/modules/googlecloud.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ This file is generated! See scripts/docs_collector.py

beta[]

This is a module for Google Cloud logs. It supports reading VPC flow logs that
have been exported from Stackdriver to a Google Pub/Sub topic sink.
This is a module for Google Cloud logs. It supports reading VPC flow
and firewall logs that have been exported from Stackdriver to a
Google Pub/Sub topic sink.

include::../include/what-happens.asciidoc[]

Expand Down Expand Up @@ -60,6 +61,51 @@ exist it will be created.

Path to a JSON file containing the credentials and key used to subscribe.

:fileset_ex!:

:fileset_ex: firewall

[float]
==== `firewall` fileset settings

Example config:

[source,yaml]
----
- module: googleclcoud
firewall:
enabled: true
var.project_id: my-gcp-project-id
var.topic: googlecloud-vpc-flowlogs
var.subscription_name: filebeat-googlecloud-vpc-flowlogs-sub
var.credentials_file: ${path.config}/gcp-service-account-xyz.json
var.keep_original_message: false
----

include::../include/var-paths.asciidoc[]

*`var.project_id`*::

Google Cloud project ID.

*`var.topic`*::

Google Cloud Pub/Sub topic name.

*`var.subscription_name`*::

Google Cloud Pub/Sub topic subscription name. If the subscription does not
exist it will be created.

*`var.credentials_file`*::

Path to a JSON file containing the credentials and key used to subscribe.

*`var.keep_original_message`*::

Flag to control whether the original message is stored in the `log.original`
field. Defaults to `false`, meaning the original message is not saved.

:has-dashboards!:

:fileset_ex!:
Expand Down
Loading

0 comments on commit 4a66f0b

Please sign in to comment.