Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Update HAProxy module to follow ECS #10558

Merged
merged 3 commits into from
Feb 5, 2019
Merged

[Metricbeat] Update HAProxy module to follow ECS #10558

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b10a87d
[Metricbeat] Update HAProxy module to follow ECS
ruflin Feb 5, 2019
0eb3e20
make hound happy
ruflin Feb 5, 2019
93727d2
add process field to check
ruflin Feb 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Update a few elasticsearch.* fields to map to ECS. {pull}10350[10350]
- Update a few logstash.* fields to map to ECS. {pull}10350[10350]
- Update a few kibana.* fields to map to ECS. {pull}10350[10350]
- Update haproxy.* fields to map to ECS. {pull}10558[10558]

*Packetbeat*

Expand Down
12 changes: 12 additions & 0 deletions dev-tools/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1654,6 +1654,18 @@

## Modules

### HAProxy

- from: haproxy.stat.process_id
to: process.pid
alias: true
beat: metricbeat

- from: haproxy.stat.service_name
to: process.name
alias: true
beat: metricbeat

### Mongodb

- from: mongodb.status.version
Expand Down
8 changes: 6 additions & 2 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9902,7 +9902,9 @@ Component type (0=frontend, 1=backend, 2=server, or 3=socket/listener).
*`haproxy.stat.process_id`*::
+
--
type: integer
type: alias
alias to: process.pid
Process ID (0 for first instance, 1 for second, and so on).
Expand All @@ -9912,7 +9914,9 @@ Process ID (0 for first instance, 1 for second, and so on).
*`haproxy.stat.service_name`*::
+
--
type: keyword
type: alias
alias to: process.name
Service name (FRONTEND for frontend, BACKEND for backend, or any name for server/listener).
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/haproxy/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions metricbeat/module/haproxy/stat/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
},
"in.bytes": 0,
"out.bytes": 0,
"process_id": 1,
"proxy": {
"id": 2,
"name": "stat"
Expand Down Expand Up @@ -57,7 +56,6 @@
"server": {
"id": 0
},
"service_name": "FRONTEND",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webmat I renamed it to service_name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree with this mapping.

In HAProxy you define frontends which map 1:1 with a vhost/ssl configuration. Then you map this frontend to one or multiple backends, like your main pool of app server, and perhaps a fallback.

So frontend is an internal concept to HAProxy, and one HAProxy process can service many of them. So haproxy.service_name is not at all a process.name.

I would have disagreed less with mapping it to service.name, but I still don't think this would have been a good idea.

Actually I was hoping to look into adding support for various kinds of proxies to ECS. The concept of public entrypoints mapping to backends is generic enough to warrant inclusion in ECS.

I think we should revert this. I can take care of it if you agree.

"session": {
"current": 0,
"limit": 25000,
Expand All @@ -74,6 +72,10 @@
"metricset": {
"name": "stat"
},
"process": {
"name": "FRONTEND",
"pid": 1
},
"service": {
"address": "127.0.0.1:14567",
"type": "haproxy"
Expand Down
8 changes: 6 additions & 2 deletions metricbeat/module/haproxy/stat/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,16 @@
Component type (0=frontend, 1=backend, 2=server, or 3=socket/listener).
- name: process_id
type: integer
type: alias
path: process.pid
migration: true
description: >
Process ID (0 for first instance, 1 for second, and so on).
- name: service_name
type: keyword
type: alias
path: process.name
migration: true
description: >
Service name (FRONTEND for frontend, BACKEND for backend, or any name for server/listener).
Expand Down
19 changes: 17 additions & 2 deletions metricbeat/module/haproxy/stat/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package stat
import (
"reflect"

"github.com/elastic/beats/libbeat/common"
s "github.com/elastic/beats/libbeat/common/schema"
c "github.com/elastic/beats/libbeat/common/schema/mapstrstr"
"github.com/elastic/beats/metricbeat/mb"
Expand Down Expand Up @@ -137,8 +138,22 @@ func eventMapping(info []*haproxy.Stat, r mb.ReporterV2) {
source[typeOfT.Field(i).Name] = f.Interface()
}

event := mb.Event{}
event.MetricSetFields, _ = schema.Apply(source)
fields, _ := schema.Apply(source)
event := mb.Event{
RootFields: common.MapStr{},
}

if processID, err := fields.GetValue("process_id"); err == nil {
event.RootFields.Put("process.pid", processID)
fields.Delete("process_id")
}

if processName, err := fields.GetValue("service_name"); err == nil {
event.RootFields.Put("process.name", processName)
fields.Delete("service_name")
}

event.MetricSetFields = fields
r.Event(event)
}
}
2 changes: 1 addition & 1 deletion metricbeat/tests/system/test_haproxy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ def _test_stat(self):

for evt in output:
print(evt)
self.assertItemsEqual(self.de_dot(HAPROXY_FIELDS), evt.keys(), evt)
self.assertItemsEqual(self.de_dot(HAPROXY_FIELDS + ["process"]), evt.keys(), evt)
self.assert_fields_are_documented(evt)

@unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")
Expand Down