Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438

sayden · 2020-09-30T21:49:39Z

Ready to test with Kibana

Most of the fields will work with aliases but the fields that previously were on index_stats.* will now be in the same subpath but always under elasticsearch.index root path (and without index_stats at the beginning)

elasticmachine · 2020-09-30T21:49:56Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-09-30T21:49:57Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-09-30T22:08:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21438 updated
Start Time: 2020-12-01T12:53:13.089+0000
Duration: 61 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2244
Skipped	524
Total	2768

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2244
Skipped	524
Total	2768

chrisronline · 2020-10-23T18:03:04Z

I'm seeing this error when running MB:

2020-10-23T13:59:06.418-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [elasticsearch.index.primaries.indexing.throttle_time.ms] for field alias [index_stats.primaries.indexing.throttle_time_in_millis]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [elasticsearch.index.primaries.indexing.throttle_time.ms] for field alias [index_stats.primaries.indexing.throttle_time_in_millis]: an alias must refer to an existing field in the mappings."},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [elasticsearch.index.primaries.indexing.throttle_time.ms] for field alias [index_stats.primaries.indexing.throttle_time_in_millis]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [elasticsearch.index.primaries.indexing.throttle_time.ms] for field alias [index_stats.primaries.indexing.throttle_time_in_millis]: an alias must refer to an existing field in the mappings."},"status":400}

sayden · 2020-10-28T17:51:01Z

Okay! Fixed, sorry for that. I still don't understand very well why those errors do not appear to me because this was tested with ES to produce the data.json.

chrisronline

LGTM!

andresrc · 2020-10-29T17:27:55Z

@ycombinator I think this is ready for your final review, thanks!

metricbeat/module/elasticsearch/index/_meta/fields.yml

ycombinator · 2020-10-30T03:46:34Z

metricbeat/module/elasticsearch/index/data.go

-		event.MetricSetFields, err = schema.Apply(index)
+		// Convert struct to common.Mapstr by passing it to JSON first so we can store the data in the root of the
+		// metricset level
+		byt2, err := json.Marshal(idx)


Consider replacing byt2 with something more descriptive or at least less arbitrary?

Yeah, good catch, I change its name to something more meaningful

ycombinator · 2020-10-30T03:51:10Z

metricbeat/module/elasticsearch/index/data.go

+		event.MetricSetFields.Put("total.store.size.bytes", idx.Total.Store.SizeInBytes)
+		event.MetricSetFields.Put("total.segments.memory.bytes", idx.Total.Segments.MemoryInBytes)


I'm curious why these two fields need special treatment here. Shouldn't they get marshalled & unmarshalled above along with all the other fields?

Frankly, I don't remember either but I have double checked it and I think that it was an attempt to rename the fields to follow the naming conventions, that I later realized that I had way too many to change and then I regret and I forgot to remove those two.

I finally removed them

Ha! No, forget that. They were original fields https://github.com/elastic/beats/pull/21438/files#diff-397cb07191b7b507fd7ca773b66d961954da4c2bfc5694ab8cf997e1cf991037R44

Instead of creating an alias I just duplicated those fields. I'll add the alias.

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

elasticmachine · 2020-11-05T21:17:36Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	7
Passed	1958
Skipped	108
Total	2073

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / metricbeat-goIntegTest / TestGetAllIndices – elasticsearch
Name: Build&Test / metricbeat-unitTest / [build failed] – index
Name: Build&Test / metricbeat-windows-windows-2019 / [build failed] – index
Name: Build&Test / x-pack/metricbeat-build / test_export_index_pattern – x-pack.metricbeat.tests.system.test_xpack_base.Test
Name: Build&Test / x-pack/metricbeat-build / test_export_index_pattern_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test
Name: Build&Test / x-pack/metricbeat-windows-windows-2019 / test_export_index_pattern – x-pack.metricbeat.tests.system.test_xpack_base.Test
Name: Build&Test / x-pack/metricbeat-windows-windows-2019 / test_export_index_pattern_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test

metricbeat/module/elasticsearch/_meta/fields.yml

ycombinator

Left a minor comment about a TODO. Other than that, code changes LGTM.

However, tests are failing in CI and the failures look related to this PR. So please fix those before merging.

…without xpack.enabled flag (elastic#20615)

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go

…csearch/index_xpack_flag

elasticmachine · 2020-11-19T10:08:48Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…csearch/index_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/testing.go

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/_meta/fields.yml # metricbeat/module/elasticsearch/fields.go

sayden · 2020-11-20T14:19:00Z

/test metricbeat

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go

…xpack.enabled flag (elastic#21438)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 30, 2020

sayden self-assigned this Sep 30, 2020

sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Sep 30, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 30, 2020

sayden changed the title ~~Make beats/index metricset work for Stack Monitoring without xpack.enabled flag~~ Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag Oct 28, 2020

chrisronline approved these changes Oct 29, 2020

View reviewed changes

ycombinator reviewed Oct 30, 2020

View reviewed changes

metricbeat/module/elasticsearch/index/_meta/fields.yml Outdated Show resolved Hide resolved

ycombinator reviewed Oct 30, 2020

View reviewed changes

sayden added 2 commits November 5, 2020 12:15

Merge branch 'master' into feature-stack-monitoring-mb-ecs

b1903d4

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

Make update after rebasing and fixing conflicts in fields.yml

abc4f69

sayden force-pushed the feature/mb/elasticsearch/index_xpack_flag branch from f500817 to c0ed397 Compare November 5, 2020 20:47

sayden requested a review from ycombinator November 5, 2020 20:49

ycombinator reviewed Nov 8, 2020

View reviewed changes

metricbeat/module/elasticsearch/_meta/fields.yml Outdated Show resolved Hide resolved

ycombinator approved these changes Nov 8, 2020

View reviewed changes

sayden added 6 commits November 10, 2020 12:02

Make elasticsearch/index_summary metricset work for Stack Monitoring …

58e2a7d

…without xpack.enabled flag (elastic#20615)

Atomic commit

97fcea2

Fix the paths of some aliases

abd63cd

Remove TODOs, refactor and fixes

5c3bf32

Create aliases for the 2 missing fields

8e156b8

Rebase and fix conflict

0c0ded2

sayden force-pushed the feature/mb/elasticsearch/index_xpack_flag branch from c0ed397 to 0c0ded2 Compare November 10, 2020 16:45

sayden added 3 commits November 10, 2020 20:27

Add missing field and make update

71035e0

Fix failing test

0024721

Add missing fields

6012455

sayden force-pushed the feature-stack-monitoring-mb-ecs branch from 58e2a7d to 9de5959 Compare November 12, 2020 11:44

sayden added 6 commits November 18, 2020 23:21

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

84c5013

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

Fix mappings. Add data.json test

5f25adc

Add a tester to use with the http helper

5d917ad

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

e2a1cee

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

98af618

…csearch/index_xpack_flag

Fix mapping issues

7a4e7c1

sayden requested review from a team as code owners November 19, 2020 10:08

botelastic bot added the Team:Ingest Management label Nov 19, 2020

sayden added 7 commits November 19, 2020 11:16

Remove file

2321108

Remove another file

19f4823

Fix broken test

e3a4e41

Remove duplicated fields from mapping

dfe8094

Try to fix linting

976a373

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

83e0e43

…csearch/index_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/testing.go

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

bedff6f

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/_meta/fields.yml # metricbeat/module/elasticsearch/fields.go

sayden added 2 commits December 1, 2020 13:42

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

36f118d

…csearch/index_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go

make update

298cbe2

sayden merged commit ae189ad into elastic:feature-stack-monitoring-mb-ecs Dec 1, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make elasticsearch/index metricset work for Stack Monitoring without …

7df22a0

…xpack.enabled flag (elastic#21438)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438

Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438

sayden commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

elasticmachine commented Sep 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

chrisronline commented Oct 23, 2020 •

edited

Loading

sayden commented Oct 28, 2020

chrisronline left a comment •

edited

Loading

andresrc commented Oct 29, 2020

ycombinator Oct 30, 2020

sayden Nov 3, 2020

ycombinator Oct 30, 2020

sayden Nov 3, 2020

sayden Nov 3, 2020

sayden Nov 3, 2020 •

edited

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

Genuine test errors

ycombinator left a comment

elasticmachine commented Nov 19, 2020

sayden commented Nov 20, 2020

		event.MetricSetFields.Put("total.store.size.bytes", idx.Total.Store.SizeInBytes)
		event.MetricSetFields.Put("total.segments.memory.bytes", idx.Total.Segments.MemoryInBytes)

Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438

Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438

Conversation

sayden commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

elasticmachine commented Sep 30, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

chrisronline commented Oct 23, 2020 • edited Loading

sayden commented Oct 28, 2020

chrisronline left a comment • edited Loading

Choose a reason for hiding this comment

andresrc commented Oct 29, 2020

ycombinator Oct 30, 2020

Choose a reason for hiding this comment

sayden Nov 3, 2020

Choose a reason for hiding this comment

ycombinator Oct 30, 2020

Choose a reason for hiding this comment

sayden Nov 3, 2020

Choose a reason for hiding this comment

sayden Nov 3, 2020

Choose a reason for hiding this comment

sayden Nov 3, 2020 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Nov 5, 2020 • edited by jenkins-beats-ci bot Loading

🐛 Flaky test report

Test stats 🧪

Genuine test errors

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 19, 2020

sayden commented Nov 20, 2020

elasticmachine commented Sep 30, 2020 •

edited by jenkins-beats-ci bot

Loading

chrisronline commented Oct 23, 2020 •

edited

Loading

chrisronline left a comment •

edited

Loading

sayden Nov 3, 2020 •

edited

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading