Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099

sayden · 2020-09-15T15:23:58Z

Ready to test in Kibana. data.json is included to help troubleshooting and it has been generating testing the Metricbeat binary directly with Elasticsearch

elasticmachine · 2020-09-15T15:24:00Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-09-15T15:24:01Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-09-15T15:39:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21099 updated
Start Time: 2020-12-01T11:31:31.855+0000
Duration: 59 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2258
Skipped	520
Total	2778

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2258
Skipped	520
Total	2778

chrisronline · 2020-09-16T18:20:57Z

I'm seeing this while running this PR:

2020-09-16T14:19:17.647-0400	ERROR	runtime/map_faststr.go:204	recovered from panic while fetching 'elasticsearch/cluster_stats' for host 'localhost:9200'. Recovering, but please report this.	{"panic": "assignment to entry in nil map", "stack": "github.com/elastic/beats/v7/libbeat/logp.Recover\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/libbeat/logp/global.go:101\nruntime.gopanic\n\t/usr/local/Cellar/go/1.13.7/libexec/src/runtime/panic.go:679\nruntime.mapassign_faststr\n\t/usr/local/Cellar/go/1.13.7/libexec/src/runtime/map_faststr.go:204\ngithub.jparrowsec.cn/elastic/beats/v7/libbeat/common.MapStr.Put\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/libbeat/common/mapstr.go:199\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/module/elasticsearch/cluster_stats.eventMapping\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/module/elasticsearch/cluster_stats/data_xpack.go:229\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/module/elasticsearch/cluster_stats.(*MetricSet).Fetch\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/module/elasticsearch/cluster_stats/cluster_stats.go:70\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:256\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:224\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:208\ngithub.jparrowsec.cn/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1\n\t/Users/chris/dev/repos/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:147"}

chrisronline · 2020-09-24T13:51:37Z

I'm seeing this error now:

{"type":"mapper_parsing_exception","reason":"failed to parse field [elasticsearch.cluster.stats.nodes.total] of type [long] in document with id 'CaJUwHQBA9bBTy50HQpt'. Preview of field's value: '{total=1}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: (byte[])\"{\"@timestamp\":\"2020-09-24T13:36:11.140Z\",\"elasticsearch\":{\"cluster\":{\"name\":\"elasticsearch\",\"id\":\"PChw572_S52efCun1M0eNg\",\"stats\":{\"indices\":{\"store\":{\"size\":{\"bytes\":1211431477},\"reserved\":{\"bytes\":0}},\"query_cache\":{\"hit\":{\"total\":0},\"miss\":{\"total\":26},\"cache\":{\"total\":0},\"evictions\":0,\"cache_size\":{\"bytes\":0},\"memory_size\":{\"bytes\":0},\"total\":26},\"segments\":{\"total\":396,\"memory\":{\"bytes\":5437784,\"doc_values\":{\"bytes\":3400816},\"fixed_bit_set\":{\"bytes\":316224},\"points\":{\"bytes\":0},\"terms\":{\"ve\"[truncated 7233 bytes]; line: 1, column: 6281]"}}

Are you able to run this successfully against ES?

chrisronline · 2020-10-02T19:31:35Z

I'm seeing this error on startup still:

2020-10-02T15:31:19.433-0400	WARN	[elasticsearch]	elasticsearch/client.go:407	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfd5fde073c27530, ext:48941485, loc:(*time.Location)(0x8416f00)}, Meta:null, Fields:{"agent":{"ephemeral_id":"7971d271-a0a5-4774-8e5c-ec770c1db6e5","id":"5ca7fc45-dad9-4ed5-afa0-ff17fd83a2e1","name":"cbr-mbp.lan","type":"metricbeat","version":"8.0.0"},"ecs":{"version":"1.5.0"},"elasticsearch":{"cluster":{"id":"OQ5paLC_Rn2qEVfCyRVg6Q","name":"elasticsearch","stats":{"cluster_name":"elasticsearch","indices":{"docs":{"deleted":{"total":0},"total":7},"fielddata":{"memory":{"bytes":0}},"query_cache":{"cache":{"total":0.000000},"cache_size":{"bytes":0},"evictions":0,"hit":{"total":0.000000},"memory_size":{"bytes":0},"miss":{"total":0.000000},"total":0},"segments":{"max_unsafe_auto_id":{"ms":-1},"memory":{"bytes":6500,"doc_values":{"bytes":380},"fixed_bit_set":{"bytes":0},"index_writer":{"bytes":0},"norms":{"bytes":0},"points":{"bytes":0},"stored_fields":{"bytes":2440},"terms":{"bytes":3680,"vectors":{"bytes":0}},"version_map":{"bytes":0}},"total":5},"shards":{"index":{"primaries":{"avg":1.000000,"max":1.000000,"min":1.000000},"replication":{"avg":0.000000,"max":0.000000,"min":0.000000},"shards":{"avg":1.000000,"max":1.000000,"min":1.000000}},"primaries":1,"replication":0,"total":1},"store":{"reserved":{"bytes":0},"size":{"bytes":18686}},"total":1},"license":{"cluster_needs_tls":false,"issue_date":"2020-10-02T19:30:16.399Z","issue_date_in_millis":1601667016399,"issued_to":"elasticsearch","issuer":"elasticsearch","max_nodes":1000,"start_date_in_millis":-1,"status":"active","type":"basic","uid":"50fbff47-0ccf-4eb0-8afb-e1df722163b6"},"nodes":{"coordinating_only":{"total":0},"data":{"total":1},"fs":{"available":{"bytes":222692048896},"free":{"bytes":239005847552},"total":{"bytes":499963174912}},"ingest":{"pipelines":{"total":1},"processor_stats":{"gsub":{"count":0.000000,"current":0.000000,"failed":0.000000,"time_in_millis":0.000000},"script":{"count":0.000000,"current":0.000000,"failed":0.000000,"time_in_millis":0.000000}},"total":1},"jvm":{"max_uptime":{"ms":71164},"memory":{"heap":{"max":{"bytes":1073741824},"used":{"bytes":101822472}}},"threads":{"total":68}},"master":{"total":1},"ml":{"total":1},"remote_cluster_client":{"total":1},"total":1,"transform":{"total":1},"voting_only":{"total":0}},"stack":{"apm":{"found":false},"xpack":{"ccr":{"auto_follow_patterns_count":0.000000,"available":false,"enabled":true,"follower_indices_count":0.000000},"data_streams":{"available":true,"data_streams":0.000000,"enabled":true,"indices_count":0.000000},"frozen_indices":{"available":true,"enabled":true,"indices_count":0.000000},"graph":{"available":false,"enabled":true},"ilm":{"policy":{"stats":[{"indices_managed":0.000000,"phases":{"delete":{"actions":["delete"],"min_age":7776000000.000000},"hot":{"actions":["rollover"],"min_age":0.000000}}},{"indices_managed":0.000000,"phases":{"hot":{"actions":["rollover"],"min_age":0.000000}}},{"indices_managed":0.000000,"phases":{"delete":{"actions":["delete"],"min_age":604800000.000000}}},{"indices_managed":0.000000,"phases":{"hot":{"actions":["rollover"],"min_age":0.000000}}},{"indices_managed":0.000000,"phases":{"hot":{"actions":["rollover"],"min_age":0.000000}}},{"indices_managed":0.000000,"phases":{"hot":{"actions":["rollover"],"min_age":0.000000}}},{"indices_managed":0.000000,"phases":{"delete":{"actions":["delete"],"min_age":7776000000.000000},"hot":{"actions":["rollover"],"min_age":0.000000}}}],"total":7}},"logstash":{"available":false,"enabled":true},"ml":{"available":false,"enabled":true,"jobs":{"total":0},"node":{"total":1}},"monitoring":{"available":true,"collection_enabled":false,"enabled":true,"enabled_exporters":{"local":1.000000}},"rollup":{"available":true,"enabled":true},"searchable_snapshots":{"available":false,"enabled":true,"indices_count":0.000000},"security":{"anonymous":{"enabled":false},"api_key_service":{"enabled":true},"audit":{"enabled":false},"available":true,"enabled":true,"fips_140":{"enabled":false},"ipfilter":{"http":false,"transport":false},"realms":{"active_directory":{"available":false,"enabled":false},"file":{"available":true,"cache":[{"size":0.000000}],"enabled":true,"name":["default_file"],"order":[-2147483647.000000],"size":[0.000000]},"kerberos":{"available":false,"enabled":false},"ldap":{"available":false,"enabled":false},"native":{"available":true,"cache":[{"size":0.000000}],"enabled":true,"name":["default_native"],"order":[-2147483646.000000],"size":[0.000000]},"oidc":{"available":false,"enabled":false},"pki":{"available":false,"enabled":false},"saml":{"available":false,"enabled":false}},"role_mapping":{"native":{"enabled":0.000000,"size":0.000000}},"roles":{"dls":{"bit_set_cache":{"count":0.000000,"memory":"0b","memory_in_bytes":0.000000}},"file":{"dls":false,"fls":false,"size":0.000000},"native":{"dls":false,"fls":false,"size":0.000000}},"ssl":{"http":{"enabled":true},"transport":{"enabled":false}},"token_service":{"enabled":true}},"slm":{"available":true,"enabled":true},"spatial":{"available":true,"enabled":true},"transform":{"available":true,"enabled":true},"vectors":{"available":true,"dense_vector_dims_avg_count":0.000000,"dense_vector_fields_count":0.000000,"enabled":true},"voting_only":{"available":true,"enabled":true}}},"state":{"cluster_uuid":"OQ5paLC_Rn2qEVfCyRVg6Q","master_node":"oqZqInokQGWbYdpwNwIEJA","nodes":{"oqZqInokQGWbYdpwNwIEJA":{"attributes":{"ml.machine_memory":"34359738368","ml.max_open_jobs":"20","transform.node":"true","xpack.installed":"true"},"ephemeral_id":"LPslIylaTwGjNCHms4g8DA","name":"cbr-mbp.lan","transport_address":"127.0.0.1:9300"}},"nodes_hash":-112262078,"state_uuid":"X6RT9q6fRluvMosidF4pdQ","status":"green","version":41.000000},"status":"green","version":"8.0.0-SNAPSHOT"}}},"event":{"dataset":"elasticsearch.cluster.stats","duration":336449522,"module":"elasticsearch"},"host":{"architecture":"x86_64","hostname":"cbr-mbp.lan","id":"0E799C52-1A23-58FB-8293-749EDE93385A","ip":["fe80::c2f:e735:1bac:b22d","192.168.86.227","fe80::aede:48ff:fe00:1122","fe80::d1:2983:3330:e64e","169.254.116.221","fe80::c135:20d0:94d9:dde2","fe80::fca8:b5:391d:542","fe80::2fe2:834:5714:9d4e","fe80::f36c:9877:dbc0:95fc"],"mac":["68:5b:35:cc:cb:1e","ac:de:48:00:11:22","3a:f9:d3:be:e5:4f","38:f9:d3:be:e5:4f","82:41:7e:c3:9c:01","82:41:7e:c3:9c:00","82:41:7e:c3:9c:05","82:41:7e:c3:9c:04","82:41:7e:c3:9c:01","0a:f9:d3:be:e5:4f","6e:a0:7b:dd:71:d9","6e:a0:7b:dd:71:d9"],"name":"cbr-mbp.lan","os":{"build":"19H2","family":"darwin","kernel":"19.6.0","name":"Mac OS X","platform":"darwin","version":"10.15.7"}},"metricset":{"name":"cluster_stats","period":10000},"service":{"address":"localhost:9200","type":"elasticsearch"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [elasticsearch.cluster.stats.nodes.master] of type [long] in document with id 'kyDM6nQBVnm6CwgnHOYF'. Preview of field's value: '{total=1}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: (byte[])\"{\"@timestamp\":\"2020-10-02T19:31:13.868Z\",\"agent\":{\"version\":\"8.0.0\",\"ephemeral_id\":\"7971d271-a0a5-4774-8e5c-ec770c1db6e5\",\"id\":\"5ca7fc45-dad9-4ed5-afa0-ff17fd83a2e1\",\"name\":\"cbr-mbp.lan\",\"type\":\"metricbeat\"},\"metricset\":{\"name\":\"cluster_stats\",\"period\":10000},\"service\":{\"address\":\"localhost:9200\",\"type\":\"elasticsearch\"},\"elasticsearch\":{\"cluster\":{\"name\":\"elasticsearch\",\"id\":\"OQ5paLC_Rn2qEVfCyRVg6Q\",\"stats\":{\"nodes\":{\"jvm\":{\"threads\":{\"total\":68},\"max_uptime\":{\"ms\":71164},\"memory\":{\"heap\":{\"used\"[truncated 5663 bytes]; line: 1, column: 855]"}}

chrisronline · 2020-10-16T15:35:22Z

@sayden

I can now run this PR and it indexes properly! Thanks for fixing that!

I'm seeing an issue with the document though.

In existing .monitoring-es-* indices, we have:

POST .monitoring-es-*/_search?filter_path=hits.hits._source.cluster_stats.nodes.count
{
  "size": 1,
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "term": {
      "type": {
        "value": "cluster_stats"
      }
    }
  }
}

which returns:

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "cluster_stats" : {
            "nodes" : {
              "count" : {
                "total" : 3,
                "coordinating_only" : 0,
                "data" : 2,
                "data_cold" : 0,
                "data_content" : 0,
                "data_hot" : 0,
                "data_warm" : 0,
                "ingest" : 2,
                "master" : 3,
                "ml" : 0,
                "remote_cluster_client" : 3,
                "transform" : 2,
                "voting_only" : 1
              }
            }
          }
        }
      }
    ]
  }
}

I'm not seeing that in the metricbeat-* documents. I see elasticsearch.cluster.stats.nodes.total as a number but I don't see an overall count. Is this by design? If so, that's fine and I can just adjust the code to look in either spot.

sayden · 2020-10-28T15:40:25Z

The total vs count thing was my mistake, I have reverted all those fields to count again (which is the correct naming).

The error is more a misconception: You'll never find a hits.hits._source.cluster_stats.nodes.count now because that path doesn't exists in the stored JSON document. The only way to reach that info is to use hits.hits._source.elasticsearch.cluster.stats.nodes.count. Example:

POST metricbeat-*/_search?filter_path=hits.hits._source.elasticsearch.cluster.stats.nodes.count
{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "elasticsearch" : {
            "cluster" : {
              "stats" : {
                "nodes" : {
                  "count" : 1
                }
              }
            }
          }
        }
      },
      {
        "_source" : {
          "elasticsearch" : {
            "cluster" : {
              "stats" : {
                "nodes" : {
                  "count" : 1
                }
              }
            }
          }
        }
      }
    ]
  }
}

Another solution is to create a mapping, which is included as an example in the PR now. But you'll have to "query" this info:

POST metricbeat-*/_search
{
  "_source": "elasticsearch.cluster.stats.nodes.count", 
  "query": {
    "term": {
      "cluster_stats.nodes.count": {
        "value": 1
      }
    }
  }
}

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 17,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "metricbeat-8.0.0-2020.10.28-000001",
        "_id" : "vljIb3UBKtS08QdByHUG",
        "_score" : 1.0,
        "_source" : {
          "elasticsearch" : {
            "cluster" : {
              "stats" : {
                "nodes" : {
                  "count" : 1
                }
              }
            }
          }
        }
      }
    ]
  }
}

Note that we query the term cluster_stats.nodes.count but we request only the path in the source that is elasticsearch.cluster.stats.nodes.count, which is the real path in the doc.

I also removed some data from the incoming data from Elasticsearch to save some space. I have included it again just in case.

Sorry for the long explanation 😅 I just wanted to make sure that I explained it properly.

EDIT: Forgot to mention: In the mappings https://gist.github.com/chrisronline/3b123c85d03b97ec3cbf63d9d72b3e2c there was no mention to cluster_stats metrics and that's why I didn't include it at the beginning. I know it can be a pain to modify those paths in Kibana now but the only way to avoid this is through aliases in the fields that are accessed using a query, but it doesn't work when they are accessed directly.

sayden · 2020-10-28T15:59:25Z

I forgot to push the commit that brings back the data at nodes.count

chrisronline · 2020-10-29T14:43:18Z

I'm seeing an error running this now:

2020-10-29T10:43:02.451-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping: Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]"}},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping: Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [count] has unsupported parameters: [properties : {total={type=long}, data={type=long}, master={type=long}}]"}},"status":400}.

sayden · 2020-11-03T16:03:24Z

Chris, can you briefly describe a step by step way to reproduce this error? Somehow it's happening in all metricsets but I cannot manage to reproduce it (I'm usually just guessing what the mapping issue is). The way I do it is:

Start a new Elasticsearch and Kibana container for each metricset (I remove the running container every time I start with a different metricset)
go build in the metricbeat folder
./metricbeat modules disable system && .'/metricbeat modules enable elasticsearch
Then, I configure the elasticsearch module (in the modules.d folder) to activate only the metricset being tested
I usually comment the processors in metricbeat.yml to remove some noise of the add_*_metadata
Finally I start metricbeat with ./metricbeat -e -d "*"

From there, I don't get any error and I usually copy-paste the generated event from the line processing/processors.go:187 Publish event: { but I don't get any error.

chrisronline · 2020-11-05T17:59:21Z

My steps which go against the same ES cluster

Pull down your PR
cd metricbeat
mage build
./metricbeat modules enable elasticsearch-xpack (same error with elasticsearch module too)
Delete all existing MB ES resources:

DELETE metricbeat-8.0.0* 
DELETE _template/metricbeat-8.0.0

./metricbeat -c metricbeat.dev.yml -e

Am I doing something wrong here?

sayden · 2020-11-05T20:22:32Z

It must be me the one who is doing something wrong but I can't manage to understand why because, esentially, we are doing the same thing. You delete MB ES resources and I docker rm -f elasticsearch on each test. I'm wondering if it's reusing the same docker volume or something similar.

Anyways, the mapping error was fixed.

elasticmachine · 2020-11-05T21:29:10Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	2
Passed	2242
Skipped	502
Total	2746

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/metricbeat-build / test_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test
Name: Build&Test / x-pack/metricbeat-build / test_template – x-pack.metricbeat.tests.system.test_xpack_base.Test

chrisronline

LGTM!

metricbeat/module/elasticsearch/cluster_stats/_meta/data.json

metricbeat/module/elasticsearch/cluster_stats/data.go

ycombinator · 2020-11-12T02:47:57Z

metricbeat/module/elasticsearch/cluster_stats/data.go

+					"stats": c.Ifc("policy_stats"),
+				},
+			}),
+			//"watcher": c.Ifc("watcher"),


Why is this field commented out?

I forgot to add a comment for double check it's not required (I was hoping it wasn't). If Chris hasn't complain I guess it's not so I have removed it 😄

Is this data contained in the stack_stats field?

It wasn't included when you first tested it, @chrisronline I guess it's safe to remove it

ycombinator · 2020-11-12T02:54:02Z

metricbeat/module/elasticsearch/cluster_stats/data_test.go

 )

 func TestMapper(t *testing.T) {
-	elasticsearch.TestMapperWithInfo(t, "./_meta/test/cluster_stats.*.json", eventMapping)
+	// TODO This test must be skipped until we find a reasonable way to test the event mapping when it has to do extra
+	// HTTP calls


Could you mock an HTTP server using http.NewServeMux? IIRC we are doing this in a few places in the Beats codebase if you want to try and follow some examples.

Ah yes, the problem I had with this test is the helper.HTTP which is not easy to use in tests. Actually it's not used in any test file apart from some fake test in rabbitmq. The cleanest way to fix it is refactoring the helper as you can see in this branch #22573 with the solution applied as example.

Understood, the problem is that you need to pass in the Metricset and that's not easy to construct in tests.

I took a look at the refactoring PR (#22573) and it LGTM.

…csearch/cluster_stats_xpack_flag

sayden · 2020-11-18T18:52:25Z

Okay I have a final version now. I had to remove 2 fields that could be affected according to this mapping https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/monitoring-es.json#L503

cluster_state.nodes
cluster_state.shards

The reason is that an alias cannot point to an object type. According to docs "The target must be a concrete field, and not an object or another field alias."

Take a final look and tell me if I can do something as a workaround

chrisronline · 2020-11-18T19:50:45Z

could you confirm that Telemetry code in Kibana looks at the stack_stats fields from type: cluster_stats documents in .monitoring-es-* indices?

Yes -> here and here

If so, then we will need to preserve this information as part of the changes in this PR, and add a field alias for it so Telemetry code continues to work as before.

We don't need the fields aliased right? We just read them from source for telemetry purposes.

BTW has Telemetry code in Kibana been updated to look at .monitoring-* and metricbeat-* indices, like Stack Monitoring code?

Not yet, but it will be done for elastic/kibana#73864

sayden · 2020-11-18T22:18:17Z

/test metricbeat

ycombinator · 2020-11-19T00:46:44Z

If so, then we will need to preserve this information as part of the changes in this PR, and add a field alias for it so Telemetry code continues to work as before.

We don't need the fields aliased right? We just read them from source for telemetry purposes.

Gotcha. Yeah, if we just read them from source then we don't need the aliases. But then I imagine Telemetry code will need to be updated to also look at the new non-aliased field paths? Will this also be done as part of elastic/kibana#73864?

…csearch/cluster_stats_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/elasticsearch_integration_test.go # metricbeat/module/elasticsearch/fields.go

sayden · 2020-11-19T20:36:25Z

/test metricbeat

…csearch/cluster_stats_xpack_flag

…csearch/cluster_stats_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/elasticsearch_integration_test.go # metricbeat/module/elasticsearch/fields.go

…csearch/cluster_stats_xpack_flag

… without xpack.enabled flag (elastic#21099)

sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Sep 15, 2020

sayden self-assigned this Sep 15, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 15, 2020

sayden changed the base branch from master to feature-stack-monitoring-mb-ecs September 15, 2020 15:24

sayden force-pushed the feature/mb/elasticsearch/cluster_stats_xpack_flag branch from 9a43848 to 0a93e93 Compare November 5, 2020 20:31

chrisronline approved these changes Nov 9, 2020

View reviewed changes

sayden force-pushed the feature/mb/elasticsearch/cluster_stats_xpack_flag branch 2 times, most recently from e7a761b to 0b57912 Compare November 10, 2020 16:35

sayden requested a review from ycombinator November 11, 2020 20:28

ycombinator reviewed Nov 12, 2020

View reviewed changes

metricbeat/module/elasticsearch/cluster_stats/_meta/data.json Show resolved Hide resolved

ycombinator reviewed Nov 12, 2020

View reviewed changes

metricbeat/module/elasticsearch/cluster_stats/data.go Outdated Show resolved Hide resolved

ycombinator reviewed Nov 12, 2020

View reviewed changes

sayden added 7 commits November 18, 2020 12:15

Fix linting

dd931e0

Add a TestData function

3089e56

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

470ac23

…csearch/cluster_stats_xpack_flag

Fix mapping issues

b3d3b2a

Reduce size of cluster_stats by removing state key

6029bce

Fix mapping conflict

e40d727

Remove conflicting fields in mapping

961a9ee

Include cluster_state.nodes in the JSON just in case

a11bb53

sayden added 3 commits November 19, 2020 09:47

Update mock data and data.json

7a83026

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

b64816c

…csearch/cluster_stats_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/elasticsearch_integration_test.go # metricbeat/module/elasticsearch/fields.go

Fix conflicts

9881044

sayden added 11 commits November 19, 2020 22:58

Fix lint

c8196d3

Fix input json file

ef523ac

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

0b95938

…csearch/cluster_stats_xpack_flag

Fix TestMapper test

e43a6ec

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

675e8b6

…csearch/cluster_stats_xpack_flag

Reorder imports

172df98

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

386f088

…csearch/cluster_stats_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/elasticsearch_integration_test.go # metricbeat/module/elasticsearch/fields.go

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

05ee64b

…csearch/cluster_stats_xpack_flag

Try to fix linting

f1f8aea

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

8611892

…csearch/cluster_stats_xpack_flag

Make check from a linux machine to fix the possible linting problems

fd21fc0

sayden merged commit 2b5b420 into elastic:feature-stack-monitoring-mb-ecs Dec 1, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make elasticsearch/cluster_stats metricset work for Stack Monitoring…

998ec9c

… without xpack.enabled flag (elastic#21099)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099

Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099

sayden commented Sep 15, 2020 •

edited

Loading

elasticmachine commented Sep 15, 2020

elasticmachine commented Sep 15, 2020

elasticmachine commented Sep 15, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

chrisronline commented Sep 16, 2020

chrisronline commented Sep 24, 2020

chrisronline commented Oct 2, 2020

chrisronline commented Oct 16, 2020

sayden commented Oct 28, 2020 •

edited

Loading

sayden commented Oct 28, 2020

chrisronline commented Oct 29, 2020

sayden commented Nov 3, 2020

chrisronline commented Nov 5, 2020

sayden commented Nov 5, 2020

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

Genuine test errors

chrisronline left a comment

ycombinator Nov 12, 2020

sayden Nov 12, 2020

chrisronline Nov 12, 2020

sayden Nov 12, 2020

ycombinator Nov 12, 2020

sayden Nov 12, 2020

ycombinator Nov 12, 2020

sayden commented Nov 18, 2020

chrisronline commented Nov 18, 2020

sayden commented Nov 18, 2020

ycombinator commented Nov 19, 2020

sayden commented Nov 19, 2020

Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099

Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099

Conversation

sayden commented Sep 15, 2020 • edited Loading

elasticmachine commented Sep 15, 2020

elasticmachine commented Sep 15, 2020

elasticmachine commented Sep 15, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

chrisronline commented Sep 16, 2020

chrisronline commented Sep 24, 2020

chrisronline commented Oct 2, 2020

chrisronline commented Oct 16, 2020

sayden commented Oct 28, 2020 • edited Loading

sayden commented Oct 28, 2020

chrisronline commented Oct 29, 2020

sayden commented Nov 3, 2020

chrisronline commented Nov 5, 2020

sayden commented Nov 5, 2020

elasticmachine commented Nov 5, 2020 • edited by jenkins-beats-ci bot Loading

🐛 Flaky test report

Test stats 🧪

Genuine test errors

chrisronline left a comment

Choose a reason for hiding this comment

ycombinator Nov 12, 2020

Choose a reason for hiding this comment

sayden Nov 12, 2020

Choose a reason for hiding this comment

chrisronline Nov 12, 2020

Choose a reason for hiding this comment

sayden Nov 12, 2020

Choose a reason for hiding this comment

ycombinator Nov 12, 2020

Choose a reason for hiding this comment

sayden Nov 12, 2020

Choose a reason for hiding this comment

ycombinator Nov 12, 2020

Choose a reason for hiding this comment

sayden commented Nov 18, 2020

chrisronline commented Nov 18, 2020

sayden commented Nov 18, 2020

ycombinator commented Nov 19, 2020

sayden commented Nov 19, 2020

sayden commented Sep 15, 2020 •

edited

Loading

elasticmachine commented Sep 15, 2020 •

edited by jenkins-beats-ci bot

Loading

sayden commented Oct 28, 2020 •

edited

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading