Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to create query - field expansion matches too many fields error on Kibana for metricbeats dashboard #15863

Closed
bhavyarm opened this issue Jan 5, 2018 · 7 comments
Assignees
Labels
blocker bug Fixes for quality problems that affect the customer experience Feature:Dashboard Dashboard related features Feature:Visualizations Generic visualization features (in case no more specific feature label is available) v7.0.0

Comments

@bhavyarm
Copy link
Contributor

bhavyarm commented Jan 5, 2018

Kibana version: 7.0.0-alpha1 snapshot

Elasticsearch version: 7.0.0-alpha1 snapshot

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Description of the problem including expected versus actual behavior: When I try to open a metricbeat system overview dashboard with the timepicker set to 7 days and above - Kibana errors out with
field expansion matches too many fields, limit: 1024, got: 1174"}}}]},"status":400}. Please note - Last 15 minutes, Last 30 minutes, Last 1 hour, Last 4 hours, Last 12 hours, Last 24 hours in timepicker doesn't make Kibana go coucou.

Steps to reproduce:

  1. Install elasticsearch,kibana, metricbeat
  2. Load metricbeat dashboards, run metricbeat - create metricbeat index pattern
  3. Set the timepicker to 7 days - Kibana gives failed to create query error

Errors in browser console (if relevant):


Visualize: failed to create query: { "bool" : { "must" : [ { "query_string" : { "query" : "", "default_field" : "", "fields" : [ ], "type" : …

more
Less Info
OK
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"default_field\" : \"*\",\n          \"fields\" : [ ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"auto_generate_synonyms_phrase_query\" : true,\n          \"fuzzy_transpositions\" : true,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"default_field\" : \"*\",\n          \"fields\" : [ ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"auto_generate_synonyms_phrase_query\" : true,\n          \"fuzzy_transpositions\" : true,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : null,\n            \"to\" : null,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"Z15W8qaFTWOYsV4NokMlpw","index":"metricbeat-7.0.0-alpha1-2018.01.03"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"metricbeat-7.0.0-alpha1-2018.01.03","node":"4a8hklVXQWqJie7drTr-qA","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"default_field\" : \"*\",\n          \"fields\" : [ ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"auto_generate_synonyms_phrase_query\" : true,\n          \"fuzzy_transpositions\" : true,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"default_field\" : \"*\",\n          \"fields\" : [ ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"auto_generate_synonyms_phrase_query\" : true,\n          \"fuzzy_transpositions\" : true,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : null,\n            \"to\" : null,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"Z15W8qaFTWOYsV4NokMlpw","index":"metricbeat-7.0.0-alpha1-2018.01.03","caused_by":{"type":"illegal_argument_exception","reason":"field expansion matches too many fields, limit: 1024, got: 1174"}}}]},"status":400}
    at http://localhost:5601/bundles/kibana.bundle.js?v=16438:45:655958
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16438:56:19076)
    at http://localhost:5601/bundles/commons.bundle.js?v=16438:56:18464
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16438:56:18422)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16438:45:655536)
    at http://localhost:5601/bundles/kibana.bundle.js?v=16438:45:644886
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16438:35:132456)
    at http://localhost:5601/bundles/commons.bundle.js?v=16438:35:133349
    at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16438:35:144239)

Provide logs and/or server output (if relevant):

ES logs:

[2018-01-05T06:31:27,177][DEBUG][o.e.a.s.TransportSearchAction] [4a8hklV] All shards failed for phase: [query]
org.elasticsearch.index.query.QueryShardException: failed to create query: {
  "bool" : {
    "must" : [
      {
        "query_string" : {
          "query" : "*",
          "default_field" : "*",
          "fields" : [ ],
          "type" : "best_fields",
          "default_operator" : "or",
          "max_determinized_states" : 10000,
          "enable_position_increments" : true,
          "fuzziness" : "AUTO",
          "fuzzy_prefix_length" : 0,
          "fuzzy_max_expansions" : 50,
          "phrase_slop" : 0,
          "analyze_wildcard" : true,
          "escape" : false,
          "auto_generate_synonyms_phrase_query" : true,
          "fuzzy_transpositions" : true,
          "boost" : 1.0
        }
      },
      {
        "query_string" : {
          "query" : "*",
          "default_field" : "*",
          "fields" : [ ],
          "type" : "best_fields",
          "default_operator" : "or",
          "max_determinized_states" : 10000,
          "enable_position_increments" : true,
          "fuzziness" : "AUTO",
          "fuzzy_prefix_length" : 0,
          "fuzzy_max_expansions" : 50,
          "phrase_slop" : 0,
          "analyze_wildcard" : true,
          "escape" : false,
          "auto_generate_synonyms_phrase_query" : true,
          "fuzzy_transpositions" : true,
          "boost" : 1.0
        }
      },
      {
        "range" : {
          "@timestamp" : {
            "from" : null,
            "to" : null,
            "include_lower" : true,
            "include_upper" : true,
            "boost" : 1.0
          }
        }
      }
    ],
    "adjust_pure_negative" : true,
    "boost" : 1.0
  }
}
	at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:320) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:303) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:707) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:557) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:533) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:329) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:315) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:311) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:998) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:635) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.lang.IllegalArgumentException: field expansion matches too many fields, limit: 1024, got: 1174
	at org.elasticsearch.index.search.QueryParserHelper.checkForTooManyFields(QueryParserHelper.java:194) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.index.search.QueryParserHelper.resolveMappingField(QueryParserHelper.java:188) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.index.search.QueryParserHelper.resolveMappingField(QueryParserHelper.java:144) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.index.search.QueryStringQueryParser.<init>(QueryStringQueryParser.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.jav

error

@bhavyarm bhavyarm added Feature:Dashboard Dashboard related features Feature:Visualizations Generic visualization features (in case no more specific feature label is available) bug Fixes for quality problems that affect the customer experience blocker labels Jan 5, 2018
@bhavyarm bhavyarm added bug Fixes for quality problems that affect the customer experience and removed bug Fixes for quality problems that affect the customer experience blocker labels Jan 8, 2018
@bhavyarm
Copy link
Contributor Author

bhavyarm commented Jan 8, 2018

Old existing bug. Just keeping this here for Kibana reference:elastic/beats#5275

@thomasneirynck
Copy link
Contributor

@simianhacker suggests setting the default field of index query to something like :beat.*, this should work.

@simianhacker
Copy link
Member

simianhacker commented Jan 20, 2018

I'm getting this on discover as well, the problem is kibana is setting the default_field in the query_string query instead of letting fallback to the default (which is the index setting index.query.default_field and then *). The fix is to remove default_field from the query_string query of the request.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

@Bargs @lukasolson

{
    "_source": {
        "excludes": []
    },
    "aggs": {
        "2": {
            "date_histogram": {
                "field": "@timestamp",
                "interval": "30s",
                "min_doc_count": 1,
                "time_zone": "America/Phoenix"
            }
        }
    },
    "docvalue_fields": [
        "@timestamp",
        "ceph.monitor_health.last_updated",
        "docker.container.created",
        "docker.healthcheck.event.end_date",
        "docker.healthcheck.event.start_date",
        "docker.image.created",
        "kubernetes.container.start_time",
        "kubernetes.event.metadata.timestamp.created",
        "kubernetes.event.metadata.timestamp.deleted",
        "kubernetes.node.start_time",
        "kubernetes.pod.start_time",
        "kubernetes.system.start_time",
        "mongodb.status.background_flushing.last_finished",
        "mongodb.status.local_time",
        "postgresql.activity.backend_start",
        "postgresql.activity.query_start",
        "postgresql.activity.state_change",
        "postgresql.activity.transaction_start",
        "postgresql.bgwriter.stats_reset",
        "postgresql.database.stats_reset",
        "system.process.cpu.start_time"
    ],
    "highlight": {
        "fields": {
            "*": {}
        },
        "fragment_size": 2147483647,
        "post_tags": [
            "@/kibana-highlighted-field@"
        ],
        "pre_tags": [
            "@kibana-highlighted-field@"
        ]
    },
    "query": {
        "bool": {
            "filter": [],
            "must": [
                {
                    "query_string": {
                        "analyze_wildcard": true,
                        "default_field": "*",
                        "query": "metricset.name:node"
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "format": "epoch_millis",
                            "gte": 1516405485930,
                            "lte": 1516406385930
                        }
                    }
                }
            ],
            "must_not": [],
            "should": []
        }
    },
    "script_fields": {},
    "size": 500,
    "sort": [
        {
            "@timestamp": {
                "order": "desc",
                "unmapped_type": "boolean"
            }
        }
    ],
    "stored_fields": [
        "*"
    ],
    "version": true
}

@Bargs
Copy link
Contributor

Bargs commented Jan 22, 2018

@simianhacker Users should be able to fix this themselves by removing default_field from the query:queryString:options advanced setting. I'm not sure there's a better way to handle this at the moment. We set default_field to * by default so users with a mix of indices with and without an _all field will experience consistent highlighting across indices.

@Bargs
Copy link
Contributor

Bargs commented Jan 23, 2018

Ok, so I confirmed with the ES team that _all will be completely gone in 7.0, so we don't need this setting anymore. I've created a ticket to track it #16232

@thomasneirynck
Copy link
Contributor

can this be considered a dupe of #16232?

@Bargs
Copy link
Contributor

Bargs commented Jan 24, 2018

I'd leave them both open, this describes the bug, the issue I created describes my proposed solution. If for some reason we decided we needed a different solution and closed that other ticket, I'd still want to have this issue open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker bug Fixes for quality problems that affect the customer experience Feature:Dashboard Dashboard related features Feature:Visualizations Generic visualization features (in case no more specific feature label is available) v7.0.0
Projects
None yet
Development

No branches or pull requests

5 participants