Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Null Pointer when using tophits on double nested field #33163

Closed
dvdeurse opened this issue Aug 27, 2018 · 1 comment · Fixed by #33180
Closed

Null Pointer when using tophits on double nested field #33163

dvdeurse opened this issue Aug 27, 2018 · 1 comment · Fixed by #33180
Labels
>bug :Search/Search Search-related issues that do not fall into other categories

Comments

@dvdeurse
Copy link

Elasticsearch version (bin/elasticsearch --version): 6.3.2

Plugins installed: []

JVM version (java -version): 1.8.0_171

OS version (uname -a if on a Unix-like system): Alpine 3.8

Description of the problem including expected versus actual behavior:

Prerequisites to reproduce the problem:

  • a mapping with a double nested structure, where at least two fields are present (in the deepest nested object)
  • instances where only one of these two fields appear (so one of the two fields is never present in the index)
  • a tophits aggregation where you request the field that never appears as one of the source fields.

The result is a failing shard with a nullpointer as reason. You don't see this problem when you only have one level of nesting or when there are instances that contain both fields.

Steps to reproduce:

  1. Create the index + mapping:
curl -X PUT "localhost:9200/test?pretty" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "_doc" : {
            "properties" : {
                "a" : { 
                    "type" : "nested" ,
                    "include_in_root": true,
                    "properties": {
                        "b": {
                            "type" : "nested" ,
                            "include_in_root": true,
                            "properties": {
                                "c": {
                                    "type": "keyword"
                                },
                                "d": {
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
'
  1. Add one document (without the 'd' field):
curl -X PUT "localhost:9200/test/_doc/1?pretty&refresh" -H 'Content-Type: application/json' -d'
{
    "a" : [
        {
            "b": [
                {
                    "c": "foo1"
                },
                {
                    "c": "foo2"
                }
            ]
        }
    ]
}
'
  1. Execute the tophits aggregation, where you ask to include the 'd' field
curl -X POST "localhost:9200/test/_search?pretty" -H 'Content-Type: application/json' -d'
{
   "aggs":{
      "a":{
         "nested":{
            "path":"a"
         },
         "aggregations":{
            "b":{
               "nested":{
                  "path":"a.b"
               },
               "aggregations":{
                  "tophits":{
                     "top_hits":{
                        "from":0,
                        "size":1,
                        "_source":{
                           "includes":[
                              "a.b.d"
                           ],
                           "excludes":[]
                        }
                     }
                  }
               }
            }
         }
      }
   }
}
'

The result of the last command is as follows:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "test",
        "node" : "1lzL7jmZS0W3JNkjfe0h5g",
        "reason" : {
          "type" : "null_pointer_exception",
          "reason" : null
        }
      }
    ],
    "caused_by" : {
      "type" : "null_pointer_exception",
      "reason" : null,
      "caused_by" : {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    }
  },
  "status" : 500
}

Provide logs (if relevant):
Additional logs seen in the server logs:

[2018-08-27T11:20:18,534][WARN ][r.suppressed             ] path: /test/_search, params: {pretty=, index=test}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1196) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1180) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.elasticsearch.ElasticsearchException$1
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.2.jar:6.3.2]
	... 26 more
Caused by: java.lang.NullPointerException
	at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.getNestedSource(FetchSourceSubPhase.java:83) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:58) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.metrics.tophits.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:168) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:128) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregation(NestedAggregator.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:128) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregation(NestedAggregator.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:130) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:324) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:357) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.2.jar:6.3.2]
	... 9 more
@dvdeurse dvdeurse changed the title Nullpointer when using tophits on double nested field Null Pointer when using tophits on double nested field Aug 27, 2018
jimczi added a commit to jimczi/elasticsearch that referenced this issue Aug 27, 2018
@jimczi
Fix nested _source retrieval with includes/excludes
0db3524 
If an exclude or an include clause removes an entry to a nested field in the original source at query time,
the creation of nested hits fails with an NPE. This change fixes this exception and replaces the nested document
source with an empty map.

Closes elastic#33163
Closes elastic#33170
@jimczi jimczi mentioned this issue Aug 27, 2018
Merged
@jimczi jimczi added >bug :Search/Search Search-related issues that do not fall into other categories labels Aug 27, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search-aggs

jimczi added a commit that referenced this issue Aug 30, 2018
@jimczi
Fix nested _source retrieval with includes/excludes (#33180)
1404dd2 
If an exclude or an include clause removes an entry to a nested field in the original source at query time,
the creation of nested hits fails with an NPE. This change fixes this exception and replaces the nested document
source with an empty map.

Closes #33163
Closes #33170
jimczi added a commit that referenced this issue Aug 30, 2018
@jimczi
Fix nested _source retrieval with includes/excludes (#33180)
32b63fd 
If an exclude or an include clause removes an entry to a nested field in the original source at query time,
the creation of nested hits fails with an NPE. This change fixes this exception and replaces the nested document
source with an empty map.

Closes #33163
Closes #33170
jimczi added a commit that referenced this issue Aug 30, 2018
@jimczi
Fix nested _source retrieval with includes/excludes (#33180)
27af04d 
If an exclude or an include clause removes an entry to a nested field in the original source at query time,
the creation of nested hits fails with an NPE. This change fixes this exception and replaces the nested document
source with an empty map.

Closes #33163
Closes #33170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Search/Search Search-related issues that do not fall into other categories
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix nested _source retrieval with includes/excludes jimczi/elasticsearch
3 participants
@dvdeurse @elasticmachine @jimczi