PUT /_settings causes NullPointerException if a system index is present #67147

rudolf · 2021-01-07T11:26:30Z

Versions affected: 7.12.0, 8.0.0

Steps to reproduce:

# Perform a reindex so that the .tasks index is created
$ curl -X PUT "elastic:changeme@localhost:9220/my-index?pretty" -H 'Content-Type: application/json'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "my-index"
}
$ curl -X POST "elastic:changeme@localhost:9220/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d '{"source": {"index": "my-index"}, "dest": {"index": "my-index-2"}}'

# Update all indices' settings
$ curl -X PUT "elastic:changeme@localhost:9220/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index": {
      "number_of_replicas": 0
    }
}
'

API response

{
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : "Cannot invoke \"String.equals(Object)\" because \"expectedValue\" is null"
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : "Cannot invoke \"String.equals(Object)\" because \"expectedValue\" is null"
  },
  "status" : 500
}

Stack trace:

  │ info [r.suppressed] [rudolf-elastic-mbp.lan] path: /_settings, params: {pretty=}
   │      java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "expectedValue" is null
   │            at org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.checkForSystemIndexViolations(TransportUpdateSettingsAction.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.masterOperation(TransportUpdateSettingsAction.java:93) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.masterOperation(TransportUpdateSettingsAction.java:51) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:162) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:176) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:162) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:57) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:88) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:160) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:178) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:381) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:128) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:176) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:76) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:376) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:128) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:176) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:376) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:128) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:176) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
   │            at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.el

Related: #65604

The text was updated successfully, but these errors were encountered:

gwbrown · 2021-01-08T00:26:45Z

I took a quick look at this and this no longer reproduces, probably because some changes around .tasks were reverted today (#67162) and .tasks is no longer an automatically-managed system index.

However, if you run the commands above against an ES instance running with assertions enabled, it will trigger an assertion and kill the node (updating the number of replicas on an index with auto_expand_replicas does this on every version I tried, going back to at least 6.8.10 though).

I do believe this is still an issue, but it no longer reproduces with the commands given above. The problem appears to be pretty clear, but I haven't yet reproduced it to be 100% certain. I'll have a PR up fairly shortly once I've confirmed it fixes the issue.

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

elasticmachine · 2021-01-13T21:12:20Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

gwbrown · 2021-01-13T21:12:46Z

@rudolf I've merged the fix for the NPE to all current branches, is there anything left to be done on this issue?

rudolf · 2021-01-14T12:01:13Z

No, thanks for fixing this!

rudolf added >bug >regression needs:triage Requires assignment of a team area label labels Jan 7, 2021

rudolf mentioned this issue Jan 7, 2021

Failing ES Promotion: migration actions createIndex resolves right after waiting for an index status to be green if the index already existed elastic/kibana#87160

Closed

rudolf added a commit to rudolf/kibana that referenced this issue Jan 7, 2021

Workaround elastic/elasticsearch#67147

9d581f0

dnhatn mentioned this issue Jan 7, 2021

PUT #67146

Closed

gwbrown mentioned this issue Jan 8, 2021

Fix NPE when updating settings on managed system indices #67200

Merged

rudolf added a commit to elastic/kibana that referenced this issue Jan 11, 2021

Workaround elastic/elasticsearch#67147 (#87615)

3ef7bd3

Co-authored-by: Kibana Machine <[email protected]>

rudolf mentioned this issue Jan 11, 2021

[7.x] Workaround https://github.com/elastic/elasticsearch/issues/67147 (#87615) elastic/kibana#87922

Merged

rudolf added a commit to elastic/kibana that referenced this issue Jan 13, 2021

Workaround elastic/elasticsearch#67147 (#87615) (#87922)

3f76cc4

Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

gwbrown added the :Core/Infra/Core Core issues without another label label Jan 13, 2021

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jan 13, 2021

rudolf closed this as completed Jan 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PUT /_settings causes NullPointerException if a system index is present #67147

PUT /_settings causes NullPointerException if a system index is present #67147

rudolf commented Jan 7, 2021

gwbrown commented Jan 8, 2021

elasticmachine commented Jan 13, 2021

gwbrown commented Jan 13, 2021

rudolf commented Jan 14, 2021

PUT /_settings causes NullPointerException if a system index is present #67147

PUT /_settings causes NullPointerException if a system index is present #67147

Comments

rudolf commented Jan 7, 2021

gwbrown commented Jan 8, 2021

elasticmachine commented Jan 13, 2021

gwbrown commented Jan 13, 2021

rudolf commented Jan 14, 2021