Trying to set a password in ES 8.0 fails for a security index that was created in 7.0 #38740

joshdover · 2019-02-11T16:18:53Z

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 8.0.0-SNAPSHOT

Plugins installed: []

JVM version (java -version): openjdk version "11.0.2" 2019-01-15

OS version (uname -a if on a Unix-like system): Darwin Kernel Version 18.2.0

Description of the problem including expected versus actual behavior:

I should be able to update passwords for users created in 7.0 on a 8.0 cluster.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

Start a 7.0 node with Security enabled, create a user
Shutdown node
Start an ES 8.0 node with data from the 7.0 node (also with Security enabled)
Try to update a user's password using the PUT /_security/user/<username>/_password API
Get this response:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]"},"status":400}

When looking at the mappings for the .security index it appears to only have a single type (_doc). My guess is that the logic for putting the mappings internally in ES is using doc rather than just no type at all.

Provide logs (if relevant):

09:55:12      │ info [o.e.a.a.i.m.p.TransportPutMappingAction] [kibana-ci-immutable-ubuntu-1549898516931060281] failed to put mappings on indices [[[.reindexed-v7-security-6/MHu7LjQqRB2eD4DkS2b_xg]]], type [doc]
09:55:12      │      java.lang.IllegalArgumentException: Rejecting mapping update to [.reindexed-v7-security-6] as the final mapping would have more than 1 type: [_doc, doc]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:449) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:398) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:331) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:315) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
09:55:12      │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
09:55:12      │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
09:55:12      │      	at java.lang.Thread.run(Thread.java:834) [?:?]

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-02-11T16:19:22Z

Pinging @elastic/es-security

tvernum · 2019-02-12T02:24:15Z

I can't reproduce this using the standard snapshots. I suspect it is something to do with the builds that Kibana is running (there may still be a real bug, it's just that your listed reproduction steps aren't enough to trigger it).

I don't understand where this index name comes from: .reindexed-v7-security-6
That's not the name of the default security index, and I'm not aware of anything that creates that index. Do you know why your CI has it?

The security code uses doc for its type. If your .security index has _doc, then something strange has happened.

My steps were

downloaded the latest snapshots of 7.0.0-beta1 and 8.0.0-SNAPSHOT (details below)

Set the 7.0.0 config to

node.name: node1
cluster.name: es7-8-test
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.watcher.enabled: false
xpack.monitoring.enabled: false

set the bootstrap password for 7.0.0
started ES 7.0.0

Created a user

PUT /_security/user/foo
{
  "password": "magic123", "roles": []
}

Shutdown that node
Copied the 7.0.0 config and the data directory to the 8.0.0 directory
Set the bootstrap password for 8.0.0
Started 8.0.0
Change the user's password

PUT /_security/user/foo/_password
{
  "password": "hunter"
}

ES 7.0.0 build details

commit https://github.com/elastic/elasticsearch/commits/9fae0a4
build_id 7.0.0-beta1-715764f6
build start_time Mon, 11 Feb 2019 18:08:19 GMT
package elasticsearch-7.0.0-beta1-darwin-x86_64.tar.gz

ES 8.0.0 build details

commit https://github.com/elastic/elasticsearch/commits/4e9b1cf
build_id 8.0.0-fcd45d56
build start_time "Sat, 9 Feb 2019 07:39:22 GMT
package elasticsearch-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz

joshdover · 2019-02-12T19:07:58Z

Great, this is helpful. I think I have found the issue.

The security code uses doc for its type. If your .security index has _doc, then something strange has happened.

The mapping with type _doc gets created when using the Upgrade Assistant in 8.0 to reindex 7.x indices in preparation for 9.0. Because the Upgrade Assistant uses the typeless API in 7.0+ (which I believe is the only way to create new indices), I'm guessing Elasticsearch creates a mapping with _doc under the hood. Seems to me the internal mappings for .security need to be updated to use _doc or no type at all.

The only change needed in your steps to reproduce the issue would be between steps (9) and (10). Before Step 10:

Startup Kibana 8.0 against ES 8.0
Start the reindex. Currently this has to be done via API b/c the Elasticsearch Deprecations API does not mark any indices in 8.0 as need to be reindexed for the next Lucene version, so the UI does not show them.
- curl -u elastic:changeme -XPOST localhost:5601/api/upgrade_assistant/reindex/.security-6 -H 'kbn-xsrf: true'
Wait for the reindex to complete. You'll need to wait until the reindexOp.status field of this response is equal to 1:
- curl -u elastic:changeme -XGET localhost:5601/api/upgrade_assistant/reindex/.security-6
Look at the new mappings for .security, verify you see _doc now instead of doc.
- curl "localhost:9200/.security/_mappings?include_type_name=true" -u elastic:changeme

tvernum · 2019-02-13T03:46:08Z

@joshdover
Thanks for your investigation. I'd like to close this issue and open a new one about the security index not being compatible with the Kibana upgrade assistant for 7.x to 8.x

It's not particularly surprising that this doesn't work - we haven't done any work in master to support security upgrades from 7 to 8.

Per #38678 (comment), we very much hope that security will stop using a regular index by the time we get to 8, so we're not likely to prioritise work that would change the type used in the security inde (or move to type-less APIs) until we have a better feel about where we're going with "system indices" in 7.x/8.0

For now can you just not run the upgrade assistant on the security index?

Blackiie · 2019-02-13T07:43:22Z

Emm ok no entiendo muy bien pero trataré de comprenderlo 😀. Gracias por la atencion!! El 12 feb. 2019 19:47, "Tim Vernum" <[email protected]> escribió:

…

@joshdover <https://github.com/joshdover> Thanks for your investigation. I'd like to close this issue and open a new one about the security index not being compatible with the Kibana upgrade assistant for 7.x to 8.x It's not particularly surprising that this doesn't work - we haven't done any work in master to support security upgrades from 7 to 8. Per #38678 <#38678>, we very much hope that security will stop using a regular index by the time we get to 8, so we're not likely to prioritise work that would change the type used in the security inde (or move to type-less APIs) until we have a better feel about where we're going with "system indices" in 7.x/8.0 For now can you just not run the upgrade assistant on the security index? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#38740 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AtTlEwea3ZAIrR_Nr0uCgUmQM75vzfvWks5vM4rbgaJpZM4a0ig9> .

joshdover · 2019-02-13T15:38:23Z

@tvernum That works for me. Just wanted to flag this early so it wasn't forgotten about. For our testing purposes, we'll exclude .security for the time being. Thanks!

joshdover added >bug :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Feb 11, 2019

joshdover mentioned this issue Feb 11, 2019

[upgrade] Update functional data archives for 8.0 elastic/kibana#30406

Merged

joshdover closed this as completed Feb 13, 2019

tvernum mentioned this issue Feb 17, 2019

Cannot run Kibana upgrade assistant (7.x=>8.0) on Security Index #39018

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trying to set a password in ES 8.0 fails for a security index that was created in 7.0 #38740

Trying to set a password in ES 8.0 fails for a security index that was created in 7.0 #38740

joshdover commented Feb 11, 2019

elasticmachine commented Feb 11, 2019

tvernum commented Feb 12, 2019

joshdover commented Feb 12, 2019

tvernum commented Feb 13, 2019 •

edited

Loading

Blackiie commented Feb 13, 2019 via email

joshdover commented Feb 13, 2019

Trying to set a password in ES 8.0 fails for a security index that was created in 7.0 #38740

Trying to set a password in ES 8.0 fails for a security index that was created in 7.0 #38740

Comments

joshdover commented Feb 11, 2019

elasticmachine commented Feb 11, 2019

tvernum commented Feb 12, 2019

joshdover commented Feb 12, 2019

tvernum commented Feb 13, 2019 • edited Loading

Blackiie commented Feb 13, 2019 via email

joshdover commented Feb 13, 2019

tvernum commented Feb 13, 2019 •

edited

Loading