.security with status yellow when cluster.routing.allocation.same_shard.host enabled #29933

elasticmachine · 2017-04-28T13:02:21Z

Original comment by @MarxDimitri:

Enabling cluster.routing.allocation.same_shard.host on deployments with multiple nodes on a single host puts .security to yellow. Tested with the version 5.3.

GET /_cat/indices
health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
…
yellow open   .security                         bpl6HgGdRcqmwZEqAAGBnw   1  11          1            0     17.3kb          2.8kb
…

GET /_cluster/allocation/explain
{
  "index": ".security",
  "shard": 0,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "REPLICA_ADDED",
    "at": "2017-04-24T11:50:54.057Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "9-hnXjLEThy-c4TbfgqUTg",
      "node_name": "esindex-il03_warm_2",
      "transport_address": "172.27.130.206:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[9-hnXjLEThy-c4TbfgqUTg], [R], s[STARTED], a[id=yutvMaxhTfCHI6lUgyvCDQ]]"
        }
      ]
    },
    {
      "node_id": "FSGDMHoIQwWtypVKi-Tf2A",
      "node_name": "esindex-il05_warm_2",
      "transport_address": "172.27.130.208:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[FSGDMHoIQwWtypVKi-Tf2A], [R], s[STARTED], a[id=ffHbyCIaT0WcNpB25Xu17g]]"
        }
      ]
    },
    {
      "node_id": "G7FHRbvPSWKgrbLdwgP5Cw",
      "node_name": "esindex-il05_warm_1",
      "transport_address": "172.27.130.208:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.208], where it already exists on node [G7FHRbvPSWKgrbLdwgP5Cw]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "I0IBpNXYQj63CtqFzXEQng",
      "node_name": "esindex-il01_hot_2",
      "transport_address": "172.27.130.198:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.198], where it already exists on node [I0IBpNXYQj63CtqFzXEQng]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "Wht2Q3m4ShCG9fDb1_WPAA",
      "node_name": "esindex-il04_warm_2",
      "transport_address": "172.27.130.207:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.207], where it already exists on node [Wht2Q3m4ShCG9fDb1_WPAA]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "XZMimYlLThSbMY3L9sYFbA",
      "node_name": "esindex-il03_warm_1",
      "transport_address": "172.27.130.206:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.206], where it already exists on node [XZMimYlLThSbMY3L9sYFbA]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "XiYyR0iMR6Kmb49DYoeuTw",
      "node_name": "esindex-il00_hot_2",
      "transport_address": "172.27.130.197:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[XiYyR0iMR6Kmb49DYoeuTw], [R], s[STARTED], a[id=0IuYqdzeRs-FXqHpzNDSjQ]]"
        }
      ]
    },
    {
      "node_id": "ZSk0zfN3SJeg09PDkK8E6Q",
      "node_name": "esindex-il01_hot_1",
      "transport_address": "172.27.130.198:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[ZSk0zfN3SJeg09PDkK8E6Q], [R], s[STARTED], a[id=yDdnf-wkSlySVhwhm2JsNA]]"
        }
      ]
    },
    {
      "node_id": "ceG6PFJ6Qq2s2ia8dAB68w",
      "node_name": "esindex-il02_hot_2",
      "transport_address": "172.27.130.199:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.199], where it already exists on node [ceG6PFJ6Qq2s2ia8dAB68w]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "eE8aD3CwTJeEEg30KbfFDg",
      "node_name": "esindex-il00_hot_1",
      "transport_address": "172.27.130.197:9301",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated on host address [172.27.130.197], where it already exists on node [eE8aD3CwTJeEEg30KbfFDg]; set cluster setting [cluster.routing.allocation.same_shard.host] to false to allow multiple nodes on the same host to hold the same shard copies"
        }
      ]
    },
    {
      "node_id": "uKD4Gy6ZRe-SolYYQNVMmw",
      "node_name": "esindex-il02_hot_1",
      "transport_address": "172.27.130.199:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "hot"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[uKD4Gy6ZRe-SolYYQNVMmw], [R], s[STARTED], a[id=Yln4Q0N6TrGD1FCuFvXgMg]]"
        }
      ]
    },
    {
      "node_id": "ui7i2lC3S7K_hvjF8_BMtw",
      "node_name": "esindex-il04_warm_1",
      "transport_address": "172.27.130.207:9300",
      "node_attributes": {
        "firezone": "l",
        "box_type": "warm"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.security][0], node[ui7i2lC3S7K_hvjF8_BMtw], [P], s[STARTED], a[id=cNLniAwgS4mrZDFDPRxc3g]]"
        }
      ]
    }
  ]
}

The text was updated successfully, but these errors were encountered:

elasticmachine · 2017-04-28T14:51:46Z

Original comment by @abeyad:

I don't follow - this is expected behavior. If you set cluster.routing.allocation.same_shard.host to true, that means even if there are two different nodes on the same host, and a shard copy exists on one of those nodes, then a replica copy won't be allocated to the other node, because they are on the same physical host. By default, the value for this setting is false, so if two nodes are run on the same host, they can each be allocated a copy of the same shard.

Does that make sense?

elasticmachine · 2017-04-28T15:10:55Z

Original comment by @tvernum:

I think the issue (which we might not consider to be a problem) is that the security template sets

    "auto_expand_replicas" : "0-all",

So (I assume, without having actually verified it myself) there's an inherent incompatibility between x-pack security and cluster.routing.allocation.same_shard.host

If you are running multiple nodes on the same host, and want to set cluster.routing.allocation.same_shard.host to true in order to ensure your replicas are physically distributed, then your security index will always be yellow.

I think the issue if there is one, is that cluster.routing.allocation.same_shard.host is a blunt instrument - it applies to all indices, even those that are set to auto expand to all nodes.
In this particularly case it looks like the desired effect would be to have it apply to the user's own indices, but be ignored by the security index.

elasticmachine · 2017-04-28T15:12:44Z

Original comment by @jaymode:

I think this ties into whether or not we should use auto expand replicas for the security index (another issue open about that).

elasticmachine · 2017-04-28T15:13:49Z

Original comment by @abeyad:

@tvernum thanks for the explanation, I follow what you're saying. Yes, I think then given auto_expand_replicas is used in this manner, then setting cluster.routing.allocation.same_shard.host to true will always guarantee yellow health for the .security index

elasticmachine · 2017-05-03T06:46:21Z

Original comment by @robin13:

There's a lot of people having multiple nodes on one machine: all of them should be using allocation awareness to avoid primary/replica on the same machine (cluster.routing.allocation.same_shard.host) so for all of them it seems that using security will be a problem...

elasticmachine · 2018-06-15T13:59:12Z

Pinging @elastic/es-distributed

ywelsch · 2018-06-29T13:29:40Z

The root cause for this is a well-known issue: auto-expand-replicas and shard allocation settings such as filtering, awareness and same_shard.host do not play well together: #2869

@jaymode We discussed this during FixitFriday and wondered if we need to solve the larger issue (linked above) or whether we could just not auto-expand the .security index to all nodes. IIRC you mentioned performance concerns a year ago or so. Is this still a current concern or could we change "auto_expand_replicas" : "0-all" for example to "auto_expand_replicas" : "0-1"? I've marked it as team-discuss for the security team (and would be happy to attend the security meeting when this is discussed)

jaymode · 2018-06-29T13:36:01Z

@ywelsch I think we should just go ahead with 0-1, especially since it can be updated by the user if necessary.

ywelsch · 2018-06-29T13:40:04Z

OK, thanks @jaymode

This change removes the use 0-all for auto expand replicas for the security index. The use of 0-all causes some unexpected behavior with certain allocation settings. This change allows us to avoid these with a default install. If necessary, the number of replicas can be tuned by the user. Closes elastic#29933 Closes elastic#29712

This change removes the use of 0-all for auto expand replicas for the security index. The use of 0-all causes some unexpected behavior with certain allocation settings. This change allows us to avoid these with a default install. If necessary, the number of replicas can be tuned by the user. Closes #29933 Closes #29712

elasticmachine added :Security/Security Security issues without another label discuss labels Apr 25, 2018

colings86 added the >bug label Apr 25, 2018

tvernum mentioned this issue May 10, 2018

Add expire-after-write to roles cache #30505

Open

DaveCTurner added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Jun 15, 2018

ywelsch added team-discuss and removed discuss labels Jun 29, 2018

jaymode self-assigned this Jun 29, 2018

jaymode removed the team-discuss label Jun 29, 2018

ywelsch added help wanted adoptme and removed help wanted adoptme labels Jun 29, 2018

tvernum mentioned this issue Jul 25, 2018

Curious case of draining a data node in the presence of X-Pack Security #32340

Closed

jaymode mentioned this issue Aug 24, 2018

Security index expands to a single replica #33131

Merged

jaymode closed this as completed in #33131 Aug 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.security with status yellow when cluster.routing.allocation.same_shard.host enabled #29933

.security with status yellow when cluster.routing.allocation.same_shard.host enabled #29933

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented May 3, 2017

elasticmachine commented Jun 15, 2018

ywelsch commented Jun 29, 2018

jaymode commented Jun 29, 2018

ywelsch commented Jun 29, 2018

.security with status yellow when cluster.routing.allocation.same_shard.host enabled #29933

.security with status yellow when cluster.routing.allocation.same_shard.host enabled #29933

Comments

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented Apr 28, 2017

elasticmachine commented May 3, 2017

elasticmachine commented Jun 15, 2018

ywelsch commented Jun 29, 2018

jaymode commented Jun 29, 2018

ywelsch commented Jun 29, 2018