Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] CLUSTER SHARDS command returns "empty array" in slots section #784

Closed
patademahesh opened this issue Jul 14, 2024 · 6 comments · Fixed by #790
Closed

[BUG] CLUSTER SHARDS command returns "empty array" in slots section #784

patademahesh opened this issue Jul 14, 2024 · 6 comments · Fixed by #790
Assignees
@hpatro

Comments

@patademahesh
Copy link

We are running a 6-node Valkey cluster (version 7.2.5) in a docker environment with 1 replica. When we stop one of the master nodes in the cluster, the CLUSTER SHARDS command returns empty slots for that specific shard.

Output with an empty array.

1) 1) "slots"
   2) 1) (integer) 5461
      2) (integer) 10922
   3) "nodes"
   4) 1)  1) "id"
          2) "d3b881c0c7d7643d507315ce628eedbcc9941479"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.6"
          7) "endpoint"
          8) "172.18.0.6"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 476
         13) "health"
         14) "online"
      2)  1) "id"
          2) "a779c9422551aa92ad0f986b66014eb2eb50f609"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.7"
          7) "endpoint"
          8) "172.18.0.7"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 476
         13) "health"
         14) "online"
2) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "516fdb677583963b8ec91723422f4ecaa59d0afe"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.5"
          7) "endpoint"
          8) "172.18.0.5"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 476
         13) "health"
         14) "online"
      2)  1) "id"
          2) "900fe89950abac7ac427e265809d96816c51ecf9"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.2"
          7) "endpoint"
          8) "172.18.0.2"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 476
         13) "health"
         14) "online"
3) 1) "slots"
   2) (empty array)
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.4"
          7) "endpoint"
          8) "172.18.0.4"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 434
         13) "health"
         14) "fail"
      2)  1) "id"
          2) "8bba1340c8ffedadf25abe968aeccee3b7253cb8"
          3) "port"
          4) (integer) 6379
          5) "ip"
          6) "172.18.0.3"
          7) "endpoint"
          8) "172.18.0.3"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 434
         13) "health"
         14) "online"

Steps to reproduce the behavior and/or a minimal code sample.

  • start the cluster using attached docker-compose.txt
  • create a cluster using the below command
    docker exec -it redis-node-0 redis-cli --cluster create redis-node-0:6379 redis-node-1:6379 redis-node-2:6379 redis-node-3:6379 redis-node-4:6379 redis-node-5:6379 --cluster-replicas 1
  • stop one of the master containers and check the output of the CLUSTER SHARDS command
    docker stop redis-node-2
    docker exec -it redis-node-0 redis-cli -h redis-node-5 CLUSTER SHARDS

It should return slots correctly. Please let us know if we are doing something wrong or if this is expected behavior.
@madolson
Copy link
Member

madolson commented Jul 14, 2024 •
edited
Loading

@hpatro can you take a look? This does look like some sort of bug.

@hpatro hpatro self-assigned this Jul 15, 2024
@hpatro
Copy link
Collaborator

hpatro commented Jul 15, 2024

Taking a look.

@hpatro hpatro mentioned this issue Jul 15, 2024
Merged
@hpatro
Copy link
Collaborator

hpatro commented Jul 15, 2024

Able to reproduce it via test #790. Will look into the fix next.

PingXie pushed a commit that referenced this issue Jul 19, 2024
@hpatro
Generate correct slot information in cluster shards command on primar…
816acce 
…y failure (#790)

Fix #784 

Prior to the change, `CLUSTER SHARDS` command processing might pick a
failed primary node which won't have the slot coverage information and
the slots `output` in turn would be empty. This change finds an
appropriate node which has the slot coverage information served by a
given shard and correctly displays it as part of `CLUSTER SHARDS`
output.

 Before:
 
 ```
 1) 1) "slots"
   2)  (empty array)
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```
 
 After:
 

 ```
 1) 1) "slots"
   2) 1) 0
       2) 5461
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

---------

Signed-off-by: Harkrishn Patro <[email protected]>
PingXie pushed a commit to PingXie/valkey that referenced this issue Jul 22, 2024
@hpatro @PingXie
Generate correct slot information in cluster shards command on primar…
5055052 
…y failure (valkey-io#790)

Fix valkey-io#784

Prior to the change, `CLUSTER SHARDS` command processing might pick a
failed primary node which won't have the slot coverage information and
the slots `output` in turn would be empty. This change finds an
appropriate node which has the slot coverage information served by a
given shard and correctly displays it as part of `CLUSTER SHARDS`
output.

 Before:

 ```
 1) 1) "slots"
   2)  (empty array)
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

 After:

 ```
 1) 1) "slots"
   2) 1) 0
       2) 5461
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

---------

Signed-off-by: Harkrishn Patro <[email protected]>
@PingXie PingXie mentioned this issue Jul 22, 2024
Merged
PingXie pushed a commit that referenced this issue Jul 23, 2024
@hpatro @PingXie
Generate correct slot information in cluster shards command on primar…
3aaa129 
…y failure (#790)

Fix #784

Prior to the change, `CLUSTER SHARDS` command processing might pick a
failed primary node which won't have the slot coverage information and
the slots `output` in turn would be empty. This change finds an
appropriate node which has the slot coverage information served by a
given shard and correctly displays it as part of `CLUSTER SHARDS`
output.

 Before:

 ```
 1) 1) "slots"
   2)  (empty array)
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

 After:

 ```
 1) 1) "slots"
   2) 1) 0
       2) 5461
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

---------

Signed-off-by: Harkrishn Patro <[email protected]>
@patademahesh
Copy link
Author

Thank you @PingXie @hpatro

When this will be released?

@madolson
Copy link
Member

When this will be released?

For 7.2, in the upcoming week or so. We had a release planned.

hwware pushed a commit to hwware/valkey that referenced this issue Jul 25, 2024
@hpatro @hwware
Generate correct slot information in cluster shards command on primar…
55b9d91 
…y failure (valkey-io#790)

Fix valkey-io#784 

Prior to the change, `CLUSTER SHARDS` command processing might pick a
failed primary node which won't have the slot coverage information and
the slots `output` in turn would be empty. This change finds an
appropriate node which has the slot coverage information served by a
given shard and correctly displays it as part of `CLUSTER SHARDS`
output.

 Before:
 
 ```
 1) 1) "slots"
   2)  (empty array)
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```
 
 After:
 

 ```
 1) 1) "slots"
   2) 1) 0
       2) 5461
   3) "nodes"
   4) 1)  1) "id"
          2) "2936f22a490095a0a851b7956b0a88f2b67a5d44"
          ...
          9) "role"
         10) "master"
         ...
         13) "health"
         14) "fail"
 ```

---------

Signed-off-by: Harkrishn Patro <[email protected]>
@PingXie
Copy link
Member

PingXie commented Aug 16, 2024

Thank you @PingXie @hpatro

When this will be released?

It has been released as part of 7.2.6

@rueian rueian mentioned this issue Nov 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hpatro hpatro

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Generate correct slot information in cluster shards command on primary failure hpatro/valkey
4 participants
@patademahesh @PingXie @hpatro @madolson