kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-22.2 @ 6dd9210156809e27bb02063c518e43599e42d28d:

          | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:938
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
          | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:183
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
          | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:205
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
          | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
          | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
          | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1108
          | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
          | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1161
          | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1
          | 	github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-fastbuild/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9619
          | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
          | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
          | google.golang.org/grpc.chainUnaryInterceptors.func1.1
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1117
          | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
          | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:263
          | google.golang.org/grpc.chainUnaryInterceptors.func1.1
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1120
          | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
          | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:73
          | google.golang.org/grpc.chainUnaryInterceptors.func1.1
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1120
          | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
          | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:232
          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
          | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:341
          | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
          | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:230
          | google.golang.org/grpc.chainUnaryInterceptors.func1.1
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1120
          | google.golang.org/grpc.chainUnaryInterceptors.func1
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1122
          | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
          | 	github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-fastbuild/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9621
          | google.golang.org/grpc.(*Server).processUnaryRPC
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1283
          | google.golang.org/grpc.(*Server).handleStream
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1620
          | google.golang.org/grpc.(*Server).serveStreams.func1.2
          | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:922
          | runtime.goexit
          | 	GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (4) unable to find store 4 in range r52:/Table/{52-100/"cc4f70cc"} [(n1,s1):6, (n2,s2):2, (n3,s3):3, next=7, gen=25]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:113: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1347279745
--- FAIL: TestKVNemesisMultiNode (11.83s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvnemesis: TestKVNemesisMultiNode failed #101721 kv/kvnemesis: TestKVNemesisMultiNode failed [C-bug C-test-failure O-robot T-kv-replication branch-master]
• kv/kvnemesis: TestKVNemesisMultiNode failed #100417 kv/kvnemesis: TestKVNemesisMultiNode failed [C-bug C-test-failure O-robot T-kv branch-release-23.1]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-28393

[nvanbenschoten]

error applying x.AdminChangeReplicas(ctx, /Table/100/"b4ecb781", [{REMOVE_VOTER n1,s1}]) // unable to find store 4 in range r52:/Table/{52-100/"cc4f70cc"} [(n1,s1):6, (n2,s2):2, (n3,s3):3, next=7, gen=25]: unable to find store 4 in range r52:/Table/{52-100/"cc4f70cc"} [(n1,s1):6, (n2,s2):2, (n3,s3):3, next=7, gen=25]

This will be fixed by #100957.

[nvanbenschoten]

Closed by #100957.