kv/kvserver: TestRaftRemoveRace failed

[cockroach-teamcity]

kv/kvserver.TestRaftRemoveRace failed with artifacts on master @ 9b43dc50d468171bc9919dc684b7274b63e13da2:

=== RUN   TestRaftRemoveRace
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/d351c1750c446bc1442d58db9c0ceef9/logTestRaftRemoveRace4047806013
    test_log_scope.go:79: use -show-logs to present logs inline
    client_raft_test.go:2963: could not remove [n3,s3] replicas from range containing /Table/Max: AdminChangeReplicas error: cannot remove learner while snapshot is in flight
        (1) AdminChangeReplicas error: cannot remove learner while snapshot is in flight
          | -- cause hidden behind barrier
          | AdminChangeReplicas error: cannot remove learner while snapshot is in flight
          | (1) attached stack trace
          |   -- stack trace:
          |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).changeReplicas
          |   | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:764
          |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RemoveVoters
          |   | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:920
          |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RemoveVotersOrFatal
          |   | 	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:928
          |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRaftRemoveRace
          |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:2963
          |   | testing.tRunner
          |   | 	GOROOT/src/testing/testing.go:1446
          |   | runtime.goexit
          |   | 	GOROOT/src/runtime/asm_arm64.s:1172
          | Wraps: (2) AdminChangeReplicas error
          | Wraps: (3) forced error mark
          |   | "replication change in progress"
          |   | github.com/cockroachdb/errors/withstack/*withstack.withStack::
          | Wraps: (4)
          |   | (opaque error wrapper)
          |   | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          |   | reportable 0:
          |   |
          |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.init
          |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:1309
          |   | runtime.doInit
          |   | 	GOROOT/src/runtime/proc.go:6348
          |   | runtime.doInit
          |   | 	GOROOT/src/runtime/proc.go:6325
          |   | runtime.main
          |   | 	GOROOT/src/runtime/proc.go:233
          |   | runtime.goexit
          |   | 	GOROOT/src/runtime/asm_arm64.s:1172
          | Wraps: (5) cannot remove learner while snapshot is in flight
          | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *markers.withMark (4) *errbase.opaqueWrapper (5) *errutil.leafError
        Error types: (1) *barriers.barrierErr
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/d351c1750c446bc1442d58db9c0ceef9/logTestRaftRemoveRace4047806013
--- FAIL: TestRaftRemoveRace (6.15s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-25417

[erikgrinaker]

Resolved by #99099.