Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #61322

Closed
cockroach-teamcity opened this issue Mar 2, 2021 · 16 comments · Fixed by #62580
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #61322

cockroach-teamcity opened this issue Mar 2, 2021 · 16 comments · Fixed by #62580
Assignees
@nvanbenschoten
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@1b18fc1fa8d9194d5e7f9aaa857425d80f0e4970:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode786161560
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:85: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"b81ab7eb", 2) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:618
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:353
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:100
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:81
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"b81ab7eb", 2) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:87: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode786161560
--- FAIL: TestKVNemesisMultiNode (0.60s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 2, 2021
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@6de4313ec216161c79fe725fcc31fc87ef1804ea:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode579305741
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:85: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"e6c3fa5e", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1614755980.855514634,0 epo=1 pro=1614755980.855325030,0 with repl=(n1,s1):1 seq=0 start=1614755980.886729612,0 exp=<nil> pro=1614755980.886729612,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n4,s4):2 seq=2 start=1614755980.855514634,0 epo=1 pro=1614755980.855325030,0 with repl=(n1,s1):1 seq=0 start=1614755980.886729612,0 exp=<nil> pro=1614755980.886729612,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:100
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:81
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"e6c3fa5e", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1614755980.855514634,0 epo=1 pro=1614755980.855325030,0 with repl=(n1,s1):1 seq=0 start=1614755980.886729612,0 exp=<nil> pro=1614755980.886729612,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n4,s4):2 seq=2 start=1614755980.855514634,0 epo=1 pro=1614755980.855325030,0 with repl=(n1,s1):1 seq=0 start=1614755980.886729612,0 exp=<nil> pro=1614755980.886729612,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:87: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode579305741
--- FAIL: TestKVNemesisMultiNode (0.77s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@37cdf01299861235888af3411e699480f5a2d670:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode198946516
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:85: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"e93d9dc4", 3) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:100
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:81
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"e93d9dc4", 3) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:87: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode198946516
--- FAIL: TestKVNemesisMultiNode (0.63s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@b703e663da8ededaee2e28fc39a24e3880ae54cf:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode387236614
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:85: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"2085254b", 3) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:100
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:81
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"2085254b", 3) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:87: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode387236614
--- FAIL: TestKVNemesisMultiNode (0.64s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@a69e6549a71f5a0e83eb13509001f4d7351050fb:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode049091563
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"3b23d7fd", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1615100868.961781083,0 epo=1 pro=1615100868.961420487,0 with repl=(n1,s1):1 seq=0 start=1615100868.999518092,0 exp=<nil> pro=1615100868.999518092,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n4,s4):2 seq=2 start=1615100868.961781083,0 epo=1 pro=1615100868.961420487,0 with repl=(n1,s1):1 seq=0 start=1615100868.999518092,0 exp=<nil> pro=1615100868.999518092,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"3b23d7fd", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1615100868.961781083,0 epo=1 pro=1615100868.961420487,0 with repl=(n1,s1):1 seq=0 start=1615100868.999518092,0 exp=<nil> pro=1615100868.999518092,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n4,s4):2 seq=2 start=1615100868.961781083,0 epo=1 pro=1615100868.961420487,0 with repl=(n1,s1):1 seq=0 start=1615100868.999518092,0 exp=<nil> pro=1615100868.999518092,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode049091563
--- FAIL: TestKVNemesisMultiNode (1.17s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@93c38e97ddb9d18721eed850180c103fe3c45547:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode842178868
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"a4d28c77", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1615276193.079342606,0 epo=1 pro=1615276193.078941519,0 with repl=(n1,s1):1 seq=0 start=1615276193.081319832,0 exp=<nil> pro=1615276193.081319832,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1615276193.079342606,0 epo=1 pro=1615276193.078941519,0 with repl=(n1,s1):1 seq=0 start=1615276193.081319832,0 exp=<nil> pro=1615276193.081319832,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"a4d28c77", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1615276193.079342606,0 epo=1 pro=1615276193.078941519,0 with repl=(n1,s1):1 seq=0 start=1615276193.081319832,0 exp=<nil> pro=1615276193.081319832,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1615276193.079342606,0 epo=1 pro=1615276193.078941519,0 with repl=(n1,s1):1 seq=0 start=1615276193.081319832,0 exp=<nil> pro=1615276193.081319832,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode842178868
--- FAIL: TestKVNemesisMultiNode (5.36s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@fc68d927f946103a757888fcbd3ac20904c57a6b:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode728807589
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"47c09f46", 4) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"47c09f46", 4) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode728807589
--- FAIL: TestKVNemesisMultiNode (1.15s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@bdff5338ca725bf1cfddf7e3f648bbf02ab42999:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode003609470
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"5b4746aa", 4) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"5b4746aa", 4) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode003609470
--- FAIL: TestKVNemesisMultiNode (1.13s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@erikgrinaker erikgrinaker assigned erikgrinaker and unassigned tbg Mar 14, 2021
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@f3c8f6ed74494f93bd1eb01b1d7569127f127906:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode396686203
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"148a0682", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1615790437.893944621,0 epo=1 pro=1615790437.893016762,0 with repl=(n1,s1):1 seq=0 start=1615790437.916012208,0 exp=<nil> pro=1615790437.916012208,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1615790437.893944621,0 epo=1 pro=1615790437.893016762,0 with repl=(n1,s1):1 seq=0 start=1615790437.916012208,0 exp=<nil> pro=1615790437.916012208,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"148a0682", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1615790437.893944621,0 epo=1 pro=1615790437.893016762,0 with repl=(n1,s1):1 seq=0 start=1615790437.916012208,0 exp=<nil> pro=1615790437.916012208,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1615790437.893944621,0 epo=1 pro=1615790437.893016762,0 with repl=(n1,s1):1 seq=0 start=1615790437.916012208,0 exp=<nil> pro=1615790437.916012208,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode396686203
--- FAIL: TestKVNemesisMultiNode (1.33s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@3d19b2cf6b290a152b23722fc32e995eed3b437b:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode775967295
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"48ca8b46", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1616221348.715285078,0 epo=1 pro=1616221348.715072682,0 with repl=(n1,s1):1 seq=0 start=1616221348.887276887,0 exp=<nil> pro=1616221348.887276887,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1616221348.715285078,0 epo=1 pro=1616221348.715072682,0 with repl=(n1,s1):1 seq=0 start=1616221348.887276887,0 exp=<nil> pro=1616221348.887276887,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"48ca8b46", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1616221348.715285078,0 epo=1 pro=1616221348.715072682,0 with repl=(n1,s1):1 seq=0 start=1616221348.887276887,0 exp=<nil> pro=1616221348.887276887,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1616221348.715285078,0 epo=1 pro=1616221348.715072682,0 with repl=(n1,s1):1 seq=0 start=1616221348.887276887,0 exp=<nil> pro=1616221348.887276887,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode775967295
--- FAIL: TestKVNemesisMultiNode (5.16s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@893643b63ea0b1cfa4888c6b73b5c68a9c100c3a:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode111655860
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"b6d5dab1", 2) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"b6d5dab1", 2) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode111655860
--- FAIL: TestKVNemesisMultiNode (1.12s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@3f9f18ebf27dc0fdbf7fa0fce639be1513c12623:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode295798159
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"d64828a0", 3) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"d64828a0", 3) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:505
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode295798159
--- FAIL: TestKVNemesisMultiNode (1.15s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten nvanbenschoten self-assigned this Mar 25, 2021
@nvanbenschoten nvanbenschoten mentioned this issue Mar 25, 2021
Merged
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@d891594d3c998f153b88f631e3c89ac7d12c2a6e:

          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"84745278", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.899911531,0 exp=<nil> pro=1617000610.899911531,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.899911531,0 exp=<nil> pro=1617000610.899911531,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"6ff672f7", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.905154847,0 exp=<nil> pro=1617000610.905154847,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.905154847,0 exp=<nil> pro=1617000610.905154847,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"6ff672f7", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.905154847,0 exp=<nil> pro=1617000610.905154847,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.905154847,0 exp=<nil> pro=1617000610.905154847,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"723c45b3", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.929346329,0 exp=<nil> pro=1617000610.929346329,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.929346329,0 exp=<nil> pro=1617000610.929346329,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"723c45b3", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.929346329,0 exp=<nil> pro=1617000610.929346329,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1617000610.874884348,0 epo=1 pro=1617000610.874670825,0 with repl=(n1,s1):1 seq=0 start=1617000610.929346329,0 exp=<nil> pro=1617000610.929346329,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode967354694
--- FAIL: TestKVNemesisMultiNode (4.96s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@erikgrinaker erikgrinaker removed their assignment Mar 30, 2021
@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@9891b8db5ea2e9339a04fd2731a16658103e969f:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode010515030
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"c63f363f", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617344475.331209739,0 epo=1 pro=1617344475.330256239,0 with repl=(n1,s1):1 seq=0 start=1617344475.476365348,0 exp=<nil> pro=1617344475.476365348,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n3,s3):2 seq=2 start=1617344475.331209739,0 epo=1 pro=1617344475.330256239,0 with repl=(n1,s1):1 seq=0 start=1617344475.476365348,0 exp=<nil> pro=1617344475.476365348,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"c63f363f", 1) // cannot replace lease repl=(n3,s3):2 seq=2 start=1617344475.331209739,0 epo=1 pro=1617344475.330256239,0 with repl=(n1,s1):1 seq=0 start=1617344475.476365348,0 exp=<nil> pro=1617344475.476365348,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n3,s3):2 seq=2 start=1617344475.331209739,0 epo=1 pro=1617344475.330256239,0 with repl=(n1,s1):1 seq=0 start=1617344475.476365348,0 exp=<nil> pro=1617344475.476365348,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode010515030
--- FAIL: TestKVNemesisMultiNode (5.57s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@df07592e26735be7362ef2b815ee163ead00739e:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode059915257
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"7c52eb95", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.628373299,0 exp=<nil> pro=1617432664.628373299,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.628373299,0 exp=<nil> pro=1617432664.628373299,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"7c52eb95", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.628373299,0 exp=<nil> pro=1617432664.628373299,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.628373299,0 exp=<nil> pro=1617432664.628373299,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"6625d470", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.701637883,0 exp=<nil> pro=1617432664.701637883,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.701637883,0 exp=<nil> pro=1617432664.701637883,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"6625d470", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.701637883,0 exp=<nil> pro=1617432664.701637883,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n4,s4):2 seq=2 start=1617432664.579909696,0 epo=1 pro=1617432664.579689746,0 with repl=(n1,s1):1 seq=0 start=1617432664.701637883,0 exp=<nil> pro=1617432664.701637883,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode059915257
--- FAIL: TestKVNemesisMultiNode (1.15s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@651184b3cfbd7c6d336bb49e7152e3d48ea06416:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode942691456
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"724d8a2a", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617777779.042326013,0 epo=1 pro=1617777779.041851333,0 with repl=(n1,s1):1 seq=0 start=1617777779.231175602,0 exp=<nil> pro=1617777779.231175602,0: couldn't request lease for (n1,s1):1: liveness record not found in cache: cannot replace lease repl=(n4,s4):2 seq=2 start=1617777779.042326013,0 epo=1 pro=1617777779.041851333,0 with repl=(n1,s1):1 seq=0 start=1617777779.231175602,0 exp=<nil> pro=1617777779.231175602,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"724d8a2a", 1) // cannot replace lease repl=(n4,s4):2 seq=2 start=1617777779.042326013,0 epo=1 pro=1617777779.041851333,0 with repl=(n1,s1):1 seq=0 start=1617777779.231175602,0 exp=<nil> pro=1617777779.231175602,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Wraps: (3) cannot replace lease repl=(n4,s4):2 seq=2 start=1617777779.042326013,0 epo=1 pro=1617777779.041851333,0 with repl=(n1,s1):1 seq=0 start=1617777779.231175602,0 exp=<nil> pro=1617777779.231175602,0: couldn't request lease for (n1,s1):1: liveness record not found in cache
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *roachpb.LeaseRejectedError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode942691456
--- FAIL: TestKVNemesisMultiNode (5.14s)
More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 2c50690c86b0de434946bb74f0e63aea1907ea20:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode848721057
    test_log_scope.go:74: use -show-logs to present logs inline
    kvnemesis_test.go:92: failure:
        error applying x.TransferLeaseOperation(ctx, /Table/50/"95ea6651", 4) // replica not found in RangeDescriptor: replica not found in RangeDescriptor
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).failIfError
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:621
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.(*validator).processOp
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:356
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.Validate
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/validator.go:74
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.RunNemesis
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis.go:104
          | github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.TestKVNemesisMultiNode
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:88
          | testing.tRunner
          | 	/usr/local/go/src/testing/testing.go:1123
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) error applying x.TransferLeaseOperation(ctx, /Table/50/"95ea6651", 4) // replica not found in RangeDescriptor
        Wraps: (3)
          | (opaque error wrapper)
          | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
          | reportable 0:
          |
          | github.com/cockroachdb/cockroach/pkg/roachpb.init
          | 	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/metadata_replicas.go:493
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5652
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.doInit
          | 	/usr/local/go/src/runtime/proc.go:5647
          | runtime.main
          | 	/usr/local/go/src/runtime/proc.go:191
          | runtime.goexit
          | 	/usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) replica not found in RangeDescriptor
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError
    kvnemesis_test.go:94: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode848721057
--- FAIL: TestKVNemesisMultiNode (1.12s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-parallel=4

See this test on roachdash

Improve this report!

craig bot pushed a commit that referenced this issue Apr 8, 2021
@nvanbenschoten
Merge #62580
c41bd9c 
62580: kv: deflake multi-node kvnemesis r=nvanbenschoten a=nvanbenschoten

Fixes #61322.
Prep to address #59062.

#### kv: don't consider Subsume error as assertion failures

These were added in b15e3dd, but that was incorrect. It is valid for a merge that races with another merge or split to hit these case, as the Subsume request is non-transactional.

Also, allow `RHS range bounds do not match` errors in kvnemesis. This is a valid error for the reasons mentioned above.

#### kv: mark error from Replica.GetSnapshot with errMarkSnapshotError

Without it, we see the following errors in kvnemesis:
```
Wraps: (2) error applying x.AdminChangeReplicas(ctx, /Table/50/"244956da", [{ADD_VOTER n2,s2}]) // [n1,s1,r36/1:/Table/{40-50/"ba86b…}]: failed to generate LEARNER_INITIAL snapshot: couldn't find range descriptor
```

#### kv: allow two more errors with TransferLeaseOperation

This deflakes TestKVNemesisMultiNode. The test had recently gotten flakier, which I bisected back to 3fe1992. It appears that the changes in that commit made it more likely to hit this `replica not found in RangeDescriptor`, which is returned after a TransferLease request has acquired latches, in addition to the existing `unable to find store \d+ in range` error, which is returned before a TransferLease request has acquired latches.

Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in f7d2018 Apr 8, 2021
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 9, 2021
@nvanbenschoten
kv: allow two more errors with TransferLeaseOperation
5608853 
Fixes cockroachdb#61322.

This deflakes TestKVNemesisMultiNode. The test had recently gotten more
flaky, which I bisected back to 3fe1992. It appears that the changes in
that commit made it more likely to hit this `replica not found in RangeDescriptor`,
which is returned after a TransferLease request has acquired latches, in
addition to the existing `unable to find store \d+ in range` error, which
is returned before a TransferLease request has acquired latches.
@nvanbenschoten nvanbenschoten mentioned this issue Apr 9, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv: deflake multi-node kvnemesis nvanbenschoten/cockroach
4 participants
@erikgrinaker @tbg @nvanbenschoten @cockroach-teamcity