Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: store_raft.go:647: log.Fatal: raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: repl=(n1,s1... #103658

Closed
cockroach-teamcity opened this issue May 19, 2023 · 1 comment
Closed

sentry: store_raft.go:647: log.Fatal: raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: repl=(n1,s1... #103658

cockroach-teamcity opened this issue May 19, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 19, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/4193922203/?referrer=webhooks_plugin

Panic message:

store_raft.go:647: log.Fatal: raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: repl=(n1,s1):1 seq=2 start=1684255601.937166900,0 epo=2 pro=1684521780.332246100,0, req: TruncateLog [/System/tsd,/Min), applying at LAI: 1478.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:647
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:392
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
| raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: repl=(n1,s1):1 seq=2 start=1684255601.937166900,0 epo=2 pro=1684521780.332246100,0, req: TruncateLog [/System/tsd,/Min), applying at LAI: 1478.
| (1) assertion failure
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:738
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:118
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:188
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1005
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:392
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
| | runtime.goexit
| | GOROOT/src/runtime/asm_amd64.s:1594
| Wraps: (3) raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: repl=(n1,s1):1 seq=2 start=1684255601.937166900,0 epo=2 pro=1684521780.332246100,0, req: TruncateLog [/System/tsd,/Min), applying at LAI: 1478.
| | Closed timestamp was set by req: <unknown; not leaseholder or not lease request> under lease: ; applied at LAI: 0. Batch idx: 0.
| | This assertion will fire again on restart; to ignore run with env var COCKROACH_RAFT_CLOSEDTS_ASSERTIONS_ENABLED=false
| | Raft log tail:
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) log.Fatal: raft closed timestamp regression in cmd: 6bce641e86002f53 (term: 8, index: 3731); batch state: 1684521806.564273000,0, command: 1684492223.646065300,0, lease: ...

Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/358e0d87912365b8976c55ab9b3292e999cf720d/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L646-L648 in pkg/kv/kvserver.(*Store).processReady
https://github.com/cockroachdb/cockroach/blob/358e0d87912365b8976c55ab9b3292e999cf720d/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L391-L393 in pkg/kv/kvserver.(*raftSchedulerShard).worker
https://github.com/cockroachdb/cockroach/blob/358e0d87912365b8976c55ab9b3292e999cf720d/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L298-L300 in pkg/kv/kvserver.(*raftScheduler).Start.func2

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 358e0d8

sp.UpdateGoroutineIDToCurrent()
f(ctx)
}()
in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 647
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 392
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 299
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
Tag Value
Cockroach Release v23.1.0
Cockroach SHA: 358e0d8
Platform linux amd64
Distribution CCL
Environment v23.1.0
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-28115
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels May 19, 2023
@yuzefovich
Copy link
Member

dup of #90682

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity