Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: clearrange/checks=true/rangeTs=false failed #104011

Closed
cockroach-teamcity opened this issue May 28, 2023 · 2 comments
Closed

roachtest: clearrange/checks=true/rangeTs=false failed #104011

cockroach-teamcity opened this issue May 28, 2023 · 2 comments
Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. X-duplicate Closed as a duplicate of another issue.
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 28, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.clearrange/checks=true/rangeTs=false failed with artifacts on release-22.2 @ afe2d46600257f6c022ce89168bebdc0023c8215:

test artifacts and logs in: /artifacts/clearrange/checks=true/rangeTs=false/run_1
(cluster.go:1980).Run: output in run_092527.798282916_n1-10_cockroach-workload-r: ./cockroach workload run kv --concurrency=32 --duration=1h returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 10: dead (exit status 7)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-28308
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 28, 2023
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone May 28, 2023
@blathers-crl blathers-crl bot added the T-storage Storage Team label May 28, 2023
@jbowens
Copy link
Collaborator

jbowens commented May 30, 2023

Another consistency-checker SysBytes divergence, this one on 22.2.
teamcity-10281171-1685251195-34-n10cpu16-0010> F230528 09:46:15.083795 5179860 kv/kvserver/replica_consistency.go:274 ⋮ [n10,merge,s10,r1372/1:‹×›] 1109 found a delta of {ContainsEstimates:0 LastUpdateNanos:1685267155807924024 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:10 SysCount:0 AbortSpanBytes:0} stack trace: goroutine 5179860 [running]:

Related to #104078.

@jbowens jbowens added A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels May 30, 2023
@blathers-crl
Copy link

blathers-crl bot commented May 30, 2023

cc @cockroachdb/replication

@jbowens jbowens removed the T-storage Storage Team label May 30, 2023
@erikgrinaker erikgrinaker added X-duplicate Closed as a duplicate of another issue. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 31, 2023
jbowens added a commit to jbowens/cockroach that referenced this issue May 31, 2023
@jbowens
roachtest: partially disable clearrange checks=true configuration
c3fe09b 
Previously, the checks=true configuration of the clearrange roachtest would
fatal on stats divergences. Due to cockroachdb#93896, this test can fatal with a SysBytes
divergence. This is fixed on master, but the fix will not be backported to 23.1
or 22.2. Disable the enforcement of consistent stats on this branch.

Fixes cockroachdb#104078.
Informs cockroachdb#104011.
Epic: none
Release note: none
Release justification: non-production code changes
@jbowens jbowens mentioned this issue May 31, 2023
Merged
jbowens added a commit to jbowens/cockroach that referenced this issue May 31, 2023
@jbowens
roachtest: partially disable clearrange checks=true configuration
92ad2dc 
Previously, the checks=true configuration of the clearrange roachtest would
fatal on stats divergences. Due to cockroachdb#93896, this test can fatal with a SysBytes
divergence. This is fixed on master, but the fix will not be backported to 23.1
or 22.2. Disable the enforcement of consistent stats on this branch.

Fixes cockroachdb#104011.
Epic: none
Release note: none
Release justification: non-production code changes
@jbowens jbowens mentioned this issue May 31, 2023
Merged
@jbowens jbowens closed this as completed May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. X-duplicate Closed as a duplicate of another issue.
Projects
None yet
Milestone
22.2
Development

No branches or pull requests
3 participants
@erikgrinaker @jbowens @cockroach-teamcity