pkg/upgrade/upgrademanager/upgrademanager_test: TestPrecondition failed

[cockroach-teamcity]

pkg/upgrade/upgrademanager/upgrademanager_test.TestPrecondition failed on release-24.1 @ a4743bb42931fb7b9652f4f01bb3a93d7c8ff018:

=== RUN   TestPrecondition
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestPrecondition1542160416
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #107397 (expected label: C-bug)

pkg/upgrade/upgrademanager/upgrademanager_test_test/pkg/upgrade/upgrademanager/manager_external_test.go:758: (TestPrecondition.func7)
	NOTICE: .ClusterSettings() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().ClusterSettings() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization

pkg/upgrade/upgrademanager/upgrademanager_test_test/pkg/upgrade/upgrademanager/manager_external_test.go:758: (TestPrecondition.func7)
	NOTICE: .ClusterSettings() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().ClusterSettings() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization

pkg/upgrade/upgrademanager/upgrademanager_test_test/pkg/upgrade/upgrademanager/manager_external_test.go:758: (TestPrecondition.func7)
	NOTICE: .ClusterSettings() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().ClusterSettings() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization

Parameters:

• attempt=1
• race=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/release-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-46718

[srosenberg]

Cluster is up around 11:58,

I250122 11:58:57.265332 65228 upgrade/upgrademanager/manager.go:167 ⋮ [T1,Vsystem,n1] 58  running permanent upgrades up to version: 23.1

The attempted v1 tenant upgrade doesn't happen until 12:09,

I250122 12:09:45.269976 129481 upgrade/upgrademanager/manager.go:760 ⋮ [T1,Vsystem,n1,client=127.0.0.1:53888,hostssl,user=root,migration-mgr] 2065  running Upgrade to 23.1-upgrading-to-23.2-step-002: "v1"

The injected precondition error is 5 minutes later,

E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677  MIGRATION job 1040309508549476353: stepping through state reverting with unexpected error: running migration for 23.1-upgrading-to-23.2-step-002: boom
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +(1) attached stack trace
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  -- stack trace:
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/upgrade/upgradejob.resumer.Resume
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/upgrade/upgradejob/upgrade_job.go:125
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | [...repeated from below...]
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +Wraps: (2) running migration for 23.1-upgrading-to-23.2-step-002
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +Wraps: (3) forced error mark
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | ‹"permanent job error"›
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +Wraps: (4) attached stack trace
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  -- stack trace:
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | pkg/upgrade/upgrademanager/upgrademanager_test_test.TestPrecondition.func5.TestPrecondition.func3.2
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        pkg/upgrade/upgrademanager/upgrademanager_test_test/pkg/upgrade/upgrademanager/manager_external_test.go:693
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/upgrade.(*TenantUpgrade).Run
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/upgrade/tenant_upgrade.go:106
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/upgrade/upgradejob.resumer.Resume
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/upgrade/upgradejob/upgrade_job.go:120
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1593
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1594
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:451
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:285
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  | runtime.goexit
E250122 12:14:06.222970 145909 jobs/registry.go:1551 ⋮ [T1,Vsystem,n1] 2677 +  |        src/runtime/asm_amd64.s:1695

Eventually the test times out... lots of other indicators in the log that this is likely a flake due to resource contention.

[blathers-crl]

Based on the specified backports for linked PR #140024, I applied the following new label(s) to this issue: branch-release-23.1, branch-release-23.2, branch-release-24.2, branch-release-24.3, branch-release-25.1. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}