Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiregion: v23.1.6: duplicate region × found in super region × #108556

Closed
cockroach-sentry opened this issue Aug 10, 2023 · 2 comments · Fixed by #110083
Closed

multiregion: v23.1.6: duplicate region × found in super region × #108556

cockroach-sentry opened this issue Aug 10, 2023 · 2 comments · Fixed by #110083
Assignees
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Aug 10, 2023

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4386539462/?referrer=webhooks_plugin

Panic Message:

region_config.go:464: duplicate region × found in super region ×
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion.ValidateSuperRegions
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion/region_config.go:464
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion.ValidateRegionConfig
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion/region_config.go:391
  | github.com/cockroachdb/cockroach/pkg/sql.SynthesizeRegionConfig
  | 	github.com/cockroachdb/cockroach/pkg/sql/region_util.go:1455
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).refreshZoneConfigsForTables
  | 	github.com/cockroachdb/cockroach/pkg/sql/region_util.go:1022
  | github.com/cockroachdb/cockroach/pkg/sql.(*planner).addSuperRegion
  | 	github.com/cockroachdb/cockroach/pkg/sql/alter_database.go:1908
  | github.com/cockroachdb/cockroach/pkg/sql.(*alterDatabaseAddSuperRegion).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/alter_database.go:1601
  | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:519
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:299
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  | github.com/cockroachdb/cockroach/pkg/sql.startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:522
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:174
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:183
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:94
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:103
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:244
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:278
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:325
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1913
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1645
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1648
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2103
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1656
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:964
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2986
  | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141
Wraps: (4) duplicate region × found in super region ×
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: duplicate region × found in super region ×
region_config.go:464: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError
Stacktrace (expand for inline code snippets):

}
err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)

} else {
err = op(ctx)
}

err = ex.execWithProfiling(ctx, ast, preparedStmt, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, portal, pinfo, res, canAutoCommit)
return err

if err = ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

}
err = ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
}

)
}()

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}

if err := f.init(ctx); err != nil {
f.pushError(err)

func (f *BatchFlowCoordinator) init(ctx context.Context) error {
return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)

}()
operation()
return retErr

return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
})

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.init)
bic.stopwatch.Stop()

}()
operation()
return retErr

func (bic *batchInfoCollector) init() {
bic.Input.Init(bic.ctx)
}

ctx = c.StartInternal(ctx, "columnarizer" /* name */)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)

cockroach/pkg/sql/plan.go

Lines 521 to 523 in 3bacdbe

}
return walkPlan(params.ctx, plan, o)
}

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err

}
v.visitInternal(plan, name)
return plan

cockroach/pkg/sql/walk.go

Lines 298 to 300 in 3bacdbe

}
}

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 3bacdbe

}
v.err = v.observer.leaveNode(name, plan)
}()

cockroach/pkg/sql/plan.go

Lines 518 to 520 in 3bacdbe

}
return n.startExec(params)
},

}
return params.p.addSuperRegion(params.ctx, n.desc, typeDesc, n.n.Regions, n.n.SuperRegionName, tree.AsStringWithFQNames(n.n, params.Ann()))
}

// Update all regional and regional by row tables.
return p.refreshZoneConfigsForTables(
ctx,

regionConfig, err := SynthesizeRegionConfig(ctx, p.txn, desc.GetID(), p.Descriptors())
if err != nil {

if err := multiregion.ValidateRegionConfig(regionConfig); err != nil {
return multiregion.RegionConfig{}, err

var err error
ValidateSuperRegions(config.SuperRegions(), config.SurvivalGoal(), config.Regions(), func(validateErr error) {
if err == nil {

if found {
err := errors.AssertionFailedf("duplicate region %s found in super region %s", region, superRegion.SuperRegionName)
errorHandler(err)

pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 141
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2986
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 142
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 964
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1656
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 2103
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1648
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1645
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1913
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 325
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 278
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init at line 244
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init.func1 at line 245
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 103
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).init at line 94
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 183
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 174
pkg/sql/plan.go in pkg/sql.startExec at line 522
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 299
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 519
pkg/sql/alter_database.go in pkg/sql.(*alterDatabaseAddSuperRegion).startExec at line 1601
pkg/sql/alter_database.go in pkg/sql.(*planner).addSuperRegion at line 1908
pkg/sql/region_util.go in pkg/sql.(*planner).refreshZoneConfigsForTables at line 1022
pkg/sql/region_util.go in pkg/sql.SynthesizeRegionConfig at line 1455
pkg/sql/catalog/multiregion/region_config.go in pkg/sql/catalog/multiregion.ValidateRegionConfig at line 391
pkg/sql/catalog/multiregion/region_config.go in pkg/sql/catalog/multiregion.ValidateSuperRegions at line 464

Tags

Tag Value
Command demo
Environment v23.1.6
Go Version go1.19.10
Platform darwin arm64
Distribution CCL
Cockroach Release v23.1.6
Cockroach SHA 3bacdbe
# of CPUs 10
# of Goroutines 3376

Jira issue: CRDB-30518

@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Aug 10, 2023
@yuzefovich yuzefovich changed the title Sentry: region_config.go:464: duplicate region × found in super region × (1) Wraps: (2) assertion failure Wraps: (3) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/s... multiregion: v23.1.6: duplicate region × found in super region × Aug 25, 2023
@blathers-crl blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Aug 25, 2023
@rafiss
Copy link
Collaborator

rafiss commented Aug 29, 2023

Simple repro:

./cockroach demo --global --nodes=9

> ALTER DATABASE defaultdb set PRIMARY REGION 'us-east1';
> alter database defaultdb add super region 's' values 'us-east1', 'europe-west1', 'europe-west1';

*
* ERROR: Queued as error df939b19f2ae4392a091bf1b53a3f8f0
*
ERROR: internal error: duplicate region europe-west1 found in super region s
SQLSTATE: XX000
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion/region_config.go:464: ValidateSuperRegions()
github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion/region_config.go:391: ValidateRegionConfig()
github.com/cockroachdb/cockroach/pkg/sql/region_util.go:1455: SynthesizeRegionConfig()
github.com/cockroachdb/cockroach/pkg/sql/region_util.go:1022: refreshZoneConfigsForTables()
github.com/cockroachdb/cockroach/pkg/sql/alter_database.go:1908: addSuperRegion()
github.com/cockroachdb/cockroach/pkg/sql/alter_database.go:1601: startExec()
github.com/cockroachdb/cockroach/pkg/sql/plan.go:519: func2()
github.com/cockroachdb/cockroach/pkg/sql/walk.go:112: func1()
github.com/cockroachdb/cockroach/pkg/sql/walk.go:299: visitInternal()
github.com/cockroachdb/cockroach/pkg/sql/walk.go:79: visit()
github.com/cockroachdb/cockroach/pkg/sql/walk.go:43: walkPlan()
github.com/cockroachdb/cockroach/pkg/sql/plan.go:522: startExec()
github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:174: Start()
github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:183: Init()
github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:94: init()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92: CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:103: Init()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245: func1()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92: CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:244: init()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:278: Run()
github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:325: Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902: Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1913: PlanAndRun()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1645: func3()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1648: PlanAndRunAll()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2103: execWithDistSQLEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1656: dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:964: execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:142: func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2986: execWithProfiling()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:141: execStmt()

fqazi added a commit to fqazi/cockroach that referenced this issue Sep 6, 2023
Previously, when duplicate regions would exist within
a super region an internal assertion would be generated.
This was inadequate since an assertion failure would
lead to a stack dump. To address this, this patch
will generate a proper pgerror.

Fixes: cockroachdb#108556

Release note: None
craig bot pushed a commit that referenced this issue Sep 6, 2023
110083: sql: return pgerror for duplicate regions in super region r=fqazi a=fqazi

Previously, when duplicate regions would exist within a super region an internal assertion would be generated. This was inadequate since an assertion failure would lead to a stack dump. To address this, this patch
will generate a proper pgerror.

Fixes: #108556

Release note: None

Co-authored-by: Faizan Qazi <[email protected]>
@craig craig bot closed this as completed in 0ebc0ce Sep 6, 2023
blathers-crl bot pushed a commit that referenced this issue Sep 6, 2023
Previously, when duplicate regions would exist within
a super region an internal assertion would be generated.
This was inadequate since an assertion failure would
lead to a stack dump. To address this, this patch
will generate a proper pgerror.

Fixes: #108556

Release note: None
@exalate-issue-sync exalate-issue-sync bot reopened this Feb 12, 2024
@fqazi
Copy link
Collaborator

fqazi commented May 1, 2024

Merged via: #110083

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants