Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] #140044

cockroach-sentry · 2025-01-29T17:13:32Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6256445744/?referrer=webhooks_plugin

Panic Message:

datum.go:1216: expected *DString, found tree.dNull
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.MustBeDString
  | 	pkg/sql/sem/tree/datum.go:1216
  | github.com/cockroachdb/cockroach/pkg/server.(*adminServer).dataDistributionHelper
  | 	pkg/server/admin.go:3057
  | github.com/cockroachdb/cockroach/pkg/server.(*adminServer).DataDistribution
  | 	pkg/server/admin.go:2928
  | github.com/cockroachdb/cockroach/pkg/server/serverpb._Admin_DataDistribution_Handler.func1
  | 	bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go:6281
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:104
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	external/org_golang_google_grpc/server.go:1177
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	pkg/rpc/context.go:165
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	external/org_golang_google_grpc/server.go:1177
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	pkg/rpc/auth.go:90
  | google.golang.org/grpc.getChainUnaryHandler.func1
  | 	external/org_golang_google_grpc/server.go:1177
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	pkg/rpc/context.go:132
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	pkg/util/stop/stopper.go:349
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	pkg/rpc/context.go:130
  | google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
  | 	external/org_golang_google_grpc/server.go:1168
  | github.com/cockroachdb/cockroach/pkg/server/serverpb._Admin_DataDistribution_Handler
  | 	bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go:6283
  | google.golang.org/grpc.(*Server).processUnaryRPC
  | 	external/org_golang_google_grpc/server.go:1358
  | google.golang.org/grpc.(*Server).handleStream
  | 	external/org_golang_google_grpc/server.go:1735
  | google.golang.org/grpc.(*Server).serveStreams.func1.1
  | 	external/org_golang_google_grpc/server.go:970
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1695
Wraps: (4) expected *DString, found tree.dNull
Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: expected *DString, found tree.dNull
datum.go:1216: *withstack.withStack (top exception)
*assert.withAssertionFailure
panic.go:770: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1735
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1358
bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go in pkg/server/serverpb._Admin_DataDistribution_Handler at line 6283
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1 at line 1168
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 90
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 104
bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go in pkg/server/serverpb._Admin_DataDistribution_Handler.func1 at line 6281
pkg/server/admin.go in pkg/server.(*adminServer).DataDistribution at line 2928
pkg/server/admin.go in pkg/server.(*adminServer).dataDistributionHelper at line 3057
pkg/sql/sem/tree/datum.go in pkg/sql/sem/tree.MustBeDString at line 1216
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1735
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processUnaryRPC at line 1358
bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go in pkg/server/serverpb._Admin_DataDistribution_Handler at line 6283
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1 at line 1168
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 130
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 349
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 132
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 90
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 165
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainUnaryHandler.func1 at line 1177
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 104
bazel-out/k8-opt/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go in pkg/server/serverpb._Admin_DataDistribution_Handler.func1 at line 6281
pkg/server/admin.go in pkg/server.(*adminServer).DataDistribution at line 2928
pkg/server/admin.go in pkg/server.(*adminServer).dataDistributionHelper at line 3057
pkg/sql/sem/tree/datum.go in pkg/sql/sem/tree.MustBeDString at line 1216

Tags

Tag	Value
Command	server
Environment	v25.1.0-beta.2
Go Version	go1.22.8 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v25.1.0-beta.2
Cockroach SHA	`b13cf51`
# of CPUs	16
# of Goroutines	19952

Jira issue: CRDB-46973

The text was updated successfully, but these errors were encountered:

blathers-crl · 2025-01-29T17:13:37Z

CC'ing via the CODEOWNERS-based sentry heuristic:

@cockroachdb/sql-foundations

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

xinhaoz · 2025-01-29T19:35:35Z

This occured when attempting to read raw_config_sql from crdb_internal.zones which can be null:

cockroach/pkg/sql/crdb_internal.go

Lines 4958 to 4959 in ef17976

    
             raw_config_sql       STRING, -- this column can be NULL if there is no specifier syntax 
        
                                          -- possible (e.g. the object was deleted).

cockroach/pkg/server/admin.go

Lines 3046 to 3057 in bf897af

    
           	zoneConfigsQuery) 
        
           if err != nil { 
        
           	return nil, err 
        
           } 
        
           // We have to make sure to close the iterator since we might return from the 
        
           // for loop early (before Next() returns false). 
        
           defer func(it isql.Rows) { retErr = errors.CombineErrors(retErr, it.Close()) }(it) 
        
           for hasNext, err = it.Next(ctx); hasNext; hasNext, err = it.Next(ctx) { 
        
           	row := it.Cur() 
        
           	target := string(tree.MustBeDString(row[0])) 
        
           	zcSQL := tree.MustBeDString(row[1])

xinhaoz · 2025-01-29T19:37:36Z

Seeing as how this started appearing in 25.1 I'll look into if there were any recent zone config changes and if this is an expected state or not.

xinhaoz · 2025-01-29T19:55:52Z

I think it could be related to changes in #137584 where we changed the raw_config_sql to be NULL instead of the empty string or an invalid sql string in more cases. I'll look into the cluster state a bit more but I think it's valid to just update the admin api and convert this MustBeDString check since we were probably just returning empty string in those cases before.

In the data distribution handler we were attempting to read a `raw_sql_config` on `crd_internal.zones` using `MustBeDString` which panics if the value is null. This column is nullable so we should not make this assertion. Fixes: cockroachdb#140044 Release note (bug fix): Data distribution page in advanced debug will no longer crash if there are null values for `raw_sql_config` in `crdb_internal.zons`.

In the data distribution handler we were attempting to read a `raw_sql_config` on `crd_internal.zones` using `MustBeDString` which panics if the value is null. This column is nullable. This commit addresses this by adding a filter to the query so that we only get results with non-null values for the column. Fixes: cockroachdb#140044 Release note (bug fix): Data distribution page in advanced debug will no longer crash if there are null values for `raw_sql_config` in `crdb_internal.zons`.

In the data distribution handler we were attempting to read a `raw_sql_config` on `crdb_internal.zones` using `MustBeDString` which panics if the value is null. This column is nullable. We now allow null values to be read and make the response value an empty string in that case. Fixes: cockroachdb#140044 Release note (bug fix): Data distribution page in advanced debug will no longer crash if there are null values for `raw_sql_config` in `crdb_internal.zones`.

cockroach-sentry added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Jan 29, 2025

blathers-crl bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner labels Jan 29, 2025

rafiss changed the title ~~Sentry: datum.go:1216: expected *DString, found tree.dNull(1) [in (*adminServer).DataDistribution]~~ Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] Jan 29, 2025

rafiss added T-observability and removed T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jan 29, 2025

dhartunian added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. GA-blocker branch-release-25.1 labels Jan 29, 2025

dhartunian assigned xinhaoz Jan 29, 2025

xinhaoz linked a pull request Jan 29, 2025 that will close this issue

server: do not use MustBeDString on nullable result col #140066

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] #140044

Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] #140044

cockroach-sentry commented Jan 29, 2025 •

edited by cockroach-jira-scripts

Loading

blathers-crl bot commented Jan 29, 2025

xinhaoz commented Jan 29, 2025

xinhaoz commented Jan 29, 2025

xinhaoz commented Jan 29, 2025 •

edited

Loading

Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] #140044

Sentry: expected DString, found tree.dNull [in (*adminServer).DataDistribution] #140044

Comments

cockroach-sentry commented Jan 29, 2025 • edited by cockroach-jira-scripts Loading

Tags

blathers-crl bot commented Jan 29, 2025

xinhaoz commented Jan 29, 2025

xinhaoz commented Jan 29, 2025

xinhaoz commented Jan 29, 2025 • edited Loading

cockroach-sentry commented Jan 29, 2025 •

edited by cockroach-jira-scripts

Loading

xinhaoz commented Jan 29, 2025 •

edited

Loading