Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [mismatched type in violatesFkConstraintsHelper query] #119288

Closed
cockroach-teamcity opened this issue Feb 16, 2024 · 27 comments · Fixed by #119653
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-noreuse Prevent automatic commenting from CI test failures
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 16, 2024

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 617bf347978dcc0d711399b1a76402d7f88de958:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['6383303776005880229306.14565726'::DECIMAL = '1999-08-20 22:33:07.000795':::TIMESTAMP]) AND count(*) = 0 from schema_w1_11.table_w0_31\n\t    WHERE cȍl31_w0_36 = ('1999-08-20 22:33:07.000795':::TIMESTAMP)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <decimal> = <timestamp> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 113
Total Schema Statements Succeeded = 113
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 67
Total Transactions Rolled Back = 56
Total Transactions Executed = 123
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_8_of_30/test.outputs/logTestWorkload675587457
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload675587457
--- FAIL: TestWorkload (22.44s)

Parameters:

  • attempt=1
  • run=8
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-36128

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Feb 16, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Feb 16, 2024
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY [(-2.96182736880654676E+27):::DECIMAL = e'BW~jp\\'\\x0e\\x11R'::NAME:::NAME]) AND count(*) = 0 from public.table_w1_1\n\t    WHERE \"co\\\\xbel1_w1_9\" = (e'BW~jp\\'\\x0e\\x11R'::NAME:::NAME)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: false = ANY ARRAY[(-2.96182736880654676E+27):::DECIMAL = e'BW~jp\'\x0e\x11R'::NAME:::NAME]: unsupported comparison operator: <decimal> = <name> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 12
Total Schema Statements Succeeded = 12
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 6
Total Transactions Rolled Back = 9
Total Transactions Executed = 15
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload783464542
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload783464542
--- FAIL: TestWorkload (23.76s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss changed the title pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [mismatched type in violatesFkConstraintsHelper query] Feb 16, 2024
@rafiss rafiss added P-1 Issues/test failures with a fix SLA of 1 month and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 16, 2024
@annrpom annrpom self-assigned this Feb 16, 2024
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 1340a1e0419ce5df9ae05f5a362a8bbedbcc0a07:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['-0.7179127791224806'::FLOAT8 = 'yjvau'::VARCHAR]) AND count(*) = 0 from schema_w1_42.table_w0_32\n\t    WHERE col32_w0_44 = ('yjvau'::VARCHAR)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <float> = <varchar> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 88
Total Schema Statements Succeeded = 88
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 41
Total Transactions Rolled Back = 57
Total Transactions Executed = 98
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload282689912
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload282689912
--- FAIL: TestWorkload (71.93s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 04f0416d526a43741d22fd03966758dcccdeb79f:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY [ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL],ARRAY[(-7.595952600872596537E+37):::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL],ARRAY[(-62828.30572770897655):::DECIMAL,(-6.588003786479220899E+28):::DECIMAL,1.0:::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL]]) AND count(*) = 0 from schema_w1_41.table_w0_42\n\t    WHERE \"col4*2,_w0_47\" = (ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL])\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <string> = <decimal[]> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 101
Total Schema Statements Succeeded = 101
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 63
Total Transactions Rolled Back = 72
Total Transactions Executed = 135
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3856467008
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3856467008
--- FAIL: TestWorkload (75.74s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@annrpom
Copy link
Contributor

annrpom commented Feb 22, 2024

Here is the table that is leading to this failure in the most recent test run ^, along with the query generated by violatesFkConstriantsHelper that gets us in this fail state:

[email protected]:26257/demoapp/movr> show create schema_w1_41.table_w0_42;                                                                      
         table_name        |                                                                                                create_statement
---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  schema_w1_41.table_w0_42 | CREATE TABLE schema_w1_41.table_w0_42 (
                           |     "c\u000col42_w0_43" DECIMAL[] NOT NULL,
                           |     col42_w0_44 NAME NULL,
                           |     col42_w0_45 BIT(10) NULL,
                           |     "c(ol42_w0_46" STRING NOT NULL,
                           |     "col4*2,_w0_47" STRING NULL AS (lower(col42_w0_44)) STORED,
                           |     "c  ol4%q  2_w0_48" STRING NOT NULL AS (lower(col42_w0_44)) VIRTUAL,
                           |     CONSTRAINT table_w0_42_pkey PRIMARY KEY ("c  ol4%q  2_w0_48" ASC, "c\u000col42_w0_43" ASC, "c(ol42_w0_46" DESC),
                           |     CONSTRAINT "table_w0_42_col4*2,_w0_47_table_w0_42_c(ol42_w0_46_fk" FOREIGN KEY ("c(ol42_w0_46") REFERENCES schema_w1_41.table_w0_42("col4*2,_w0_47") ON DELETE CASCADE ON UPDATE CASCADE,
                           |     CONSTRAINT "table_w0_42_c\u000col42_w0_43_table_w0_42_c\u000col42_w0_43_fk" FOREIGN KEY ("c\u000col42_w0_43") REFERENCES schema_w1_41.table_w0_42("c\u000col42_w0_43") ON DELETE CASCADE ON UPDATE CASCADE,
                           |     UNIQUE INDEX "table_w0_42_col4*2,_w0_47_key" ("col4*2,_w0_47" DESC),
                           |     INDEX "table_w0_42_c  ol4%q  2_w0_48_idx" ("c  ol4%q  2_w0_48" DESC) VISIBILITY 0.03,
                           |     INVERTED INDEX table_w0_42_col42_w0_44_idx (col42_w0_44 gin_trgm_ops),
                           |     INDEX "table_w0_42_c(ol42_w0_46_col42_w0_45_c\u000col42_w0_43_c  ol4%q  2_w0_48_idx" ("c(ol42_w0_46" ASC, col42_w0_45 ASC, "c\u000col42_w0_43" ASC, "c  ol4%q  2_w0_48" DESC) VISIBILITY 0.33,
                           |     UNIQUE INDEX "table_w0_42_c\u000col42_w0_43_key" ("c\u000col42_w0_43" DESC) STORING (col42_w0_44, "col4*2,_w0_47"),
                           |     INDEX "table_w0_42_col4*2,_w0_47_c(ol42_w0_46_idx" ("col4*2,_w0_47" ASC, "c(ol42_w0_46" ASC),
                           |     INDEX "table_w0_42_col4*2,_w0_47_c\u000col42_w0_43_idx" ("col4*2,_w0_47" ASC, "c\u000col42_w0_43" ASC) STORING (col42_w0_44, col42_w0_45),
                           |     FAMILY "fam_0_c\u000col42_w0_43_col4*2,_w0_47_c(ol42_w0_46" ("c\u000col42_w0_43", "col4*2,_w0_47", "c(ol42_w0_46"),
                           |     FAMILY fam_1_col42_w0_45 (col42_w0_45),
                           |     FAMILY fam_2_col42_w0_44 (col42_w0_44)
                           | )
(1 row)


SELECT false = ANY (ARRAY [ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL],ARRAY[(-7.595952600872596537E+37):::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL],ARRAY[(-62828.30572770897655):::DECIMAL,(-6.588003786479220899E+28):::DECIMAL,1.0:::DECIMAL] = ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL]]) 
AND count(*) = 0 from schema_w1_41.table_w0_42
WHERE "col4*2,_w0_47" = (ARRAY[(-86.45467643078375079):::DECIMAL,5.663854075949911563E+22:::DECIMAL,(-4.473327843027759606E+24):::DECIMAL,(-58650.36833044207356):::DECIMAL]);

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 24414b864f65c6544ea987f7970fa1c192ca3310:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['''hITBFxwm''':::TSQUERY = e'\\bP\\x01Y\\x10\\x13)\\x07j'::VARCHAR,'''aZA'' <-> ''K'' & ''VqjCt''':::TSQUERY = e'\\bP\\x01Y\\x10\\x13)\\x07j'::VARCHAR]) AND count(*) = 0 from schema_w1_27.table_w0_60\n\t    WHERE \"\\\\u76FEcol60_w0_65\" = (e'\\bP\\x01Y\\x10\\x13)\\x07j'::VARCHAR)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: false = ANY ARRAY[e'\'hITBFxwm\'':::TSQUERY = e'\bP\x01Y\x10\x13)\x07j'::VARCHAR, e'\'aZA\' <-> \'K\' & \'VqjCt\'':::TSQUERY = e'\bP\x01Y\x10\x13)\x07j'::VARCHAR]: unsupported comparison operator: <tsquery> = <varchar> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 93
Total Schema Statements Succeeded = 93
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 48
Total Transactions Rolled Back = 53
Total Transactions Executed = 101
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2441337405
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2441337405
--- FAIL: TestWorkload (68.59s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

    "result": true
   },
   {
    "query": "SELECT ARRAY[parent.table_schema, parent.table_name, parent.column_name, child.column_name] FROM (SELECT conname, conkey, confkey, conrelid, confrelid FROM pg_constraint WHERE (contype = 'f') AND (conrelid = 'schema_w0_21.table_w0_49'::REGCLASS::INT8)) AS con JOIN (SELECT constraint_name FROM information_schema.table_constraints WHERE ((table_schema = 'schema_w0_21') AND (table_name = 'table_w0_49')) AND (crdb_internal.is_constraint_active('schema_w0_21.table_w0_49', constraint_name) = true)) AS tc ON conname = tc.constraint_name JOIN (SELECT column_name, ordinal_position, column_default FROM information_schema.columns WHERE (table_schema = 'schema_w0_21') AND (table_name = 'table_w0_49')) AS child ON conkey[1] = child.ordinal_position JOIN (SELECT pc.oid, cols.table_schema, cols.table_name, cols.column_name, cols.ordinal_position FROM pg_class AS pc JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid JOIN information_schema.columns AS cols ON ((pc.relname = cols.table_name) AND (pn.nspname = cols.table_schema))) AS parent ON ((con.confkey[1] = parent.ordinal_position) AND (con.confrelid = parent.oid)) WHERE child.column_name != 'rowid'",
    "result": null
   }
  ],
  "previousStatements": [
   "CREATE SCHEMA IF NOT EXISTS schema_w1_46",
   "INSERT INTO schema_w0_21.table_w0_49 (col49_w0_50,\" co%pl49_w0_51\",\"c%vol49 _w0_52\",\"cOl49_w0_53\",\"co\u000cl49_w0_54\",\"co l49_w0_55\",col49_w0_56,col49_w0_57,\"col\t49_w0_58\",col49_w0_59) VALUES ('d9733985-fb3d-45c7-a2f2-c7e20602273a':::UUID,'1995-05-18':::DATE,(-8):::INT8,'01060000E0E61000000800000001030000C0010000000D0000006A76BF5D6DCC63C080DBB802E81439C01441EC0DAEFCE4C10A638A4E160AFBC1F5E6B73BE94363C091279F1CE81C44C00269295FE527F6413CC8503A6108E4C1B73F39CE6CEB50C05B5241DE263755C0E84C783716B3E741FEF080E3EE4DEAC1EA175CD1085B5240126F8D01F2F854C040E87F0B0917B541A400D3E8047AF5C1DAD3C277665A6040CE7F161B23DB50C0F4F0457ECF94FEC1F64EA487504AFA41E8C311751555664000E8353617C3DABF2CEAAFF8348AE1C1BF74E572EDCDF0C1F80682D689B85140EC89974F24693440E6A142869530E6C17848968A290CF141D4EE4182A8F35E403C1C70C40B4044407CF44F9CD7B2E941E4D1A62D2C78E541D03709E7681D20C06C60FE7763F53B402E41E4E693C7F0419422ED5C6BFCF241F8880D3F9FAC51C008442868278D4440D4D7B464B830F541D0D548EC8369DAC1386FF23E346261C08C44B94D69375640C2301700BFE7014218CEE2E23747E1419F8D15E74E2C62C0F45F0E7E9D4D33403BB868B54474FAC1FD094927634AFEC16A76BF5D6DCC63C080DBB802E81439C01441EC0DAEFCE4C10A638A4E160AFBC101030000C0010000000A000000416BCF72200862C080C63B2A1CD51D4070896F757EEBCDC15ACD6D2EE4E4F54162A3DF20D48A60C0804413911432104058821E9E7222FE4120FC1EFF3D62D24159E3337911DD55C05062BE0BF8AB1AC0B89DF0D82AEFF0C1A2311E88ED60F9C105151F052CE45AC0DAFA36CB59F655C0B8279C78884A0142B88A0EF4195FFDC180D41F22F56E37407CD2D2BE99DA31403CCFF50128F5FA41208201BAB153E941EA5FC56FD68D61406A40643535A541402EB4A3C9BD51E5C1ECC7382853BDF8C156CB68C99487634030A2B8E1B1A8534028F4E59A0705E8C166D54F56058D01428079CF29C03CF03FCAC33B4FBC325240807272C6AA1493C1BAD74FAB014FECC182CF2012D8B35FC076CA115ED04753409094242E1952C4C1707D4B4146320042416BCF72200862C080C63B2A1CD51D4070896F757EEBCDC15ACD6D2EE4E4F54101030000C0010000000A0000003523A6F348B061C0C08B07A4F5CF28409068223B4D8BDF417264A5B49D7501C2675CE5F49B0866C0B89B87C63A0D27C09090F679CE61DFC1B010DDCCABA4ED4100FE2BD0455B48C0C829645E2E8236C054CD696150F1DAC122A86AA3190CF141486C92613D6F3AC08E4141ADFE874FC0709946B78256F541CC44F6413B2FFE41AC29F42C3EC96040D82F14A7B5F14540BA132CDEFFB2FBC1082A55587DB6E1C1862A607D7E0E4BC038B83A3EE901354000A3A26ABE39B3417CD37CADE0C6E041ECCF43E1315A43C0C086F5293F034D40A02AE9061725B841F05EFDC48B4BE9C1A3A0B9CD265961C0C61FB72EE51D5040F841F099D635D341BCE4374E1EB6FC415F879B5A54FE55C0E82510784DB33A4080FB7293665792C1369653FCE51502423523A6F348B061C0C08B07A4F5CF28409068223B4D8BDF417264A5B49D7501C201030000C0010000000B000000D8201AFD42AF5AC01C6CF58860AC45C0F4D7EAE6CECAEBC17CBF25D11D9AF8C18EBA98A9487F62C04233499C9EF94EC062DCD790E79AF8C162AE67751701F4C1C2619DA765DF4EC06633427ACA074CC04EE3BB4772A400C2F813BF54040FDA417450864A7BFE454020C547FE9F6A53C0A108A2B6ED0701C28E9F804A2B19F741E8A39F51AD4F6240C4973142A40433C0F0D1719DE6B2D7C1A09EBB839A90C341EAF434618C94604038382F6DF58D31404965D311D4E7F9C1F6AF46749CFDE6C1845291242E724C40A097BB440D7917404EB3903FC35D01422F56D018DF7DFFC1584145F36C0C5440E4161D9254BF46403A61948D0829EBC1C2ADCAAB259FF1C11061D56B572533409E5826CEEB2B514081F596EC9A9300C2D4ED5F29679BF64186C92E402C4D60C0A2AA3382367F534000C7FBA54755F741E0FB98D8B412EF41D8201AFD42AF5AC01C6CF58860AC45C0F4D7EAE6CECAEBC17CBF25D11D9AF8C101030000C001000000060000001C6793B9DE8461C00A5598838A3752C0AC1209478D98F9C1C0804A573532ADC10082F527FAB6F43F304FA195132551C04D568CD0B55A02C2D4069804ED00F841B05A9A67FC1B6340553B3719A9C153C0BAB85E2B5A19004280DEBA6878B4C4C144917B1E1CC75B400085243A43AA02C05CE2241E9E050042202A232847E0F7C1147EDB0BF9DC47401066AAB3B75A4B40750663ABFF1EFEC130FD2A840D59DCC11C6793B9DE8461C00A5598838A3752C0AC1209478D98F9C1C0804A573532ADC101030000C001000000050000007375F4534CA054C01468D070732F3EC0F46DB759F4EFF8C1A0620799A5ABF4418C5DEF9B9ECD51C0DA99C44A930041C060FF9049D19FF641A67774BEFB7DE1C1E027B716C92B24C00CF2C7AA528538C0828456E16B61EBC1947D4343B14BF641305E4DF0498341C0E07A54A5572C23C06CBA24A7F655E3415803B0DEA4E0E3C17375F4534CA054C01468D070732F3EC0F46DB759F4EFF8C1A0620799A5ABF44101030000C0010000000B00000087B0A92FF40260C0748EE358E1443FC0E055FA5E7E4BFDC13A075E9AB87FF341008C14206A6261405ABA06C7DAD44EC00EDF8AF4E06AFAC17EE546DFFA6D02422CA6D97DF66E5B40386C24A5C1772CC0F89768803BEDE141503D6F8C9CD1C8C104196A97E01E65408499CE2D45893A4080ED6BE5FC5CF6C1CD070A65862DF2C118B3D9B899E63740B04EFE9A7AD844405F3217BDAF1BFCC152910AFA773CF44120C6D982B9202540B0C9CD5AFBE836404244FE0DB862F94130BE8DD7D0DFC5C17031D738A67E2A4012DA4101BC63514050920D1D5E3EEF410465B5EAB00501C200686F5B35A4B9BF38B85232B4F54D40801F2CBCA917C44112D6B0A0D50FFDC1651A2D8D5CFC5BC00C506C966B874740DA49F33111A6F5C169854149767100C20A5B39DFEB9660C0F8A6FDE560FC4440BEA476E9FF3DE9C160C744518C77C9C187B0A92FF40260C0748EE358E1443FC0E055FA5E7E4BFDC13A075E9AB87FF34101030000C0010000000A000000E90818BF1F7850C0F388F65AA76047C0302B82BCEA6EFC4140500377C104A7C142C37F95A6ED50C0F3064EEFCFFB51C0C05674BBE7A8F3416C497D09F2F0DDC1083EBF8FA4BB3DC020D364BF148614C025C8C8641421F4C12EF4E88CC2BBF3C1FC72AADB254D4A40270DB8405F2249C08549267828C8FDC1A47591312123DCC1F45C32CFB33E4040804406AADA60F73F7875A371BCD6FDC18CF00951192AE941BC73996E31984040F08C053E1C03344020453778298DF341DC5AAF063FEAFD419A3929416F0B504048C0159ABF325240E84A46BA1220F54100CD43635EFDEE410DE9B817836157C0D6787A567B0E504030AA3B5FA695CF416A958C9C8D52F241A56B563F223B62C0844C9BF5A0615140165FD40B459CE6C1650C1055AC32FFC1E90818BF1F7850C0F388F65AA76047C0302B82BCEA6EFC4140500377C104A7C1':::GEOGRAPHY,'1995-03-08 21:31:33.000362+00':::TIMESTAMPTZ,'00:00:00':::TIME,'?QIt\u003c'::STRING,e'hx;l\\x0fcK3':::REFCURSOR,(-1.4237642288208008):::FLOAT8,'20 years 1 mon 179 days 19:54:19.50648':::INTERVAL),('ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,'1991-06-06':::DATE,61:::INT8,'01060000E0E61000000200000001030000C00100000007000000AFA0A2F51AED5AC04ADD9A8043B843C0086793785679DAC15D7C214A6A73F8C1E996BF3D045551C0579AC654E74252C094E650FE5443EC4100CFE6D22A28D2411C4743363F5562404833F4499B7C27C070D35A8E68ADC1C130BCC9A9CE9A00C2B41F60B77EE66240E0867BC3088819C040D38CB11389B54194B5B55FC432E1C108C37932EE4E584090802907AF321D40BD2409F2C293FDC1A4688DD4BB52ECC158508EC87A3B6640F48FAA40DA1251409A7777B86686024214D235F22303EC41AFA0A2F51AED5AC04ADD9A8043B843C0086793785679DAC15D7C214A6A73F8C101030000C00100000004000000EB02967C437765C09852DDB5C66D3040F0A2453FA59CEFC133CA1EE6F39301C2B86FA7E83C9F38C058FBF1FBFBBD2740D4E7101AA720EF41CFAF241ACA97F4C1AD219E37F56962C05CE02A24B0DF37408217F4B31B2BE7C1888FA0B67F19FF41EB02967C437765C09852DDB5C66D3040F0A2453FA59CEFC133CA1EE6F39301C2':::GEOGRAPHY,'294276-12-31 23:59:59.999999+00':::TIMESTAMPTZ,'02:20:20.906666':::TIME,e'\\x06K\\x12\\x11p'::STRING,'7':::REFCURSOR,0.2880493402481079:::FLOAT8,'-81 years -8 mons -656 days -11:42:51.689754':::INTERVAL)"
  ]
 }
}
    schema_change_external_test.go:132: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:132
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: integer out of range (SQLSTATE 22003)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3122
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
        	            	Wraps: (5) ERROR: integer out of range (SQLSTATE 22003)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 67
Total Schema Statements Succeeded = 67
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 38
Total Transactions Rolled Back = 54
Total Transactions Executed = 92
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload23076323
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload23076323
--- FAIL: TestWorkload (53.68s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY [1114:::OID = e'~\\x01h\\x1bGN'::STRING]) AND count(*) = 0 from schema_w0_6.table_w0_22\n\t    WHERE \"C o\"\"l\\\\U000318FD22_w0_34\" = (e'~\\x01h\\x1bGN'::STRING)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: false = ANY ARRAY[1114:::OID = e'~\x01h\x1bGN'::STRING]: unsupported comparison operator: <oid> = <string> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 58
Total Schema Statements Succeeded = 58
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 35
Total Transactions Rolled Back = 34
Total Transactions Executed = 69
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_14_of_30/test.outputs/logTestWorkload4128754569
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload4128754569
--- FAIL: TestWorkload (12.26s)

Parameters:

  • attempt=1
  • run=14
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 140ecc0c19c46204bb04a70ea0e513c45a26ad0d:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['12:57:12.869434-13:56':::TIMETZ = '12:57:12.869434-13:56':::TIMETZ]) AND count(*) = 0 from schema_w0_1.table_w0_2\n\t    WHERE \"co%l2_w0_6\" = ('12:57:12.869434-13:56':::TIMETZ)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <string> = <timetz> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 16
Total Schema Statements Succeeded = 16
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 6
Total Transactions Rolled Back = 6
Total Transactions Executed = 12
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2982225966
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2982225966
--- FAIL: TestWorkload (28.36s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@annrpom
Copy link
Contributor

annrpom commented Feb 26, 2024

ah - i see what is going on. in the violatesFkConstraintsHelper, we get the parent value that happens in the same insert by going and looking it up in the other rows - which requires us to find the index by referring to our columnNameToIndexMap map. somehow, this map doesn't contain the proper information of all columns in the table; thus, whenever we try to lookup the parent name by string, we get no results == 0. so we end up comparing the first column's values to the child values instead of the parent values to the child values

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 7bb52a7d1c75d5adfdfa53e5fcff6f5e6497408f:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['010100008087BB2A8BEA14FFC166F3E3D50A33004246E28CFD435502C2':::GEOMETRY = '010100008087BB2A8BEA14FFC166F3E3D50A33004246E28CFD435502C2':::GEOMETRY,'01020000C0050000009B0D80B0B99A02C2F9FC3AC7A3D2FAC1E6423AFD7405FA41C85230F11D340142503D73BD9B5501C259FA06A8B9E000C268FF51A6D35901427C110A67D382FBC168E3278F101EFB41A0BCA9688CEFBC4160473C14A8E1D941D78C583957D6FFC1D0129A06CBD6E341E044B60B3E82B0414A251CF950AFF641401A2514498ED0C1AA382D22D5BDF74140410B3E8A74F341F01536EF0F9EC8C1789941E1B438FD41':::GEOMETRY = '010100008087BB2A8BEA14FFC166F3E3D50A33004246E28CFD435502C2':::GEOMETRY,'0103000000010000000500000000000000000000000000000000000000000000000000F03F0000000000000000000000000000F03F000000000000F03F0000000000000000000000000000F03F00000000000000000000000000000000':::GEOMETRY = '010100008087BB2A8BEA14FFC166F3E3D50A33004246E28CFD435502C2':::GEOMETRY]) AND count(*) = 0 from schema_w0_24.table_w0_40\n\t    WHERE \"col4(0_w0_57\" = ('010100008087BB2A8BEA14FFC166F3E3D50A33004246E28CFD435502C2':::GEOMETRY)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <string> = <geometry> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 98
Total Schema Statements Succeeded = 98
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 59
Total Transactions Rolled Back = 52
Total Transactions Executed = 111
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3917684841
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3917684841
--- FAIL: TestWorkload (82.23s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ fed794450894a7d1197c763ed26f75d773b93070:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY [3220:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET,26:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET,16:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET]) AND count(*) = 0 from schema_w0_54.table_w0_62\n\t    WHERE \"col 62\t_w0_68\" = ('cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: false = ANY ARRAY[3220:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET, 26:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET, 16:::OID = 'cdcb:5962:69f4:7e06:c010:da0a:25fa:9fa4/5':::INET]: unsupported comparison operator: <oid> = <inet> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 152
Total Schema Statements Succeeded = 152
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 85
Total Transactions Rolled Back = 93
Total Transactions Executed = 178
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2090896781
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2090896781
--- FAIL: TestWorkload (111.83s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 14d19acfdfbc8eb5074ddb38f29ed92c40fac35b:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2921
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:196
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1650
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['1974-04-08 08:57:31.000076':::TIMESTAMP = '1974-04-08 08:57:31.000076':::TIMESTAMP,'1970-06-16 05:44:46.000509':::TIMESTAMP = '1974-04-08 08:57:31.000076':::TIMESTAMP,'1977-04-22 21:32:52.000684':::TIMESTAMP = '1974-04-08 08:57:31.000076':::TIMESTAMP]) AND count(*) = 0 from schema_w0_1.table_w0_10\n\t    WHERE \"c'ol10_w0_19\" = ('1974-04-08 08:57:31.000076':::TIMESTAMP)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <time> = <timestamp> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 30
Total Schema Statements Succeeded = 30
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 12
Total Transactions Rolled Back = 25
Total Transactions Executed = 37
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3757018903
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3757018903
--- FAIL: TestWorkload (34.74s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ d4a51117deac22a2ffa280bac5404811cd694265:

     ]
    ]
   },
   {
    "query": "SELECT (false = ANY (ARRAY[4905080500.501836859:::DECIMAL = 4905080500.501836859:::DECIMAL])) AND (count(*) = 0) FROM public.table_w1_6 WHERE \"c\"\"ol6\\\\xb9_w1_10\" = (4905080500.501836859:::DECIMAL)",
    "result": false
   }
  ],
  "previousStatements": [
   "INSERT INTO public.table_w1_6 (col6_w1_7,col6_w1_8,col6_w1_9) VALUES (4905080500.501836859:::DECIMAL,'193.161.226.13/3':::INET,'}'::\"char\")"
  ]
 }
}
    schema_change_external_test.go:132: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:132
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: insert on table "table_w1_6" violates foreign key constraint "table_w1_6_c""ol6\\xb9_w1_10_table_w1_6_col6_w1_7_fk" (SQLSTATE 23503)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3122
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
        	            	Wraps: (5) ERROR: insert on table "table_w1_6" violates foreign key constraint "table_w1_6_c""ol6\\xb9_w1_10_table_w1_6_col6_w1_7_fk" (SQLSTATE 23503)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 43
Total Schema Statements Succeeded = 43
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 18
Total Transactions Rolled Back = 25
Total Transactions Executed = 43
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload307612601
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload307612601
--- FAIL: TestWorkload (40.76s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 72646a555214c0705781e440b9df585d5eea9511:

     "schema_w1_52",
     "schema_w1_55",
     "schema_w0_29",
     "schema_w1_60"
    ]
   }
  ],
  "previousStatements": [
   "ALTER TYPE public.enum_w0_18 DROP VALUE 'saff'",
   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
   "CREATE FUNCTION udf_w0_35(IN p1 public.enum_w0_18)\n\tRETURNS VOID\n\tLANGUAGE SQL\n\tAS $$ SELECT NULL $$"
  ]
 }
}
    schema_change_external_test.go:132: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:132
        	Error:      	Received unexpected error:
        	            	***FAIL; Failed to receive an execution error when errors were expected
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3137
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***FAIL; Failed to receive an execution error when errors were expected
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.leafError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 101
Total Schema Statements Succeeded = 101
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 50
Total Transactions Rolled Back = 95
Total Transactions Executed = 145
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3223274831
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload3223274831
--- FAIL: TestWorkload (64.28s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 2e7ab27484135f6cd16199004ba9f20dce777859:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2922
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:197
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['BOX(-10 -10,10 10)':::BOX2D = B'1','BOX(-0.48495319978470036 -1.3660511892107063,0.4012508230611492 0.7218770515286164)':::BOX2D = B'1']) AND count(*) = 0 from schema_w0_40.table_w1_80\n\t    WHERE col80_w1_87 = (B'1')\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <box2d> = <varbit> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 142
Total Schema Statements Succeeded = 142
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 82
Total Transactions Rolled Back = 114
Total Transactions Executed = 196
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2972359829
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2972359829
--- FAIL: TestWorkload (132.50s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

annrpom added a commit to annrpom/cockroach that referenced this issue Mar 11, 2024
Previously, we would check if an insert violated a
foreign key constraint by checking if the values to-be
inserted conflicted with any existing values in the parent
table. At one point, we had to grab the parent column's
index by looking it up in a map; however this map did not
contain details of every column - just the non-generated
ones. This results in some logical errors, but in cases
where the incorrectly referenced column type was different,
we saw flakes like `ERROR: unsupported comparison operator: <string> =
<timetz>`. This patch ensures we use the right map, along with
adding a check to ensure a value that we look up in the map
actually exists before me assume it does.

Epic: none
Fixes: cockroachdb#119288

Release note: None
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 431d56840cacdc86638ccb48e9f3bdc16897d341:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:454
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2922
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:197
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:441
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['4714-11-24 BC':::DATE = '4714-11-24 BC':::DATE,'1982-12-24':::DATE = '4714-11-24 BC':::DATE,'1986-03-09':::DATE = '4714-11-24 BC':::DATE]) AND count(*) = 0 from schema_w1_16.table_w0_45\n\t    WHERE \"co\fl45_w0_59\" = ('4714-11-24 BC':::DATE)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: <string> = <date> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 67
Total Schema Statements Succeeded = 67
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 37
Total Transactions Rolled Back = 49
Total Transactions Executed = 86
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_3_of_30/test.outputs/logTestWorkload2157339838
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2157339838
--- FAIL: TestWorkload (15.24s)

Parameters:

  • attempt=1
  • run=3
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ 431d56840cacdc86638ccb48e9f3bdc16897d341:

     "schema_w1_46",
     "schema_w0_58",
     "schema_w0_62",
     "schema_w1_64"
    ]
   }
  ],
  "previousStatements": [
   "DROP SEQUENCE schema_w0_51.seq_w0_64",
   "ALTER TYPE public.enum_w1_58 DROP VALUE 's'",
   "CREATE FUNCTION udf_w0_66(IN p1 public.enum_w1_58)\n\tRETURNS VOID\n\tLANGUAGE SQL\n\tAS $$ SELECT NULL $$"
  ]
 }
}
    schema_change_external_test.go:132: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:132
        	Error:      	Received unexpected error:
        	            	***FAIL; Failed to receive an execution error when errors were expected
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3138
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:545
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***FAIL; Failed to receive an execution error when errors were expected
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.leafError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 132
Total Schema Statements Succeeded = 132
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 71
Total Transactions Rolled Back = 112
Total Transactions Executed = 183
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload4294388477
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload4294388477
--- FAIL: TestWorkload (84.02s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@exalate-issue-sync exalate-issue-sync bot added P-2 Issues/test failures with a fix SLA of 3 months P-1 Issues/test failures with a fix SLA of 1 month and removed P-1 Issues/test failures with a fix SLA of 1 month P-2 Issues/test failures with a fix SLA of 3 months labels Mar 13, 2024
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 455b16592df7d8efd121b3ba1256fb477e227564:

        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:495
        	            	  | [...repeated from below...]
        	            	Wraps: (4) ***UNEXPECTED ERROR; Failed to generate a random operation
        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:167
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/query_util.go:149
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:88
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraintsHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1202
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).violatesFkConstraints
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/error_screening.go:1121
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).insertRow
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2922
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:197
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:482
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:586
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:121
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (6) CollectOne: Query: "\n\t    SELECT false = ANY (ARRAY ['19:37:14.808015+06:11':::TIMETZ = e'7?,\\b['::STRING,'17:33:11.4988+02:26':::TIMETZ = e'7?,\\b['::STRING,'01:32:39.57083+14:52':::TIMETZ = e'7?,\\b['::STRING]) AND count(*) = 0 from schema_w0_4.table_w0_14\n\t    WHERE \"c ol1 4_w0_31\" = (e'7?,\\b['::STRING)\n\t" []
        	            	Wraps: (7) ERROR: unsupported comparison operator: false = ANY ARRAY['19:37:14.808015+06:11':::TIMETZ = e'7?,\b['::STRING, '17:33:11.4988+02:26':::TIMETZ = e'7?,\b['::STRING, '01:32:39.57083+14:52':::TIMETZ = e'7?,\b['::STRING]: unsupported comparison operator: <timetz> = <string> (SQLSTATE 22023)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 46
Total Schema Statements Succeeded = 46
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 27
Total Transactions Rolled Back = 31
Total Transactions Executed = 58
    schema_change_external_test.go:98: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_27_of_30/test.outputs/logTestWorkload315216621
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload315216621
--- FAIL: TestWorkload (13.15s)

Parameters:

  • attempt=1
  • run=27
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

rafiss added a commit to rafiss/cockroach that referenced this issue Mar 15, 2024
Until cockroachdb#119288 is resolved, this should be disabled.

Release note: None
@rafiss rafiss added the X-noreuse Prevent automatic commenting from CI test failures label Mar 15, 2024
craig bot pushed a commit that referenced this issue Mar 15, 2024
120454: roachtest: fix bundle collection on failure in tpchvec/mt-shared-process r=yuzefovich a=yuzefovich

Previously, we hard-coded using the connection to the system tenant when collecting the stmt bundle in case the test is failing, and this is now fixed.

Additionally, the output of EXPLAIN ANALYZE (DEBUG) doesn't contain the AdminUI link (present in the system tenant) that we used to download the bundle, so this commit refactors the code to use the CLI client instead, that is common for both system and application tenants.

Fixes: #120397.

Release note: None

120575: workload/schemachange: disable foreign keys due to flakes r=rafiss a=rafiss

Until #119288 is resolved, this should be disabled.

informs #119288
Release note: None

Co-authored-by: Yahor Yuzefovich <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
annrpom added a commit to annrpom/cockroach that referenced this issue Mar 19, 2024
Previously, we would check if an insert violated a
foreign key constraint by checking if the values to-be
inserted conflicted with any existing values in the parent
table. At one point, we had to grab the parent column's
index by looking it up in a map; however this map did not
contain details of every column - just the non-generated
ones. This results in some logical errors, but in cases
where the incorrectly referenced column type was different,
we saw flakes like `ERROR: unsupported comparison operator: <string> =
<timetz>`. This patch ensures we use the right map, along with
adding a check to ensure a value that we look up in the map
actually exists before me assume it does.

Epic: none
Fixes: cockroachdb#119288

Release note: None
annrpom added a commit to annrpom/cockroach that referenced this issue Mar 21, 2024
Previously, we saw flakes like `ERROR: unsupported comparison
operator: <string> = <timetz>` occurring in the rsw because at times,
we were indexing a column name in a map without any guards. This meant
that if any improperly quoted column name was being looked up in the
map, we would just return the default for that key not existing in our
map (0). This caused us to refer an arbitrary column's values when
validating FK constraints, causing a mismatch that we see above.

We now have checks to guard us from making this mistake again, along
with proper quotes around column names.

In addition, alter FK statements and inserts in the same explicit txn
did not bode well - as the query to get our fkConstraints relied on
the crdb_internal.is_constraint_active builtin, which does not work in
the legacy schema changer: cockroachdb#120702. In this case, we do not need the
builtin to filter for our FK constraints.

Epic: none
Fixes: cockroachdb#119288

Release note: None
craig bot pushed a commit that referenced this issue Mar 22, 2024
119653: workload/schemachange: deflake violatesFkConstraintsHelper in workload r=rafiss a=annrpom

Previously, we saw flakes like `ERROR: unsupported comparison operator: <string> = <timetz>` occurring in the rsw because at times, we were indexing a column name in a map without any guards. This meant that if any improperly quoted column name was being looked up in the map, we would just return the default for that key not existing in our map (0). This caused us to refer an arbitrary column's values when validating FK constraints, causing a mismatch that we see above.

We now have checks to guard us from making this mistake again, along with proper quotes around column names.

In addition, alter FK statements and inserts in the same explicit txn did not bode well - as the query to get our `fkConstraints` relied on the `crdb_internal.is_constraint_active` builtin, which does not work in the legacy schema changer: #120702. In this case, we do not need the builtin to filter for our FK constraints.

Epic: none
Fixes: #119288

Release note: None

Co-authored-by: Annie Pompa <[email protected]>
@craig craig bot closed this as completed in c4bea7f Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-noreuse Prevent automatic commenting from CI test failures
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants