Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: cdc/pubsub-sink/assume-role failed #85374

Closed
cockroach-teamcity opened this issue Jul 31, 2022 · 16 comments · Fixed by #88289
Closed

roachtest: cdc/pubsub-sink/assume-role failed #85374

cockroach-teamcity opened this issue Jul 31, 2022 · 16 comments · Fixed by #88289
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-cdc
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 31, 2022

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 0352fbee5e6933e8bca28aae4b136bcd495ab227:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_073444.689975139_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220731 07:34:45.911560 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220731 07:34:45.913945 64 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.14:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.14 user=root database=tpcc``: dial error (dial tcp 10.142.0.14:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6222
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-18212

Epic CRDB-11732

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jul 31, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Jul 31, 2022
@blathers-crl blathers-crl bot added the T-cdc label Jul 31, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ b83e83b7e54fabf6ebd3a42cff67f4b8c5b7b863:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_074417.261515267_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220801 07:44:18.406802 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220801 07:44:18.408885 111 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.56:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.56 user=root database=tpcc``: dial error (dial tcp 10.142.0.56:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6222
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@HonoreDB
Copy link
Contributor

HonoreDB commented Aug 1, 2022

"failed to initialize the load generator" is a test flake, not an a real failure, so I'll remove the release-blocker tag, but we should still triage this.

@HonoreDB HonoreDB removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 1, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 773f7d4445ce3e0e806b7a182adba70a0f270f19:

		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_071113.719576588_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  | I220812 07:11:14.923700 1 workload/cli/run.go:427  [-] 1  creating load generator...
		  | I220812 07:11:14.926801 116 workload/pgx_helpers.go:79  [-] 2  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.132:26257: connect: connection refused]
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.132 user=root database=tpcc``: dial error (dial tcp 10.142.0.132:26257: connect: connection refused)
		  |
		  | stdout:
		  | Initializing 2 connections...
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,cdc.go:287,cdc.go:829,test_runner.go:896: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:829
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6240
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1597
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:177
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_072421.278700404_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:287,cdc.go:816,test_runner.go:896: monitor failure: monitor task failed: dial tcp 34.138.240.36:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:287
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:816
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.138.240.36:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ c080f7ac2f5be11aedd787dceba77cca7df18a16:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1602
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:176
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_072358.006431375_n4_workload_run_tpcc
		Wraps: (3) ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}  returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:897: monitor failure: monitor task failed: dial tcp 35.237.211.226:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.237.211.226:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     3.0s        0            3.0            1.0     65.0     71.3     71.3     71.3 orderStatus
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     3.0s        0            1.0            0.3     60.8     60.8     60.8     60.8 stockLevel
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:794,test_runner.go:917: monitor failure: monitor task failed: read tcp 172.17.0.3:41914 -> 34.138.23.29:26257: read: connection reset by peer
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func8
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:794
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) read tcp 172.17.0.3:41914 -> 34.138.23.29:26257
		Wraps: (6) read
		Wraps: (7) connection reset by peer
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:

		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     3.0s        0            1.0            0.3     71.3     71.3     71.3     71.3 delivery
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  |     3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  |     4.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 delivery
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		  |     4.0s        0            4.0            1.0     16.8     35.7     35.7     35.7 payment
		  |     4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (4) secondary error attachment
		  | UNCLASSIFIED_PROBLEM: context canceled
		  | (1) UNCLASSIFIED_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run tpcc --warehouses=1 --duration=30m  {pgurl:1-3}
		  |   | ``````
		  | Wraps: (3) context canceled
		  | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	monitor.go:127,cdc.go:300,cdc.go:794,test_runner.go:917: monitor failure: monitor task failed: read tcp 172.17.0.3:48362 -> 34.148.95.11:26257: read: connection reset by peer
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func8
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:794
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) read tcp 172.17.0.3:48362 -> 34.148.95.11:26257
		Wraps: (6) read
		Wraps: (7) connection reset by peer
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 5862e9b3c1e24b3f644716789c96108a8e92fd71:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:917: monitor failure: monitor task failed: dial tcp 34.138.103.200:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.138.103.200:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1048,test_runner.go:947: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@HonoreDB
Copy link
Contributor

Looks like I introduced a new bug while fixing the previous one, fixing now.

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ bbcb10f982e47943de63a3a2d1faf10d3eb49e75:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.139.149.103:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.139.149.103:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 726cf22b9f06b766d857b4617dec0df18d1e5cd0:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.148.134.64:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.148.134.64:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 34dc56fbb5789b39be47b110bf22332c7f5654f6:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.229.118.106:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.229.118.106:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 89f4ad907a1756551bd6864c3e8516eeff6b0e0a:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.185.108.152:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.185.108.152:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ 51c8aae748d338549400c047796c6c9b892527da:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 35.196.58.108:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 35.196.58.108:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ ff13325e9368c4e8dd9a4d5cf4aa2ad2f33e9ac0:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.74.155.56:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.74.155.56:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cdc/pubsub-sink/assume-role failed with artifacts on master @ a0bfa6dafcc206301d3a21887c374db63b377075:

test artifacts and logs in: /artifacts/cdc/pubsub-sink/assume-role/run_1
	monitor.go:127,cdc.go:300,cdc.go:820,test_runner.go:928: monitor failure: monitor task failed: dial tcp 34.139.191.22:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:300
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func9
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:820
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) dial tcp 34.139.191.22:26257
		Wraps: (6) connect
		Wraps: (7) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants