Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands/#01 #18963

Closed
cockroach-teamcity opened this issue Oct 2, 2017 · 1 comment
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands/#01 #18963

cockroach-teamcity opened this issue Oct 2, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#366829:

--- FAIL: test/TestRefreshPendingCommands (0.000s)
Test ended in panic.

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I171002 21:35:20.062293 44331 storage/store.go:1210  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171002 21:35:20.062353 44331 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:53176" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171002 21:35:20.062410 45009 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171002 21:35:20.062435 45010 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171002 21:35:20.065627 44331 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171002 21:35:20.065796 44331 storage/store.go:1210  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171002 21:35:20.066113 44331 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:51821" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171002 21:35:20.065863 45025 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:53176
W171002 21:35:20.069613 44331 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171002 21:35:20.069721 44331 storage/store.go:1210  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171002 21:35:20.070159 44331 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:48224" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171002 21:35:20.070261 43928 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:53176
I171002 21:35:20.072242 45127 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 0595c632 at index 15
I171002 21:35:20.072767 45127 storage/store.go:3509  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 0ms
I171002 21:35:20.072930 44900 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=0595c632, encoded size=4620, 1 rocksdb batches, 5 log entries)
I171002 21:35:20.073197 44900 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171002 21:35:20.073541 45127 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171002 21:35:20.074495 45132 storage/replica.go:3072  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171002 21:35:20.075232 44905 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot fa6bf93f at index 17
I171002 21:35:20.075830 44905 storage/store.go:3509  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 0ms
I171002 21:35:20.075974 45149 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=fa6bf93f, encoded size=5667, 1 rocksdb batches, 7 log entries)
I171002 21:35:20.076286 45149 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171002 21:35:20.076703 44905 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171002 21:35:20.077502 45298 storage/raft_transport.go:455  raft transport stream to node 1 established
I171002 21:35:20.078613 45286 storage/replica.go:3072  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171002 21:35:20.355979 44908 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:20.356165 45273 storage/store.go:3247  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171002 21:35:20.356188 45302 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I171002 21:35:20.356619 45274 storage/raft_transport.go:455  raft transport stream to node 3 established
W171002 21:35:20.356788 45304 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:20.356865 45304 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:20.356945 45292 storage/store.go:3247  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I171002 21:35:20.357323 45275 storage/raft_transport.go:455  raft transport stream to node 3 established
W171002 21:35:20.357473 45279 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:20.357577 45284 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W171002 21:35:20.357648 45277 storage/raft_transport.go:494  no handler found for store 1 in response range_id:1 from_replica:<node_id:3 store_id:3 replica_id:3 > to_replica:<node_id:1 store_id:1 replica_id:1 > union:<error:<message:"store 3 was not found" transaction_restart:NONE origin_node:0 detail:<store_not_found:<store_id:3 > > now:<> > > 
W171002 21:35:20.357743 45300 storage/store.go:3247  [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W171002 21:35:20.357768 45298 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
I171002 21:35:20.358161 44331 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
W171002 21:35:20.358407 45385 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171002 21:35:20.358430 45386 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I171002 21:35:20.358588 44331 storage/client_test.go:1194  test clock advanced to: 3.600000127,0
W171002 21:35:20.359264 45470 storage/store.go:1297  [s3,r1/3:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171002 21:35:20.359280 45471 storage/store.go:1297  [s3,r1/3:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I171002 21:35:20.584481 45494 storage/client_test.go:1194  [hb] test clock advanced to: 5.400000129,0
W171002 21:35:20.584634 45495 storage/replica.go:2696  [hb,s3,r1/3:/M{in-ax}] context cancellation after 0.2s of attempting command RequestLease [/Min,/Min)
W171002 21:35:20.584792 45473 storage/node_liveness.go:358  [hb] failed node liveness heartbeat: context deadline exceeded
I171002 21:35:20.666326 45496 storage/raft_transport.go:455  raft transport stream to node 1 established
I171002 21:35:20.667586 45142 storage/replica_raftstorage.go:526  [raftsnapshot,s2,r1/2:/M{in-ax}] generated Raft snapshot f978690f at index 23
I171002 21:35:20.668321 45142 storage/store.go:3509  [raftsnapshot,s2,r1/2:/M{in-ax}] streamed snapshot to (n3,s3):3: kv pairs: 42, log entries: 2, rate-limit: 8.0 MiB/sec, 1ms
I171002 21:35:20.668535 45484 storage/replica_raftstorage.go:732  [s3,r1/3:/M{in-ax}] applying Raft snapshot at index 23 (id=f978690f, encoded size=4403, 1 rocksdb batches, 2 log entries)
I171002 21:35:20.668849 45484 storage/replica_raftstorage.go:738  [s3,r1/3:/M{in-ax}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171002 21:35:20.901402 44591 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:48224->127.0.0.1:34274: use of closed network connection
I171002 21:35:20.901461 44625 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:53176->127.0.0.1:52120: read: connection reset by peer
I171002 21:35:20.901501 44932 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:51821->127.0.0.1:37572: use of closed network connection
--- FAIL: test/TestRefreshPendingCommands/#01 (0.000s)
Test ended in panic.

------- Stdout: -------
I171002 21:35:20.904684 45554 storage/store.go:1210  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171002 21:35:20.904737 45554 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:55447" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171002 21:35:20.904944 45640 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171002 21:35:20.904964 45641 storage/store.go:1297  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171002 21:35:20.908080 45554 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171002 21:35:20.908199 45554 storage/store.go:1210  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171002 21:35:20.908272 45488 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:55447
I171002 21:35:20.908555 45554 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:44912" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171002 21:35:20.910926 45554 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171002 21:35:20.911022 45554 storage/store.go:1210  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171002 21:35:20.911057 45554 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:43470" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171002 21:35:20.911205 45795 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:55447
I171002 21:35:20.982259 45908 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 31e81e42 at index 15
I171002 21:35:20.982858 45908 storage/store.go:3509  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 0ms
I171002 21:35:20.983096 45911 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=31e81e42, encoded size=4620, 1 rocksdb batches, 5 log entries)
I171002 21:35:20.983376 45911 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171002 21:35:20.983712 45908 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171002 21:35:20.984607 45915 storage/replica.go:3072  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171002 21:35:20.985222 45918 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 20dffd8f at index 17
I171002 21:35:20.986237 45918 storage/store.go:3509  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms
I171002 21:35:20.987152 45902 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=20dffd8f, encoded size=5667, 1 rocksdb batches, 7 log entries)
I171002 21:35:20.987553 45902 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171002 21:35:20.987946 45918 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171002 21:35:20.988776 45905 storage/raft_transport.go:455  raft transport stream to node 1 established
I171002 21:35:20.990306 45935 storage/replica.go:3072  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171002 21:35:21.128725 45514 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:21.128892 45530 storage/store.go:3247  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171002 21:35:21.128915 45528 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I171002 21:35:21.129360 45942 storage/raft_transport.go:455  raft transport stream to node 3 established
W171002 21:35:21.129551 45548 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:21.129671 45546 storage/store.go:3247  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I171002 21:35:21.130060 45943 storage/raft_transport.go:455  raft transport stream to node 3 established
W171002 21:35:21.130331 45933 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W171002 21:35:21.130480 45516 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171002 21:35:21.134743 45939 storage/store.go:3247  [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W171002 21:35:21.134998 45905 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
I171002 21:35:21.135227 45554 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
SIGQUIT: quit
PC=0x17305f6 m=0 sigcode=0

goroutine 45554 [running]:
sync/atomic.(*Value).Load(...)
	/usr/local/go/src/sync/atomic/value.go:37
github.com/cockroachdb/cockroach/pkg/storage.(*Store).IsDraining(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1786
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1(0xc420102870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1097 +0x836 fp=0xc423b53fa8 sp=0xc423b53af0 pc=0x17305f6
testing.tRunner(0xc420102870, 0xc42038a8f0)
	/usr/local/go/src/testing/testing.go:746 +0xd0 fp=0xc423b53fd0 sp=0xc423b53fa8 pc=0x78eaa0
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc423b53fd8 sp=0xc423b53fd0 pc=0x6f4ec1
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4201024b0, 0x1cd2ca9, 0x1a, 0x1d97898, 0x725b01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201024b0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201024b0, 0xc4205a5b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42042a660, 0x2ae36c0, 0x1db, 0x1db, 0xc4205a5c98)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420899f18, 0xc4205a5d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc4205a5f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1010 +0xdb

goroutine 6 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2fbd400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 13 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45563 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc420e129c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1227 +0x148
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:292 +0xd2e

goroutine 45567 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2b04bc0, 0xc4208c40c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038b6a0, 0xc420d90480, 0xc4203f4ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45687 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000049)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c4420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c4420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff050, 0xc420d917a0, 0xc4202ff040)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46025 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6b50, 0xc4208c3560, 0xc423b49810)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45701 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000017)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c46c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c46c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff2d0, 0xc420d917a0, 0xc4202ff2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45675 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2b04bc0, 0xc420aa4f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202feed0, 0xc420d90480, 0xc4203dcb80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45704 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000001a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c4750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c4750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff350, 0xc420d917a0, 0xc4202ff340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45989 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc400000014)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6490, 0xc4208c3560, 0xc4202e6480)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45691 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000004d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c44e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c44e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff0f0, 0xc420d917a0, 0xc4202ff0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45977 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc400000016)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e62f0, 0xc4208c3560, 0xc4202e62e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45697 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000013)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c4600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c4600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff240, 0xc420d917a0, 0xc4202ff230)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45777 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420bbad70, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420bbad60)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420bbac80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45957 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc400000002)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b9f80, 0xc4208c3560, 0xc4202b9f70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45696 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c45d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c45d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff220, 0xc420d917a0, 0xc4202ff1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45565 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc423f1e8c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff

goroutine 45489 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func3(0x2b119c0, 0xc420e129c0, 0xc420e12820, 0xc423f5c6e0, 0xc420a55000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:255 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:254 +0xe23

goroutine 45785 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420357a00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 45566 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc42344da00, 0xc4203e47c0, 0xc420300f70, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2bb
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2b04bc0, 0xc4208c40f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42038b670, 0xc420d90480, 0xc420c4f560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45999 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e66e0, 0xc4208c3560, 0xc4202e66c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45955 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2b04bc0, 0xc420e47aa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b9f30, 0xc4208c3560, 0xc42034af40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45970 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc40000000f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e61f0, 0xc4208c3560, 0xc4202e61e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46013 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e68e0, 0xc4208c3560, 0xc4202e68d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45783 [runnable]:
internal/poll.runtime_pollWait(0x7fed856c1970, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc423ee3118, 0x72, 0xffffffffffffff00, 0x2af2340, 0x2ae6e00)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc423ee3118, 0xc4234fa000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc423ee3100, 0xc4234fa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc423ee3100, 0xc4234fa000, 0x8000, 0x8000, 0xa9c327, 0xc4206ca1c0, 0xc420c7acb0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420bec3e0, 0xc4234fa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420505620, 0xc4206ca118, 0x9, 0x9, 0xc420c7ad50, 0xaf24e6, 0xc4208c4fc0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2ae8c00, 0xc420505620, 0xc4206ca118, 0x9, 0x9, 0x9, 0x0, 0x0, 0xc420c7ae5f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2ae8c00, 0xc420505620, 0xc4206ca118, 0x9, 0x9, 0x51db401f01, 0x4d00000000000051, 0x51)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4206ca118, 0x9, 0x9, 0x2ae8c00, 0xc420505620, 0x0, 0x0, 0xc420877560, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4206ca0e0, 0x2af5600, 0xc4234f0ed0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc4234f0ea0, 0xc4234f0ed0, 0xc4234f0ed0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420357a00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 45665 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc4207fe7e0, 0x2b04bc0, 0xc4208c4180)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:253 +0x1a4
github.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1(0x2b04bc0, 0xc4208c4180)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:237 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42026bfd0, 0xc420d917a0, 0xc42026bfc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45730 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000034)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc420aa5620)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420aa5620)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff750, 0xc420d917a0, 0xc4202ff740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45690 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000004c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c44b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c44b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff0c0, 0xc420d917a0, 0xc4202ff0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45966 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc40000000b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6170, 0xc4208c3560, 0xc4202e6160)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45707 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000001d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c47e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff3b0, 0xc420d917a0, 0xc4202ff3a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45798 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420fa67d0, 0xc42031b770, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc42093ff20, 0xc42031b770, 0x5, 0x5, 0x70df4e, 0xc420417ac0, 0xc420fda640)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aee480, 0xc42093ff20, 0xc42031b770, 0x5, 0x5, 0x5, 0xc420880dc0, 0xc420417a00, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aee480, 0xc42093ff20, 0xc42031b770, 0x5, 0x5, 0x100000000000003, 0xc42070e180, 0x2)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc420417a00, 0xc42031b770, 0x5, 0x5, 0x49, 0xc42378b9b8, 0x6c2ce6)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42031b760, 0x7fffffff, 0x0, 0x0, 0xc420419bd0, 0x49, 0x6, 0xc42070e180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42031b760, 0x2b04400, 0x2fffb38, 0xc420417a00, 0x2af6f00, 0x2fffb38, 0x1be2660, 0xc4234f7a90, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc420fbb200, 0x1be2660, 0xc4234f7a90, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:648 +0x10e
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc420314d60, 0xc42378bbb0, 0xba66d0, 0xc42365e340)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:230 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(Gossip_GossipServer).Recv-fm(0xc42365e340, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:161 +0x2f
github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc42365e300, 0x2b04bc0, 0xc423f1a120, 0xc420c68130, 0xc4203e1cc0, 0xc423ebbf58, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:339 +0x474
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x2b04bc0, 0xc423f1a120)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:161 +0x96
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420314d90, 0xc420d90480, 0xc4203e1d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45716 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000026)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc420aa5350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420aa5350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff4e0, 0xc420d917a0, 0xc4202ff4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45672 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420e12d00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 45722 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000002c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc420aa54a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420aa54a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff650, 0xc420d917a0, 0xc4202ff640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45689 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc40000004b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c4480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c4480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff0a0, 0xc420d917a0, 0xc4202ff080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45683 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420e6ce10, 0xc400000045)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420e6ce00)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420d91830, 0x2b04bc0, 0xc4208c4360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc4208c4360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fefd0, 0xc420d917a0, 0xc4202fefc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45756 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b04bc0, 0xc420a0f020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ffb60, 0xc420d917a0, 0xc4203dce20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45755 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b04bc0, 0xc420a0ef00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ffb50, 0xc420d917a0, 0xc4203dce00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45962 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6080, 0xc4208c3560, 0xc4202e6060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45752 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b04bc0, 0xc42093eae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ffb20, 0xc420d917a0, 0xc4203dcda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45743 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0xc420d918b4)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc420d918a8)
	/usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc420d91830, 0x2b04bc0, 0xc420aa58c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:173 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x2b04bc0, 0xc420aa58c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3640 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ff900, 0xc420d917a0, 0xc4202ff8f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46019 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e69c0, 0xc4208c3560, 0xc4202e69a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45945 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4238c37c0, 0xc42036b990, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc423fba3c0, 0xc42036b990, 0x5, 0x5, 0x70df4e, 0xc420417ec0, 0xc4237f2ac0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aee480, 0xc423fba3c0, 0xc42036b990, 0x5, 0x5, 0x5, 0xc420357a00, 0xc420417e00, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aee480, 0xc423fba3c0, 0xc42036b990, 0x5, 0x5, 0x6dbbb9, 0x120, 0x1ca6b20)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc420417e00, 0xc42036b990, 0x5, 0x5, 0xc42001f300, 0x6b49d27d, 0x1)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42036b980, 0x7fffffff, 0xc420a6a600, 0xc420a6a758, 0xc420a6a598, 0x6a8987, 0xc4206dc360, 0x60)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42036b980, 0x2b04400, 0x2fffb38, 0xc420417e00, 0x2af6f00, 0x2fffb38, 0x1bd2440, 0xc423fba450, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc423508f20, 0x1bd2440, 0xc423fba450, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc42021c930, 0xc423fba420, 0xc423fba420, 0x1acfca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:329 +0x62
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2b11720, 0xc42021c930, 0xc420347d40, 0xc420102c30, 0x2b04bc0, 0xc423fba420, 0x18, 0xd8ebe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:487 +0x35
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x2b04bc0, 0xc423fba420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:502 +0x66
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42021c970, 0xc420d90480, 0xc423fba3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45562 [IO wait]:
internal/poll.runtime_pollWait(0x7fed856c17f0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420ba9798, 0x72, 0xffffffffffffff00, 0x2af2340, 0x2ae6e00)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420ba9798, 0xc420b88000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420ba9780, 0xc420b88000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420ba9780, 0xc420b88000, 0x8000, 0x8000, 0x1d998a8, 0x6a99b6, 0xc4233bac30)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42360e220, 0xc420b88000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc4203f6660, 0xc4202c2f18, 0x9, 0x9, 0xc4233bad50, 0xaf24e6, 0xc423f78ba0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2ae8c00, 0xc4203f6660, 0xc4202c2f18, 0x9, 0x9, 0x9, 0x0, 0x6a99b6, 0xc4233bada0)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2ae8c00, 0xc4203f6660, 0xc4202c2f18, 0x9, 0x9, 0xc4202c2f08, 0x0, 0x51)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4202c2f18, 0x9, 0x9, 0x2ae8c00, 0xc4203f6660, 0x0, 0x0, 0xc420cf0600, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4202c2ee0, 0x2af5600, 0xc420c4f3e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420c4f3b0, 0xc420c4f3e0, 0xc420c4f3e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420e129c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 45965 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc40000000a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6150, 0xc4208c3560, 0xc4202e6140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45969 [semacquire]:
sync.runtime_notifyListWait(0xc420dccd90, 0xc40000000e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420dccd80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc423740750, 0x2b04bc0, 0xc420e47d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04bc0, 0xc420e47d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e61d0, 0xc4208c3560, 0xc4202e61c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45992 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202e6520, 0xc4208c3560, 0xc4202e6510)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45903 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc420102c30, 0x2b11780, 0xc420315bd0, 0x2af4d80, 0xc420102c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:355 +0x215
github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x1c87580, 0xc420102c30, 0x2b0dee0, 0xc4208c2c60, 0xc420417b00, 0x2af6f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:378 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420a337c0, 0x2b0f620, 0xc423f5c580, 0xc420417c00, 0xc42

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 2, 2017
@a-robinson
Copy link
Contributor

#18554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@a-robinson @cockroach-teamcity