Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#01 #18965

Closed
cockroach-teamcity opened this issue Oct 3, 2017 · 1 comment
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#01 #18965

cockroach-teamcity opened this issue Oct 3, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#366918:

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I171003 01:52:34.875483 44845 storage/store.go:1209  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171003 01:52:34.875544 44845 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:43105" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171003 01:52:34.875641 44974 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171003 01:52:34.875886 44973 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171003 01:52:34.881321 44845 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171003 01:52:34.881517 44845 storage/store.go:1209  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171003 01:52:34.881588 44845 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:37629" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171003 01:52:34.882089 44988 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:43105
W171003 01:52:34.957804 44845 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171003 01:52:34.957988 44845 storage/store.go:1209  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171003 01:52:34.958126 44845 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:39683" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171003 01:52:34.959204 44174 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:43105
I171003 01:52:35.029065 45116 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot c50b7fb1 at index 15
I171003 01:52:35.029735 45116 storage/store.go:3508  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 1ms
I171003 01:52:35.029993 45150 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=c50b7fb1, encoded size=4620, 1 rocksdb batches, 5 log entries)
I171003 01:52:35.030317 45150 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171003 01:52:35.030721 45116 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171003 01:52:35.031742 45121 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171003 01:52:35.033691 45153 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 263184fa at index 17
I171003 01:52:35.034189 45153 storage/store.go:3508  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 0ms
I171003 01:52:35.034490 45288 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=263184fa, encoded size=5667, 1 rocksdb batches, 7 log entries)
I171003 01:52:35.034883 45288 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171003 01:52:35.035308 45153 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171003 01:52:35.036476 45301 storage/raft_transport.go:455  raft transport stream to node 1 established
I171003 01:52:35.038878 45290 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171003 01:52:35.112626 45312 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.112847 45310 storage/store.go:3246  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171003 01:52:35.113565 45308 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
W171003 01:52:35.113393 45312 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.113650 45312 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
I171003 01:52:35.114450 45317 storage/raft_transport.go:455  raft transport stream to node 3 established
W171003 01:52:35.114812 45292 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.115944 45265 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
I171003 01:52:35.118273 44845 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
I171003 01:52:35.119422 44845 storage/client_test.go:1194  test clock advanced to: 3.600000127,0
W171003 01:52:35.121149 45409 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171003 01:52:35.121548 45303 storage/store.go:3246  [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W171003 01:52:35.121760 45301 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
W171003 01:52:35.123086 45408 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171003 01:52:35.124132 45330 storage/store.go:3246  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171003 01:52:35.125547 45495 storage/store.go:1296  [s3,r1/3:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171003 01:52:35.125905 45496 storage/store.go:1296  [s3,r1/3:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I171003 01:52:35.350576 45279 storage/client_test.go:1194  [hb] test clock advanced to: 5.400000129,0
W171003 01:52:35.350694 45280 storage/replica.go:2695  [hb,s3,r1/3:/M{in-ax}] context cancellation after 0.2s of attempting command RequestLease [/Min,/Min)
W171003 01:52:35.350806 45498 storage/node_liveness.go:358  [hb] failed node liveness heartbeat: context deadline exceeded
I171003 01:52:35.383663 45523 storage/raft_transport.go:455  raft transport stream to node 1 established
I171003 01:52:35.385381 45524 storage/raft_transport.go:455  raft transport stream to node 3 established
I171003 01:52:35.386327 45111 storage/replica_raftstorage.go:526  [raftsnapshot,s2,r1/2:/M{in-ax}] generated Raft snapshot 50ebbe33 at index 23
I171003 01:52:35.387028 45111 storage/store.go:3508  [raftsnapshot,s2,r1/2:/M{in-ax}] streamed snapshot to (n3,s3):3: kv pairs: 42, log entries: 2, rate-limit: 8.0 MiB/sec, 1ms
I171003 01:52:35.387193 45422 storage/replica_raftstorage.go:732  [s3,r1/3:/M{in-ax}] applying Raft snapshot at index 23 (id=50ebbe33, encoded size=4403, 1 rocksdb batches, 2 log entries)
I171003 01:52:35.387556 45422 storage/replica_raftstorage.go:738  [s3,r1/3:/M{in-ax}] applied Raft snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
--- FAIL: test/TestRefreshPendingCommands (0.000s)
Test ended in panic.

--- FAIL: test/TestRefreshPendingCommands/#01 (0.000s)
Test ended in panic.

------- Stdout: -------
I171003 01:52:35.673684 44780 storage/store.go:1209  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171003 01:52:35.673743 44780 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:46397" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171003 01:52:35.674133 45618 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171003 01:52:35.674158 45619 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171003 01:52:35.678866 44780 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171003 01:52:35.680034 45746 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:46397
I171003 01:52:35.680247 44780 storage/store.go:1209  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171003 01:52:35.680305 44780 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:51342" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171003 01:52:35.683940 44780 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171003 01:52:35.684074 44780 storage/store.go:1209  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171003 01:52:35.684121 44780 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:48342" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171003 01:52:35.684647 45779 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:46397
I171003 01:52:35.754804 45893 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 6f7d11ea at index 15
I171003 01:52:35.755744 45893 storage/store.go:3508  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 1ms
I171003 01:52:35.755971 45742 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=6f7d11ea, encoded size=4620, 1 rocksdb batches, 5 log entries)
I171003 01:52:35.756365 45742 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171003 01:52:35.757767 45893 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171003 01:52:35.759782 45906 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171003 01:52:35.761120 45909 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot de543d48 at index 17
I171003 01:52:35.761840 45909 storage/store.go:3508  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms
I171003 01:52:35.762026 45925 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=de543d48, encoded size=5667, 1 rocksdb batches, 7 log entries)
I171003 01:52:35.762420 45925 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171003 01:52:35.762859 45909 storage/replica_command.go:3687  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171003 01:52:35.763764 45941 storage/raft_transport.go:455  raft transport stream to node 1 established
I171003 01:52:35.765453 45962 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171003 01:52:35.906269 45945 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.906386 45945 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.906419 45945 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.906447 45945 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171003 01:52:35.906554 45911 storage/store.go:3246  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171003 01:52:35.906575 45970 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
W171003 01:52:35.908432 46053 storage/store.go:1296  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
I171003 01:52:35.908667 44780 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
SIGQUIT: quit
PC=0x6f6341 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x2fbea30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fff2680f368, 0x6a789b, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21
runtime.futexsleep(0x2fbea30, 0x7fff00000000, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0x2fbea30)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stopm()
	/usr/local/go/src/runtime/proc.go:1670 +0xe5
runtime.findrunnable(0xc420022c00, 0x0)
	/usr/local/go/src/runtime/proc.go:2125 +0x4d2
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2245 +0x12c
runtime.park_m(0xc420450480)
	/usr/local/go/src/runtime/proc.go:2308 +0xb6
runtime.mcall(0x7fff2680f500)
	/usr/local/go/src/runtime/asm_amd64.s:286 +0x5b

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4201104b0, 0x1cd2d89, 0x1a, 0x1d97978, 0x725b01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201104b0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201104b0, 0xc42069db80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42044e480, 0x2ae36c0, 0x1db, 0x1db, 0xc42069dc98)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420697f18, 0xc42069dd80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc42069df18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1010 +0xdb

goroutine 6 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2fbdd60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 35 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45731 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc4204b5800, 0x2b04b00, 0xc42089b650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3655 +0x205
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x2b04b00, 0xc42089b650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3641 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42033a180, 0xc420df2ea0, 0xc42033a170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45696 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000031)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc42089a690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc42089a690)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abb90, 0xc420df2ea0, 0xc4202abb60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45778 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc421082840)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:949 +0x251
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:221 +0x8e3

goroutine 46037 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000036)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ecb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ecb10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd6a0, 0xc4214c6d80, 0xc4202dd690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45698 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc40000003f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc42089b920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc42089b920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abbe0, 0xc420df2ea0, 0xc4202abbd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46009 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc40000001a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd1d0, 0xc4214c6d80, 0xc4202dd1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45782 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420e48730, 0xc4202af830, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc420887f50, 0xc4202af830, 0x5, 0x5, 0x70df4e, 0xc421084fc0, 0xc420f2e940)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aee480, 0xc420887f50, 0xc4202af830, 0x5, 0x5, 0x5, 0xc421022d00, 0xc421084f00, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aee480, 0xc420887f50, 0xc4202af830, 0x5, 0x5, 0xa, 0x0, 0xa)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc421084f00, 0xc4202af830, 0x5, 0x5, 0x0, 0xc420021970, 0xc420021900)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4202af820, 0x7fffffff, 0xc420886330, 0xc400000001, 0x3001700, 0x0, 0x0, 0xc420475b00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4202af820, 0x2b04340, 0x3000498, 0xc421084f00, 0x2af6f00, 0x3000498, 0x1bc7460, 0xc421a6ccd0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc4210829a0, 0x1bc7460, 0xc421a6ccd0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc4203972b0, 0x2b04b00, 0xc420887fb0, 0xc420d52300)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:194 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2b0faa0, 0xc4203972b0, 0xc4209e0ea0, 0x2b04b00, 0xc420887fb0, 0xc420d52300, 0xc420c43f70, 0x6c2349)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x35
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x2b04b00, 0xc420887fb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xc0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420397340, 0xc421b90090, 0xc420011980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45544 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc421090680, 0xc42061c880, 0xc4202b85c0, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2bb
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2b04b00, 0xc420b308d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cef30, 0xc421b90090, 0xc420b301e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46034 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215eca80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215eca80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd630, 0xc4214c6d80, 0xc4202dd620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45712 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc40000000e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847bf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abdb0, 0xc420df2ea0, 0xc4202abda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45991 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000008)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dcdf0, 0xc4214c6d80, 0xc4202dcdd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45541 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc421022d00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1227 +0x148
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:292 +0xd2e

goroutine 45724 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc40000003a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc42089a930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc42089a930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42033a000, 0xc420df2ea0, 0xc4202abff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45327 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2b04b00, 0xc420887ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420356500, 0xc421b90090, 0xc4202c8960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45748 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420b2bae0, 0xc420339c70, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc420884720, 0xc420339c70, 0x5, 0x5, 0x70df4e, 0xc4212b20c0, 0xc421195a40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aee480, 0xc420884720, 0xc420339c70, 0x5, 0x5, 0x5, 0xc421022d00, 0xc4212b2000, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aee480, 0xc420884720, 0xc420339c70, 0x5, 0x5, 0xa, 0x0, 0xa)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4212b2000, 0xc420339c70, 0x5, 0x5, 0x0, 0xc42001e070, 0xc42001e000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc420339c60, 0x7fffffff, 0xc421177bf0, 0xc400000001, 0x3001700, 0x0, 0x0, 0xc42006cc00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc420339c60, 0x2b04340, 0x3000498, 0xc4212b2000, 0x2af6f00, 0x3000498, 0x1bc7460, 0xc421523090, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42109d8c0, 0x1bc7460, 0xc421523090, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc42030d830, 0x2b04b00, 0xc420884810, 0xc421110c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:194 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2b0faa0, 0xc42030d830, 0xc42086d520, 0x2b04b00, 0xc420884810, 0xc421110c00, 0xc420a58770, 0x6c2349)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x35
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x2b04b00, 0xc420884810)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xc0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030d900, 0xc421b90090, 0xc4201fd640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45669 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000018)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847fb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847fb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ab4d0, 0xc420df2ea0, 0xc4202ab4c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45987 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000004)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec1b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec1b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dcd00, 0xc4214c6d80, 0xc4202dccf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45705 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000047)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abcc0, 0xc420df2ea0, 0xc4202abcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45249 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).controller(0xc421082840)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:1011 +0x148
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:220 +0x8be

goroutine 44785 [IO wait]:
internal/poll.runtime_pollWait(0x7f76cad7a970, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420cad798, 0x72, 0xc42150ec00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420cad798, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420cad780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420cad780, 0xc4204a02c0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc420be21a0, 0xb29269, 0x6f1610, 0xc42150ee28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc420be21a0, 0x1d98cb8, 0xc420656280, 0x2b10460, 0xc4204a02c0)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc420656280, 0x2b03140, 0xc420be21a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2b04b00, 0xc420cf7bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ced60, 0xc421b90090, 0xc420338080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45328 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2b04b00, 0xc420887d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1136 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420356520, 0xc421b90090, 0xc420356510)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46005 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000016)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd130, 0xc4214c6d80, 0xc4202dd120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46019 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000024)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd410, 0xc4214c6d80, 0xc4202dd400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45546 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2b04b00, 0xc420b30960)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1136 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cef80, 0xc421b90090, 0xc4203cef70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46014 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc40000001f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd2f0, 0xc4214c6d80, 0xc4202dd2e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45986 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dcce0, 0xc4214c6d80, 0xc4202dccd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 30156 [select, 4 minutes, locked to thread]:
runtime.gopark(0x1d99860, 0x0, 0x1cb9162, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc42046e750, 0xc4204302a0)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 45951 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2b04b00, 0xc4215ec0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dcc80, 0xc4214c6d80, 0xc42029d300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46000 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000011)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd030, 0xc4214c6d80, 0xc4202dd010)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45913 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1.1(0xc421022340, 0xc4203be8e0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1083
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1083 +0x765

goroutine 45706 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000048)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abce0, 0xc420df2ea0, 0xc4202abcd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46016 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd350, 0xc4214c6d80, 0xc4202dd330)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45994 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc40000000b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dce70, 0xc4214c6d80, 0xc4202dce60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45697 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc42089a6f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc42089a6f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abbc0, 0xc420df2ea0, 0xc4202abbb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45660 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42077f180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff

goroutine 46010 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc40000001b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd1f0, 0xc4214c6d80, 0xc4202dd1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45329 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2b04b00, 0xc420ab8e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1174 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420356540, 0xc421b90090, 0xc420356530)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45704 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000046)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847a10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847a10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abca0, 0xc420df2ea0, 0xc4202abc90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45989 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000006)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dcd90, 0xc4214c6d80, 0xc4202dcd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45646 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b04b00, 0xc420610d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202903f0, 0xc420df2ea0, 0xc42038c580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46053 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2b04b00, 0xc4215ece10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1308 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd970, 0xc4214c6d80, 0xc4215c4870)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45245 [IO wait]:
internal/poll.runtime_pollWait(0x7f76cad7adf0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42033e698, 0x72, 0xc420a89c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42033e698, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc42033e680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc42033e680, 0xc4206ba1e0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4206ba1c8, 0xb29269, 0x6f1610, 0xc420a89e28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4206ba1c8, 0x1d98cb8, 0xc420b34780, 0x2b10460, 0xc4206ba1e0)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc420b34780, 0x2b03140, 0xc4206ba1c8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2b04b00, 0xc420887920)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420396cb0, 0xc421b90090, 0xc4202af200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45659 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420f584e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 45685 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000028)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc42089a390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc42089a390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ab980, 0xc420df2ea0, 0xc4202ab970)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46055 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd9a0, 0xc4214c6d80, 0xc4202dd990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45701 [semacquire]:
sync.runtime_notifyListWait(0xc420447890, 0xc400000043)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420447880)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420df3200, 0x2b04b00, 0xc420847950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc420847950)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202abc40, 0xc420df2ea0, 0xc4202abc30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 46017 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc400000022)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202dd380, 0xc4214c6d80, 0xc4202dd360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45665 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2b04b00, 0xc420847e30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ab450, 0xc420df2ea0, 0xc4203287a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45995 [semacquire]:
sync.runtime_notifyListWait(0xc4205160d0, 0xc40000000c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4205160c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4214c6e10, 0x2b04b00, 0xc4215ec330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b04b00, 0xc4215ec330)
	/go/src/g

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 3, 2017
@a-robinson
Copy link
Contributor

#18554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@a-robinson @cockroach-teamcity