Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands #18561

Closed
cockroach-teamcity opened this issue Sep 18, 2017 · 1 comment
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands #18561

cockroach-teamcity opened this issue Sep 18, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#354599:

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I170918 19:28:22.046784 44511 storage/store.go:1208  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
W170918 19:28:22.046914 44661 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W170918 19:28:22.046950 44662 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I170918 19:28:22.047140 44511 gossip/gossip.go:315  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:47644" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W170918 19:28:22.051740 44511 gossip/gossip.go:1261  [n2] no incoming or outgoing connections
I170918 19:28:22.051844 44511 storage/store.go:1208  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I170918 19:28:22.051884 44511 gossip/gossip.go:315  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:60159" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I170918 19:28:22.052212 44767 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:47644
W170918 19:28:22.123271 44511 gossip/gossip.go:1261  [n3] no incoming or outgoing connections
I170918 19:28:22.123401 44511 storage/store.go:1208  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I170918 19:28:22.123455 44511 gossip/gossip.go:315  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:52590" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I170918 19:28:22.123798 44888 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:47644
I170918 19:28:22.194037 44918 storage/replica_raftstorage.go:516  [s1,r1/1:/M{in-ax}] generated preemptive snapshot a65b25bf at index 15
I170918 19:28:22.194596 44918 storage/store.go:3500  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 0ms
I170918 19:28:22.194832 44921 storage/replica_raftstorage.go:722  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=a65b25bf, encoded size=5681, 1 rocksdb batches, 5 log entries)
I170918 19:28:22.195123 44921 storage/replica_raftstorage.go:728  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170918 19:28:22.195452 44918 storage/replica_command.go:3664  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I170918 19:28:22.196369 44925 storage/replica.go:3056  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I170918 19:28:22.197032 44928 storage/replica_raftstorage.go:516  [s1,r1/1:/M{in-ax}] generated preemptive snapshot a6994d9b at index 17
I170918 19:28:22.198043 44928 storage/store.go:3500  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms
I170918 19:28:22.198208 44799 storage/replica_raftstorage.go:722  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=a6994d9b, encoded size=7228, 1 rocksdb batches, 7 log entries)
I170918 19:28:22.198504 44799 storage/replica_raftstorage.go:728  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170918 19:28:22.198973 44928 storage/replica_command.go:3664  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I170918 19:28:22.199840 44537 storage/raft_transport.go:455  raft transport stream to node 1 established
I170918 19:28:22.201661 44908 storage/replica.go:3056  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W170918 19:28:22.344309 44965 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 19:28:22.344468 44953 storage/store.go:3237  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170918 19:28:22.344493 44951 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I170918 19:28:22.345054 44954 storage/raft_transport.go:455  raft transport stream to node 3 established
W170918 19:28:22.345451 44958 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 19:28:22.345538 44958 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 19:28:22.345641 44956 storage/store.go:3237  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170918 19:28:22.346117 44959 storage/raft_transport.go:455  raft transport stream to node 3 established
W170918 19:28:22.347954 45053 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W170918 19:28:22.347990 45054 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W170918 19:28:22.348561 45075 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 19:28:22.348698 44961 storage/store.go:3237  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170918 19:28:22.348809 44511 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
SIGQUIT: quit
PC=0x6f5151 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x2f89790, 0x0, 0x0, 0x0, 0x0, 0xc420a97000, 0x0, 0x0, 0x7ffc7a0deb38, 0x6a66ab, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21
runtime.futexsleep(0x2f89790, 0xc400000000, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0x2f89790)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stopm()
	/usr/local/go/src/runtime/proc.go:1670 +0xe5
runtime.findrunnable(0xc42001f300, 0x0)
	/usr/local/go/src/runtime/proc.go:2125 +0x4d2
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2245 +0x12c
runtime.park_m(0xc420454000)
	/usr/local/go/src/runtime/proc.go:2308 +0xb6
runtime.mcall(0x7ffc7a0decd0)
	/usr/local/go/src/runtime/asm_amd64.s:286 +0x5b

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201123c0, 0x1cab37e, 0x1a, 0x1d6ec40, 0x724901)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201123c0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201123c0, 0xc420595b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42021a180, 0x2a98620, 0x1d8, 0x1d8, 0x58)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc42058df18, 0xc420595d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420595f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1002 +0xdb

goroutine 6 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2f88ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1004 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:580 +0xbf

goroutine 14 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45027 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000002b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3adb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3adb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212d80, 0xc4205b0870, 0xc420212d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45042 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000003a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3be60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3be60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420213100, 0xc4205b0870, 0xc4202130f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44742 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000030)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce57a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce57a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374a10, 0xc424525680, 0xc420374a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44753 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc40000003b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce59b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce59b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374ca0, 0xc424525680, 0xc420374c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44715 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000015)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5200)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203744e0, 0xc424525680, 0xc4203744d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45040 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000038)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3bda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3bda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202130c0, 0xc4205b0870, 0xc4202130b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44710 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000010)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce50e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce50e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374420, 0xc424525680, 0xc420374400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44573 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2ab9820, 0xc422a566c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1120 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ceb00, 0xc424524120, 0xc4203ceaf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44592 [IO wait]:
internal/poll.runtime_pollWait(0x7f44bb92d970, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420496698, 0x72, 0xc424238c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420496698, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420496680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420496680, 0xc420b340d8, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc420b340a0, 0xb064f9, 0x6f0420, 0xc424238e28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc420b340a0, 0x1d6ff60, 0xc4228103c0, 0x2ac5680, 0xc420b340d8)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4228103c0, 0x2ab7e60, 0xc420b340a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2ab9820, 0xc4208a28a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42022ab20, 0xc424524120, 0xc420306da0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45033 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000031)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3b320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212f70, 0xc4205b0870, 0xc420212f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44891 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4207c7590, 0xc4203077b0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc4208a31d0, 0xc4203077b0, 0x5, 0x5, 0x70cd5e, 0xc4205a49c0, 0xc4240b24c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aa3220, 0xc4208a31d0, 0xc4203077b0, 0x5, 0x5, 0x5, 0xc422c9fa00, 0xc4205a4900, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aa3220, 0xc4208a31d0, 0xc4203077b0, 0x5, 0x5, 0xa, 0x0, 0xa)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4205a4900, 0xc4203077b0, 0x5, 0x5, 0x0, 0xc420020670, 0xc420020600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4203077a0, 0x7fffffff, 0xc4207e8e70, 0xc400000001, 0x2fcc3a0, 0x0, 0x0, 0xc420455080)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4203077a0, 0x2ab9060, 0x2fcb178, 0xc4205a4900, 0x2aabc60, 0x2fcb178, 0x1ba0e00, 0xc4245a2280, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc4208e2580, 0x1ba0e00, 0xc4245a2280, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc42022bfe0, 0x2ab9820, 0xc4208a3260, 0xc42072f380)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:194 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2ac3f40, 0xc42022bfe0, 0xc42412dd40, 0x2ab9820, 0xc4208a3260, 0xc42072f380, 0xc422a6e770, 0x6c1159)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x35
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x2ab9820, 0xc4208a3260)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xc0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420434040, 0xc424524120, 0xc420b90d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44685 [IO wait]:
internal/poll.runtime_pollWait(0x7f44bb92d8b0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42423be98, 0x72, 0xffffffffffffff00, 0x2aa70a0, 0x2a9bd10)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42423be98, 0xc422a64000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42423be80, 0xc422a64000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc42423be80, 0xc422a64000, 0x8000, 0x8000, 0xa9b197, 0xc420727180, 0xc4203fd904)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc420a98428, 0xc422a64000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420371e60, 0xc4207270d8, 0x9, 0x9, 0x1d70c30, 0xc422875d60, 0x69bfc2)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2a9dae0, 0xc420371e60, 0xc4207270d8, 0x9, 0x9, 0x9, 0xc420414320, 0x80000000006, 0x20)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2a9dae0, 0xc420371e60, 0xc4207270d8, 0x9, 0x9, 0x0, 0xc422875e28, 0x6c1af6)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4207270d8, 0x9, 0x9, 0x2a9dae0, 0xc420371e60, 0x0, 0xc400000000, 0xc422875e6d, 0xc422a3bbc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4207270a0, 0x2aaa460, 0xc422a3bbc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc422ce4690, 0xc422a3bbc0, 0xc422a3bbc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4229421a0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 44768 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func3(0x2ac55c0, 0xc422c9fa00, 0xc422c9f380, 0xc422878580, 0xc42055a900)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:255 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:254 +0xe23

goroutine 44747 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000035)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5890)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5890)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374ad0, 0xc424525680, 0xc420374ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44739 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc40000002d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce56e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce56e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203749b0, 0xc424525680, 0xc4203749a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44729 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000023)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374830, 0xc424525680, 0xc420374800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44722 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc40000001c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374720, 0xc424525680, 0xc420374710)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44761 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x2ab9820, 0xc422ce5b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1178 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374e30, 0xc424525680, 0xc420359380)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44696 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000042)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce4e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce4e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374080, 0xc424525680, 0xc420374070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44721 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc40000001b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374650, 0xc424525680, 0xc420374640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44888 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc42412dd40, 0x2ab9820, 0xc4208a2fc0, 0xc42072f380, 0x2ac3f40, 0xc42022bfe0, 0xc424524120, 0xc4203684d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x350
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x2ab9820, 0xc4208a2fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x3cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42022bf80, 0xc424524120, 0xc420b90bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45044 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000003c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3bf80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3bf80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420213140, 0xc4205b0870, 0xc420213130)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44762 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2ab9820, 0xc422ce5b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1307 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374e40, 0xc424525680, 0xc4229deaa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44996 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000000c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3a750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3a750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202124a0, 0xc4205b0870, 0xc420212470)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44790 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4245055c0, 0x2ab9820, 0xc420410db0, 0xbe682575832a291a, 0x49183ce7b, 0x2f86d00, 0xc420572040, 0xc424525680, 0x0, 0xc42001e000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:208 +0x226
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x2ab9820, 0xc420410db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:275 +0x335
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375690, 0xc424525680, 0xc42039e5e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44513 [semacquire]:
sync.runtime_notifyListWait(0xc420728230, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420728220)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420728140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 44699 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000045)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce4ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce4ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374110, 0xc424525680, 0xc420374100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44712 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374480, 0xc424525680, 0xc420374470)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 29941 [select, 4 minutes, locked to thread]:
runtime.gopark(0x1d70b08, 0x0, 0x1c91953, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc422cd1750, 0xc4204c6780)
	/usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 45032 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000030)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3af00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212f30, 0xc4205b0870, 0xc420212f10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45017 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000021)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3aba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3aba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212a80, 0xc4205b0870, 0xc420212a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45013 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000001d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3aa80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3aa80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212950, 0xc4205b0870, 0xc420212940)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45068 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ab9820, 0xc42075c540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420213bd0, 0xc4205b0870, 0xc420437660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44813 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2ab9820, 0xc4208a3290)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1158 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42022ade0, 0xc424524120, 0xc42022add0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45043 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000003b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3bef0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3bef0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420213120, 0xc4205b0870, 0xc420213110)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44569 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc422c9fa00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 44895 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc424142200, 0x2ac3fa0, 0xc420434210, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:198 +0x61d
github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x1c538a0, 0xc424142200, 0x2ac1c00, 0xc422ac6900, 0x2, 0x1c795a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:211 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420728280, 0x2ac3280, 0xc420536580, 0xc4205a4a00, 0xc4246aabd0, 0x2a7d760, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:891 +0x943
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420728280, 0x2ac3280, 0xc420536580, 0xc4205a4a00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:985 +0x14c1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc420350040, 0xc420728280, 0x2ac3280, 0xc420536580, 0xc4205a4a00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:551 +0x9f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:549 +0xa1

goroutine 45024 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000028)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212ca0, 0xc4205b0870, 0xc420212c70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45015 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc40000001f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212a30, 0xc4205b0870, 0xc420212a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44758 [semacquire]:
sync.runtime_Semacquire(0xc424525794)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc424525788)
	/usr/local/go/src/sync/waitgroup.go:131 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc424525710, 0x2ab9820, 0xc422ce5ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:173 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x2ab9820, 0xc422ce5ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3631 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374d60, 0xc424525680, 0xc420374d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44679 [IO wait]:
internal/poll.runtime_pollWait(0x7f44bb92df70, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42423bc98, 0x72, 0xc42003ec00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42423bc98, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc42423bc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc42423bc80, 0xc420a98420, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc420a98408, 0xb064f9, 0x6f0420, 0xc42003ee28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc420a98408, 0x1d6ff60, 0xc420729040, 0x2ac5680, 0xc420a98420)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc420729040, 0x2ab7e60, 0xc420a98408, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2ab9820, 0xc422ce4360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42028fc90, 0xc424524120, 0xc420358ee0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44702 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000048)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce4f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce4f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203741a0, 0xc424525680, 0xc420374180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44787 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ab9820, 0xc420411020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375660, 0xc424525680, 0xc42039e560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44711 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000011)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce5110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce5110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374460, 0xc424525680, 0xc420374430)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44764 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2ab9820, 0xc422ce5c50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1307 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374e60, 0xc424525680, 0xc4229df270)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45056 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc420112a50, 0x2ab9760, 0xc4209f7900, 0xc420112aa0, 0x1, 0xc420a46780, 0x3, 0x3, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1144 +0x69a
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc420112a50, 0x2ab9760, 0xc4209f7900, 0x1, 0xc420a46780, 0x3, 0x3, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:382 +0x2db
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc420112a50, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420992e00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:446 +0x17b
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc420112a50, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420992e00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:930 +0x4f9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc420112a50, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420992e00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:795 +0xae8
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc420112a50, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420992a80, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:597 +0x3ad
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc4209927e0, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4209929a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:444 +0x1f1
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc420436440, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4209929a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:545 +0x2ac
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc42070ab00, 0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:930 +0x315
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send-fm(0x2ab9760, 0xc4209f6dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:497 +0x83
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x2ab9760, 0xc4209f6dc0, 0xc42058f668, 0xc424589500, 0x0, 0xc42058f748)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:454 +0x101
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc42070ab00, 0x2ab9760, 0xc4209f6dc0, 0xc424589500, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:497 +0xd1
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLivenessAttempt.func1(0x2ab9760, 0xc4209f6dc0, 0xc42070ab00, 0x1, 0xc4203a6500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:692 +0x33f
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x2ab9760, 0xc4209f6dc0, 0xc42070ab00, 0xc4203a6500, 0x7, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:507 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc42070ab00, 0x2ab9760, 0xc4209f6dc0, 0xc4209f0101, 0xc420213330, 0xc422818700, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:741 +0xf6
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc420436440, 0x2ab9760, 0xc4209f6dc0, 0xc420436bc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:506 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLivenessAttempt(0xc422b6a700, 0x2ab9820, 0xc424146630, 0xc4201b2300, 0x0, 0xc424369bd8, 0xc420572050, 0x7b)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:668 +0xe4
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLiveness(0xc422b6a700, 0x2ab9820, 0xc424146630, 0xc4201b2300, 0x0, 0xc424369bd8, 0x2f86d00, 0x30003)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:643 +0xce
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).heartbeatInternal(0xc422b6a700, 0x2ab9820, 0xc424146630, 0x0, 0x4000000000000001, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:456 +0x3dd
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1.1(0xc422b6a700, 0xc422adbe88, 0x2faf080, 0x3b9aca00, 0x4000000000000000, 0x0, 0x0, 0xc422818600, 0xc424369e2f)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:353 +0x35b
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2ab9820, 0xc424146570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:364 +0x31c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202132e0, 0xc4205b0870, 0xc420a46230)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45020 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000024)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3ac60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3ac60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212b30, 0xc4205b0870, 0xc420212b20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44695 [semacquire]:
sync.runtime_notifyListWait(0xc422972190, 0xc400000041)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc422972180)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424525710, 0x2ab9820, 0xc422ce4e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422ce4e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420374060, 0xc424525680, 0xc420374020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44989 [semacquire]:
sync.runtime_notifyListWait(0xc4209f6c90, 0xc400000005)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4209f6c80)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4205b0900, 0x2ab9820, 0xc422b3a540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc422b3a540)
	/go/src/github.com/cockroachdb

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Sep 18, 2017
@jordanlewis
Copy link
Member

#18554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jordanlewis @cockroach-teamcity