Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_missing_log_output.tcl #19397

Closed
cockroach-teamcity opened this issue Oct 20, 2017 · 0 comments
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_missing_log_output.tcl #19397

cockroach-teamcity opened this issue Oct 20, 2017 · 0 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#383703:

--- FAIL: test/TestRefreshPendingCommands (0.000s)
Test ended in panic.

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I171020 06:06:17.601051 45394 storage/store.go:1213  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
W171020 06:06:17.601180 45481 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I171020 06:06:17.601225 45394 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:42683" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171020 06:06:17.601620 45480 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171020 06:06:17.605250 45394 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171020 06:06:17.605390 45394 storage/store.go:1213  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171020 06:06:17.605439 45394 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:56488" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171020 06:06:17.605539 45514 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:42683
W171020 06:06:17.609691 45394 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171020 06:06:17.609810 45394 storage/store.go:1213  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171020 06:06:17.609861 45394 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:57489" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171020 06:06:17.610355 44166 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:42683
I171020 06:06:17.613167 45233 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 8ee72d90 at index 15
I171020 06:06:17.615970 45233 storage/store.go:3550  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 3ms
I171020 06:06:17.616166 45724 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=8ee72d90, encoded size=4652, 1 rocksdb batches, 5 log entries)
I171020 06:06:17.616479 45724 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171020 06:06:17.616838 45233 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171020 06:06:17.618037 45766 storage/replica.go:3080  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171020 06:06:17.618708 45769 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot c103aaeb at index 17
I171020 06:06:17.619213 45769 storage/store.go:3550  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 0ms
I171020 06:06:17.619420 45610 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=c103aaeb, encoded size=5699, 1 rocksdb batches, 7 log entries)
I171020 06:06:17.619764 45610 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171020 06:06:17.620213 45769 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171020 06:06:17.621113 45795 storage/raft_transport.go:455  raft transport stream to node 1 established
I171020 06:06:17.622388 45612 storage/replica.go:3080  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171020 06:06:17.896252 45801 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171020 06:06:17.896491 45500 storage/store.go:3288  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171020 06:06:17.896517 45799 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I171020 06:06:17.897117 45752 storage/raft_transport.go:455  raft transport stream to node 3 established
W171020 06:06:17.897493 45756 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171020 06:06:17.897573 45756 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171020 06:06:17.897672 45754 storage/store.go:3288  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I171020 06:06:17.898160 45757 storage/raft_transport.go:455  raft transport stream to node 3 established
W171020 06:06:17.900087 45627 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171020 06:06:17.900226 45759 storage/store.go:3288  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I171020 06:06:17.900481 45394 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
W171020 06:06:17.903804 45883 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
SIGQUIT: quit
PC=0x713e81 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x316cdb0, 0x0, 0x0, 0x0, 0xc400000000, 0x0, 0x0, 0x0, 0x7ffd20c61400, 0x6c53db, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21
runtime.futexsleep(0x316cdb0, 0x0, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0x316cdb0)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stopm()
	/usr/local/go/src/runtime/proc.go:1670 +0xe5
runtime.gcstopm()
	/usr/local/go/src/runtime/proc.go:1874 +0xb7
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2209 +0x2c0
runtime.goschedImpl(0xc425ace480)
	/usr/local/go/src/runtime/proc.go:2323 +0xff
runtime.gopreempt_m(0xc425ace480)
	/usr/local/go/src/runtime/proc.go:2351 +0x36
runtime.newstack(0x0)
	/usr/local/go/src/runtime/stack.go:1042 +0x2d1
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:415 +0x86

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201c43c0, 0x1de8e3c, 0x1a, 0x1eaa210, 0x743601)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201c43c0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201c43c0, 0xc420599b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4201a9100, 0x2c83200, 0x1dc, 0x1dc, 0xc420599c98)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc4208aff18, 0xc420599d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420599f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1010 +0xdb

goroutine 6 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x316c0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 15 [syscall, 4 minutes]:
os/signal.signal_recv(0x6972707265676e69)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45879 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc42047cc00, 0x2ca4c80, 0xc420a61920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3730 +0x1a3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x2ca4c80, 0xc420a61920)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3716 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf4b0, 0xc425bf3170, 0xc4203cf4a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45511 [IO wait]:
internal/poll.runtime_pollWait(0x7fb89fb12bb0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42367a818, 0x72, 0xffffffffffffff00, 0x2c92040, 0x2c868f0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42367a818, 0xc4260fa000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42367a800, 0xc4260fa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc42367a800, 0xc4260fa000, 0x8000, 0x8000, 0x6b9d8c, 0xc425dfd800, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42099a128, 0xc4260fa000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42089dbc0, 0xc42092e1f8, 0x9, 0x9, 0xc4229d2a90, 0xc4229d2b18, 0xb6c067)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2c88640, 0xc42089dbc0, 0xc42092e1f8, 0x9, 0x9, 0x9, 0xc42070c5d8, 0xc4207a44b0, 0x4b)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c88640, 0xc42089dbc0, 0xc42092e1f8, 0x9, 0x9, 0x4b, 0x4b, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42092e1f8, 0x9, 0x9, 0x2c88640, 0xc42089dbc0, 0x0, 0x4b00000000, 0x4b, 0x4b)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42092e1c0, 0x4b, 0x4b, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc426108690, 0xc4261086c0, 0xc4261086c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:613 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc425b7e300, 0xc426108840, 0x1eab5b8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:389 +0x30d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4200ba280, 0x2caf240, 0xc425b7e300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:548 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4200ba280, 0x2cb08c0, 0xc42099a128, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:541 +0x463
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4200ba280, 0x2cb08c0, 0xc42099a128)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:508 +0x497
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:477 +0x5b9

goroutine 45829 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ced00, 0xc425bf3170, 0xc4203cecf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45544 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc40000003c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426109170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426109170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204384b0, 0xc42373a3f0, 0xc420438490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45639 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc425dc0000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1318 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:293 +0xdd2

goroutine 45560 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc40000004c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2cf30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2cf30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438700, 0xc42373a3f0, 0xc4204386f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45564 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000010)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2cff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2cff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438780, 0xc42373a3f0, 0xc420438770)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45553 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000045)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2cd80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2cd80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438620, 0xc42373a3f0, 0xc420438610)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45869 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000018)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf2e0, 0xc425bf3170, 0xc4203cf2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45871 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc40000001a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61290)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61290)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf340, 0xc425bf3170, 0xc4203cf320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45838 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cee30, 0xc425bf3170, 0xc4203cee20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45866 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000001)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf260, 0xc425bf3170, 0xc4203cf250)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45854 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000009)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf0e0, 0xc425bf3170, 0xc4203cf0d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45824 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ceae0, 0xc425bf3170, 0xc4203cead0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45532 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000029)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426108e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426108e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204382d0, 0xc42373a3f0, 0xc4204382c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45522 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4209e4000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1093 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:805 +0x808

goroutine 45857 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc40000000c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf140, 0xc425bf3170, 0xc4203cf130)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45527 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2ca4c80, 0xc426108d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438230, 0xc42373a3f0, 0xc420372360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45761 [runnable]:
time.startTimer(0xc420a6a488)
	/usr/local/go/src/runtime/time.go:70 +0x3a
time.(*Timer).Reset(0xc420a6a480, 0x12a05f200, 0xc420a6a480)
	/usr/local/go/src/time/sleep.go:128 +0x83
github.com/cockroachdb/cockroach/pkg/util/timeutil.(*Timer).Reset(0xc4204370e0, 0x12a05f200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/timeutil/timer.go:81 +0xeb
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc420309340, 0x2ca4c80, 0xc420818b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:252 +0xbb
github.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1(0x2ca4c80, 0xc420818b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:237 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce220, 0xc425bf3170, 0xc4203ce210)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45528 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000020)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426108d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426108d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438250, 0xc42373a3f0, 0xc420438240)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45891 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1.1(0xc420378d00, 0xc42033a190, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077 +0x765

goroutine 45402 [IO wait]:
internal/poll.runtime_pollWait(0x7fb89fb12eb0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420111798, 0x72, 0xffffffffffffff00, 0x2c92040, 0x2c868f0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420111798, 0xc42096a000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420111780, 0xc42096a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420111780, 0xc42096a000, 0x8000, 0x8000, 0x869, 0x0, 0xc4209ffcb0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4238ba308, 0xc42096a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420101740, 0xc422a801f8, 0x9, 0x9, 0xc4209ffd50, 0xb6d4e6, 0xc4209fc3f0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2c88640, 0xc420101740, 0xc422a801f8, 0x9, 0x9, 0x9, 0x0, 0x0, 0xc4209ffe5f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c88640, 0xc420101740, 0xc422a801f8, 0x9, 0x9, 0x86039685b01, 0xbd00000000000860, 0x860)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc422a801f8, 0x9, 0x9, 0x2c88640, 0xc420101740, 0x0, 0x0, 0xc425f0e400, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc422a801c0, 0x2c953c0, 0xc42295dd70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc42295dd40, 0xc42295dd70, 0xc42295dd70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:613 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420379860)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1195 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:261 +0xbf8

goroutine 45313 [chan receive]:
testing.(*T).Run(0xc425bd00f0, 0x0, 0x0, 0xc425ee6040, 0x7ac57d)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands(0xc425bd00f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1010 +0xfe
testing.tRunner(0xc425bd00f0, 0x1eaa210)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 45719 [IO wait]:
internal/poll.runtime_pollWait(0x7fb89fb127f0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4206d4b18, 0x72, 0xffffffffffffff00, 0x2c92040, 0x2c868f0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4206d4b18, 0xc423814000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4206d4b00, 0xc423814000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4206d4b00, 0xc423814000, 0x8000, 0x8000, 0x6c76dc, 0xc422b0ae70, 0xc42366caf0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc422ac00f8, 0xc423814000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc42051bf80, 0xc4203f90d8, 0x9, 0x9, 0xc42366ca90, 0xc42366cb18, 0xb6c067)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2c88640, 0xc42051bf80, 0xc4203f90d8, 0x9, 0x9, 0x9, 0xc422b0aea8, 0xc422b18140, 0x4b)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c88640, 0xc42051bf80, 0xc4203f90d8, 0x9, 0x9, 0x4b, 0x4b, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203f90d8, 0x9, 0x9, 0x2c88640, 0xc42051bf80, 0x0, 0x4b00000000, 0x4b, 0x4b)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203f90a0, 0x4b, 0x4b, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc4237c3f20, 0xc4237c3f50, 0xc4237c3f50, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:613 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc425aea780, 0xc423912180, 0x1eab5b8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:389 +0x30d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc426114500, 0x2caf240, 0xc425aea780)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:548 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc426114500, 0x2cb08c0, 0xc422ac00f8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:541 +0x463
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc426114500, 0x2cb08c0, 0xc422ac00f8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:508 +0x497
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:477 +0x5b9

goroutine 45395 [chan receive]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x2ca4c80, 0xc42295c960)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425ee60e0, 0xc422b04000, 0xc4203a0400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45568 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000014)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2d0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2d0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438810, 0xc42373a3f0, 0xc420438800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45758 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func4(0x2cb0800, 0xc425dc0000, 0xc422aaa700, 0xc422a109a0, 0xc4237ee700)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:260 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:259 +0xe23

goroutine 45580 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000021)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2d2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2d2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438ab0, 0xc42373a3f0, 0xc420438aa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45841 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ceea0, 0xc425bf3170, 0xc4203cee90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45627 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc420abaa00, 0xc4203a19b0, 0x5, 0x5, 0x5a, 0xc422929b48, 0x6e0548)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:130 +0x28b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420abaa00, 0xc4203a19b0, 0x5, 0x5, 0x3, 0xc42001e000, 0xc42001e070)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:119 +0x67
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc422b0af60, 0xc4203a19b0, 0x5, 0x5, 0x72bace, 0xc422b1f0c0, 0xc4209b0900)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:395 +0x55
io.ReadAtLeast(0x2c8e040, 0xc422b0af60, 0xc4203a19b0, 0x5, 0x5, 0x5, 0xc425aea780, 0xc422b1f000, 0x5)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c8e040, 0xc422b0af60, 0xc4203a19b0, 0x5, 0x5, 0x1090cb6, 0x2c88700, 0xc422929cc8)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc422b1f000, 0xc4203a19b0, 0x5, 0x5, 0xb82d56, 0x1b4ed80, 0x5)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:379 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4203a19a0, 0x7fffffff, 0x1eab600, 0xc4237be910, 0xc42020ada8, 0xc422929d40, 0x6, 0xc42020ad80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:266 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4203a19a0, 0x2ca4480, 0x31b00b8, 0xc422b1f000, 0x2c96cc0, 0x31b00b8, 0x1ce20c0, 0xc4203a1a00, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:352 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc422b046c0, 0x1ce20c0, 0xc4203a1a00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:661 +0x10e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc425ee7750, 0xc422b0b020, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:396 +0x62
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2cb0620, 0xc425ee7750, 0xc425bd0870, 0x2ca4c80, 0xc422b0afc0, 0x18, 0xe61e80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:326 +0x83
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x2ca4c80, 0xc422b0afc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:349 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425ee7770, 0xc422b04000, 0xc422b0af90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45851 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000006)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf080, 0xc425bf3170, 0xc4203cf070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45856 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc40000000b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf120, 0xc425bf3170, 0xc4203cf110)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45819 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce840, 0xc425bf3170, 0xc4203ce800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45850 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000005)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf060, 0xc425bf3170, 0xc4203cf050)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45518 [running]:
	goroutine running on other thread; stack unavailable
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:550 +0xa1

goroutine 45847 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000002)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a60e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a60e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf000, 0xc425bf3170, 0xc4203cefa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45644 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2ca4c80, 0xc4209fc690)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1174 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42041d280, 0xc422b04000, 0xc42041d250)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45870 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000019)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61260)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf310, 0xc425bf3170, 0xc4203cf2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45864 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc400000013)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf220, 0xc425bf3170, 0xc4203cf210)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45514 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc425d5ef70, 0x2ca4c80, 0xc426108ae0, 0xc425b7e000, 0x2caff60, 0xc4202c9110, 0xc422b04000, 0xc42041f300, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x350
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x2ca4c80, 0xc426108ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x3cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c90d0, 0xc422b04000, 0xc425c5c100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45845 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cef60, 0xc425bf3170, 0xc4203cef40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45836 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cedf0, 0xc425bf3170, 0xc4203cede0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45537 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426108f30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426108f30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438370, 0xc42373a3f0, 0xc420438360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45875 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc40000001e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf430, 0xc425bf3170, 0xc4203cf420)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45543 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc40000003b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426109140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426109140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438480, 0xc42373a3f0, 0xc420438460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45406 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc42369a000, 0xc422958540, 0xc4202d7200, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2bb
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2ca4c80, 0xc422b4a060)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425ee6980, 0xc422b04000, 0xc42295dec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45540 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc400000038)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc426109080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc426109080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204383f0, 0xc42373a3f0, 0xc4204383e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45524 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2ca4c80, 0xc426108930)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204381d0, 0xc422b04000, 0xc420372300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45405 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42083a000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1093 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:805 +0x808

goroutine 45822 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cea60, 0xc425bf3170, 0xc4203cea20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45859 [semacquire]:
sync.runtime_notifyListWait(0xc420a6b310, 0xc40000000e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420a6b300)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc425bf3200, 0x2ca4c80, 0xc420a61050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420a61050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203cf180, 0xc425bf3170, 0xc4203cf170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45596 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2ca4c80, 0xc422b2d5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1312 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438df0, 0xc42373a3f0, 0xc422b5cb40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44166 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc425e0c9c0, 0x2ca4c80, 0xc4209fc2a0, 0xc425aea300, 0x2caff60, 0xc420328200, 0xc422b04000, 0xc420314870, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x350
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x2ca4c80, 0xc4209fc2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x3cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420328120, 0xc422b04000, 0xc42039c140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45826 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ceb80, 0xc425bf3170, 0xc4203ceb70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45833 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ced90, 0xc425bf3170, 0xc4203ced80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45640 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc422a0e000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1093 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:805 +0x808

goroutine 45832 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ced70, 0xc425bf3170, 0xc4203ced60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45812 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc422b04000, 0x2ca4c00, 0xc4200160e8, 0xc4203e6630, 0x22, 0x0, 0x0, 0xc4203ce2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0x133

goroutine 45561 [semacquire]:
sync.runtime_notifyListWait(0xc42063d790, 0xc40000004d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42063d780)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42373a480, 0x2ca4c80, 0xc422b2cf60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc422b2cf60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420438720, 0xc42373a3f0, 0xc420438710)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

gorou

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 20, 2017
tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
acceptance: de-flake test_missing_log_output
09ce934 
Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
d0e58d2 
...by filtering out all GRPC log messages.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
storage: de-flake TestRefreshPendingCommands
1b44843 
The test ran a for loop without preemption points. The loop checked a
condition that would only become true after another goroutine had been
scheduled and carried out its job.

If, with only few cores (four in my case) GC kicked in before that other
goroutine got scheduled, that loop would just run hot forever until the
test timed out, and the resulting stack dump looked quite unhelpful.

Add a small sleep so the runtime can preempt the goroutine.

The issue was harder to run into when stressing only the test, since there
was less garbage available at that point. Adding some print statements,
I accidentally made it much more likely.

Previously flaked (got stuck) within <500iters, now ran past 1.5k without
problems.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Touches cockroachdb#19367.
Fixes cockroachdb#18554.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
7130cb9 
...by filtering out all GRPC log messages. The specific offender is
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L937; this is
the same error message that we throttle in the grpcutil log bridge. It
is a warning that reads:

> "grpc: addrConn.resetTransport failed to create client transport:..."

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
7dae7e6 
...by filtering out all GRPC log messages. The specific offender is
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L937; this is
the same error message that we throttle in the grpcutil log bridge. It
is a warning that reads:

> grpc: addrConn.resetTransport failed to create client transport: ...

Turns out there are other offenders, so we just filter out any log messages
that originates from GRPC. Another example is logged from
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L696 and reads:

> Failed to dial ...: ...; please retry.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity