Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_missing_log_output.tcl #19388

Closed
cockroach-teamcity opened this issue Oct 19, 2017 · 0 comments
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands, test/TestRefreshPendingCommands/#00, acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_missing_log_output.tcl #19388

cockroach-teamcity opened this issue Oct 19, 2017 · 0 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#383473:

--- FAIL: test/TestRefreshPendingCommands (0.000s)
Test ended in panic.

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I171019 20:59:25.049214 45386 storage/store.go:1213  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171019 20:59:25.049271 45386 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:36409" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171019 20:59:25.049440 45505 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171019 20:59:25.049446 45504 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171019 20:59:25.053241 45386 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171019 20:59:25.053406 45386 storage/store.go:1213  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171019 20:59:25.053461 45386 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:60771" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171019 20:59:25.053647 45543 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:36409
W171019 20:59:25.057450 45386 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171019 20:59:25.057589 45386 storage/store.go:1213  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171019 20:59:25.057637 45386 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:48724" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171019 20:59:25.057691 45683 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:36409
I171019 20:59:25.127698 45127 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 421c4961 at index 15
I171019 20:59:25.128983 45127 storage/store.go:3550  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 1ms
I171019 20:59:25.129204 45406 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=421c4961, encoded size=4652, 1 rocksdb batches, 5 log entries)
I171019 20:59:25.129556 45406 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171019 20:59:25.130039 45127 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171019 20:59:25.131406 45132 storage/replica.go:3080  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171019 20:59:25.132248 45646 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot a4338ac5 at index 17
I171019 20:59:25.132935 45646 storage/store.go:3550  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms
I171019 20:59:25.133231 45135 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=a4338ac5, encoded size=5699, 1 rocksdb batches, 7 log entries)
I171019 20:59:25.133601 45135 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171019 20:59:25.134097 45646 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171019 20:59:25.135091 45800 storage/raft_transport.go:455  raft transport stream to node 1 established
I171019 20:59:25.136402 45806 storage/replica.go:3080  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171019 20:59:25.279119 45813 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 20:59:25.279398 45789 storage/store.go:3288  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171019 20:59:25.279522 45813 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 20:59:25.279681 45787 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I171019 20:59:25.280096 45687 storage/raft_transport.go:455  raft transport stream to node 3 established
W171019 20:59:25.280295 45417 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 20:59:25.280844 45804 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W171019 20:59:25.280991 45417 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 20:59:25.281138 45689 storage/raft_transport.go:494  no handler found for store 1 in response range_id:1 from_replica:<node_id:3 store_id:3 replica_id:3 > to_replica:<node_id:1 store_id:1 replica_id:1 > union:<error:<message:"store 3 was not found" transaction_restart:NONE origin_node:0 detail:<store_not_found:<store_id:3 > > now:<> > > 
W171019 20:59:25.281199 45689 storage/raft_transport.go:494  no handler found for store 1 in response range_id:1 from_replica:<node_id:3 store_id:3 replica_id:3 > to_replica:<node_id:1 store_id:1 replica_id:1 > union:<error:<message:"store 3 was not found" transaction_restart:NONE origin_node:0 detail:<store_not_found:<store_id:3 > > now:<> > > 
W171019 20:59:25.281263 45802 storage/store.go:3288  [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W171019 20:59:25.281286 45800 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
I171019 20:59:25.281401 45386 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
W171019 20:59:25.281428 45898 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W171019 20:59:25.281450 45897 storage/store.go:1300  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
SIGQUIT: quit
PC=0x713e81 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x316cdb0, 0x0, 0x0, 0x0, 0x0, 0xc4206a8800, 0x0, 0x0, 0x7ffddb8819e8, 0x6c53db, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21
runtime.futexsleep(0x316cdb0, 0xc400000000, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0x316cdb0)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stopm()
	/usr/local/go/src/runtime/proc.go:1670 +0xe5
runtime.gcstopm()
	/usr/local/go/src/runtime/proc.go:1874 +0xb7
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2209 +0x2c0
runtime.park_m(0xc420000180)
	/usr/local/go/src/runtime/proc.go:2308 +0xb6
runtime.mcall(0x7ffddb881b10)
	/usr/local/go/src/runtime/asm_amd64.s:286 +0x5b

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4201b63c0, 0x1de8e3c, 0x1a, 0x1eaa210, 0x316a201)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4201b63c0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4201b63c0, 0xc4206c1b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4203749e0, 0x2c83200, 0x1dc, 0x1dc, 0xc4206c1c98)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc4206a3f18, 0xc4206c1d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc4206c1f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1010 +0xdb

goroutine 6 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x316c0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 24 [syscall, 4 minutes]:
os/signal.signal_recv(0x6769633856486f6e)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45388 [semacquire]:
sync.runtime_notifyListWait(0xc4200ad130, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4200ad120)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4200ad040)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45567 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000049)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266350, 0xc4208b8000, 0xc420266340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45390 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func1(0x2ca4c80, 0xc420bd0000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:49 +0x5f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042a9c0, 0xc4205a83f0, 0xc42446bbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45385 [chan receive]:
testing.(*T).Run(0xc4209b24b0, 0x0, 0x0, 0xc42042a0d0, 0x7ac57d)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands(0xc4209b24b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1010 +0xfe
testing.tRunner(0xc4209b24b0, 0x1eaa210)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 45557 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2ca4c80, 0xc423618ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1174 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266170, 0xc4205a83f0, 0xc420266150)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45853 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc40000001b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203011b0, 0xc420e65050, 0xc4203011a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45670 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2ca4c80, 0xc4206a11d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1136 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202865c0, 0xc4205a83f0, 0xc4202865b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45563 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000045)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202662d0, 0xc4208b8000, 0xc4202662c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45848 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000015)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203010a0, 0xc420e65050, 0xc420301080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45686 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc420c82280, 0xc4203aa530, 0x5, 0x5, 0x10a863d, 0xc420929b20, 0x6e0548)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:130 +0x28b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420c82280, 0xc4203aa530, 0x5, 0x5, 0x3, 0xc420025300, 0xc420025370)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:119 +0x67
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc4206a1380, 0xc4203aa530, 0x5, 0x5, 0x72bace, 0xc420d080c0, 0xc420ab6cc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:395 +0x55
io.ReadAtLeast(0x2c8e040, 0xc4206a1380, 0xc4203aa530, 0x5, 0x5, 0x5, 0xc4206009c0, 0xc420d08000, 0xc400000005)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c8e040, 0xc4206a1380, 0xc4203aa530, 0x5, 0x5, 0xa, 0x0, 0x8)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc420d08000, 0xc4203aa530, 0x5, 0x5, 0x0, 0xc420025370, 0xc420025300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:379 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4203aa520, 0x7fffffff, 0xc4209f4de0, 0xc400000001, 0x31b1440, 0x0, 0x0, 0xc42043be00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:266 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4203aa520, 0x2ca4480, 0x31b00b8, 0xc420d08000, 0x2c96cc0, 0x31b00b8, 0x1cd6ba0, 0xc4236051d0, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:352 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc422a9a000, 0x1cd6ba0, 0xc4236051d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:403 +0xe6
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc420266f10, 0x2ca4c80, 0xc4206a14d0, 0xc422a2cd80)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:193 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2caff60, 0xc420266f10, 0xc42281ad00, 0x2ca4c80, 0xc4206a14d0, 0xc422a2cd80, 0xc42058ff70, 0x6dfe89)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x35
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x2ca4c80, 0xc4206a14d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xc0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266fd0, 0xc4205a83f0, 0xc420ab6980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45868 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000028)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420c5d020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420c5d020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420301400, 0xc420e65050, 0xc4203013f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45612 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc40000003d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266a10, 0xc4208b8000, 0xc420266a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45850 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000017)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420301120, 0xc420e65050, 0xc420301110)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45906 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ca4c80, 0xc4209cc300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42029f1b0, 0xc420e65050, 0xc42038c2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45586 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc40000001c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc4236188d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc4236188d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266660, 0xc4208b8000, 0xc420266650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45596 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000026)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266810, 0xc4208b8000, 0xc420266800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45578 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000014)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618720)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618720)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202664b0, 0xc4208b8000, 0xc4202664a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45795 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc4209b2780, 0x2, 0xc423576de0, 0xc42083e300, 0x2cb05c0, 0xc42034c210, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:513 +0x52a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc4209b2780, 0x2, 0x2ca4c80, 0xc4236cfa40, 0x1, 0x0, 0xc422aba0a0, 0xc423576de0, 0xc42083e300, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:458 +0x32a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc4209b2780, 0x2ca4c80, 0xc4236cfa40, 0x2, 0xc423576de0, 0xc42083e300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0xd0
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func1.1(0x2ca4c80, 0xc4236cfa40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:587 +0x60
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42034c1e0, 0xc4205a83f0, 0xc4236cf9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45657 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ca4c80, 0xc423619a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203c2960, 0xc4208b8000, 0xc4204006e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45573 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc40000000f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266410, 0xc4208b8000, 0xc420266400)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45606 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266950, 0xc4208b8000, 0xc420266940)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45866 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000025)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420c5cfc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420c5cfc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203013c0, 0xc420e65050, 0xc4203013b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45684 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func4(0x2cb0800, 0xc4206009c0, 0xc422bb68c0, 0xc422a9a000, 0xc420d08000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:260 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:259 +0xe23

goroutine 45913 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range(0xc422986b30, 0xc4236c5630)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:328 +0xcf
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc422989c20, 0xc423605680)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:291 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x2ca4c80, 0xc4206da6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:268 +0x1f5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42029f250, 0xc420e65050, 0xc42038c3a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45653 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ca4c80, 0xc4206a0b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203c2920, 0xc4208b8000, 0xc420400660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45893 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc422986800, 0x2ca4c80, 0xc4206da300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3730 +0x205
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x2ca4c80, 0xc4206da300)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3716 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203018b0, 0xc420e65050, 0xc4203018a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45588 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc40000001e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266710, 0xc4208b8000, 0xc420266700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45910 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42029f210, 0xc420e65050, 0xc42038c340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45887 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc40000003c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420c5d2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420c5d2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203017d0, 0xc420e65050, 0xc4203017c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45891 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc40000003f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc4206da2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc4206da2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420301870, 0xc420e65050, 0xc420301860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45416 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc4209b2780, 0x2cb0620, 0xc4202d42d0, 0x2c94a80, 0xc4209b2780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:355 +0x215
github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x1d9b840, 0xc4209b2780, 0x2cadaa0, 0xc420cead80, 0x0, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:377 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420ed6500, 0x2caf240, 0xc420e0f800, 0xc420c60500, 0xc4209f4c60, 0x2c702c0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:916 +0x943
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420ed6500, 0x2caf240, 0xc420e0f800, 0xc420c60500, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1010 +0x14c1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4204224c0, 0xc420ed6500, 0x2caf240, 0xc420e0f800, 0xc420c60500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:552 +0x9f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:550 +0xa1

goroutine 45654 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ca4c80, 0xc4206a0b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203c2930, 0xc4208b8000, 0xc420400680)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45835 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000008)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51b90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420300e80, 0xc420e65050, 0xc420300e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45592 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000022)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266790, 0xc4208b8000, 0xc420266780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45583 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000019)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618810)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202665e0, 0xc4208b8000, 0xc4202665c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45590 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000020)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc4236189c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc4236189c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266750, 0xc4208b8000, 0xc420266740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45562 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000044)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202662b0, 0xc4208b8000, 0xc4202662a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45855 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc40000001f)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51fb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51fb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420301260, 0xc420e65050, 0xc420301250)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45634 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc423610100, 0x2caffc0, 0xc42029e050, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:198 +0x61d
github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x1d8ede0, 0xc423610100, 0x2cadaa0, 0xc4227e6090, 0xc4236c55c8, 0x1144571)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:210 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4200ad2c0, 0x2caf240, 0xc423548000, 0xc420c78200, 0xc42446a9f0, 0x2c67820, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:916 +0x943
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4200ad2c0, 0x2caf240, 0xc423548000, 0xc420c78200, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1010 +0x14c1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42363c030, 0xc4200ad2c0, 0x2caf240, 0xc423548000, 0xc420c78200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:552 +0x9f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:550 +0xa1

goroutine 45564 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000046)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618480)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202662f0, 0xc4208b8000, 0xc4202662e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45403 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc422d4ef50, 0xc42033a7d0, 0x5, 0x5, 0xc4206d9988, 0xc4206d9830, 0x6e0548)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:130 +0x28b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc422d4ef50, 0xc42033a7d0, 0x5, 0x5, 0x3, 0xc42001f300, 0xc42001f370)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:119 +0x67
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc42033dd10, 0xc42033a7d0, 0x5, 0x5, 0x72bace, 0xc4201b8ac0, 0xc4205c6fc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:395 +0x55
io.ReadAtLeast(0x2c8e040, 0xc42033dd10, 0xc42033a7d0, 0x5, 0x5, 0x5, 0xc423548000, 0xc4201b8a00, 0x5)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c8e040, 0xc42033dd10, 0xc42033a7d0, 0x5, 0x5, 0xc42001f300, 0xc4203ea0f0, 0x49)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4201b8a00, 0xc42033a7d0, 0x5, 0x5, 0xc420a3fb00, 0x1eab600, 0xc422d4f090)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:379 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42033a7c0, 0x7fffffff, 0x0, 0x81a, 0x0, 0x0, 0x6, 0xc420a3fb00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:266 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42033a7c0, 0x2ca4480, 0x31b00b8, 0xc4201b8a00, 0x2c96cc0, 0x31b00b8, 0x1cf24a0, 0xc422d4f360, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:352 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc420bac6c0, 0x1cf24a0, 0xc422d4f360, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:661 +0x10e
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc420368080, 0xc4206d9bb0, 0xa17480, 0xc423610140)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:229 +0x62
github.com/cockroachdb/cockroach/pkg/gossip.(Gossip_GossipServer).Recv-fm(0xc423610140, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:161 +0x2f
github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc423610100, 0x2ca4c80, 0xc42033de60, 0xc4235300c8, 0xc4205c6f00, 0xc4206d9f58, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:339 +0x474
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x2ca4c80, 0xc42033de60)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:161 +0x96
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203680e0, 0xc4205a83f0, 0xc4205c6f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45858 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000027)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc4206da0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc4206da0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203012c0, 0xc420e65050, 0xc4203012b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45888 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc40000003d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420c5d2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420c5d2f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420301810, 0xc420e65050, 0xc4203017e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45902 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1.1(0xc4209a3d40, 0xc42032e200, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077 +0x765

goroutine 45599 [semacquire]:
sync.runtime_notifyListWait(0xc420d21b50, 0xc400000029)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420d21b40)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208b8510, 0x2ca4c80, 0xc423618b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc423618b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266870, 0xc4208b8000, 0xc420266860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45398 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc423548000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:1005 +0x251
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:226 +0x8e3

goroutine 45683 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc42281ad00, 0x2ca4c80, 0xc4206a1260, 0xc422a2cd80, 0x2caff60, 0xc420266f10, 0xc4205a83f0, 0xc4204226b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x350
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x2ca4c80, 0xc4206a1260)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x3cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420266ed0, 0xc4205a83f0, 0xc420ab6680)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45396 [IO wait]:
internal/poll.runtime_pollWait(0x7faf55c2e730, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42020e098, 0x72, 0xffffffffffffff00, 0x2c92040, 0x2c868f0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc42020e098, 0xc420882000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42020e080, 0xc420882000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc42020e080, 0xc420882000, 0x8000, 0x8000, 0xd, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc423530000, 0xc420882000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc422744000, 0xc42035bb58, 0x9, 0x9, 0x10, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2c88640, 0xc422744000, 0xc42035bb58, 0x9, 0x9, 0x9, 0x7da409, 0x2ca4c00, 0xc4200160e8)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2c88640, 0xc422744000, 0xc42035bb58, 0x9, 0x9, 0x7da91b, 0x2ca4c00, 0xc4200160e8)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42035bb58, 0x9, 0x9, 0x2c88640, 0xc422744000, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42035bb20, 0xc422a74600, 0xc420262c70, 0xc422a74600, 0x1)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420bd0150, 0xc420262c70, 0xc420262c70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:613 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc423548000, 0xc420bd02d0, 0x1eab5b8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:389 +0x30d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4200ad2c0, 0x2caf240, 0xc423548000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:548 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc4200ad2c0, 0x2cb08c0, 0xc423530000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:541 +0x463
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc4200ad2c0, 0x2cb08c0, 0xc423530000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:508 +0x497
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:477 +0x5b9

goroutine 45656 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ca4c80, 0xc423619a10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203c2950, 0xc4208b8000, 0xc4204006c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45828 [semacquire]:
sync.runtime_notifyListWait(0xc420845450, 0xc400000001)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420845440)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e650e0, 0x2ca4c80, 0xc420d51a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ca4c80, 0xc420d51a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420300d80, 0xc420e65050, 0xc420300d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45796 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func4(0x2cb0800, 0xc42036d1e0, 0xc4228bf340, 0xc4200aaf20, 0xc422a74300)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:260 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:259 +0xe23

goroutine 45827 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2ca4c80, 0xc420d51a10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420300d50, 0xc420e65050, 0xc420306340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
\n```
Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 19, 2017
tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
acceptance: de-flake test_missing_log_output
09ce934 
Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
d0e58d2 
...by filtering out all GRPC log messages.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
storage: de-flake TestRefreshPendingCommands
1b44843 
The test ran a for loop without preemption points. The loop checked a
condition that would only become true after another goroutine had been
scheduled and carried out its job.

If, with only few cores (four in my case) GC kicked in before that other
goroutine got scheduled, that loop would just run hot forever until the
test timed out, and the resulting stack dump looked quite unhelpful.

Add a small sleep so the runtime can preempt the goroutine.

The issue was harder to run into when stressing only the test, since there
was less garbage available at that point. Adding some print statements,
I accidentally made it much more likely.

Previously flaked (got stuck) within <500iters, now ran past 1.5k without
problems.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Touches cockroachdb#19367.
Fixes cockroachdb#18554.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
7130cb9 
...by filtering out all GRPC log messages. The specific offender is
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L937; this is
the same error message that we throttle in the grpcutil log bridge. It
is a warning that reads:

> "grpc: addrConn.resetTransport failed to create client transport:..."

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
tamird pushed a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg @tamird
acceptance: de-flake test_missing_log_output
7dae7e6 
...by filtering out all GRPC log messages. The specific offender is
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L937; this is
the same error message that we throttle in the grpcutil log bridge. It
is a warning that reads:

> grpc: addrConn.resetTransport failed to create client transport: ...

Turns out there are other offenders, so we just filter out any log messages
that originates from GRPC. Another example is logged from
https://github.com/grpc/grpc-go/blob/v1.7.0/clientconn.go#L696 and reads:

> Failed to dial ...: ...; please retry.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-teamcity