Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestServer/version/6, test/TestServer/version/4, test/TestServer/version, test/TestServer/version/3, test/TestServer #17869

Closed
cockroach-teamcity opened this issue Aug 23, 2017 · 1 comment
Closed

teamcity: failed tests on master: test/TestServer/version/6, test/TestServer/version/4, test/TestServer/version, test/TestServer/version/3, test/TestServer #17869

cockroach-teamcity opened this issue Aug 23, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#332800:

--- FAIL: test/TestServer/version/6 (0.000s)
Test ended in panic.

------- Stdout: -------
W170823 20:41:36.066401 4877 sql/distsqlrun/server.go:220  [n1] version mismatch in flow request: 6; this node accepts 4 through 5
--- FAIL: test/TestServer/version/4 (0.000s)
Test ended in panic.

------- Stdout: -------
I170823 20:41:36.078452 4000 util/stop/stopper.go:475  quiescing; tasks left:
1      distsqlrun.ServerImpl: sync flow
1      [async] drain
E170823 20:41:36.078638 5027 util/log/crash_reporting.go:82  [n1] a panic has occurred!
E170823 20:41:36.079690 5027 util/log/crash_reporting.go:183  [n1] Reported as error eaa0bbaa843e41358553d6f46a46a8df
panic: send on closed channel [recovered]
	panic: send on closed channel

goroutine 5027 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc42160e090, 0x7f293e7e8060, 0xc422173080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:182 +0xb2
panic(0x193ad60, 0xc4213e2710)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*RowChannel).Push(0xc422194c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x279cce0, 0xc4221b4c60, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:320 +0x1d8
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).cancel(0xc422044000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:459 +0x222
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait(0xc422044000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:387 +0x11b
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ServerImpl).RunSyncFlow.func1(0x7f293d6c0000, 0xc422034180)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/server.go:347 +0xfd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc42160e090, 0x7f293e7e8060, 0xc422173080, 0x1b1d7c4, 0x20, 0xc421c6bba0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0xf7
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ServerImpl).RunSyncFlow(0xc421dcc200, 0x27bcde0, 0xc422174070, 0x27ad820, 0xc421dcc200)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/server.go:349 +0x1c2
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun._DistSQL_RunSyncFlow_Handler(0x1ab85a0, 0xc421dcc200, 0x27b9fc0, 0xc421da4000, 0xc4217ec320, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/api.pb.go:323 +0xb7
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420ebea00, 0x27bb280, 0xc4203c82c0, 0xc420406400, 0xc42236fc20, 0x2785980, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:891 +0xa4c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420ebea00, 0x27bb280, 0xc4203c82c0, 0xc420406400, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:985 +0x153f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4201eacd0, 0xc420ebea00, 0x27bb280, 0xc4203c82c0, 0xc420406400)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:551 +0xa9
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:552 +0xa1

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4202644e0, 0x1afe578, 0xa, 0x1bc16a8, 0xc420593c01)
	/usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc4202644e0)
	/usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc4202644e0, 0xc420593da0)
	/usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc4201ed740, 0x278fb40, 0x27, 0x27, 0x2c55f60)
	/usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc420639f20, 0x2c96a80)
	/usr/local/go/src/testing/testing.go:822 +0xfc
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun_test.TestMain(0xc420593f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/main_test.go:36 +0xda
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/_test/_testmain.go:124 +0xf7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2c57b20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:990 +0x77
created by github.com/cockroachdb/cockroach/pkg/util/log.init.1
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:575 +0xd9

goroutine 15 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 4000 [runnable]:
sync.runtime_notifyListWait(0xc420ffc650, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc420ffc640)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc42160e090, 0x7f293e91c0f8, 0xc4200130f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:477 +0x1b6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc42160e090, 0x7f293e91c0f8, 0xc4200130f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:420 +0x231
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.TestServer(0xc4218a81a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/server_test.go:150 +0xf11
testing.tRunner(0xc4218a81a0, 0x1bc16a8)
	/usr/local/go/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 4245 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012f)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348880, 0xc42160e090, 0xc420348870)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 50 [select, locked to thread]:
runtime.gopark(0x1bc4238, 0x0, 0x1af9199, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc42003bf50, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc42003bf50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 4250 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000136)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348980, 0xc42160e090, 0xc420348970)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4189 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000111)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031df00, 0xc42160e090, 0xc42031def0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4233 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012e)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203485f0, 0xc42160e090, 0xc4203485e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4215 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000011a)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203482c0, 0xc42160e090, 0xc4203482b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4244 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012d)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172b10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348830, 0xc42160e090, 0xc420348820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4247 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000134)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203488e0, 0xc42160e090, 0xc4203488b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4270 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x7f293e7e8060, 0xc420b008d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x42f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203601f0, 0xc42160e090, 0xc4217b4100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4234 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000130)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348610, 0xc42160e090, 0xc420348600)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4099 [select]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc42160e090, 0xc42207a720, 0xc4200a2150)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:125 +0x3b5
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:138 +0xd0

goroutine 4221 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000011c)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221726c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221726c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203483d0, 0xc42160e090, 0xc4203483c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4197 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000101)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc420c6e930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc420c6e930)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348040, 0xc42160e090, 0xc420348030)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4285 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420b5e060)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f08d0, 0xc42160e090, 0xc4202f08c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4242 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012a)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203487f0, 0xc42160e090, 0xc4203487c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4274 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x7f293e7e8060, 0xc420b9eab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1129 +0x185
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0780, 0xc42160e090, 0xc42238c260)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4102 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x7f293e7e8060, 0xc422172000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1362 +0xaf5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202eaa60, 0xc42160e090, 0xc42236ef30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4252 [semacquire]:
sync.runtime_Semacquire(0xc4217f0e04)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4217f0df8)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4217f0d80, 0x7f293e7e8060, 0xc422172c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:173 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x7f293e7e8060, 0xc422172c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3580 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203489c0, 0xc42160e090, 0xc4203489b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4293 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x7f293e7e8060, 0xc420ac63c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:974 +0x163a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f09d0, 0xc42160e090, 0xc42238c2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4230 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000137)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348570, 0xc42160e090, 0xc420348560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4295 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420ac6ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0a50, 0xc42160e090, 0xc4202f0a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4279 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420b00c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0810, 0xc42160e090, 0xc4202f0800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4192 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000013c)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc420b00db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc420b00db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031df90, 0xc42160e090, 0xc42031df60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4217 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000117)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221725d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221725d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348320, 0xc42160e090, 0xc420348310)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4959 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/getsentry/raven-go.(*Client).worker(0xc4203bb260)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/getsentry/raven-go/client.go:482 +0x6b
created by github.com/cockroachdb/cockroach/vendor/github.com/getsentry/raven-go.(*Client).Capture.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/getsentry/raven-go/client.go:541 +0x3e

goroutine 4218 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000118)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348360, 0xc42160e090, 0xc420348330)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4236 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000132)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203486e0, 0xc42160e090, 0xc4203486d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4238 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000135)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221729f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221729f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348730, 0xc42160e090, 0xc420348720)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4239 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000129)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348760, 0xc42160e090, 0xc420348740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4271 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x7f293e7e8060, 0xc420b00900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x42f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420360200, 0xc42160e090, 0xc4217b4120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4199 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000102)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221720f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221720f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348080, 0xc42160e090, 0xc420348070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4103 [semacquire]:
sync.runtime_notifyListWait(0xc420ebfef0, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc420ebfee0)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420ebfe00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:474 +0xd7
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:461 +0x4c7

goroutine 4294 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420ac6ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0a30, 0xc42160e090, 0xc4202f0a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4206 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000109)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348180, 0xc42160e090, 0xc420348170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4202 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000105)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203480f0, 0xc42160e090, 0xc4203480e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4205 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000107)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348160, 0xc42160e090, 0xc420348150)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4117 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x7f293e7e8060, 0xc42203d3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x70
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420314800, 0xc42160e090, 0xc42202d6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4277 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420b00ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f07d0, 0xc42160e090, 0xc4202f07c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4278 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420b00c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f07f0, 0xc42160e090, 0xc4202f07e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4276 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteSummaries.func1(0x7f293e7e8060, 0xc420b5e0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:737 +0x2c2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f07b0, 0xc42160e090, 0xc42238c280)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4065 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x7f293e7e8060, 0xc420b9ea50)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:112 +0x38e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0650, 0xc42160e090, 0xc42238c240)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4275 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x7f293e7e8060, 0xc420b9eae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:88 +0x16f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f07a0, 0xc42160e090, 0xc4202f0790)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4232 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012c)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221728d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221728d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203485c0, 0xc42160e090, 0xc4203485b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4101 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc421e72000, 0x7f293e7e8060, 0xc420c6e900)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:240 +0xb4d
github.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1(0x7f293e7e8060, 0xc420c6e900)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:224 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202ea730, 0xc42160e090, 0xc4202ea720)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4115 [semacquire]:
sync.runtime_notifyListWait(0xc4202b20f0, 0xc4000000f1)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4202b20e0)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4202b2000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:474 +0xd7
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:461 +0x4c7

goroutine 4308 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420ac72f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f0c50, 0xc42160e090, 0xc4202f0c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 5030 [runnable]:
sync.runtime_Semacquire(0xc422044214)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc422044208)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait.func1(0xc422044000, 0xc421ca2900)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:381 +0x33
created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:383 +0x7c

goroutine 4213 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000114)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221724b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221724b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348270, 0xc42160e090, 0xc420348260)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4191 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000010f)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031df50, 0xc42160e090, 0xc42031df40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4226 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000121)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221727b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221727b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203484b0, 0xc42160e090, 0xc420348490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4193 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000013d)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc420b00de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc420b00de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031dfb0, 0xc42160e090, 0xc42031dfa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4224 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000011f)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348460, 0xc42160e090, 0xc420348440)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4188 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000110)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc4221723c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc4221723c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031dee0, 0xc42160e090, 0xc42031ded0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4248 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000138)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348900, 0xc42160e090, 0xc4203488f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4251 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000139)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203489a0, 0xc42160e090, 0xc420348990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4207 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000108)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203481b0, 0xc42160e090, 0xc4203481a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4243 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000012b)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348810, 0xc42160e090, 0xc420348800)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4064 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x7f293e7e8060, 0xc420b9e990)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:659 +0x464
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f05e0, 0xc42160e090, 0xc42238c1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4246 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000131)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203488a0, 0xc42160e090, 0xc420348890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4266 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x7f293e7e8060, 0xc420b00750)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x42f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203601b0, 0xc42160e090, 0xc4217b4080)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4223 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc40000011e)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172720)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172720)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348420, 0xc42160e090, 0xc420348410)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4225 [semacquire]:
sync.runtime_notifyListWait(0xc4217bde90, 0xc400000120)
	/usr/local/go/src/runtime/sema.go:298 +0x10b
sync.(*Cond).Wait(0xc4217bde80)
	/usr/local/go/src/sync/cond.go:57 +0x89
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4217f0d80, 0x7f293e7e8060, 0xc422172780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0xac
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x7f293e7e8060, 0xc422172780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420348480, 0xc42160e090, 0xc420348470)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4291 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420ac6360)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f09a0, 0xc42160e090, 0xc4202f0990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4253 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc42057bc00, 0x7f293e7e8060, 0xc422172ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3596 +0x228
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x7f293e7e8060, 0xc422172ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3582 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203489f0, 0xc42160e090, 0xc4203489e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4316 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x7f293e7e8060, 0xc420b00cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:140 +0x227
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c3200, 0xc42160e090, 0xc421b4fd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 4286 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*distSQLPlanner).initRunners.func1(0x7f293e7e8060, 0xc420b004b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:85 +0x184
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202f08f0, 0xc42160e090, 0xc4202f08e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0xf7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xad

goroutine 3867 [runnable]:
sync.runtime_Semacquire(0xc4201eacdc)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4201eacd0)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420ebea00, 0x27bb280, 0xc4203c82c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:560 +0x170
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc420ebea00, 0x27bbdc0, 0xc420a28000, 0x27a0fe0, 0xc420f14630)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:540 +0x497
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc420ebea00, 0x27bd080, 0xc4221a4000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:507 +0x4ad
created by github.com/co

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Aug 23, 2017
@tbg tbg mentioned this issue Aug 23, 2017
Merged
@knz
Copy link
Contributor

knz commented Aug 24, 2017

Dup #17851.

@knz knz closed this as completed Aug 24, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@knz @cockroach-teamcity