Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jobs: TestRegistryLifecycle failed #62939

Closed
cockroach-teamcity opened this issue Apr 1, 2021 · 2 comments · Fixed by #62962
Closed

jobs: TestRegistryLifecycle failed #62939

cockroach-teamcity opened this issue Apr 1, 2021 · 2 comments · Fixed by #62962
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(jobs).TestRegistryLifecycle failed on master@6bc054abae5d7e177666e7cc3bc736bee3a99d8b:

Fatal error:

panic: send on closed channel [recovered]
	panic: send on closed channel

Stack:

goroutine 46667 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc005860980, 0x51482e0, 0xc00019e010)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:233 +0x126
panic(0x3e71f00, 0x5060370)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/cockroachdb/cockroach/pkg/sql.(*syncIEResultChannel).unblockWriter(0xc0014c82a0, 0x5148360, 0xc0014c8330, 0x0, 0xc002158300, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal_result_channel.go:186 +0xfd
github.com/cockroachdb/cockroach/pkg/sql.(*syncIEResultChannel).nextResult(0xc0014c82a0, 0x5148360, 0xc0014c8330, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal_result_channel.go:204 +0x8c
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal(0xc0056fa480, 0x5148360, 0xc0014c8330, 0x44e04df, 0x1f, 0x516d260, 0xc0014c82a0, 0x0, 0x44839dc, 0x4, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:788 +0xd7b
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx(0xc0056fa480, 0x51482a0, 0xc005809340, 0x44e04df, 0x1f, 0x0, 0x44839dc, 0x4, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:549 +0x1d0
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).processClaimedJobs(0xc0019fb1e0, 0x51482a0, 0xc005809340, 0x50fb9e0, 0xc005bf4180, 0x13ee7d8, 0xc0010e1d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:73 +0x285
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func6(0x51482a0, 0xc005809340, 0x50fb9e0, 0xc005bf4180)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:705 +0x8e
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1.1(0x51482a0, 0xc005809340)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:666 +0x282
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10(0x51482e0, 0xc00019e010)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:811 +0x2c6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc005860980, 0x51482e0, 0xc00019e010, 0x0, 0xc0058092c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

Log preceding fatal error

=== RUN   TestRegistryLifecycle/rollback
    jobs_test.go:205: Starting resume
    jobs_test.go:213: Exiting resume
    jobs_test.go:205: Starting resume
    jobs_test.go:213: Exiting resume
    jobs_test.go:255: Starting success
    jobs_test.go:259: Exiting success

More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestRegistryLifecycle PKG=./pkg/jobs TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 1, 2021
@spaskob spaskob assigned ajwerner and unassigned spaskob Apr 1, 2021
@ajwerner
Copy link
Contributor

ajwerner commented Apr 1, 2021

Cc @yuzefovich were not out of the woods yet.

ajwerner added a commit to ajwerner/cockroach that referenced this issue Apr 1, 2021
When finishing sending on the ieSyncResultChannel we do not need to and should
not close the channel used to unblock the sender. Doing so can result in a
panic if the reader is concurrently trying to unblock the sender. We could
close that channel but there's no obvious reason to.

Fixes cockroachdb#62939

Release note: None
@ajwerner
Copy link
Contributor

ajwerner commented Apr 1, 2021

Got it #62962

craig bot pushed a commit that referenced this issue Apr 1, 2021
61948: roachtest/cdc: update status when watching changefeed latency r=andreimatei a=andreimatei

When watching the latency, we were logging every second. This patch
reduces the frequency to 1/10s and makes the test also update its status
so that the current latency high-water mark is visible in the web UI.

Release note: None

62166: sql: add roach test / workload for testing connection latencies r=rafiss a=RichardJCai

Add roach test and workload test for testing connection latencies.

Release note: None

Making it a private test right now (not available through `cockroach workload`)

Need to also add it to teamcity and make sure the results are reported in roachperf, will open PR there.

Resolves #59394


62287: cdc: Improve avro encoder performance.  r=miretskiy a=miretskiy

Add avro encoder benchmarks.

Avoid expensive allocations (maps) when encoding datums.
Improve encoder performance by ~40%, and significantly reduce
memory allocations per op.

Release Notes: None

62883: sqlsmith: add support for ALTER TYPE ... DROP VALUE r=rafiss a=arulajmani

Now that `ALTER TYPE ... DROP VALUE` is a thing, let's add it to SQL
Smith.

Closes #62812

Release note: None

62922: roachtest/tpccbench: don't require workload binary r=andreimatei a=andreimatei

tpccbench was copying around the `workload` binary, but it then doesn't
use it; it uses `cockroach workload ...`. This patch removes the
copying.

Release note: None

62925: colexec: optimize CASE, AND, and OR projection operators r=yuzefovich a=yuzefovich

Previously, in CASE, AND, and OR projection operators we would always
allocate internal selection vectors of maximum capacity in the
constructor. This is inefficient when the query processes only small
data sets. This commit makes the allocations lazy and of tight sizes
based on the batch lengths.

Addresses: #62212

Release note: None

62926: bench,sql: add logScope to some benchmarks r=yuzefovich a=yuzefovich

This commits adds `defer log.Scope(b).Close(b)` to some benchmarks in
`bench`, `colexec`, and `rowexec` packages. Note that in some cases this
might not matter, but I didn't think through each carefully since it
shouldn't do any harm.

Release note: None

62962: sql: fix bug in close of ieSyncResultChannel r=yuzefovich a=ajwerner

When finishing sending on the ieSyncResultChannel we do not need to and should
not close the channel used to unblock the sender. Doing so can result in a
panic if the reader is concurrently trying to unblock the sender. We could
close that channel but there's no obvious reason to.

Fixes #62939

Release note: None

Co-authored-by: Andrei Matei <[email protected]>
Co-authored-by: richardjcai <[email protected]>
Co-authored-by: Yevgeniy Miretskiy <[email protected]>
Co-authored-by: arulajmani <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
Co-authored-by: Andrew Werner <[email protected]>
@craig craig bot closed this as completed in 5533dae Apr 1, 2021
yuzefovich pushed a commit to yuzefovich/cockroach that referenced this issue Apr 9, 2021
When finishing sending on the ieSyncResultChannel we do not need to and should
not close the channel used to unblock the sender. Doing so can result in a
panic if the reader is concurrently trying to unblock the sender. We could
close that channel but there's no obvious reason to.

Fixes cockroachdb#62939

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants