Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed timer queue processor goroutine leak in test #5857

Merged
merged 3 commits into from
Apr 4, 2024

Conversation

shijiesheng
Copy link
Member

@shijiesheng shijiesheng commented Apr 4, 2024

What changed?

  • Correctly stop timer queue processor on InitializedState Instead, stop timergate in unit test only and added a comment for improvement
  • added basic start stop test on transfer queue processor

Why?

Previously in initialized state, stopping timer queue processor will not stop the timer gate which has a daemon loop; although this shall never happen in production, it's causing go routine leak in unit tests

How did you test it?

unit test

Potential risks

Release notes

Documentation Changes

Copy link

codecov bot commented Apr 4, 2024

Codecov Report

Merging #5857 (211f5ce) into master (992b219) will increase coverage by 0.18%.
Report is 1 commits behind head on master.
The diff coverage is n/a.

❗ Current head 211f5ce differs from pull request most recent head 4bf720c. Consider uploading reports for the commit 4bf720c to get more accurate results

Additional details and impacted files
Files Coverage Δ
...ervice/history/queue/timer_queue_processor_base.go 69.92% <ø> (ø)

... and 11 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5fde92...4bf720c. Read the comment docs.

@@ -176,7 +176,10 @@ func (t *timerQueueProcessorBase) Start() {

func (t *timerQueueProcessorBase) Stop() {
if !atomic.CompareAndSwapInt32(&t.status, common.DaemonStatusStarted, common.DaemonStatusStopped) {
return
// initialized should also be stopped to stop TimerGate
if !atomic.CompareAndSwapInt32(&t.status, common.DaemonStatusInitialized, common.DaemonStatusStopped) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good catch but instead of solving problem here we can fix TimerGate itself. It should have Start/Stop functions and shouldn't create any goroutine in constructor. That would be cleaner solution I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think the code is buggy since lifecycle of timer gate is not handled correctly here.

  • if you just initialize the timer queue processor without starting it, timer gate needs to be stopped separately
  • if you start timer queue processor and stop it, timer gate is closed.

But to make minimum change. I'll handle this in the test case. The remaining concern is whether we do create timer gate without close it in the first scenario. But it should be addressed separately. I'll just add a comment.


func TestTransferQueueProcessor_RequireStartStop(t *testing.T) {
// some goroutine leak not from this test
defer goleak.VerifyNone(t)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great idea to verify leaks in these critical components with bunch of underlying goroutines!

@coveralls
Copy link

coveralls commented Apr 4, 2024

Pull Request Test Coverage Report for Build 018eaa23-245f-4679-971e-285ef54448f8

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 124 unchanged lines in 18 files lost coverage.
  • Overall coverage decreased (-0.06%) to 66.153%

Files with Coverage Reduction New Missed Lines %
service/history/task/transfer_standby_task_executor.go 2 86.19%
service/matching/taskReader.go 2 84.88%
common/task/parallel_task_processor.go 2 93.06%
common/persistence/execution_manager.go 2 83.54%
service/history/task/transfer_active_task_executor.go 2 72.7%
service/history/execution/mutable_state_builder.go 2 70.28%
service/matching/taskListManager.go 2 80.96%
service/matching/matcher.go 2 90.72%
common/persistence/historyManager.go 2 66.67%
service/history/task/fetcher.go 4 85.05%
Totals Coverage Status
Change from base Build 018eaa1f-d5d0-4534-bee2-9fc1730e903c: -0.06%
Covered Lines: 96806
Relevant Lines: 146337

💛 - Coveralls

@shijiesheng shijiesheng enabled auto-merge (squash) April 4, 2024 17:24
@shijiesheng shijiesheng merged commit 4d34cfb into cadence-workflow:master Apr 4, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants