[broker] Cursor status has always been SwitchingLedger and pendingMarkDeleteOps has accumulated tens of thousands of requests

[poorbarcode]

Describe the bug
The pendingMarkDeleteOps has accumulated tens of thousands of requests, and the cursor status has always been SwitchingLedger, and the retention cannot be executed, resulting in the situation that the disk space cannot be reclaimed.

Screenshots

[poorbarcode]

Hi @keyboardbobo

Could you please provide the version used? If possible, you can upload the dump file ( High light which may cause data leakage of the company ).

Could you show the property ManagedCursorMXBean mbean of this Cursor ?

[keyboardbobo]

@poorbarcode My version is 2.9.2, the dump file is very large, and it is difficult to upload it on the intranet due to security reasons. The values of the ManagedCursorMXBean mbean are as follows:

persistLedgeSucceed：cellsBusy=0,base=63308,cells:null
persistLedgeFailed：cellsBusy=0,base=1,cells:null
persistZookeeperSucceed：cellsBusy=0,base=1,cells:null
persistZookeeperFailed：cellsBusy=0,base=0,cells:null
writeCursorLedgerSize：cellsBusy=0,base=3437666,cells:null
writeCursorLedgerLogicalSize：cellsBusy=0,base=1718833,cells:null
readCursorLedgerSize：cellsBusy=0,base=9,cells:null

[poorbarcode]

Hi @keyboardbobo

I now suspect that something is wrong with the pendingMarkDeletedSubmittedCount. Could you show the property ManagedCursorImpl.pendingMarkDeletedSubmittedCount ?

[keyboardbobo]

Hi @keyboardbobo

I now suspect that something is wrong with the pendingMarkDeletedSubmittedCount. Could you show the property ManagedCursorImpl.pendingMarkDeletedSubmittedCount ?

pendingMarkDeletedSubmittedCount =0

[poorbarcode]

Hi @keyboardbobo

pendingMarkDeletedSubmittedCount =0

Thanks, then it is not pendingMarkDeletedSubmittedCount's problem. Is there any other error log？

[keyboardbobo]

@poorbarcode
At that time, an abnormal scenario with a large network delay from the broker to the bookie was simulated, and there were many errors similar to the following:

Write of ledger entry to quorum failed
NotEnoughBookiesException: Not enough non-faulty bookies available
UpdateLoop(ledgerId=2780774,loopId=54e667fe) Exception updating

Then the disk space increases, and some partitions cannot be reclaim.

[poorbarcode]

Hi @keyboardbobo

there were many errors similar to the following:

Could you show more detail about the error log?

[keyboardbobo]

@poorbarcode

[keyboardbobo]

@poorbarcode I found that there are thousands of tasks in the "BookKeeperClientWorker-OrderedExecutor-59-%d" thread group, and I don't know what caused it. SafeRunnable should catch the exception, but I don't know why this happens

[poorbarcode]

Hi @keyboardbobo

I found that there are thousands of tasks in the "BookKeeperClientWorker-OrderedExecutor-59-%d" thread group, and I don't know what caused it. SafeRunnable should catch the exception, but I don't know why this happens

OrderedExecutor used in these is used in the following three scenarios:

• Write Bookie
• Read Bookie
• Send messages to the client

@hangc0276 Is it normal to have so many tasks in the queue?

[keyboardbobo]

@poorbarcode
I took a look at some thread groups. Other thread groups are all 0. Only this one has a backlog of more than 7,000 tasks.

[keyboardbobo]

@poorbarcode The stack of the three threads of this thread group is as follows:

[poorbarcode]

Hi @keyboardbobo

The stack of the three threads of this thread group is as follows:

It seems that one topic is busy while others are not, but this seems have not related to the cursor state always being SwitchingLedger or maybe I didn't understand enough to realize the problem

[keyboardbobo]

@poorbarcode I suspect that when SwitchingLedger, some tasks are blocked in the queue, which makes it impossible to complete the execution.

It seems that one topic is busy while others are not, but this seems have not related to the cursor state always being SwitchingLedger or maybe I didn't understand enough to realize the problem

[poorbarcode]

Hi @keyboardbobo

Could you provide the BK configuration?

• Ensemble size
• Write quorum size
• Ack quorum size

E.g. Ensemble size = 3, and Write quorum size = 3, When any bookie server does not work, the first write request will timeout, and other requests will backlog

[keyboardbobo]

@poorbarcode

managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=3
managedLedgerDefaultAckQuorum=2

If bookie is not working, should all thread pools be backlogged, not just that thread pool?

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[keyboardbobo]

@poorbarcode There is a new development on the question, but I'm not sure if it's the same problem:
#17967

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[poorbarcode]

This issue might be fixed by #17971