processor: fix a bug that will cause processor Tick get stuck when downstream is Kafka

[asddongmen]

What problem does this PR solve?

Issue Number: close #11340

What is changed and how it works?

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)

Deploy a cdc cluster, create two changefeeds and synchronize the same table to both Kafka and TiDB separately.

Every 20 minutes, inject CDC to network isolation:

• First, isolate cdc to Kafka network for 10 minutes.
• Then, 20 minutes later, isolate cdc to downstream TiDB network for 10 minutes.

Repeat this process six times.

Before this PR, the lag in the MySQL sink changefeed was influenced by the Kafka sink changefeed when network isolation was introduced between CDC and Kafka:

After this PR, only Kafka sink changefeed lag increasing during network isolation injection:

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that causes the processor tick to get stuck when the downstream is Kafka and becomes unreachable.```

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[asddongmen]

/test all

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 57.5747%. Comparing base (04a7d6a) to head (54b828b).

Additional details and impacted files

Components	Coverage Δ
cdc	61.3431% <0.0000%> (-0.0342%)	⬇️
dm	51.1809% <ø> (-0.0585%)	⬇️
engine	63.3526% <ø> (+0.0070%)	⬆️

Flag	Coverage Δ
unit	57.5747% <0.0000%> (-0.0401%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11339        +/-   ##
================================================
- Coverage   57.6147%   57.5747%   -0.0401%     
================================================
  Files           849        849                
  Lines        126294     126262        -32     
================================================
- Hits          72764      72695        -69     
- Misses        48121      48155        +34     
- Partials       5409       5412         +3     

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [3AceShowHand,hicqu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-06-25 06:40:09.196671537 +0000 UTC m=+701735.682160364: ☑️ agreed by hicqu.
• 2024-06-27 08:09:51.558875886 +0000 UTC m=+879918.044364718: ☑️ agreed by 3AceShowHand.

[asddongmen]

/retest

[CharlesCheung96]

/retest

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #11387.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #11388.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #11389.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #11390.