owner: fix a bug which lead to replication stopped and no error report (#1814)

[ti-chi-bot]

This is an automated cherry-pick of #1814

Bug phenomenon

The resolved TS and checkpoint TS of a change feed is stopped, but no error reported

How to confirm the bug

We can check the owner log, and find logs like this:

[2021/04/25 21:39:48.899 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.899 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.989 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]
[2021/04/25 21:39:48.989 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.989 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.049 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]
[2021/04/25 21:39:49.049 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.049 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.111 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]

Trigger conditions

A capture is offline when some table is moving by the owner.

Versions have this bug

[4.0.0, 4.0.13], 5.0.0-rc, [5.0.0, 5.0.1]

Bug mechanism

when the owner moves a table, the owner tries to remove the table from source capture and add the table to target capture.
if the target capture is offline at the same time, the owner should add this table to orphan tables and dispatch this table in the next tick.

but the owner forget to remove the invalid move table job, so the owner will add the table to orphan table every tick, leads to the other logic does not work properly

Check List

Tests

• Unit test

Release note

• fix a bug about resolved ts stopped when move a table.

[liuzix]

/lgtm

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• amyangfei
• liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

[lonng]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: ce58ce2

[ti-chi-bot]

@ti-chi-bot: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (release-5.0@8caa4e8). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             release-5.0      #1828   +/-   ##
================================================
  Coverage               ?   53.9316%           
================================================
  Files                  ?        153           
  Lines                  ?      16100           
  Branches               ?          0           
================================================
  Hits                   ?       8683           
  Misses                 ?       6496           
  Partials               ?        921