dm validator should retry on transient error and there should be no deadlock when bad thing happens

[hihihuhu]

What did you do?

there are 2 issues,

1. Error 9005: Region is unavailable is not retryable even though it could/should be transient error https://github.com/pingcap/tiflow/blob/master/dm/syncer/validate_worker.go#L170
2. there could be deadlock when errors happen, which will block any dmctl operation on the task, particularly, one cannot restart the task by dmctl stop-task/start-task because of the deadlock.

in data validator thread,

• the data validator tries to stop when encounter the first error https://github.com/pingcap/tiflow/blob/v7.1.0/dm/syncer/data_validator.go#L462
• in stopInner, it waits for all workers to be done https://github.com/pingcap/tiflow/blob/v7.1.0/dm/syncer/data_validator.go#L743
• thus, when it waits for the wg, and no one will drain the errChan

meantime, in worker threads,

• it reports error to the errChan https://github.com/pingcap/tiflow/blob/v7.1.0/dm/syncer/validate_worker.go#L170
• the bounded errChan will be filled fully because the validator doesn't drain the errors
• thus the workers waits on errChan holding the wg unreleased, which blocks the innerStop to finish

What did you expect to see?

1. region is not available could be retried
2. there is no deadlock and the task could be restart by stop-task/start-task in any cases

What did you see instead?

1. region is not available stops the validator
2. deadlock happens when a lot of workers report error

Versions of the cluster

master

current status of DM cluster (execute query-status <task-name> in dmctl)

{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": true,
            "msg": "",
            "sourceStatus": {
                "source": "tidbmigration-sample-src",
                "worker": "test-single-cell-dm-worker-0",
                "result": null,
                "relayStatus": null
            },
            "subTaskStatus": [
                {
                    "name": "tidbmigration-sample-task",
                    "stage": "Running",
                    "unit": "Sync",
                    "result": null,
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "19635499",
                        "totalTps": "41072",
                        "recentTps": "12483",
                        "masterBinlog": "(mysql-bin-changelog.061817, 57798824)",
                        "masterBinlogGtid": "",
                        "syncerBinlog": "(mysql-bin-changelog.061817, 55744546)",
                        "syncerBinlogGtid": "00000000-0000-0000-0000-000000000000:0",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false,
                        "binlogType": "remote",
                        "secondsBehindMaster": "0",
                        "blockDDLOwner": "",
                        "conflictMsg": "",
                        "totalRows": "19635499",
                        "totalRps": "41072",
                        "recentRps": "12483"
                    },
                    "validation": {
                        "task": "tidbmigration-sample-task",
                        "source": "tidbmigration-sample-src",
                        "mode": "full",
                        "stage": "Running",
                        "validatorBinlog": "(mysql-bin-changelog.061762, 77247898)",
                        "validatorBinlogGtid": "00000000-0000-0000-0000-000000000000:0",
                        "result": {
                            "isCanceled": false,
                            "errors": [
                                {
                                    "ErrCode": 43005,
                                    "ErrClass": "validator",
                                    "ErrScope": "internal",
                                    "ErrLevel": "high",
                                    "Message": "failed to validate row change",
                                    "RawCause": "Error 9005: Region is unavailable",
                                    "Workaround": ""
                                }
                            ],
                            "detail": null
                        },
                        "processedRowsStatus": "insert/update/delete: 1673826/1673561/371420",
                        "pendingRowsStatus": "insert/update/delete: 17259/15936/3015",
                        "errorRowsStatus": "new/ignored/resolved: 0/0/0"
                    }
                }
            ]
        }
    ]
}

[hihihuhu]

another example of retryable error is

...
                    "validation": {
                        "task": "tidbmigration-sample-task",
                        "source": "tidbmigration-sample-src",
                        "mode": "full",
                        "stage": "Stopped",
                        "validatorBinlog": "(mysql-bin-changelog.113098, 21486296)",
                        "validatorBinlogGtid": "00000000-0000-0000-0000-000000000000:0",
                        "result": {
                            "isCanceled": false,
                            "errors": [
                                {
                                    "ErrCode": 43005,
                                    "ErrClass": "validator",
                                    "ErrScope": "internal",
                                    "ErrLevel": "high",
                                    "Message": "failed to validate row change",
                                    "RawCause": "Error 1105: no available connections",
                                    "Workaround": ""
                                }
                            ],
                            "detail": null
                        },
                        "processedRowsStatus": "insert/update/delete: 2455061263/2454996366/545577884",
                        "pendingRowsStatus": "insert/update/delete: 690/668/160",
                        "errorRowsStatus": "new/ignored/resolved: 0/0/0"
                    }

[D3Hunter]

the deadlock issue is similar to desc in #7241 (comment), but it's on validator.lock, not wg