sinkManage should not retry on sink error infinitely.

[asddongmen]

What did you do?

1. Create a changefeed that replicating table t1, t2.
2. Drop table t1 in the downstream.

What did you expect to see?

After a period of time, sink error should be reported to owner.

What did you see instead?

Changefeed stucks, but changefeed's status keep normal, processor retry to restart sink again and again.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

master

[mayjiang0203]

Agree with that, but I wonder if it is possible to show the use of the sink error message in changefeed status, and also change normal to such as "normal with errors".

Hit this failure in ticdc log.

[2023/06/27 13:47:19.780 +08:00] [WARN] [manager.go:270] ["Sink manager backend sink fails"] [namespace=default] [changefeed=dr-autosyn-changefeed] [error="[CDC:ErrMySQLTxnError]MySQL txn error: Error 1146: Table 'ecasdb.dpl_event' doesn't exist"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: Error 1146: Table 'ecasdb.dpl_event' doesn't exist\[ngithub.jparrowsec.cn/pingcap/errors.AddStack\n\tgithub.jparrowsec.cn/pingcap/[email protected]/errors.go:174\ngithub.jparrowsec.cn/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.jparrowsec.cn/pingcap/[email protected]/normalize.go:164\ngithub.jparrowsec.cn/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.jparrowsec.cn/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).multiStmtExecute\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:631\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:703\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/metrics.(*Statistics).RecordBatchExecution\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/metrics/statistics.go:74\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:691\ngithub.jparrowsec.cn/pingcap/tiflow/pkg/retry.run\n\tgithub.jparrowsec.cn/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.jparrowsec.cn/pingcap/tiflow/pkg/retry.Do\n\tgithub.jparrowsec.cn/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:680\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:171\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).doFlush\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:179\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).runLoop\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:135\ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.newSink.func1\n\tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/txn_sink.go:106\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594](http://ngithub.jparrowsec.cn/pingcap/errors.AddStack/n/tgithub.jparrowsec.cn/pingcap/[email protected]/errors.go:174/ngithub.jparrowsec.cn/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.jparrowsec.cn/pingcap/[email protected]/normalize.go:164/ngithub.jparrowsec.cn/pingcap/tiflow/pkg/errors.WrapError/n/tgithub.jparrowsec.cn/pingcap/tiflow/pkg/errors/helper.go:34/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).multiStmtExecute/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:631/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:703/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/metrics.(*Statistics).RecordBatchExecution/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/metrics/statistics.go:74/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:691/ngithub.jparrowsec.cn/pingcap/tiflow/pkg/retry.run/n/tgithub.jparrowsec.cn/pingcap/tiflow/pkg/retry/retry_with_opt.go:57/ngithub.jparrowsec.cn/pingcap/tiflow/pkg/retry.Do/n/tgithub.jparrowsec.cn/pingcap/tiflow/pkg/retry/retry_with_opt.go:34/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:680/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).Flush/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:171/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).doFlush/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:179/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).runLoop/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:135/ngithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn.newSink.func1/n/tgithub.jparrowsec.cn/pingcap/tiflow/cdc/sinkv2/eventsink/txn/txn_sink.go:106/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/tgolang.org/x/[email protected]/errgroup/errgroup.go:75/nruntime.goexit/n/truntime/asm_amd64.s:1594)"]

while changefeed still show it is normal.

[root@tiup-0 ~]# tiup cdc:v6.5.3 cli changefeed list  --server=http://ticdc1-peer.e2e-dr-auto-sync-5r-pref-tps-1809605-1-603:8300/
Starting component `cdc`: /root/.tiup/components/cdc/v6.5.3/cdc /root/.tiup/components/cdc/v6.5.3/cdc cli changefeed list --server=http://ticdc1-peer.e2e-dr-auto-sync-5r-pref-tps-1809605-1-603:8300/
[
  {
    "id": "dr-autosyn-changefeed",
    "namespace": "default",
    "summary": {
      "state": "normal",
      "tso": 442446779705196803,
      "checkpoint": "2023-06-27 01:28:40.726",
      "error": null
    }
  }
]