BR restore failed: error="cannot find rewrite rule: [BR:Restore:ErrRestoreInvalidRewrite]invalid rewrite rule"

[Tammyxia]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• restore sysbench data from minio to TiDB cluster upstream and downstream separately :

2022-06-25T18:08:31.177Z INFO host/host.go:30 Execute command {"command": " /br restore full "-s" "s3://nfs/lixia-tools-workload/BR-sysbench-tmp?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" "-u" "http://downstream-pd.cdc-testbed-tps-813939-1-87:2379\" "--check-requirements=false"", "timeout": "1h", "resource name": "br"}

2022-06-25T18:08:31.181Z INFO host/host.go:30 Execute command {"command": " /br restore full "-s" "s3://nfs/lixia-tools-workload/BR-sysbench-tmp?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" "-u" "http://upstream-pd.cdc-testbed-tps-813939-1-87:2379\" "--check-requirements=false"", "timeout": "1h", "resource name": "br"}

2. What did you expect to see? (Required)

3. What did you see instead (Required)

• restore to upstream cluster successfully, but restore to downstream cluster failed:
  Error: cannot find rewrite rule: [BR:Restore:ErrRestoreInvalidRewrite]invalid rewrite rule\n", "error": "command terminated with exit code 1"

4. What is your TiDB version? (Required)

["Welcome to Backup & Restore (BR)"] [release-version=v6.1.0-nightly] [git-hash=ce7b541b2d3efd0120f9bec62db3f5088c723482] [git-branch=heads/refs/tags/v6.1.0-nightly] [go-version=go1.18.2] [utc-build-time="2022-06-24 11:01:21"] [race-enabled=false]

[3pointer]

looks like the backup files has some problems.

[overvenus]

This issue is caused by restoring to an existing table and the table lacks an index.

Reproduced with following steps

# Create a table with an index.
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=4000 --mysql-user=root \
        --mysql-db=test --tables=1 --table-size=10 --create_secondary=true \
        oltp_update_index prepare

# Backup the table
br backup full ...

# Drop the table and create it again without index.
mysql -h 127.0.0.1 -P 4000 -u root -E -e 'drop table test.sbtest1'
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=4000 --mysql-user=root \
        --mysql-db=test --tables=1 --table-size=10 --create_secondary=false \
        oltp_update_index prepare

# Restore
br restore full ...
...
[2023/06/15 18:36:13.819 +08:00] [INFO] [collector.go:77] ["Full Restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0] [split-region=511.73µs] [restore-ranges=4]
Error: cannot find rewrite rule: [BR:Restore:ErrRestoreInvalidRewrite]invalid rewrite rule

[benmaoer]

There should be a pre-recovery check to ensure that the target cluster does not already have the table being restored.

[Leavrth]

If this PR cherry-pick to release-7.1, please also cherry-pick #57030.