From ccf30348e954c487d4f99d3356d45ad234bb25b9 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Wed, 22 Jun 2022 13:48:16 +0800 Subject: [PATCH 1/8] ticdc: adjust TOC and reorganize troubleshooting&faq --- TOC.md | 10 +- faq/migration-tidb-faq.md | 2 +- migrate-from-tidb-to-tidb.md | 2 +- production-deployment-using-tiup.md | 1 + ...-betwwen-primary-and-secondary-clusters.md | 2 +- ticdc/deploy-ticdc.md | 2 +- ticdc/ticdc-alert-rules.md | 2 +- ticdc/ticdc-faq.md | 255 ++++++++++++++++++ ticdc/ticdc-overview.md | 7 +- ticdc/troubleshoot-ticdc.md | 253 +---------------- 10 files changed, 278 insertions(+), 258 deletions(-) create mode 100644 ticdc/ticdc-faq.md diff --git a/TOC.md b/TOC.md index a46bf57697db2..dbee00780e8d6 100644 --- a/TOC.md +++ b/TOC.md @@ -486,12 +486,14 @@ - [Deploy](/ticdc/deploy-ticdc.md) - [Maintain](/ticdc/manage-ticdc.md) - [Troubleshoot](/ticdc/troubleshoot-ticdc.md) + - [FAQ](/ticdc/ticdc-faq.md) - [Monitor](/ticdc/monitor-ticdc.md) - [Alert Rules](/ticdc/ticdc-alert-rules.md) - - [TiCDC OpenAPI](/ticdc/ticdc-open-api.md) - - [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) - - [TiCDC Avro Protocol](/ticdc/ticdc-avro-protocol.md) - - [TiCDC Canal-JSON Protocol](/ticdc/ticdc-canal-json.md) + - Reference + - [TiCDC OpenAPI](/ticdc/ticdc-open-api.md) + - [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + - [TiCDC Avro Protocol](/ticdc/ticdc-avro-protocol.md) + - [TiCDC Canal-JSON Protocol](/ticdc/ticdc-canal-json.md) - [Integrate TiDB with Confluent Platform](/ticdc/integrate-confluent-using-ticdc.md) - [Glossary](/ticdc/ticdc-glossary.md) - [Dumpling](/dumpling-overview.md) diff --git a/faq/migration-tidb-faq.md b/faq/migration-tidb-faq.md index 5b46e424ff38d..36e339979001b 100644 --- a/faq/migration-tidb-faq.md +++ b/faq/migration-tidb-faq.md @@ -13,7 +13,7 @@ For the frequently asked questions about migration-related tools, click the corr - [TiDB Binlog FAQ](/tidb-binlog/tidb-binlog-faq.md) - [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md) - [TiDB Data Migration (DM) FAQs](/dm/dm-faq.md) -- [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) +- [TiCDC FAQs](/ticdc/ticdc-faq.md) ## Full data export and import diff --git a/migrate-from-tidb-to-tidb.md b/migrate-from-tidb-to-tidb.md index 6c8b355c22f48..e926a572776d4 100644 --- a/migrate-from-tidb-to-tidb.md +++ b/migrate-from-tidb-to-tidb.md @@ -238,7 +238,7 @@ After setting up the environment, you can use the backup and restore functions o 3. Enable GC. - In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/troubleshoot-ticdc.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). + In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/ticdc-faq.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). {{< copyable "sql" >}} diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index b20f8cab44a8f..583c4ee7b1eae 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -431,4 +431,5 @@ If you have deployed [TiFlash](/tiflash/tiflash-overview.md) along with the TiDB If you have deployed [TiCDC](/ticdc/ticdc-overview.md) along with the TiDB cluster, see the following documents: - [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) +- [TiCDC FAQs](/ticdc/ticdc-faq.md) - [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) diff --git a/replicate-betwwen-primary-and-secondary-clusters.md b/replicate-betwwen-primary-and-secondary-clusters.md index ad25150dcda26..40044e0805af2 100644 --- a/replicate-betwwen-primary-and-secondary-clusters.md +++ b/replicate-betwwen-primary-and-secondary-clusters.md @@ -255,7 +255,7 @@ After setting up the environment, you can use the backup and restore functions o 3. Enable GC. - In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/troubleshoot-ticdc.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). + In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/ticdc-faq.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). {{< copyable "sql" >}} diff --git a/ticdc/deploy-ticdc.md b/ticdc/deploy-ticdc.md index c2f10b9ef1eb8..4775762a82e1b 100644 --- a/ticdc/deploy-ticdc.md +++ b/ticdc/deploy-ticdc.md @@ -55,7 +55,7 @@ The following are descriptions of options available in the `cdc server` command: - `pd`: A comma-separated list of PD endpoints. - `config`: The address of the configuration file that TiCDC uses (optional). This option is supported since TiCDC v5.0.0. This option can be used in the TiCDC deployment since TiUP v1.4.0. - `data-dir`: Specifies the directory that TiCDC uses when it needs to use disks to store files. Unified Sorter uses this directory to store temporary files. It is recommended to ensure that the free disk space for this directory is greater than or equal to 500 GiB. For more details, see [Unified Sorter](/ticdc/manage-ticdc.md#unified-sorter). If you are using TiUP, you can configure `data_dir` in the [`cdc_servers`](/tiup/tiup-cluster-topology-reference.md#cdc_servers) section, or directly use the default `data_dir` path in `global`. -- `gc-ttl`: The TTL (Time To Live) of the service level `GC safepoint` in PD set by TiCDC, and the duration that the replication task can suspend, in seconds. The default value is `86400`, which means 24 hours. Note: Suspending of the TiCDC replication task affects the progress of TiCDC GC safepoint, which means that it affects the progress of upstream TiDB GC, as detailed in [Complete Behavior of TiCDC GC safepoint](/ticdc/troubleshoot-ticdc.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). +- `gc-ttl`: The TTL (Time To Live) of the service level `GC safepoint` in PD set by TiCDC, and the duration that the replication task can suspend, in seconds. The default value is `86400`, which means 24 hours. Note: Suspending of the TiCDC replication task affects the progress of TiCDC GC safepoint, which means that it affects the progress of upstream TiDB GC, as detailed in [Complete Behavior of TiCDC GC safepoint](/ticdc/ticdc-faq.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). - `log-file`: The path to which logs are output when the TiCDC process is running. If this parameter is not specified, logs are written to the standard output (stdout). - `log-level`: The log level when the TiCDC process is running. The default value is `"info"`. - `ca`: Specifies the path of the CA certificate file in PEM format for TLS connection (optional). diff --git a/ticdc/ticdc-alert-rules.md b/ticdc/ticdc-alert-rules.md index 6cded5900b4ad..e0dab477a2b47 100644 --- a/ticdc/ticdc-alert-rules.md +++ b/ticdc/ticdc-alert-rules.md @@ -182,7 +182,7 @@ Warning alerts are a reminder for an issue or error. - Solution: - There are many possible root causes. See [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). + There are many possible root causes. See [Troubleshoot TiCDC](/ticdc/). ### `ticdc_memory_abnormal` diff --git a/ticdc/ticdc-faq.md b/ticdc/ticdc-faq.md new file mode 100644 index 0000000000000..d93fdb402e7ad --- /dev/null +++ b/ticdc/ticdc-faq.md @@ -0,0 +1,255 @@ +--- +title: FAQs +summary: Learn the FAQs you might encounter when you use TiCDC. +--- + +# FAQs + +This document introduces the common questions that you might encounter when using TiCDC. + +> **Note:** +> +> In this document, the PD address specified in `cdc cli` commands is `--pd=http://10.0.10.25:2379`. When you use the command, replace the address with your actual PD address. + +## How do I choose `start-ts` when creating a task in TiCDC? + +The `start-ts` of a replication task corresponds to a Timestamp Oracle (TSO) in the upstream TiDB cluster. TiCDC requests data from this TSO in a replication task. Therefore, the `start-ts` of the replication task must meet the following requirements: + +- The value of `start-ts` is larger than the `tikv_gc_safe_point` value of the current TiDB cluster. Otherwise, an error occurs when you create a task. +- Before starting a task, ensure that the downstream has all data before `start-ts`. For scenarios such as replicating data to message queues, if the data consistency between upstream and downstream is not required, you can relax this requirement according to your application need. + +If you do not specify `start-ts`, or specify `start-ts` as `0`, when a replication task is started, TiCDC gets a current TSO and starts the task from this TSO. + +## Why can't some tables be replicated when I create a task in TiCDC? + +When you execute `cdc cli changefeed create` to create a replication task, TiCDC checks whether the upstream tables meet the [replication restrictions](/ticdc/ticdc-overview.md#restrictions). If some tables do not meet the restrictions, `some tables are not eligible to replicate` is returned with a list of ineligible tables. You can choose `Y` or `y` to continue creating the task, and all updates on these tables are automatically ignored during the replication. If you choose an input other than `Y` or `y`, the replication task is not created. + +## How do I view the state of TiCDC replication tasks? + +To view the status of TiCDC replication tasks, use `cdc cli`. For example: + +{{< copyable "shell-regular" >}} + +```shell +cdc cli changefeed list --pd=http://10.0.10.25:2379 +``` + +The expected output is as follows: + +```json +[{ + "id": "4e24dde6-53c1-40b6-badf-63620e4940dc", + "summary": { + "state": "normal", + "tso": 417886179132964865, + "checkpoint": "2020-07-07 16:07:44.881", + "error": null + } +}] +``` + +* `checkpoint`: TiCDC has replicated all data before this timestamp to downstream. +* `state`: The state of this replication task: + * `normal`: The task runs normally. + * `stopped`: The task is stopped manually or encounters an error. + * `removed`: The task is removed. + +> **Note:** +> +> This feature is introduced in TiCDC 4.0.3. + +## What is `gc-ttl` in TiCDC? + +Since v4.0.0-rc.1, PD supports external services in setting the service-level GC safepoint. Any service can register and update its GC safepoint. PD ensures that the key-value data later than this GC safepoint is not cleaned by GC. + +When the replication task is unavailable or interrupted, this feature ensures that the data to be consumed by TiCDC is retained in TiKV without being cleaned by GC. + +When starting the TiCDC server, you can specify the Time To Live (TTL) duration of GC safepoint by configuring `gc-ttl`. You can also [use TiUP to modify](/ticdc/manage-ticdc.md#modify-ticdc-configuration-using-tiup) `gc-ttl`. The default value is 24 hours. In TiCDC, this value means: + +- The maximum time the GC safepoint is retained at the PD after the TiCDC service is stopped. +- The maximum time a replication task can be suspended after the task is interrupted or manually stopped. If the time for a suspended replication task is longer than the value set by `gc-ttl`, the replication task enters the `failed` status, cannot be resumed, and cannot continue to affect the progress of the GC safepoint. + +The second behavior above is introduced in TiCDC v4.0.13 and later versions. The purpose is to prevent a replication task in TiCDC from suspending for too long, causing the GC safepoint of the upstream TiKV cluster not to continue for a long time and retaining too many outdated data versions, thus affecting the performance of the upstream cluster. + +> **Note:** +> +> In some scenarios, for example, when you use TiCDC for incremental replication after full replication with Dumpling/BR, the default 24 hours of `gc-ttl` may not be sufficient. You need to specify an appropriate value for `gc-ttl` when you start the TiCDC server. + +## What is the complete behavior of TiCDC garbage collection (GC) safepoint? + +If a replication task starts after the TiCDC service starts, the TiCDC owner updates the PD service GC safepoint with the smallest value of `checkpoint-ts` among all replication tasks. The service GC safepoint ensures that TiCDC does not delete data generated at that time and after that time. If the replication task is interrupted, or manually stopped, the `checkpoint-ts` of this task does not change. Meanwhile, PD's corresponding service GC safepoint is not updated either. + +If the replication task is suspended longer than the time specified by `gc-ttl`, the replication task enters the `failed` status and cannot be resumed. The PD corresponding service GC safepoint will continue. + +The Time-To-Live (TTL) that TiCDC sets for a service GC safepoint is 24 hours, which means that the GC mechanism does not delete any data if the TiCDC service can be recovered within 24 hours after it is interrupted. + +## How to understand the relationship between the TiCDC time zone and the time zones of the upstream/downstream databases? + +||Upstream time zone| TiCDC time zone|Downstream time zone| +| :-: | :-: | :-: | :-: | +| Configuration method | See [Time Zone Support](/configure-time-zone.md) | Configured using the `--tz` parameter when you start the TiCDC server | Configured using the `time-zone` parameter in `sink-uri` | +| Description | The time zone of the upstream TiDB, which affects DML operations of the timestamp type and DDL operations related to timestamp type columns.| TiCDC assumes that the upstream TiDB's time zone is the same as the TiCDC time zone configuration, and performs related operations on the timestamp column.| The downstream MySQL processes the timestamp in the DML and DDL operations according to the downstream time zone setting.| + + > **Note:** + > + > Be careful when you set the time zone of the TiCDC server, because this time zone is used for converting the time type. Keep the upstream time zone, TiCDC time zone, and the downstream time zone consistent. The TiCDC server chooses its time zone in the following priority: + > + > - TiCDC first uses the time zone specified using `--tz`. + > - When `--tz` is not available, TiCDC tries to read the time zone set using the `TZ` environment variable. + > - When the `TZ` environment variable is not available, TiCDC uses the default time zone of the machine. + +## What is the default behavior of TiCDC if I create a replication task without specifying the configuration file in `--config`? + +If you use the `cdc cli changefeed create` command without specifying the `-config` parameter, TiCDC creates the replication task in the following default behaviors: + +- Replicates all tables except system tables +- Enables the Old Value feature +- Skips replicating tables that do not contain [valid indexes](/ticdc/ticdc-overview.md#restrictions) + +## Does TiCDC support outputting data changes in the Canal format? + +Yes. To enable Canal output, specify the protocol as `canal` in the `--sink-uri` parameter. For example: + +{{< copyable "shell-regular" >}} + +```shell +cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&protocol=canal" --config changefeed.toml +``` + +> **Note:** +> +> * This feature is introduced in TiCDC 4.0.2. +> * TiCDC currently supports outputting data changes in the Canal format only to MQ sinks such as Kafka and Pulsar. + +For more information, refer to [Create a replication task](/ticdc/manage-ticdc.md#create-a-replication-task). + +## Why does the latency from TiCDC to Kafka become higher and higher? + +* Check [how do I view the state of TiCDC replication tasks](#how-do-i-view-the-state-of-ticdc-replication-tasks). +* Adjust the following parameters of Kafka: + + * Increase the `message.max.bytes` value in `server.properties` to `1073741824` (1 GB). + * Increase the `replica.fetch.max.bytes` value in `server.properties` to `1073741824` (1 GB). + * Increase the `fetch.message.max.bytes` value in `consumer.properties` to make it larger than the `message.max.bytes` value. + +## When TiCDC replicates data to Kafka, does it write all the changes in a transaction into one message? If not, on what basis does it divide the changes? + +No. According to the different distribution strategies configured, TiCDC divides the changes on different bases, including `default`, `row id`, `table`, and `ts`. + +For more information, refer to [Replication task configuration file](/ticdc/manage-ticdc.md#task-configuration-file). + +## When TiCDC replicates data to Kafka, can I control the maximum size of a single message in TiDB? + +Yes. You can set the `max-message-bytes` parameter to control the maximum size of data sent to the Kafka broker each time (optional, `10MB` by default). You can also set `max-batch-size` to specify the maximum number of change records in each Kafka message. Currently, the setting only takes effect when Kafka's `protocol` is `open-protocol` (optional, `16` by default). + +## When TiCDC replicates data to Kafka, does a message contain multiple types of data changes? + +Yes. A single message might contain multiple `update`s or `delete`s, and `update` and `delete` might co-exist. + +## When TiCDC replicates data to Kafka, how do I view the timestamp, table name, and schema name in the output of TiCDC Open Protocol? + +The information is included in the key of Kafka messages. For example: + +```json +{ + "ts":, + "scm":, + "tbl":, + "t":1 +} +``` + +For more information, refer to [TiCDC Open Protocol event format](/ticdc/ticdc-open-protocol.md#event-format). + +## When TiCDC replicates data to Kafka, how do I know the timestamp of the data changes in a message? + +You can get the unix timestamp by moving `ts` in the key of the Kafka message by 18 bits to the right. + +## How does TiCDC Open Protocol represent `null`? + +In TiCDC Open Protocol, the type code `6` represents `null`. + +| Type | Code | Output Example | Note | +|:--|:--|:--|:--| +| Null | 6 | `{"t":6,"v":null}` | | + +For more information, refer to [TiCDC Open Protocol column type code](/ticdc/ticdc-open-protocol.md#column-type-code). + +## How can I tell if a Row Changed Event of TiCDC Open Protocol is an `INSERT` event or an `UPDATE` event? + +If the Old Value feature is not enabled, you cannot tell whether a Row Changed Event of TiCDC Open Protocol is an `INSERT` event or an `UPDATE` event. If the feature is enabled, you can determine the event type by the fields it contains: + +* `UPDATE` event contains both `"p"` and `"u"` fields +* `INSERT` event only contains the `"u"` field +* `DELETE` event only contains the `"d"` field + +For more information, refer to [Open protocol Row Changed Event format](/ticdc/ticdc-open-protocol.md#row-changed-event). + +## How much PD storage does TiCDC use? + +TiCDC uses etcd in PD to store and regularly update the metadata. Because the time interval between the MVCC of etcd and PD's default compaction is one hour, the amount of PD storage that TiCDC uses is proportional to the amount of metadata versions generated within this hour. However, in v4.0.5, v4.0.6, and v4.0.7, TiCDC has a problem of frequent writing, so if there are 1000 tables created or scheduled in an hour, it then takes up all the etcd storage and returns the `etcdserver: mvcc: database space exceeded` error. You need to clean up the etcd storage after getting this error. See [etcd maintaince space-quota](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) for details. It is recommended to upgrade your cluster to v4.0.9 or later versions. + +## Does TiCDC support replicating large transactions? Is there any risk? + +TiCDC provides partial support for large transactions (more than 5 GB in size). Depending on different scenarios, the following risks might exist: + +- When TiCDC's internal processing capacity is insufficient, the replication task error `ErrBufferReachLimit` might occur. +- When TiCDC's internal processing capacity is insufficient or the throughput capacity of TiCDC's downstream is insufficient, out of memory (OOM) might occur. + +If you encounter an error above, it is recommended to use BR to restore the incremental data of large transactions. The detailed operations are as follows: + +1. Record the `checkpoint-ts` of the changefeed that is terminated due to large transactions, use this TSO as the `--lastbackupts` of the BR incremental backup, and execute [incremental data backup](/br/use-br-command-line-tool.md#back-up-incremental-data). +2. After backing up the incremental data, you can find a log record similar to `["Full backup Failed summary : total backup ranges: 0, total success: 0, total failed: 0"] [BackupTS=421758868510212097]` in the BR log output. Record the `BackupTS` in this log. +3. [Restore the incremental data](/br/use-br-command-line-tool.md#restore-incremental-data). +4. Create a new changefeed and start the replication task from `BackupTS`. +5. Delete the old changefeed. + +## `enable-old-value` is set to `true` when I create a TiCDC replication task, but `INSERT`/`UPDATE` statements from the upstream become `REPLACE INTO` after being replicated to the downstream + +When a changefeed is created in TiCDC, the `safe-mode` setting defaults to `true`, which generates the `REPLACE INTO` statement to execute for the upstream `INSERT`/`UPDATE` statements. + +Currently, users cannot modify the `safe-mode` setting, so this issue currently has no solution. + +## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? + +Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: + +{{< copyable "sql" >}} + +```sql +mysql root@127.0.0.1:test> show create table test; ++-------+----------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+----------------------------------------------------------------------------------+ +| test | CREATE TABLE `test` ( | +| | `id` int(11) NOT NULL, | +| | `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | +| | PRIMARY KEY (`id`) | +| | ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | ++-------+----------------------------------------------------------------------------------+ +1 row in set +``` + +From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. + +Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. + +## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? + +When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: + +- `Select` +- `Index` +- `Insert` +- `Update` +- `Delete` +- `Create` +- `Drop` +- `Alter` +- `Create View` + +If you need to replicate `recover table` to the downstream TiDB, you should have the `Super` permission. + +## Why does TiCDC use disks? When does TiCDC write to disks? Does TiCDC use memory buffer to improve replication performance? + +When the upstream write traffic is at business peak hours, the downstream may fail to consume data in a timely manner, resulting in data pile-up. TiCDC uses disks to process the data that is piled up. TiCDC needs to write data to disks during normal operation. However, this is not usually the bottleneck for replication throughput and replication latency, given that writing to disks latency within a hundred milliseconds. tiCDC also makes use of memory to boost accelerated reads from disk to improve synchronous performance. diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index f2911b9174e37..744806fcc32b2 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -92,7 +92,7 @@ Currently, the following scenarios are not supported: - The TiKV cluster that uses RawKV alone. - The [DDL operation `CREATE SEQUENCE`](/sql-statements/sql-statement-create-sequence.md) and the [SEQUENCE function](/sql-statements/sql-statement-create-sequence.md#sequence-function) in TiDB. When the upstream TiDB uses `SEQUENCE`, TiCDC ignores `SEQUENCE` DDL operations/functions performed upstream. However, DML operations using `SEQUENCE` functions can be correctly replicated. -TiCDC only provides partial support for scenarios of large transactions in the upstream. For details, refer to [FAQ: Does TiCDC support replicating large transactions? Is there any risk?](/ticdc/troubleshoot-ticdc.md#does-ticdc-support-replicating-large-transactions-is-there-any-risk). +TiCDC only provides partial support for scenarios of large transactions in the upstream. For details, refer to [FAQ: Does TiCDC support replicating large transactions? Is there any risk?](/ticdc/ticdc-faq.md#does-ticdc-support-replicating-large-transactions-is-there-any-risk). > **Note:** > @@ -149,6 +149,7 @@ Since v5.3.0, TiCDC supports [global temporary tables](/temporary-tables.md#glob If the upstream cluster contains a global temporary table, the downstream TiDB cluster is expected to be v5.3.0 or a later version. Otherwise, an error occurs during the replication process. -## Troubleshoot TiCDC +## TiCDC FAQs and troubleshooting -For details, refer to [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). +- To learn the FAQs of TiCDC, see [TiCDC FAQs](/ticdc/ticdc-faq.md). +- To learn how to troubleshoot TiCDC, see [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index 15e35e0ba066d..ecb5b11292167 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -6,59 +6,12 @@ aliases: ['/docs/dev/ticdc/troubleshoot-ticdc/'] # Troubleshoot TiCDC -This document introduces the common issues and errors that you might encounter when using TiCDC, and the corresponding maintenance and troubleshooting methods. +This document introduces the common errors you might encounter when using TiCDC, and the corresponding maintenance and troubleshooting methods. > **Note:** > > In this document, the PD address specified in `cdc cli` commands is `--pd=http://10.0.10.25:2379`. When you use the command, replace the address with your actual PD address. -## How do I choose `start-ts` when creating a task in TiCDC? - -The `start-ts` of a replication task corresponds to a Timestamp Oracle (TSO) in the upstream TiDB cluster. TiCDC requests data from this TSO in a replication task. Therefore, the `start-ts` of the replication task must meet the following requirements: - -- The value of `start-ts` is larger than the `tikv_gc_safe_point` value of the current TiDB cluster. Otherwise, an error occurs when you create a task. -- Before starting a task, ensure that the downstream has all data before `start-ts`. For scenarios such as replicating data to message queues, if the data consistency between upstream and downstream is not required, you can relax this requirement according to your application need. - -If you do not specify `start-ts`, or specify `start-ts` as `0`, when a replication task is started, TiCDC gets a current TSO and starts the task from this TSO. - -## Why can't some tables be replicated when I create a task in TiCDC? - -When you execute `cdc cli changefeed create` to create a replication task, TiCDC checks whether the upstream tables meet the [replication restrictions](/ticdc/ticdc-overview.md#restrictions). If some tables do not meet the restrictions, `some tables are not eligible to replicate` is returned with a list of ineligible tables. You can choose `Y` or `y` to continue creating the task, and all updates on these tables are automatically ignored during the replication. If you choose an input other than `Y` or `y`, the replication task is not created. - -## How do I view the state of TiCDC replication tasks? - -To view the status of TiCDC replication tasks, use `cdc cli`. For example: - -{{< copyable "shell-regular" >}} - -```shell -cdc cli changefeed list --pd=http://10.0.10.25:2379 -``` - -The expected output is as follows: - -```json -[{ - "id": "4e24dde6-53c1-40b6-badf-63620e4940dc", - "summary": { - "state": "normal", - "tso": 417886179132964865, - "checkpoint": "2020-07-07 16:07:44.881", - "error": null - } -}] -``` - -* `checkpoint`: TiCDC has replicated all data before this timestamp to downstream. -* `state`: The state of this replication task: - * `normal`: The task runs normally. - * `stopped`: The task is stopped manually or encounters an error. - * `removed`: The task is removed. - -> **Note:** -> -> This feature is introduced in TiCDC 4.0.3. - ## TiCDC replication interruptions ### How do I know whether a TiCDC replication task is interrupted? @@ -70,7 +23,7 @@ The expected output is as follows: ### How do I know whether the replication task is stopped manually? -You can know whether the replication task is stopped manually by executing `cdc cli`. For example: +You can know whether the replication task is stopped manually by executing `cdc cli`. For example: {{< copyable "shell-regular" >}} @@ -103,16 +56,16 @@ A replication task might be interrupted in the following known scenarios: 2. Use the new task configuration file and add the `ignore-txn-start-ts` parameter to skip the transaction corresponding to the specified `start-ts`. 3. Stop the old replication task via HTTP API. Execute `cdc cli changefeed create` to create a new task and specify the new task configuration file. Specify `checkpoint-ts` recorded in step 1 as the `start-ts` and start a new task to resume the replication. -- In TiCDC v4.0.13 and earlier versions, when TiCDC replicates the partitioned table, it might encounter an error that leads to replication interruption. +- In TiCDC v4.0.13 and earlier versions, when TiCDC replicates the partitioned table, it might encounter an error that leads to replication interruption. - In this scenario, TiCDC saves the task information. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of `gc-ttl`. - Handling procedures: - 1. Pause the replication task by executing `cdc cli changefeed pause -c `. + 1. Pause the replication task by executing `cdc cli changefeed pause -c `. 2. Wait for about one munite, and then resume the replication task by executing `cdc cli changefeed resume -c `. ### What should I do to handle the OOM that occurs after TiCDC is restarted after a task interruption? -- Update your TiDB cluster and TiCDC cluster to the latest versions. The OOM problem has already been resolved in **v4.0.14 and later v4.0 versions, v5.0.2 and later v5.0 versions, and the latest versions**. +- Update your TiDB cluster and TiCDC cluster to the latest versions. The OOM problem has already been resolved in **v4.0.14 and later v4.0 versions, v5.0.2 and later v5.0 versions, and the latest versions**. - In the above updated versions, you can enable the Unified Sorter to help you sort data in the disk when the system memory is insufficient. To enable this function, you can pass `--sort-engine=unified` to the `cdc cli` command when creating a replication task. For example: @@ -134,34 +87,9 @@ cdc cli changefeed update -c --sort-engine="unified" --sort-dir= > > + Since v4.0.9, TiCDC supports the unified sorter engine. > + TiCDC (the 4.0 version) does not support dynamically modifying the sorting engine yet. Make sure that the changefeed has stopped before modifying the sorter settings. -> + `sort-dir` has different behaviors in different versions. Refer to [compatibility notes for`sort-dir` and `data-dir`](/ticdc/ticdc-overview.md#compatibility-notes-for-sort-dir-and-data-dir), and configure it with caution. +> + `sort-dir` has different behaviors in different versions. Refer to [compatibility notes for`sort-dir` and `data-dir`](/ticdc/ticdc-overview.md#compatibility-notes-for-sort-dir-and-data-dir), and configure it with caution. > + Currently, the unified sorter is an experimental feature. When the number of tables is too large (>=100), the unified sorter might cause performance issues and affect replication throughput. Therefore, it is not recommended to use it in a production environment. Before you enable the unified sorter, make sure that the machine of each TiCDC node has enough disk capacity. If the total size of unprocessed data changes might exceed 1 TB, it is not recommend to use TiCDC for replication. -## What is `gc-ttl` in TiCDC? - -Since v4.0.0-rc.1, PD supports external services in setting the service-level GC safepoint. Any service can register and update its GC safepoint. PD ensures that the key-value data later than this GC safepoint is not cleaned by GC. - -When the replication task is unavailable or interrupted, this feature ensures that the data to be consumed by TiCDC is retained in TiKV without being cleaned by GC. - -When starting the TiCDC server, you can specify the Time To Live (TTL) duration of GC safepoint by configuring `gc-ttl`. You can also [use TiUP to modify](/ticdc/manage-ticdc.md#modify-ticdc-configuration-using-tiup) `gc-ttl`. The default value is 24 hours. In TiCDC, this value means: - -- The maximum time the GC safepoint is retained at the PD after the TiCDC service is stopped. -- The maximum time a replication task can be suspended after the task is interrupted or manually stopped. If the time for a suspended replication task is longer than the value set by `gc-ttl`, the replication task enters the `failed` status, cannot be resumed, and cannot continue to affect the progress of the GC safepoint. - -The second behavior above is introduced in TiCDC v4.0.13 and later versions. The purpose is to prevent a replication task in TiCDC from suspending for too long, causing the GC safepoint of the upstream TiKV cluster not to continue for a long time and retaining too many outdated data versions, thus affecting the performance of the upstream cluster. - -> **Note:** -> -> In some scenarios, for example, when you use TiCDC for incremental replication after full replication with Dumpling/BR, the default 24 hours of `gc-ttl` may not be sufficient. You need to specify an appropriate value for `gc-ttl` when you start the TiCDC server. - -## What is the complete behavior of TiCDC garbage collection (GC) safepoint? - -If a replication task starts after the TiCDC service starts, the TiCDC owner updates the PD service GC safepoint with the smallest value of `checkpoint-ts` among all replication tasks. The service GC safepoint ensures that TiCDC does not delete data generated at that time and after that time. If the replication task is interrupted, or manually stopped, the `checkpoint-ts` of this task does not change. Meanwhile, PD's corresponding service GC safepoint is not updated either. - -If the replication task is suspended longer than the time specified by `gc-ttl`, the replication task enters the `failed` status and cannot be resumed. The PD corresponding service GC safepoint will continue. - -The Time-To-Live (TTL) that TiCDC sets for a service GC safepoint is 24 hours, which means that the GC mechanism does not delete any data if the TiCDC service can be recovered within 24 hours after it is interrupted. - ## How do I handle the `Error 1298: Unknown or incorrect time zone: 'UTC'` error when creating the replication task or replicating data to MySQL? This error is returned when the downstream MySQL does not load the time zone. You can load the time zone by running [`mysql_tzinfo_to_sql`](https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html). After loading the time zone, you can create tasks and replicate data normally. @@ -220,135 +148,14 @@ If the downstream is a special MySQL environment (a public cloud RDS or some MyS > > In China, CST usually stands for China Standard Time. -## How to understand the relationship between the TiCDC time zone and the time zones of the upstream/downstream databases? - -||Upstream time zone| TiCDC time zone|Downstream time zone| -| :-: | :-: | :-: | :-: | -| Configuration method | See [Time Zone Support](/configure-time-zone.md) | Configured using the `--tz` parameter when you start the TiCDC server | Configured using the `time-zone` parameter in `sink-uri` | -| Description | The time zone of the upstream TiDB, which affects DML operations of the timestamp type and DDL operations related to timestamp type columns.| TiCDC assumes that the upstream TiDB's time zone is the same as the TiCDC time zone configuration, and performs related operations on the timestamp column.| The downstream MySQL processes the timestamp in the DML and DDL operations according to the downstream time zone setting.| - - > **Note:** - > - > Be careful when you set the time zone of the TiCDC server, because this time zone is used for converting the time type. Keep the upstream time zone, TiCDC time zone, and the downstream time zone consistent. The TiCDC server chooses its time zone in the following priority: - > - > - TiCDC first uses the time zone specified using `--tz`. - > - When `--tz` is not available, TiCDC tries to read the time zone set using the `TZ` environment variable. - > - When the `TZ` environment variable is not available, TiCDC uses the default time zone of the machine. - -## What is the default behavior of TiCDC if I create a replication task without specifying the configuration file in `--config`? - -If you use the `cdc cli changefeed create` command without specifying the `-config` parameter, TiCDC creates the replication task in the following default behaviors: - -* Replicates all tables except system tables -* Enables the Old Value feature -* Skips replicating tables that do not contain [valid indexes](/ticdc/ticdc-overview.md#restrictions) - ## How do I handle the incompatibility issue of configuration files caused by TiCDC upgrade? Refer to [Notes for compatibility](/ticdc/manage-ticdc.md#notes-for-compatibility). -## Does TiCDC support outputting data changes in the Canal format? - -Yes. To enable Canal output, specify the protocol as `canal` in the `--sink-uri` parameter. For example: - -{{< copyable "shell-regular" >}} - -```shell -cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&protocol=canal" --config changefeed.toml -``` - -> **Note:** -> -> * This feature is introduced in TiCDC 4.0.2. -> * TiCDC currently supports outputting data changes in the Canal format only to MQ sinks such as Kafka and Pulsar. - -For more information, refer to [Create a replication task](/ticdc/manage-ticdc.md#create-a-replication-task). - -## Why does the latency from TiCDC to Kafka become higher and higher? - -* Check [how do I view the state of TiCDC replication tasks](#how-do-i-view-the-state-of-ticdc-replication-tasks). -* Adjust the following parameters of Kafka: - - * Increase the `message.max.bytes` value in `server.properties` to `1073741824` (1 GB). - * Increase the `replica.fetch.max.bytes` value in `server.properties` to `1073741824` (1 GB). - * Increase the `fetch.message.max.bytes` value in `consumer.properties` to make it larger than the `message.max.bytes` value. - -## When TiCDC replicates data to Kafka, does it write all the changes in a transaction into one message? If not, on what basis does it divide the changes? - -No. According to the different distribution strategies configured, TiCDC divides the changes on different bases, including `default`, `row id`, `table`, and `ts`. - -For more information, refer to [Replication task configuration file](/ticdc/manage-ticdc.md#task-configuration-file). - -## When TiCDC replicates data to Kafka, can I control the maximum size of a single message in TiDB? - -Yes. You can set the `max-message-bytes` parameter to control the maximum size of data sent to the Kafka broker each time (optional, `10MB` by default). You can also set `max-batch-size` to specify the maximum number of change records in each Kafka message. Currently, the setting only takes effect when Kafka's `protocol` is `open-protocol` (optional, `16` by default). - -## When TiCDC replicates data to Kafka, does a message contain multiple types of data changes? - -Yes. A single message might contain multiple `update`s or `delete`s, and `update` and `delete` might co-exist. - -## When TiCDC replicates data to Kafka, how do I view the timestamp, table name, and schema name in the output of TiCDC Open Protocol? - -The information is included in the key of Kafka messages. For example: - -```json -{ - "ts":, - "scm":, - "tbl":
, - "t":1 -} -``` - -For more information, refer to [TiCDC Open Protocol event format](/ticdc/ticdc-open-protocol.md#event-format). - -## When TiCDC replicates data to Kafka, how do I know the timestamp of the data changes in a message? - -You can get the unix timestamp by moving `ts` in the key of the Kafka message by 18 bits to the right. - -## How does TiCDC Open Protocol represent `null`? - -In TiCDC Open Protocol, the type code `6` represents `null`. - -| Type | Code | Output Example | Note | -|:--|:--|:--|:--| -| Null | 6 | `{"t":6,"v":null}` | | - -For more information, refer to [TiCDC Open Protocol column type code](/ticdc/ticdc-open-protocol.md#column-type-code). - ## The `start-ts` timestamp of the TiCDC task is quite different from the current time. During the execution of this task, replication is interrupted and an error `[CDC:ErrBufferReachLimit]` occurs Since v4.0.9, you can try to enable the unified sorter feature in your replication task, or use the BR tool for an incremental backup and restore, and then start the TiCDC replication task from a new time. -## How can I tell if a Row Changed Event of TiCDC Open Protocol is an `INSERT` event or an `UPDATE` event? - -If the Old Value feature is not enabled, you cannot tell whether a Row Changed Event of TiCDC Open Protocol is an `INSERT` event or an `UPDATE` event. If the feature is enabled, you can determine the event type by the fields it contains: - -* `UPDATE` event contains both `"p"` and `"u"` fields -* `INSERT` event only contains the `"u"` field -* `DELETE` event only contains the `"d"` field - -For more information, refer to [Open protocol Row Changed Event format](/ticdc/ticdc-open-protocol.md#row-changed-event). - -## How much PD storage does TiCDC use? - -TiCDC uses etcd in PD to store and regularly update the metadata. Because the time interval between the MVCC of etcd and PD's default compaction is one hour, the amount of PD storage that TiCDC uses is proportional to the amount of metadata versions generated within this hour. However, in v4.0.5, v4.0.6, and v4.0.7, TiCDC has a problem of frequent writing, so if there are 1000 tables created or scheduled in an hour, it then takes up all the etcd storage and returns the `etcdserver: mvcc: database space exceeded` error. You need to clean up the etcd storage after getting this error. See [etcd maintaince space-quota](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) for details. It is recommended to upgrade your cluster to v4.0.9 or later versions. - -## Does TiCDC support replicating large transactions? Is there any risk? - -TiCDC provides partial support for large transactions (more than 5 GB in size). Depending on different scenarios, the following risks might exist: - -- When TiCDC's internal processing capacity is insufficient, the replication task error `ErrBufferReachLimit` might occur. -- When TiCDC's internal processing capacity is insufficient or the throughput capacity of TiCDC's downstream is insufficient, out of memory (OOM) might occur. - -If you encounter an error above, it is recommended to use BR to restore the incremental data of large transactions. The detailed operations are as follows: - -1. Record the `checkpoint-ts` of the changefeed that is terminated due to large transactions, use this TSO as the `--lastbackupts` of the BR incremental backup, and execute [incremental data backup](/br/use-br-command-line-tool.md#back-up-incremental-data). -2. After backing up the incremental data, you can find a log record similar to `["Full backup Failed summary : total backup ranges: 0, total success: 0, total failed: 0"] [BackupTS=421758868510212097]` in the BR log output. Record the `BackupTS` in this log. -3. [Restore the incremental data](/br/use-br-command-line-tool.md#restore-incremental-data). -4. Create a new changefeed and start the replication task from `BackupTS`. -5. Delete the old changefeed. - ## When the downstream of a changefeed is a database similar to MySQL and TiCDC executes a time-consuming DDL statement, all other changefeeds are blocked. How should I handle the issue? 1. Pause the execution of the changefeed that contains the time-consuming DDL statement. Then you can see that other changefeeds are no longer blocked. @@ -390,19 +197,13 @@ To fix the error, take the following steps: ## The `[tikv:9006]GC life time is shorter than transaction duration, transaction starts at xx, GC safe point is yy` error is reported when I use TiCDC to create a changefeed -Solution: You need to execute the `pd-ctl service-gc-safepoint --pd ` command to query the current GC safepoint and service GC safepoint. If the GC safepoint is smaller than the `start-ts` of the TiCDC replication task (changefeed), you can directly add the `--disable-gc-check` option to the `cdc cli create changefeed` command to create a changefeed. +You need to run the `pd-ctl service-gc-safepoint --pd ` command to query the current GC safepoint and service GC safepoint. If the GC safepoint is smaller than the `start-ts` of the TiCDC replication task (changefeed), you can directly add the `--disable-gc-check` option to the `cdc cli create changefeed` command to create a changefeed. If the result of `pd-ctl service-gc-safepoint --pd ` does not have `gc_worker service_id`: - If your PD version is v4.0.8 or earlier, refer to [PD issue #3128](https://github.com/tikv/pd/issues/3128) for details. - If your PD is upgraded from v4.0.8 or an earlier version to a later version, refer to [PD issue #3366](https://github.com/tikv/pd/issues/3366) for details. -## `enable-old-value` is set to `true` when I create a TiCDC replication task, but `INSERT`/`UPDATE` statements from the upstream become `REPLACE INTO` after being replicated to the downstream - -When a changefeed is created in TiCDC, the `safe-mode` setting defaults to `true`, which generates the `REPLACE INTO` statement to execute for the upstream `INSERT`/`UPDATE` statements. - -Currently, users cannot modify the `safe-mode` setting, so this issue currently has no solution. - ## When I use TiCDC to replicate messages to Kafka, Kafka returns the `Message was too large` error For TiCDC v4.0.8 or earlier versions, you cannot effectively control the size of the message output to Kafka only by configuring the `max-message-bytes` setting for Kafka in the Sink URI. To control the message size, you also need to increase the limit on the bytes of messages to be received by Kafka. To add such a limit, add the following configuration to the Kafka server configuration. @@ -434,43 +235,3 @@ If you want to skip this DDL statement that goes wrong, set the start-ts of the cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 415241823337054210 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` - -## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? - -Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: - -{{< copyable "sql" >}} - -```sql -mysql root@127.0.0.1:test> show create table test; -+-------+----------------------------------------------------------------------------------+ -| Table | Create Table | -+-------+----------------------------------------------------------------------------------+ -| test | CREATE TABLE `test` ( | -| | `id` int(11) NOT NULL, | -| | `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | -| | PRIMARY KEY (`id`) | -| | ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | -+-------+----------------------------------------------------------------------------------+ -1 row in set -``` - -From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. - -Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. - -## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? - -When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: - -- `Select` -- `Index` -- `Insert` -- `Update` -- `Delete` -- `Create` -- `Drop` -- `Alter` -- `Create View` - -If you need to replicate `recover table` to the downstream TiDB, you should have the `Super` permission. From 2b7219ed8ec8b92734fc4a211bdd501686b88a88 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Wed, 22 Jun 2022 14:17:13 +0800 Subject: [PATCH 2/8] Add two FAQs --- ticdc/ticdc-faq.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/ticdc/ticdc-faq.md b/ticdc/ticdc-faq.md index d93fdb402e7ad..e0c3a897d91d7 100644 --- a/ticdc/ticdc-faq.md +++ b/ticdc/ticdc-faq.md @@ -204,12 +204,6 @@ If you encounter an error above, it is recommended to use BR to restore the incr 4. Create a new changefeed and start the replication task from `BackupTS`. 5. Delete the old changefeed. -## `enable-old-value` is set to `true` when I create a TiCDC replication task, but `INSERT`/`UPDATE` statements from the upstream become `REPLACE INTO` after being replicated to the downstream - -When a changefeed is created in TiCDC, the `safe-mode` setting defaults to `true`, which generates the `REPLACE INTO` statement to execute for the upstream `INSERT`/`UPDATE` statements. - -Currently, users cannot modify the `safe-mode` setting, so this issue currently has no solution. - ## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: @@ -234,6 +228,12 @@ From the result, you can see that the table schema before and after the replicat Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. +## `enable-old-value` is set to `true` when I create a TiCDC replication task, but `INSERT`/`UPDATE` statements from the upstream become `REPLACE INTO` after being replicated to the downstream + +When a changefeed is created in TiCDC, the `safe-mode` setting defaults to `true`, which generates the `REPLACE INTO` statement to execute for the upstream `INSERT`/`UPDATE` statements. + +Currently, users cannot modify the `safe-mode` setting, so this issue currently has no solution. + ## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: @@ -252,4 +252,8 @@ If you need to replicate `recover table` to the downstream TiDB, you should have ## Why does TiCDC use disks? When does TiCDC write to disks? Does TiCDC use memory buffer to improve replication performance? -When the upstream write traffic is at business peak hours, the downstream may fail to consume data in a timely manner, resulting in data pile-up. TiCDC uses disks to process the data that is piled up. TiCDC needs to write data to disks during normal operation. However, this is not usually the bottleneck for replication throughput and replication latency, given that writing to disks latency within a hundred milliseconds. tiCDC also makes use of memory to boost accelerated reads from disk to improve synchronous performance. +When upstream write traffic is at peak hours, the downstream may fail to consume all data in a timely manner, resulting in data pile-up. TiCDC uses disks to process the data that is piled up. TiCDC needs to write data to disks during normal operation. However, this is not usually the bottleneck for replication throughput and replication latency, given that writing to disks only results in latency within a hundred milliseconds. TiCDC also uses memory to accelerate reading data from disks to improve replication performance. + +## Why does replication using TiCDC stall or even stop after data restore using TiDB Lightning and BR? + +Currently, TiCDC is not yet fully compatible with TiDB Lightning and BR. Therefore, please avoid using TiDB Lightning and BR on tables that are replicated by TiCDC. From d404e98b3750497377e07d84ed6fa796fa2c84f9 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Wed, 22 Jun 2022 14:37:29 +0800 Subject: [PATCH 3/8] change FAQ to FAQs --- TOC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TOC.md b/TOC.md index dbee00780e8d6..f8878a2e6206b 100644 --- a/TOC.md +++ b/TOC.md @@ -486,7 +486,7 @@ - [Deploy](/ticdc/deploy-ticdc.md) - [Maintain](/ticdc/manage-ticdc.md) - [Troubleshoot](/ticdc/troubleshoot-ticdc.md) - - [FAQ](/ticdc/ticdc-faq.md) + - [FAQs](/ticdc/ticdc-faq.md) - [Monitor](/ticdc/monitor-ticdc.md) - [Alert Rules](/ticdc/ticdc-alert-rules.md) - Reference From e20a17ef537a36bdf2c2177ddfd26e84e842ebd4 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Wed, 22 Jun 2022 14:39:34 +0800 Subject: [PATCH 4/8] fix a link --- ticdc/ticdc-alert-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-alert-rules.md b/ticdc/ticdc-alert-rules.md index e0dab477a2b47..6cded5900b4ad 100644 --- a/ticdc/ticdc-alert-rules.md +++ b/ticdc/ticdc-alert-rules.md @@ -182,7 +182,7 @@ Warning alerts are a reminder for an issue or error. - Solution: - There are many possible root causes. See [Troubleshoot TiCDC](/ticdc/). + There are many possible root causes. See [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). ### `ticdc_memory_abnormal` From 69a4d31e051069816bf9dc8eb6f2abfb6ea6a2dd Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Mon, 27 Jun 2022 12:23:11 +0800 Subject: [PATCH 5/8] sync from Chinese modification --- TOC.md | 9 +++++---- production-deployment-using-tiup.md | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/TOC.md b/TOC.md index f8878a2e6206b..6695cbd445304 100644 --- a/TOC.md +++ b/TOC.md @@ -485,16 +485,17 @@ - [Overview](/ticdc/ticdc-overview.md) - [Deploy](/ticdc/deploy-ticdc.md) - [Maintain](/ticdc/manage-ticdc.md) + - Monitor and Alert + - [Monitor](/ticdc/monitor-ticdc.md) + - [Alert Rules](/ticdc/ticdc-alert-rules.md) - [Troubleshoot](/ticdc/troubleshoot-ticdc.md) - - [FAQs](/ticdc/ticdc-faq.md) - - [Monitor](/ticdc/monitor-ticdc.md) - - [Alert Rules](/ticdc/ticdc-alert-rules.md) - Reference - [TiCDC OpenAPI](/ticdc/ticdc-open-api.md) - [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) - [TiCDC Avro Protocol](/ticdc/ticdc-avro-protocol.md) - [TiCDC Canal-JSON Protocol](/ticdc/ticdc-canal-json.md) - - [Integrate TiDB with Confluent Platform](/ticdc/integrate-confluent-using-ticdc.md) + - [Integrate TiDB with Confluent Platform](/ticdc/integrate-confluent-using-ticdc.md) + - [FAQs](/ticdc/ticdc-faq.md) - [Glossary](/ticdc/ticdc-glossary.md) - [Dumpling](/dumpling-overview.md) - sync-diff-inspector diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index 583c4ee7b1eae..8eaf1ec22e6ac 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -431,5 +431,5 @@ If you have deployed [TiFlash](/tiflash/tiflash-overview.md) along with the TiDB If you have deployed [TiCDC](/ticdc/ticdc-overview.md) along with the TiDB cluster, see the following documents: - [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) -- [TiCDC FAQs](/ticdc/ticdc-faq.md) - [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md) +- [TiCDC FAQs](/ticdc/ticdc-faq.md) From b950251091ff7ad490960f7c184858decfe58df7 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Mon, 27 Jun 2022 14:35:22 +0800 Subject: [PATCH 6/8] Apply suggestions from code review Co-authored-by: Lilian Lee --- ticdc/ticdc-faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ticdc/ticdc-faq.md b/ticdc/ticdc-faq.md index e0c3a897d91d7..88a6287f8513a 100644 --- a/ticdc/ticdc-faq.md +++ b/ticdc/ticdc-faq.md @@ -1,9 +1,9 @@ --- -title: FAQs +title: TiCDC FAQs summary: Learn the FAQs you might encounter when you use TiCDC. --- -# FAQs +# TiCDC FAQs This document introduces the common questions that you might encounter when using TiCDC. From 8ff6c8b5ccda5de93834be3c5a6ea332e5f43546 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Mon, 27 Jun 2022 14:39:10 +0800 Subject: [PATCH 7/8] Update TOC.md Co-authored-by: Lilian Lee --- TOC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TOC.md b/TOC.md index 6695cbd445304..91a2e3b7c7049 100644 --- a/TOC.md +++ b/TOC.md @@ -486,7 +486,7 @@ - [Deploy](/ticdc/deploy-ticdc.md) - [Maintain](/ticdc/manage-ticdc.md) - Monitor and Alert - - [Monitor](/ticdc/monitor-ticdc.md) + - [Monitoring Metrics](/ticdc/monitor-ticdc.md) - [Alert Rules](/ticdc/ticdc-alert-rules.md) - [Troubleshoot](/ticdc/troubleshoot-ticdc.md) - Reference From ab551d5795bd58d44573cdeacb753d502c689a74 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Mon, 27 Jun 2022 15:42:01 +0800 Subject: [PATCH 8/8] trigger pull check --- migrate-from-tidb-to-tidb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migrate-from-tidb-to-tidb.md b/migrate-from-tidb-to-tidb.md index e926a572776d4..3a543f0fdfddd 100644 --- a/migrate-from-tidb-to-tidb.md +++ b/migrate-from-tidb-to-tidb.md @@ -238,7 +238,7 @@ After setting up the environment, you can use the backup and restore functions o 3. Enable GC. - In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/ticdc-faq.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). + In incremental migration using TiCDC, GC only removes history data that is replicated. Therefore, after creating a changefeed, you need to run the following command to enable GC. For details, see [What is the complete behavior of TiCDC garbage collection (GC) safepoint?](/ticdc/ticdc-faq.md#what-is-the-complete-behavior-of-ticdc-garbage-collection-gc-safepoint). {{< copyable "sql" >}}