Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: fix some error #12624

Merged
merged 4 commits into from
Jan 11, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions migrate-from-tidb-to-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,12 +160,12 @@ summary: 了解如何将数据从 TiDB 集群迁移至与 MySQL 兼容的数据
在上游集群中,执行以下命令创建从上游到下游集群的同步链路:

```shell
tiup ctl:<cluster-version> cdc changefeed create --pd=http://127.0.0.1:2379 --sink-uri="mysql://root:@127.0.0.1:3306" --changefeed-id="upstream-to-downstream" --start-ts="434217889191428107"
tiup ctl:<cluster-version> cdc changefeed create --server=http://127.0.0.1:2379 --sink-uri="mysql://root:@127.0.0.1:3306" --changefeed-id="upstream-to-downstream" --start-ts="434217889191428107"
```

以上命令中:

- `--pd`:实际的上游集群的地址
- `--server`:TiCDC 集群任意一节点地址
- `--sink-uri`:同步任务下游的地址
- `--changefeed-id`:同步任务的 ID,格式需要符合正则表达式 `^[a-zA-Z0-9]+(\-[a-zA-Z0-9]+)*$`
- `--start-ts`:TiCDC 同步的起点,需要设置为实际的备份时间点,也就是[第 2 步:迁移全量数据](/migrate-from-tidb-to-mysql.md#第-2-步迁移全量数据)中 “备份数据” 提到的 BackupTS
Expand Down
8 changes: 4 additions & 4 deletions migrate-from-tidb-to-tidb.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,12 @@ aliases: ['/zh/tidb/dev/incremental-replication-between-clusters/']
在上游集群中,执行以下命令创建从上游到下游集群的同步链路:

```shell
tiup cdc cli changefeed create --pd=http://172.16.6.122:2379 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="upstream-to-downstream" --start-ts="431434047157698561"
tiup cdc cli changefeed create --server=http://172.16.6.122:8300 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="upstream-to-downstream" --start-ts="431434047157698561"
```

以上命令中:

- `--pd`:实际的上游集群的地址
- `--server`:TiCDC 集群中任意一个节点的地址
- `--sink-uri`:同步任务下游的地址
- `--changefeed-id`:同步任务的 ID,格式需要符合正则表达式 ^[a-zA-Z0-9]+(\-[a-zA-Z0-9]+)*$
- `--start-ts`:TiCDC 同步的起点,需要设置为实际的备份时间点,也就是[第 2 步:迁移全量数据](/migrate-from-tidb-to-mysql.md#第-2-步迁移全量数据)中 “备份数据” 提到的 BackupTS
Expand Down Expand Up @@ -274,7 +274,7 @@ aliases: ['/zh/tidb/dev/incremental-replication-between-clusters/']

```shell
# 停止旧集群到新集群的 changefeed
tiup cdc cli changefeed pause -c "upstream-to-downstream" --pd=http://172.16.6.122:2379
tiup cdc cli changefeed pause -c "upstream-to-downstream" --server=http://172.16.6.122:8300

# 查看 changefeed 状态
tiup cdc cli changefeed list
Expand All @@ -297,7 +297,7 @@ aliases: ['/zh/tidb/dev/incremental-replication-between-clusters/']
2. 创建下游到上游集群的 changefeed。由于此时上下游数据是一致的,且没有新数据写入,因此可以不指定 start-ts,默认为当前时间:

```shell
tiup cdc cli changefeed create --pd=http://172.16.6.125:2379 --sink-uri="mysql://root:@172.16.6.122:4000" --changefeed-id="downstream -to-upstream"
tiup cdc cli changefeed create --server=http://172.16.6.125:8300 --sink-uri="mysql://root:@172.16.6.122:4000" --changefeed-id="downstream -to-upstream"
```

3. 将写业务迁移到下游集群,观察一段时间后,等新集群表现稳定,便可以弃用原集群。
6 changes: 3 additions & 3 deletions replicate-between-primary-and-secondary-clusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,12 +235,12 @@ summary: 了解如何配置一个 TiDB 集群以及该集群的 TiDB 或 MySQL
在上游集群中,执行以下命令创建从上游到下游集群的同步链路:

```shell
tiup cdc cli changefeed create --pd=http://172.16.6.122:2379 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="primary-to-secondary" --start-ts="431434047157698561"
tiup cdc cli changefeed create --server=http://172.16.6.122:8300 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="primary-to-secondary" --start-ts="431434047157698561"
```

以上命令中:

- `--pd`:实际的上游集群的地址
- `--server`:TiCDC 集群任意一节点的地址
- `--sink-uri`:同步任务下游的地址
- `--start-ts`:TiCDC 同步的起点,需要设置为实际的备份时间点(也就是[第 2 步:迁移全量数据](#第-2-步迁移全量数据)提到的 BackupTS)

Expand Down Expand Up @@ -314,5 +314,5 @@ tiup cdc redo apply --storage "s3://redo?access-key=minio&secret-access-key=mini

```shell
# 创建 changefeed
tiup cdc cli changefeed create --pd=http://172.16.6.122:2379 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="primary-to-secondary"
tiup cdc cli changefeed create --server=http://172.16.6.122:8300 --sink-uri="mysql://root:@172.16.6.125:4000" --changefeed-id="primary-to-secondary"
```
6 changes: 3 additions & 3 deletions replicate-data-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ aliases: ['/zh/tidb/dev/replicate-incremental-data-to-kafka/']
2. 创建一个 changefeed,将增量数据输出到 Kafka:

```shell
tiup ctl:<cluster-version> cdc changefeed create --pd="http://127.0.0.1:2379" --sink-uri="kafka://127.0.0.1:9092/kafka-topic-name?protocol=canal-json" --changefeed-id="kafka-changefeed" --config="changefeed.conf"
tiup ctl:<cluster-version> cdc changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092/kafka-topic-name?protocol=canal-json" --changefeed-id="kafka-changefeed" --config="changefeed.conf"
```

- 如果命令执行成功,将会返回被创建的 changefeed 的相关信息,包含被创建的 changefeed 的 ID 以及相关信息,内容如下:
Expand All @@ -74,13 +74,13 @@ aliases: ['/zh/tidb/dev/replicate-incremental-data-to-kafka/']
生产环境下 Kafka 集群通常有多个 broker 节点,你可以在 sink-uri 中配置多个 broker 的访问地址,这有助于提升 changefeed 到 Kafka 集群访问的稳定性,当部分被配置的 Kafka 节点故障的时候,changefeed 依旧可以正常工作。假设 Kafka 集群中有 3 个 broker 节点,地址分别为 127.0.0.1:9092 / 127.0.0.2:9092 / 127.0.0.3:9092,可以参考如下 sink-uri 创建 changefeed:

```shell
tiup ctl:<cluster-version> cdc changefeed create --pd="http://127.0.0.1:2379" --sink-uri="kafka://127.0.0.1:9092,127.0.0.2:9092,127.0.0.3:9092/kafka-topic-name?protocol=canal-json&partition-num=3&replication-factor=1&max-message-bytes=1048576" --config="changefeed.conf"
tiup ctl:<cluster-version> cdc changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092,127.0.0.2:9092,127.0.0.3:9092/kafka-topic-name?protocol=canal-json&partition-num=3&replication-factor=1&max-message-bytes=1048576" --config="changefeed.conf"
```

3. Changefeed 创建成功后,执行如下命令,查看 changefeed 的状态:

```shell
tiup ctl:<cluster-version> cdc changefeed list --pd="http://127.0.0.1:2379"
tiup ctl:<cluster-version> cdc changefeed list --server="http://127.0.0.1:8300"
```

可以参考[管理 Changefeed](/ticdc/ticdc-manage-changefeed.md),对 changefeed 状态进行管理。
Expand Down
4 changes: 2 additions & 2 deletions ticdc/deploy-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ tiup cluster upgrade <cluster-name> <cluster-version> --transfer-timeout 600

## 使用 TiUP 变更 TiCDC 集群配置

本节介绍如何使用 TiUP 的 [`tiup cluster edit-config`](/tiup/tiup-component-cluster-edit-config.md) 命令来修改 TiCDC 的配置。在以下例子中,假设需要把 TiCDC 的 `gc-ttl` 从默认值 `86400` 修改为 `3600`,即 1 小时。
本节介绍如何使用 TiUP 的 [`tiup cluster edit-config`](/tiup/tiup-component-cluster-edit-config.md) 命令来修改 TiCDC 的配置。在以下例子中,假设需要把 TiCDC 的 `gc-ttl` 从默认值 `86400` 修改为 `172800`,即 48 小时。

1. 执行 `tiup cluster edit-config` 命令,注意将 `<cluster-name>` 替换成实际的集群名:

Expand All @@ -131,7 +131,7 @@ tiup cluster upgrade <cluster-name> <cluster-version> --transfer-timeout 600
pump: {}
drainer: {}
cdc:
gc-ttl: 86400
gc-ttl: 172800
```

以上把 TiCDC 的 `gc-ttl` 的值设置为 24 小时。
Expand Down
6 changes: 3 additions & 3 deletions ticdc/integrate-confluent-using-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Confluent 是一个兼容 Apache Kafka 的数据流平台,能够访问、存
2. 创建一个 changefeed,将增量数据输出到 Confluent Cloud:

```shell
tiup ctl:<cluster-version> cdc changefeed create --pd="http://127.0.0.1:2379" --sink-uri="kafka://<broker_endpoint>/ticdc-meta?protocol=avro&replication-factor=3&enable-tls=true&auto-create-topic=true&sasl-mechanism=plain&sasl-user=<broker_api_key>&sasl-password=<broker_api_secret>" --schema-registry="https://<schema_registry_api_key>:<schema_registry_api_secret>@<schema_registry_endpoint>" --changefeed-id="confluent-changefeed" --config changefeed.conf
tiup ctl:<cluster-version> cdc changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://<broker_endpoint>/ticdc-meta?protocol=avro&replication-factor=3&enable-tls=true&auto-create-topic=true&sasl-mechanism=plain&sasl-user=<broker_api_key>&sasl-password=<broker_api_secret>" --schema-registry="https://<schema_registry_api_key>:<schema_registry_api_secret>@<schema_registry_endpoint>" --changefeed-id="confluent-changefeed" --config changefeed.conf
```

将如下字段替换为[第 2 步:创建 Access Key Pair](#第-2-步创建-access-key-pair)中创建和记录的值:
Expand All @@ -114,7 +114,7 @@ Confluent 是一个兼容 Apache Kafka 的数据流平台,能够访问、存
其中 `<schema_registry_api_secret>` 需要经过 [HTML URL 编码](https://www.w3schools.com/tags/ref_urlencode.asp)后再替换,替换完毕后示例如下:

```shell
tiup ctl:<cluster-version> cdc changefeed create --pd="http://127.0.0.1:2379" --sink-uri="kafka://xxx-xxxxx.ap-east-1.aws.confluent.cloud:9092/ticdc-meta?protocol=avro&replication-factor=3&enable-tls=true&auto-create-topic=true&sasl-mechanism=plain&sasl-user=L5WWA4GK4NAT2EQV&sasl-password=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --schema-registry="https://7NBH2CAFM2LMGTH7:[email protected]" --changefeed-id="confluent-changefeed" --config changefeed.conf
tiup ctl:<cluster-version> cdc changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://xxx-xxxxx.ap-east-1.aws.confluent.cloud:9092/ticdc-meta?protocol=avro&replication-factor=3&enable-tls=true&auto-create-topic=true&sasl-mechanism=plain&sasl-user=L5WWA4GK4NAT2EQV&sasl-password=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --schema-registry="https://7NBH2CAFM2LMGTH7:[email protected]" --changefeed-id="confluent-changefeed" --config changefeed.conf
```

- 如果命令执行成功,将会返回被创建的 changefeed 的相关信息,包含被创建的 changefeed 的 ID 以及相关信息,内容如下:
Expand All @@ -130,7 +130,7 @@ Confluent 是一个兼容 Apache Kafka 的数据流平台,能够访问、存
3. Changefeed 创建成功后,执行如下命令,查看 changefeed 的状态:

```shell
tiup ctl:<cluster-version> cdc changefeed list --pd="http://127.0.0.1:2379"
tiup ctl:<cluster-version> cdc changefeed list --server="http://127.0.0.1:8300"
```

可以参考[管理 Changefeed](/ticdc/ticdc-manage-changefeed.md),对 changefeed 状态进行管理。
Expand Down
2 changes: 1 addition & 1 deletion ticdc/monitor-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ aliases: ['/zh/tidb/dev/ticdc-grafana-dashboard']
本文档的对指标的介绍基于以下同步任务,即使用默认配置同步数据到 MySQL。

```shell
cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="mysql://root:[email protected]:3306/" --changefeed-id="simple-replication-task"
cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://root:[email protected]:3306/" --changefeed-id="simple-replication-task"
```

下图显示了 TiCDC Dashboard 各监控面板:
Expand Down
6 changes: 3 additions & 3 deletions ticdc/ticdc-avro-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Avro 是由 [Apache Avro™](https://avro.apache.org/) 定义的一种数据交
使用 Avro 时的配置样例如下所示:

```shell
cdc cli changefeed create --pd=http://127.0.0.1:2379 --changefeed-id="kafka-avro" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-avro" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
```

```shell
Expand All @@ -37,7 +37,7 @@ dispatchers = [
配置样例如下所示:

```shell
cdc cli changefeed create --pd=http://127.0.0.1:2379 --changefeed-id="kafka-avro-enable-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro&enable-tidb-extension=true" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-avro-enable-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro&enable-tidb-extension=true" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
```

```shell
Expand Down Expand Up @@ -201,7 +201,7 @@ Column 数据格式即 Key/Value 数据格式中的 `{{ColumnValueBlock}}` 部
配置样例如下所示:

```shell
cdc cli changefeed create --pd=http://127.0.0.1:2379 --changefeed-id="kafka-avro-string-option" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro&avro-decimal-handling-mode=string&avro-bigint-unsigned-handling-mode=string" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-avro-string-option" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=avro&avro-decimal-handling-mode=string&avro-bigint-unsigned-handling-mode=string" --schema-registry=http://127.0.0.1:8081 --config changefeed_config.toml
```

```shell
Expand Down
4 changes: 2 additions & 2 deletions ticdc/ticdc-canal-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Event 分为三类:
使用 `Canal-JSON` 时的配置样例如下所示:

```shell
cdc cli changefeed create --pd=http://127.0.0.1:2379 --changefeed-id="kafka-canal-json" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json"
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json"
```

## TiDB 扩展字段
Expand All @@ -33,7 +33,7 @@ Canal-JSON 协议本是为 MySQL 设计的,其中并不包含 TiDB 专有的 C
配置样例如下所示:

```shell
cdc cli changefeed create --pd=http://127.0.0.1:2379 --changefeed-id="kafka-canal-json-enable-tidb-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json&enable-tidb-extension=true"
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json-enable-tidb-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json&enable-tidb-extension=true"
```

## Message 格式定义
Expand Down
2 changes: 1 addition & 1 deletion ticdc/ticdc-manage-changefeed.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ cdc cli changefeed create --server=http://10.0.10.25:8300 --sink-uri="mysql://ro
```shell
Create changefeed successfully!
ID: simple-replication-task
Info: {"upstream_id":7178706266519722477,"namespace":"default","id":"simple-replication-task","sink_uri":"mysql://root:[email protected]:4000/?time-zone=","create_time":"2022-12-19T15:05:46.679218+08:00","start_ts":438156275634929669,"engine":"unified","config":{"case_sensitive":true,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":3600000000000,"filter":{"rules":["test.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storag
Info: {"upstream_id":7178706266519722477,"namespace":"default","id":"simple-replication-task","sink_uri":"mysql://root:[email protected]:4000/?time-zone=","create_time":"2022-12-19T15:05:46.679218+08:00","start_ts":438156275634929669,"engine":"unified","config":{"case_sensitive":true,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":3600000000000,"filter":{"rules":["test.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storage":""}},"state":"normal","creator_version":"v6.5.0"}
```

## 查询同步任务列表
Expand Down
2 changes: 1 addition & 1 deletion ticdc/ticdc-sink-to-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Sink URI 用于指定 TiCDC 目标系统的连接信息,遵循以下格式:
一个通用的配置样例如下所示:

```shell
--sink-uri="mysql://root:[email protected]:3306/?worker-count=16&max-txn-row=5000&transaction-atomicity=table"
--sink-uri="mysql://root:[email protected]:3306/?worker-count=32&max-txn-row=2048"
```

URI 中可配置的参数如下:
Expand Down