tool sync from kafka to mysql #441
Conversation
july2993
force-pushed
the
hjh/ntools
branch
2 times, most recently
from
dbe3d66 to
681458d
Compare
|
is it reviewable now? |
|
@GregoryIan @WangXiangUSTC PTAL |
|
@kennytm @WangXiangUSTC @GregoryIan PTAL |
arbiter/README.md
Outdated
| Arbiter | ||
| ========== | ||
|
|
||
| **Arbiter** 是一个从 Kafka 获取 Binlog 增量同步数据到 TiDB 的工具. |
There was a problem hiding this comment.
Chinese first? after LGTM add add Englist?
There was a problem hiding this comment.
ok,but you use english document to replace it? write more english, speak more english 😄
There was a problem hiding this comment.
add eng version while 44b1b1c
|
question about binlog component: do we filter binlog at drainer , or drainer and arbiter? |
drainer support, arbiter don't support now |
| return 0, 0, errors.NotFoundf("no checkpoint for: %s", c.topicName) | ||
| } | ||
|
|
||
| return |
There was a problem hiding this comment.
need check rows.Err()?
There was a problem hiding this comment.
no need, we use QueryRow see https://golang.org/pkg/database/sql/#DB.QueryRow
arbiter/README.md
Outdated
| - ts: the timestamp checkpoint | ||
| - status: | ||
| * 0 | ||
| if quit normally, all Binlog data <= ts has synced to downstream. |
There was a problem hiding this comment.
how about status = 0 and quit unexpectedly?
There was a problem hiding this comment.
we guarantee
if status = 0, all Binlog data <= ts has synced to downstream and no later data synced partially.
arbiter/README_CN.md
Outdated
|
|
||
| 写下游需时的直方图。标签: | ||
|
|
||
| * **type**: `exec` `commit` 执行 SQL 跟提交时的花时。 |
There was a problem hiding this comment.
use 耗时 for simple?
|
LGTM |
|
/run-all-tests |
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
|
/run-all-tests |
1 similar comment
|
/run-all-tests |
What problem does this PR solve?
relate issue: https://internal.pingcap.net/jira/browse/TOOL-771
you can read READMD first
What is changed and how it works?
add a tool named
arbiternowthis pr also fix
make checkwiil add to CI if this is mergedCheck List
Tests
run TiDB + drainer ( dest-type = kafka)
run Arbiter Server (kill -9 per minute and restart)
continue run tests/binlog again and again
Code changes
Side effects
Related changes