Support -T/--tables-list option #81

Win-Man · 2020-05-24T06:51:23Z

Support -T/--tables-list option : issue#64

Test:

prepare data

create database db1;
create database db2;
use db1;
create table t1(id int primary key auto_increment,name varchar(20));
insert into t1(name) values("zhangsan"),("lisi"),("wangwu");
create table t2(id int primary key auto_increment,age int);
insert into t2(age) values(10),(20),(30);
create view user as select t1.id,t1.name,t2.age from t1 left join t2 on t1.id = t2.id;

use db2;
create table t1(id int primary key auto_increment,name varchar(20));
insert into t1(name) values("Jack"),("Mark"),("Tom");
create table t2(id int primary key auto_increment,age int);
insert into t2(age) values(5),(15),(25);

dumpling test cases

./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -o dumpling_B_db1

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -o dumpling_B_db1
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:55:36.726 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:55:36.727 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]

$ ll dumpling_B_db1
total 48
drwxr-xr-x  8 shengang  staff  256  5 24 14:55 .
drwxr-xr-x  4 shengang  staff  128  5 24 14:55 ..
-rwxr-xr-x  1 shengang  staff   65  5 24 14:55 db1-schema-create.sql
-rwxr-xr-x  1 shengang  staff  200  5 24 14:55 db1.t1-schema.sql
-rwxr-xr-x  1 shengang  staff   95  5 24 14:55 db1.t1.0.sql
-rwxr-xr-x  1 shengang  staff  195  5 24 14:55 db1.t2-schema.sql
-rwxr-xr-x  1 shengang  staff   77  5 24 14:55 db1.t2.0.sql
-rwxr-xr-x  1 shengang  staff  140  5 24 14:55 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -T t1,t2 -o dumpling_B_db1_T_t1_t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -T t1,t2 -o dumpling_B_db1_T_t1_t2
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:56:19.006 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:56:19.006 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]

$ ll dumpling_B_db1_T_t1_t2
total 48
drwxr-xr-x  8 shengang  staff  256  5 24 14:56 .
drwxr-xr-x  5 shengang  staff  160  5 24 14:56 ..
-rwxr-xr-x  1 shengang  staff   65  5 24 14:56 db1-schema-create.sql
-rwxr-xr-x  1 shengang  staff  200  5 24 14:56 db1.t1-schema.sql
-rwxr-xr-x  1 shengang  staff   95  5 24 14:56 db1.t1.0.sql
-rwxr-xr-x  1 shengang  staff  195  5 24 14:56 db1.t2-schema.sql
-rwxr-xr-x  1 shengang  staff   77  5 24 14:56 db1.t2.0.sql
-rwxr-xr-x  1 shengang  staff  140  5 24 14:56 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -T db1.t1,db1.t2 -o dumpling_B_db1_T_db1.t1_db2._t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -B db1 -T db1.t1,db1.t2 -o dumpling_B_db1_T_db1.t1_db2._t2
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:56:52.769 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:56:52.769 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]

$ ll dumpling_B_db1_T_db1.t1_db2._t2
total 48
drwxr-xr-x  8 shengang  staff  256  5 24 14:56 .
drwxr-xr-x  6 shengang  staff  192  5 24 14:56 ..
-rwxr-xr-x  1 shengang  staff   65  5 24 14:56 db1-schema-create.sql
-rwxr-xr-x  1 shengang  staff  200  5 24 14:56 db1.t1-schema.sql
-rwxr-xr-x  1 shengang  staff   95  5 24 14:56 db1.t1.0.sql
-rwxr-xr-x  1 shengang  staff  195  5 24 14:56 db1.t2-schema.sql
-rwxr-xr-x  1 shengang  staff   77  5 24 14:56 db1.t2.0.sql
-rwxr-xr-x  1 shengang  staff  140  5 24 14:56 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -B db1,db2 -T t1 -o dumpling_B_db1_db2_T_t1

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -B db1,db2 -T t1 -o dumpling_B_db1_db2_T_t1
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:57:41.821 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:57:41.821 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]
dump failed: SHOW CREATE DATABASE db1,db2: err = Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 25 near ",db2"
goroutine 1 [running]:
runtime/debug.Stack(0x16144e0, 0xc0001aa020, 0x1615280)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/pingcap/dumpling/v4/export.withStack(0x16144e0, 0xc0001aa020, 0xc0000ca070, 0xc000028320)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/error.go:40 +0x8d
github.com/pingcap/dumpling/v4/export.simpleQuery(0xc0001820c0, 0xc000028320, 0x1c, 0xc000127ab0, 0x1, 0xc000028320)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/sql.go:367 +0x152
github.com/pingcap/dumpling/v4/export.ShowCreateDatabase(0xc0001820c0, 0x7ffeefbff94f, 0x7, 0xc000127bd0, 0x1428e94, 0x7ffeefbff960, 0x17)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/sql.go:38 +0x10c
github.com/pingcap/dumpling/v4/export.dumpDatabases(0x161e960, 0xc0000ca070, 0xc00009e300, 0xc0001820c0, 0x161cd20, 0xc00012c088, 0x0, 0x400)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/dump.go:117 +0x39a
github.com/pingcap/dumpling/v4/export.Dump(0xc00009e300, 0x0, 0x0)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/dump.go:99 +0x83d
main.main()
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/cmd/dumpling/main.go:136 +0x10e0

$ ll dumpling_B_db1_db2_T_t1
total 8
drwxr-xr-x  3 shengang  staff   96  5 24 14:57 .
drwxr-xr-x  7 shengang  staff  224  5 24 14:57 ..
-rwxr-xr-x  1 shengang  staff  102  5 24 14:57 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -T db1.t1,db1.t2 -o dumpling_T_db1.t1_db2.t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -T db1.t1,db1.t2 -o dumpling_T_db1.t1_db2.t2
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:58:16.656 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:58:16.656 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]

$ ll dumpling_T_db1.t1_db2.t2
total 40
drwxr-xr-x  7 shengang  staff  224  5 24 14:58 .
drwxr-xr-x  8 shengang  staff  256  5 24 14:58 ..
-rwxr-xr-x  1 shengang  staff  200  5 24 14:58 db1.t1-schema.sql
-rwxr-xr-x  1 shengang  staff   95  5 24 14:58 db1.t1.0.sql
-rwxr-xr-x  1 shengang  staff  195  5 24 14:58 db1.t2-schema.sql
-rwxr-xr-x  1 shengang  staff   77  5 24 14:58 db1.t2.0.sql
-rwxr-xr-x  1 shengang  staff  140  5 24 14:58 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -T t1,t2 -o dumpling_T_t1_t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -T t1,t2 -o dumpling_T_t1_t2
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:58:46.364 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:58:46.364 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]
dump failed: SHOW CREATE TABLE .t2: err = Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 19 near ".t2"
goroutine 7 [running]:
runtime/debug.Stack(0x16144e0, 0xc000138240, 0x1615280)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/pingcap/dumpling/v4/export.withStack(0x16144e0, 0xc000138240, 0xc0000e6068, 0xc0000285a0)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/error.go:40 +0x8d
github.com/pingcap/dumpling/v4/export.simpleQuery(0xc0001b2000, 0xc0000285a0, 0x15, 0xc0001f1da8, 0x2, 0xc0000285a0)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/sql.go:367 +0x152
github.com/pingcap/dumpling/v4/export.ShowCreateTable(0xc0001b2000, 0x0, 0x0, 0x7ffeefbff95a, 0x2, 0xc000032118, 0x0, 0x0, 0xc000058e60)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/sql.go:51 +0x154
github.com/pingcap/dumpling/v4/export.dumpTable(0x161e960, 0xc0000e6068, 0xc0000bc180, 0xc0001b2000, 0x0, 0x0, 0xc00000e660, 0x161cd20, 0xc0000b8040, 0x0, ...)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/dump.go:166 +0x340
github.com/pingcap/dumpling/v4/export.dumpDatabases.func1(0x0, 0x0)
	/Users/shengang/Documents/002-workspace/git-workspace/pingcap/dumpling/v4/export/dump.go:136 +0x166
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00013cf30, 0xc000158370)
	/Users/shengang/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/shengang/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

$ ll dumpling_T_t1_t2
total 8
drwxr-xr-x  3 shengang  staff   96  5 24 14:58 .
drwxr-xr-x  9 shengang  staff  288  5 24 14:58 ..
-rwxr-xr-x  1 shengang  staff  102  5 24 14:58 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -B db2 -T db1.t1,db1.t2 -o dumpling_B_db2_T_db1.t1_db2._t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -B db2 -T db1.t1,db1.t2 -o dumpling_B_db2_T_db1.t1_db2._t2
Release version:
Git commit hash: 0eb24b8cb0ae3a2823b77c8e6efd34870a940c6c
Git branch:      sg-table-list
Build timestamp: 2020-05-24 06:30:27Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 14:59:20.782 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 14:59:20.782 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]

$ ll dumpling_B_db2_T_db1.t1_db2._t2
total 48
drwxr-xr-x   8 shengang  staff  256  5 24 14:59 .
drwxr-xr-x  10 shengang  staff  320  5 24 14:59 ..
-rwxr-xr-x   1 shengang  staff   65  5 24 14:59 db1-schema-create.sql
-rwxr-xr-x   1 shengang  staff  200  5 24 14:59 db1.t1-schema.sql
-rwxr-xr-x   1 shengang  staff   95  5 24 14:59 db1.t1.0.sql
-rwxr-xr-x   1 shengang  staff  195  5 24 14:59 db1.t2-schema.sql
-rwxr-xr-x   1 shengang  staff   77  5 24 14:59 db1.t2.0.sql
-rwxr-xr-x   1 shengang  staff  140  5 24 14:59 metadata

./dumpling -u root -P 14000 -H 172.16.5.189 -T db1.t1,db2.t2 -o dumpling_T_db1.t1_db2.t2

$ ./dumpling -u root -P 14000 -H 172.16.5.189 -T db1.t1,db2.t2 -o dumpling_T_db1.t1_db2.t2
Release version:
Git commit hash: 22471656da09ee550f08530558c9fae53ae2d808
Git branch:      sg-table-list
Build timestamp: 2020-05-24 07:14:37Z
Go version:      go version go1.13 darwin/amd64

[2020/05/24 15:16:02.797 +08:00] [INFO] [config.go:117] ["detect server type"] [type=TiDB]
[2020/05/24 15:16:02.798 +08:00] [INFO] [config.go:135] ["detect server version"] [version=4.0.0-rc.2]
[2020/05/24 15:16:02.798 +08:00] [INFO] [prepare.go:92] ["dbTables:map[db1:[0xc0000f0780] db2:[0xc0000f0800]]"]

$ ll dumpling_T_db1.t1_db2.t2
total 40
drwxr-xr-x   7 shengang  staff  224  5 24 15:16 .
drwxr-xr-x  11 shengang  staff  352  5 24 15:16 ..
-rwxr-xr-x   1 shengang  staff  200  5 24 15:16 db1.t1-schema.sql
-rwxr-xr-x   1 shengang  staff   95  5 24 15:16 db1.t1.0.sql
-rwxr-xr-x   1 shengang  staff  194  5 24 15:16 db2.t2-schema.sql
-rwxr-xr-x   1 shengang  staff   76  5 24 15:16 db2.t2.0.sql
-rwxr-xr-x   1 shengang  staff  140  5 24 15:16 metadata

codecov-commenter · 2020-05-24T06:54:41Z

Codecov Report

Merging #81 into master will decrease coverage by 0.06%.
The diff coverage is 8.82%.

@@            Coverage Diff             @@
##           master      #81      +/-   ##
==========================================
- Coverage   56.66%   56.59%   -0.07%     
==========================================
  Files          16       16              
  Lines        1576     1615      +39     
==========================================
+ Hits          893      914      +21     
- Misses        608      626      +18     
  Partials       75       75

kennytm · 2020-05-24T08:41:46Z

thanks for the PR, but we are planning to support #64 using pingcap/tidb-tools#341 rather than adding just -T.

Win-Man · 2020-05-24T10:49:41Z

Thank for your reply. If you are planning to support table-filter rather than adding just -T, how soon will dumpling support it?

Support -T/--tables-list option

2247165

kennytm mentioned this pull request May 27, 2020

Implement --filter and --case-sensitive #82

Merged

kennytm closed this in #82 May 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support -T/--tables-list option #81

Support -T/--tables-list option #81

Win-Man commented May 24, 2020 •

edited

Loading

codecov-commenter commented May 24, 2020

kennytm commented May 24, 2020

Win-Man commented May 24, 2020

Support -T/--tables-list option #81

Support -T/--tables-list option #81

Conversation

Win-Man commented May 24, 2020 • edited Loading

prepare data

dumpling test cases

codecov-commenter commented May 24, 2020

Codecov Report

kennytm commented May 24, 2020

Win-Man commented May 24, 2020

Win-Man commented May 24, 2020 •

edited

Loading