syncer(dm) : fix default collation with upstream in create table statement

[WizardXiao]

What problem does this PR solve?

fix https://github.com/pingcap/ticdc/issues/3420

in this case:
sql1: create database if not exists test
sql2: create database test CHARACTER SET=utf8mb4
sql3: create table test.t1 (id int)
sql4: create table test.t1 (id int) CHARSET=utf8mb4

if syncer send the same ddl to downstream tidb, database or table will has diffrent collation from upstream mysql, because the default collation is different between tidb and mysql.

What is changed and how it works?

it will adjust collation when sync create database or create table statement.

for database, it will add collation by binlog query event and handle ddl like these:
sql1: create database if not exists test COLLATE = utf8mb4_general_ci
sql2: create database test CHARACTER SET=utf8mb4 COLLATE = utf8mb4_general_ci

for table has no charset and collation , as we can not get collation from binlog, so there will have issue about timeliness, because dm can set start position at increment mode. so we will check and add warning log in dm.
in this case, downstream db will use collation from database for table, so user can use dm sync dabase ddl sql, but if not, user should make sure dabase's collation is right if create dabase manually. so the ddl will be handled like that:
sql3: create table test.t1 (id int) and warning log detect create table risk which use implicit charset and collation

for table has charset but no collation, we will add default collation by SHOW CHARACTER SET, so the dll will be handled like that:
sql4: create table test.t1 (id int) CHARSET=utf8mb4 COLLATE = utf8mb4_general_ci

Check List

Tests

• Unit test

Release note

Fix a bug when upstream use implicit collation in create database or create table. 
It will add collation when there is no charset and collation in create database and it will not add collation in create table as table can inherit database. 
By the way, it will not directly alter downstream schema, so that user need make sure consistence when create database or table manually.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• glorv
• lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[WizardXiao]

/run-all-tests

[WizardXiao]

/run-all-tests

[lance6716]

also we should check dump/load phase will correctly handle collation.

[lance6716]

by SELECT TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES we get currently collation of the table, but what we want is get collation at the replication time. (maybe if we continue replication we'll see alter collations?) please check if we can achieve that.

we have done some work of get SQL mode from the header of query event, maybe this would help
https://github.com/pingcap/ticdc/blob/7a5634bbc0b7b53f7fb70afe4c193fba6ab199b1/dm/pkg/binlog/event/util.go#L359

[Ehco1996]

by SELECT TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES we get currently collation of the table, but what we want is get collation at the replication time. (maybe if we continue replication we'll see alter collations?) please check if we can achieve that.

we have done some work of get SQL mode from the header of query event, maybe this would help

https://github.com/pingcap/ticdc/blob/7a5634bbc0b7b53f7fb70afe4c193fba6ab199b1/dm/pkg/binlog/event/util.go#L359

seems we can get replication-time-connection from binlog QueryEvent ptal @WizardXiao
https://dev.mysql.com/doc/internals/en/query-event.html

[WizardXiao]

by SELECT TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES we get currently collation of the table, but what we want is get collation at the replication time. (maybe if we continue replication we'll see alter collations?) please check if we can achieve that.
we have done some work of get SQL mode from the header of query event, maybe this would help
https://github.com/pingcap/ticdc/blob/7a5634bbc0b7b53f7fb70afe4c193fba6ab199b1/dm/pkg/binlog/event/util.go#L359

seems we can get replication-time-connection from binlog QueryEvent ptal @WizardXiao https://dev.mysql.com/doc/internals/en/query-event.html

ok, tanks, i will try.

[lichunzhu]

Do we need to do the same thing to columns and databases?

[WizardXiao]

Do we need to do the same thing to columns and databases?

i think this pr will only support tables and if anything go well, we may think about columns and databases later.

[WizardXiao]

After some test about mysql binlog, there is a problem when we use Binlog::QUERY_EVENT:status_vars to make sure the timeliness of Collation, because there is no Q_CHARSET_DATABASE_CODE which is characterset and collation of the schema in Mysql 8.0.26. so maybe we need diffrent way to handle full or incremental phase, and maybe will sacrifice some timeliness.

[Ehco1996]

> because there is no Q_CHARSET_DATABASE_CODE which is characterset and collation of the schema in Mysql 8.0.26.

could you please add the doc link here? i can not find this in mysql doc 😅

> we need diffrent way to handle handle full or incremental phase

this is also confuses me, in full pahse, loader directy execute create-table-sql that exported by dumpling with the collation from source 🤔, so we just need to adjust collation in incremental phase

[lance6716]

because there is no Q_CHARSET_DATABASE_CODE which is characterset and collation of the schema in Mysql 8.0.26.

Not 8.0.26, almost every modern versions
https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/include/statement_events.h#L313

[lance6716]

this is also confuses me, in full pahse, loader directy execute create-table-sql that exported by dumpling with the collation from source 🤔, so we just need to adjust collation in incremental phase

SHOW CREATE TABLE will not write collation when it's the default collation of the charset. please see below comments

https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/sql/sql_show.cc#L1983-L1993

[Ehco1996]

this is also confuses me, in full pahse, loader directy execute create-table-sql that exported by dumpling with the collation from source 🤔, so we just need to adjust collation in incremental phase

SHOW CREATE TABLE will not write collation when it's the default collation of the charset. please see below comments

https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/sql/sql_show.cc#L1983-L1993

thanks! i think maybe adjust charset and collation for dumpling and syncer with INFORMATION_SCHEMA.TABLES is enough?

• dumpling: there is no need to get collation-at-the-replication-time
• syncer: when a create-table-sql-without-collation comes from binlog, use the current collation of the upstream table as the default collation there does not seem to be a problem with the correctness

this may cause correctness problems when default collation is different between upstream and downstream (select result order)

[glorv]

@WizardXiao I think this PR deserve a release note as it's an important bugfix.

[dveeden]

I assume this works fine with the New framework for collations enabled? Do we need to add tests for that?

[WizardXiao]

/run-all-tests

[WizardXiao]

I assume this works fine with the New framework for collations enabled? Do we need to add tests for that?
Current approach will sync collation to tidb and will not check collation conf in tidb, so if new_collation_enabled is false, tidb may throw error, and user can set new_collation_enabled be true.

[WizardXiao]

/run-verify-ci

[lance6716]

I assume this works fine with the New framework for collations enabled? Do we need to add tests for that?

This PR only handle the increment phase, pingcap/tidb#30292 handles dump/load phase. A testing PR will be submitted after including them two.

[WizardXiao]

/run-all-tests

[WizardXiao]

/run-all-tests

[glorv]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 89113bf

[ti-chi-bot]

@WizardXiao: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3753.