Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syncer(dm) : fix default collation with upstream in create table‘s columns #3769

Merged
merged 22 commits into from
Dec 17, 2021
Merged

syncer(dm) : fix default collation with upstream in create table‘s columns #3769

merged 22 commits into from
Dec 17, 2021

Conversation

WizardXiao
Copy link
Contributor

@WizardXiao WizardXiao commented Dec 7, 2021 •
edited
Loading

What problem does this PR solve?

fix https://github.com/pingcap/ticdc/issues/3420

What is changed and how it works?

it will add column's collation when collumn just has character set by show character set.

Check List

Tests

  • Unit test
  • Integration test

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 7, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Ehco1996
  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 7, 2021
@WizardXiao WizardXiao added the area/dm Issues or PRs related to DM. label Dec 7, 2021
@WizardXiao
Copy link
Contributor Author

i will add integration_test later.

@WizardXiao WizardXiao added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 7, 2021
@WizardXiao
Copy link
Contributor Author

/run-all-tests

@lance6716
Copy link
Contributor

lance6716 commented Dec 9, 2021 •
edited
Loading

found a bug that in GetServerCollationByStatusVars we treat collation number as 1-byte because TiDB does so, but there are collation numbers larger than 255 ⬇️

https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/strings/ctype-uca.cc#L11356

we can use SHOW COLLATION to know every Id of collations

@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 10, 2021
@WizardXiao WizardXiao removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2021
@WizardXiao
Copy link
Contributor Author

/run-all-tests

@WizardXiao
Copy link
Contributor Author

/run-all-tests

@WizardXiao
Copy link
Contributor Author

found a bug that in GetServerCollationByStatusVars we treat collation number as 1-byte because TiDB does so, but there are collation numbers larger than 255 ⬇️

https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/strings/ctype-uca.cc#L11356

we can use SHOW COLLATION to know every Id of collations

As SHOW COLLATION has different columns between mysql 5.7.22 and 8.0.21. so we can use INFORMATION_SCHEMA.COLLATIONS instead.

@WizardXiao
Copy link
Contributor Author

/run-all-tests

@WizardXiao
Copy link
Contributor Author

/run-all-tests

1 similar comment
@WizardXiao
Copy link
Contributor Author

/run-all-tests

@WizardXiao
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 15, 2021
@WizardXiao
Copy link
Contributor Author

/run-all-tests

@WizardXiao
Copy link
Contributor Author

i simplify integration test by "all" mode. please take a look. thank you. /cc @lance6716 @Ehco1996

lance6716
lance6716 reviewed Dec 16, 2021
View reviewed changes
Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will review later

dm/syncer/ddl.go Outdated Show resolved Hide resolved
dm/tests/sync_collation/run.sh Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 16, 2021
@lance6716
Copy link
Contributor

(need address comment https://github.com/pingcap/ticdc/pull/3769#discussion_r770282858 )

@WizardXiao
Copy link
Contributor Author

/run-all-tests

Ehco1996
Ehco1996 reviewed Dec 17, 2021
View reviewed changes
Copy link
Contributor

@Ehco1996 Ehco1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

dm/tests/sync_collation/conf/dm-task.yaml Outdated Show resolved Hide resolved
dm/tests/sync_collation/run.sh Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 17, 2021
@WizardXiao
Copy link
Contributor Author

/run-all-tests

@Ehco1996
Copy link
Contributor

ping @glorv do you have more suggestions?

@glorv
Copy link
Contributor

glorv commented Dec 17, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: be5e896

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 17, 2021
@ti-chi-bot ti-chi-bot merged commit fb41a3d into pingcap:master Dec 17, 2021
zhaoxinyu pushed a commit to zhaoxinyu/ticdc that referenced this pull request Dec 29, 2021
@WizardXiao @zhaoxinyu
syncer(dm) : fix default collation with upstream in create table‘s co…
e2dc007 
…lumns (pingcap#3769)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lichunzhu lichunzhu lichunzhu left review comments

@okJiang okJiang okJiang left review comments

@lance6716 lance6716 lance6716 approved these changes

@glorv glorv glorv approved these changes

@Ehco1996 Ehco1996 Ehco1996 approved these changes

Assignees
No one assigned
Labels
area/dm Issues or PRs related to DM. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

DM migrates table with incorrect collation
7 participants
@WizardXiao @ti-chi-bot @lance6716 @Ehco1996 @glorv @lichunzhu @okJiang