charset: implement utf8_unicode_ci and utf8mb4_unicode_ci collation (#18776)

[ti-srebot]

cherry-pick #18776 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/22558

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/tidb.git pr/22558:release-4.0-ba60cf5a69cf

---

What problem does this PR solve?

This is the second PR of #17596 . This PR is aim to implement utf8mb4_unicode_ci and utf8mb4_unicode_ci collation。

What is changed and how it works?

What is changed:

• add a big lookup-table, created from DUCET

How it Works:

• Implement UCA with unicode version 4.0.0 (same as mysql. see charset-unicode-sets-uca)
• Mapping: each unicode char can be convert to collation weight(s) in different levels. It may one to one, one to many(expand), many to one(contractions), many to many (contractions)
• UCA use four levels to compare two unicode strings, xxx_unicode_ci only use primary level (L1), it is ai and ci
• mysql utf8mb4_unicode_ci does not support contractions (see charset-unicode-sets-uca)

benchmark

here is benchmark for Compare and Key function

goos: linux
goarch: amd64
pkg: github.com/pingcap/tidb/util/collate
BenchmarkUtf8mb4Bin_CompareShort-8              175546567                7.12 ns/op
BenchmarkUtf8mb4GeneralCI_CompareShort-8         2656393               453 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareShort-8         2521083               409 ns/op
BenchmarkUtf8mb4Bin_CompareMid-8                160179999                7.36 ns/op
BenchmarkUtf8mb4GeneralCI_CompareMid-8             31842             36862 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareMid-8             32714             37254 ns/op
BenchmarkUtf8mb4Bin_CompareLong-8               159598683                7.38 ns/op
BenchmarkUtf8mb4GeneralCI_CompareLong-8               28          40149168 ns/op
BenchmarkUtf8mb4UnicodeCI_CompareLong-8               30          39463787 ns/op
BenchmarkUtf8mb4Bin_KeyShort-8                  25229853                45.0 ns/op
BenchmarkUtf8mb4GeneralCI_KeyShort-8             3883850               266 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyShort-8             4620993               254 ns/op
BenchmarkUtf8mb4Bin_KeyMid-8                     1491789               791 ns/op
BenchmarkUtf8mb4GeneralCI_KeyMid-8                 52681             22532 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyMid-8                 55576             21164 ns/op
BenchmarkUtf8mb4Bin_KeyLong-8                       3147            505012 ns/op
BenchmarkUtf8mb4GeneralCI_KeyLong-8                   42          28282321 ns/op
BenchmarkUtf8mb4UnicodeCI_KeyLong-8                   57          21806637 ns/op

string length short 32 Mid 2048 Long 2097152

Tests

• Unit test
• Integration test

Release note

• Implement utf8_unicode_ci and utf8mb4_unicode_ci collation

[ti-srebot]

/run-all-tests

[ti-srebot]

@xiongjiwei you're already a collaborator in bot's repo.

[bb7133]

Please fix the conflict.

[xiongjiwei]

/run-all-tests

[xiongjiwei]

/run-all-tests

[wjhuang2016]

LGTM

[bb7133]

LGTM

[bb7133]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@ti-srebot merge failed.

[qw4990]

Wait for #21877.

[qw4990]

/run-all-tests

[qw4990]

/run-all-tests

[qw4990]

/run-integration-copr-test

[xiongjiwei]

/run-unit-test