sink(ticdc): add event key output for debezium #11649

wk989898 · 2024-10-11T09:20:32Z

What problem does this PR solve?

Issue Number: close #11652

What is changed and how it works?

Debezium send event key and event value to kafka. I added event key output to this PR.
Also modify function SanitizeName for keeping same with official code.

Check List

Tests

Unit test
Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix the issue that the Key of Kafka messages has't been filled in Debezium protocol.

codecov · 2024-10-12T14:21:34Z

Codecov Report

Attention: Patch coverage is 24.44444% with 238 lines in your changes missing coverage. Please review.

Project coverage is 55.1512%. Comparing base (cd277f5) to head (2dc3baf).
Report is 6 commits behind head on master.

Additional details and impacted files

Components	Coverage Δ
cdc	`59.6168% <24.4444%> (-0.2386%)`	⬇️
dm	`50.0616% <ø> (-0.0261%)`	⬇️
engine	`53.2336% <ø> (-0.0087%)`	⬇️

Flag	Coverage Δ
unit	`55.1512% <24.4444%> (-0.1186%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11649        +/-   ##
================================================
- Coverage   55.2697%   55.1512%   -0.1186%     
================================================
  Files          1001       1001                
  Lines        136163     136472       +309     
================================================
+ Hits          75257      75266         +9     
- Misses        55417      55697       +280     
- Partials       5489       5509        +20

hongyunyan · 2024-10-16T09:37:36Z

tests/integration_tests/debezium/sql/debezium/real_test.sql

@@ -1,3 +1,5 @@
+-- set sql_mode REAL_AS_FLOAT is necessary


Why set REAL_AS_FLOAT is neccesary?

Because Debezium treats REAL as a synonym for FLOAT.

tiflow/pkg/sink/codec/debezium/codec.go

Lines 385 to 386 in 45b439b

// https://dev.mysql.com/doc/refman/8.4/en/numeric-types.html

// MySQL also treats REAL as a synonym for DOUBLE PRECISION (a nonstandard variation), unless the REAL_AS_FLOAT SQL mode is enabled.

pkg/sink/codec/debezium/codec.go

wk989898 · 2024-10-17T06:12:20Z

/retest

wk989898 · 2024-10-17T08:05:10Z

/retest

wk989898 · 2024-10-17T08:34:15Z

/retest

wk989898 · 2024-10-17T09:50:39Z

/retest

wk989898 · 2024-10-21T06:01:02Z

/retest

hongyunyan

LGTM

ti-chi-bot · 2024-10-21T06:24:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, hongyunyan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [3AceShowHand,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-10-21T06:24:06Z

[LGTM Timeline notifier]

Timeline:

2024-10-17 02:52:44.780260683 +0000 UTC m=+498761.929170515: ☑️ agreed by 3AceShowHand.
2024-10-21 06:24:05.975844173 +0000 UTC m=+245846.672634802: ☑️ agreed by hongyunyan.

wk989898 · 2025-01-02T11:53:48Z

/cherry-pick release-8.1

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-01-02T11:54:26Z

@wk989898: new pull request created to branch release-8.1: #11965.

In response to this:

/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close #11652

wk989898 · 2025-02-05T01:59:29Z

/cherry-pick release-8.1-20250130-v8.1.2

ti-chi-bot · 2025-02-05T02:00:31Z

@wk989898: new pull request created to branch release-8.1-20250130-v8.1.2: #12042.

In response to this:

/cherry-pick release-8.1-20250130-v8.1.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <[email protected]>

) close #11652

wk989898 added 3 commits October 11, 2024 16:23

update

9e049f7

update

3fe51c9

update

c92a6cc

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 11, 2024

update

c3ce204

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 11, 2024

fmt

8ae6102

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/needs-triage-completed and removed release-note-none Denotes a PR that doesn't merit a release note. labels Oct 12, 2024

flowbehappy assigned wk989898 Oct 12, 2024

fix and update test

03db995

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Oct 12, 2024

add schema test and debug

ba2f79d

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 12, 2024

wk989898 added 3 commits October 12, 2024 19:13

debug

9f50b67

update SanitizeName

6845ff8

fix

797ee4c

wk989898 added 3 commits October 13, 2024 23:08

update

04a8e05

fix test

f6a0dc3

chore

45b439b

wk989898 changed the title ~~sink(ticdc): add key message output on debezium~~ sink(ticdc): add event key output for debezium Oct 14, 2024

wk989898 requested review from breezewish, 3AceShowHand and hongyunyan October 15, 2024 03:08

hongyunyan reviewed Oct 16, 2024

View reviewed changes

wk989898 added 3 commits October 17, 2024 12:38

update

b52d4ab

fix

2a19c1e

fix

2b6262e

modify

2dc3baf

hongyunyan approved these changes Oct 21, 2024

View reviewed changes

ti-chi-bot bot added the lgtm label Oct 21, 2024

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Oct 21, 2024

ti-chi-bot bot merged commit 76ebb37 into pingcap:master Oct 21, 2024
28 checks passed

wk989898 mentioned this pull request Nov 7, 2024

sink(ticdc): fix incorrect sanitize #11728

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 2, 2025

This is an automated cherry-pick of pingcap#11649

4691d09

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jan 2, 2025

sink(ticdc): add event key output for debezium (#11649) #11965

Merged

wk989898 deleted the debezium-key branch January 6, 2025 02:25

ti-chi-bot bot pushed a commit that referenced this pull request Jan 6, 2025

sink(ticdc): add event key output for debezium (#11649) (#11965)

3e1dad2

close #11652

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Feb 5, 2025

This is an automated cherry-pick of pingcap#11649

4b10fbf

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Feb 5, 2025

sink(ticdc): add event key output for debezium (#11649) #12042

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Feb 5, 2025

This is an automated cherry-pick of pingcap#11649

a59ceeb

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Feb 5, 2025

sink(ticdc): add event key output for debezium (#11649) (#11965) #12043

Merged

wk989898 mentioned this pull request Feb 5, 2025

sink(ticdc): fix incorrect sanitize #12044

Merged

ti-chi-bot bot pushed a commit that referenced this pull request Feb 5, 2025

sink(ticdc): add event key output for debezium (#11649) (#11965) (#12043

e6447c5

) close #11652

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sink(ticdc): add event key output for debezium #11649

sink(ticdc): add event key output for debezium #11649

wk989898 commented Oct 11, 2024 •

edited

Loading

codecov bot commented Oct 12, 2024 •

edited

Loading

hongyunyan Oct 16, 2024

wk989898 Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 21, 2024

hongyunyan left a comment

ti-chi-bot bot commented Oct 21, 2024

ti-chi-bot bot commented Oct 21, 2024

wk989898 commented Jan 2, 2025

ti-chi-bot commented Jan 2, 2025

wk989898 commented Feb 5, 2025

ti-chi-bot commented Feb 5, 2025

	// https://dev.mysql.com/doc/refman/8.4/en/numeric-types.html
	// MySQL also treats REAL as a synonym for DOUBLE PRECISION (a nonstandard variation), unless the REAL_AS_FLOAT SQL mode is enabled.

sink(ticdc): add event key output for debezium #11649

sink(ticdc): add event key output for debezium #11649

Conversation

wk989898 commented Oct 11, 2024 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

codecov bot commented Oct 12, 2024 • edited Loading

Codecov Report

hongyunyan Oct 16, 2024

Choose a reason for hiding this comment

wk989898 Oct 17, 2024

Choose a reason for hiding this comment

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 17, 2024

wk989898 commented Oct 21, 2024

hongyunyan left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Oct 21, 2024

ti-chi-bot bot commented Oct 21, 2024

[LGTM Timeline notifier]

wk989898 commented Jan 2, 2025

ti-chi-bot commented Jan 2, 2025

wk989898 commented Feb 5, 2025

ti-chi-bot commented Feb 5, 2025

wk989898 commented Oct 11, 2024 •

edited

Loading

codecov bot commented Oct 12, 2024 •

edited

Loading