Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

filter (ticdc): add dml expression filter #6215

Merged
merged 14 commits into from
Jul 8, 2022
Merged

filter (ticdc): add dml expression filter #6215

merged 14 commits into from
Jul 8, 2022

Conversation

asddongmen
Copy link
Contributor

What problem does this PR solve?

Issue Number: ref #6160

What is changed and how it works?

Add strcut dmlExprFilter to pkg/filter, it is a part of cdc row filter.
Note: This PR will not change any behavior of cdc, it only:

  1. Add strcut dmlExprFilter.
  2. Add a filter config struct EventFilterRule to pkg/config/filter.go.
  3. Add a field RowChangedDatums to model.RowChangedEvent to facilitate dmlExprFilter's function.

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 


None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 7, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • nongfushanquan
  • sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 7, 2022
@asddongmen asddongmen requested review from overvenus and removed request for Rustin170506 July 7, 2022 07:41
overvenus
overvenus reviewed Jul 7, 2022
View reviewed changes
cdc/model/sink.go Outdated Show resolved Hide resolved
pkg/filter/expr_filter.go Outdated Show resolved Hide resolved
pkg/filter/expr_filter.go Outdated Show resolved Hide resolved
pkg/filter/expr_filter.go Outdated
}

// shouldSkipDML skips dml event by sql expression.
func (f *dmlExprFilter) shouldSkipDML(row *model.RowChangedEvent,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a benchmark? Since it applies to all changes, it may become a bottleneck.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

Copy link
Contributor Author

@asddongmen asddongmen Jul 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

cmd: go test -benchmem -run=^$ -bench ^BenchmarkSkipDML$ github.com/pingcap/tiflow/pkg/filter
goos: maxOS 12.3.1
goarch: arm64
cpu: Apple M1 Pro
BenchmarkSkipDML/insert-1-10              990166              1151 ns/op             768 B/op         24 allocs/op
BenchmarkSkipDML/insert-2-10             1000000              1187 ns/op             768 B/op         24 allocs/op
BenchmarkSkipDML/update-10                698208              1637 ns/op            1480 B/op         43 allocs/op
BenchmarkSkipDML/delete-10               1000000              1112 ns/op             768 B/op         24 allocs/op

cdc/model/sink.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 7, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 8, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 8, 2022 •
edited
Loading

Codecov Report

Merging #6215 (71aeabf) into master (ef587e6) will decrease coverage by 0.3493%.
The diff coverage is 63.2743%.

Flag Coverage Δ
cdc 63.9273% <67.8383%> (-0.7573%) ⬇️
dm 51.9101% <47.0238%> (+0.0210%) ⬆️
engine 62.4929% <54.3478%> (-0.0410%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #6215        +/-   ##
================================================
- Coverage   58.4193%   58.0700%   -0.3493%     
================================================
  Files           711        714         +3     
  Lines         84205      83649       -556     
================================================
- Hits          49192      48575       -617     
- Misses        30540      30704       +164     
+ Partials       4473       4370       -103

overvenus
overvenus previously approved these changes Jul 8, 2022
View reviewed changes
pkg/filter/expr_filter.go Show resolved Hide resolved
@overvenus overvenus dismissed their stale review July 8, 2022 05:39

See my comment.

@asddongmen
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 71aeabf

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 8, 2022
@asddongmen asddongmen mentioned this pull request Jul 8, 2022
Closed
5 tasks
@asddongmen
Copy link
Contributor Author

/run-integration-tests

@asddongmen
Copy link
Contributor Author

/run-all-tests

@asddongmen
Copy link
Contributor Author

/run-kafka-tests

@asddongmen asddongmen mentioned this pull request Jul 8, 2022
Closed
@asddongmen
Copy link
Contributor Author

/run-verify
/run-kafka-integration-tests

@ti-chi-bot
Copy link
Member

@asddongmen: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@asddongmen
Copy link
Contributor Author

asddongmen commented Jul 8, 2022 •
edited
Loading

/run-verify

@asddongmen
Copy link
Contributor Author

/run-integration-tests

@ti-chi-bot ti-chi-bot merged commit 117f534 into pingcap:master Jul 8, 2022
@asddongmen asddongmen deleted the add_dml_expression_filter branch July 21, 2022 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sdojjy sdojjy sdojjy approved these changes

@overvenus overvenus overvenus approved these changes

@nongfushanquan nongfushanquan nongfushanquan approved these changes

@CharlesCheung96 CharlesCheung96 Awaiting requested review from CharlesCheung96

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@asddongmen @ti-chi-bot @codecov-commenter @sdojjy @overvenus @nongfushanquan