Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sink/kafka: batch send messages #426

Merged
merged 36 commits into from
Apr 10, 2020
Merged

Conversation

5kbpers
Copy link
Contributor

@5kbpers 5kbpers commented Apr 1, 2020

Signed-off-by: 5kbpers [email protected]

What problem does this PR solve?

The throughput of kafka sink needs to be improved.

What is changed and how it works?

Batch send messages to consumer

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to be included in the release note

Signed-off-by: 5kbpers <[email protected]>
Signed-off-by: 5kbpers <[email protected]>
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

@5kbpers 5kbpers closed this Apr 1, 2020
@5kbpers 5kbpers reopened this Apr 1, 2020
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

@5kbpers 5kbpers closed this Apr 1, 2020
@5kbpers 5kbpers reopened this Apr 1, 2020
@5kbpers 5kbpers closed this Apr 1, 2020
@5kbpers 5kbpers reopened this Apr 1, 2020
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

1 similar comment
@zier-one
Copy link
Contributor

zier-one commented Apr 1, 2020

/run-kafka-tests

@5kbpers 5kbpers force-pushed the batch-kafka branch 2 times, most recently from 6dc3b0a to 2cc6079 Compare April 1, 2020 09:50
Signed-off-by: 5kbpers <[email protected]>
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

Signed-off-by: 5kbpers <[email protected]>
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

@5kbpers 5kbpers force-pushed the batch-kafka branch 2 times, most recently from deff569 to d6f4659 Compare April 1, 2020 15:27
Signed-off-by: 5kbpers <[email protected]>
@5kbpers
Copy link
Contributor Author

5kbpers commented Apr 1, 2020

/run-kafka-tests

Signed-off-by: 5kbpers <[email protected]>
@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-all-tests

@codecov-io
Copy link

codecov-io commented Apr 9, 2020

Codecov Report

Merging #426 into master will not change coverage by %.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #426   +/-   ##
===========================================
  Coverage   30.0229%   30.0229%           
===========================================
  Files            58         58           
  Lines          5669       5669           
===========================================
  Hits           1702       1702           
  Misses         3852       3852           
  Partials        115        115           

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-all-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-kafka-tests

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-integration-tests

@zier-one zier-one added the status/ptal Could you please take a look? label Apr 9, 2020
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

return nil
}

func (k *kafkaSaramaProducer) BroadcastMessage(ctx context.Context, key *model.MqMessageKey, value *model.MqMessageDDL) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be removed now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@zier-one
Copy link
Contributor

zier-one commented Apr 9, 2020

/run-all-tests

Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amyangfei amyangfei added LGT1 and removed status/ptal Could you please take a look? labels Apr 9, 2020
@amyangfei amyangfei merged commit e04a4b8 into pingcap:master Apr 10, 2020
5kbpers added a commit to 5kbpers/ticdc that referenced this pull request Aug 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
subject/performance Denotes an issue or pull request is related to replication performance.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants