Skip to content
This repository has been archived by the owner on Dec 14, 2022. It is now read-only.

Batch read #52

Merged
merged 16 commits into from
Oct 19, 2020
Merged

Batch read #52

merged 16 commits into from
Oct 19, 2020

Conversation

yjshen
Copy link
Contributor

@yjshen yjshen commented Feb 12, 2020 •
edited
Loading

Fixes #43

@yjshen yjshen requested a review from sijie as a code owner February 12, 2020 03:08
@yjshen yjshen self-assigned this Feb 18, 2020
@yjshen
Copy link
Contributor Author

yjshen commented Feb 18, 2020

This PR is ready for review. Not able to merge due to the following two issues:

  1. testcontainer based bk url could not be exposed easily. since their should be a remapping in BK client when it get the bk url from zk. currently, I haven't figure a way to work around it.
  2. Pulsar 2.5.0 breaks start and read semantic of the reader. This is tracked by [Java Reader Client] Start reader inside batch result in read first message in batch. apache/pulsar#6345

@yjshen
Copy link
Contributor Author

yjshen commented Feb 18, 2020

@sijie PTAL on the batch implementation part. The splits generation, split read, etc.
I've tested locally for PulsarInputFormatITest.

@jianyun8023 jianyun8023 self-assigned this Sep 14, 2020
@jianyun8023
Copy link
Contributor

Depend on Extract common

@jianyun8023
Merge branch 'master' into syj/batch_read
5a3d7b5 
# Conflicts:
#	common/pom.xml
#	common/src/main/java/org/apache/flink/batch/connectors/pulsar/PulsarInputSplit.java
#	pom.xml
#	pulsar-flink-1.11/pom.xml
#	pulsar-flink-1.11/src/test/java/org/apache/flink/streaming/connectors/pulsar/PulsarTestBase.java
#	pulsar-flink-1.9/pom.xml
@jianyun8023
Flink batch support
8e9fb04
@jianyun8023
Merge remote-tracking branch 'upstream/master' into syj/batch_read
44e9bc3 
# Conflicts:
#	pulsar-flink-1.11/src/test/java/org/apache/flink/streaming/connectors/pulsar/PulsarTestBase.java
@jianyun8023
fix managed-ledger-shaded
997e03f
@jianyun8023
add error log
2d4cd3d
@jianyun8023
Merge branch 'upstream-master' into syj/batch_read
dad0a2e
@jianyun8023
Merge branch 'master' into syj/batch_read
6b4af77
@jianyun8023
remove batch input test
70a2ecc
@jianyun8023
add batch input test
e466f26
@jianyun8023 jianyun8023 merged commit 981407e into streamnative:master Oct 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jiazhai jiazhai jiazhai approved these changes

@sijie sijie Awaiting requested review from sijie

Assignees

@yjshen yjshen

@jianyun8023 jianyun8023

Labels
workflow::code-completed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add batch support in flink connector
4 participants
@yjshen @jianyun8023 @jiazhai @sijie