Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add the pubsublite sink #21

Merged
merged 4 commits into from
Jul 29, 2021
Merged

feat: add the pubsublite sink #21

merged 4 commits into from
Jul 29, 2021

Conversation

palmere-google
Copy link
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

@palmere-google palmere-google requested a review from a team as a code owner July 27, 2021 15:59
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jul 27, 2021
}

@Override
public Message serialize(T value, Instant timestamp) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be used as the event timestamp field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yeah, good point! Done (in the serialize pr)


@Override
public Message serialize(T value, Instant timestamp) {
return Message.fromProto(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, use the builder methods on Message

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

import java.io.Serializable;

@AutoValue
public abstract class PubsubLiteSinkSettings<IN> implements Serializable {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use InputT

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@palmere-google palmere-google force-pushed the palmere-dev-sink branch 2 times, most recently from 6a7f71e to afa3ca5 Compare July 27, 2021 17:26
sink stuff

schema changes

remove serailzed supplier

stuff

fixes from rebasing

synchronize

formatting
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime_2.11</artifactId>
<version>1.13.0</version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit. add a property "flink.version" and use it everywhere you have 1.13.0. See pubsublite-beam-io for an example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do in another pr, tianzi suggested that I do this for pubsublite too


@Override
public synchronized void snapshotState(
org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not import these?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Intellij generated them like this and I never changed them

public abstract Builder<InputT> setTopicPath(TopicPath value);

abstract Builder<InputT> setSerializationSchema(PubsubLiteSerializationSchema<InputT> value);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the proper user interface? If so, add comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments.

return new AutoValue_PubsubLiteSinkSettings.Builder<InputT>().setSerializationSchema(schema);
}

public static PubsubLiteSinkSettings.Builder<Message> messagesBuilder() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just Builder

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, better, thanks


@AutoValue
public abstract class PubsubLiteSinkSettings<InputT> implements Serializable {
public static <InputT> PubsubLiteSinkSettings.Builder<InputT> builder(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just Builder

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@palmere-google palmere-google requested a review from a team as a code owner July 29, 2021 17:40
@palmere-google palmere-google merged commit 186c93b into master Jul 29, 2021
@palmere-google palmere-google deleted the palmere-dev-sink branch July 29, 2021 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants