Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Improve dynamic splitting in BigQuery Read API's Java Connector #24260

Open
vachan-shetty opened this issue Nov 18, 2022 · 1 comment
Open

[Feature Request]: Improve dynamic splitting in BigQuery Read API's Java Connector #24260

vachan-shetty opened this issue Nov 18, 2022 · 1 comment
Assignees
@vachan-shetty
Labels
gcp io java new feature P2

Comments

@vachan-shetty
Copy link
Contributor

What would you like to happen?

Trying to run large Dataflow pipelines which use the BQ Read API sporadically results in pipeline failure with the following error message:

worker lost contact with the service

This problem gets worse as the customer starts approaching quota limits defined in the BigQuery Read API. The connector needs to be redesigned to prevent these issues from occurring.

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp
@vachan-shetty
Copy link
Contributor Author

.take-issue

@vachan-shetty vachan-shetty mentioned this issue Feb 8, 2023
Merged
3 tasks
@Abacn Abacn mentioned this issue Mar 15, 2024
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vachan-shetty vachan-shetty

Labels
gcp io java new feature P2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vachan-shetty