BigQuery Storage Write API wrapper for Python SDK #25685

ahmedabu98 · 2023-03-01T23:26:57Z

Implementing a wrapper for Python SDK that uses Java's Storage API SchemaTransform (created in #23988) to write to BigQuery

Some features added and adjusted along the way:

Added module plugin utilities that help when creating Python-using-Java cross-language tests that use a particular expansion service (ie GCP, kinesis, etc.).
Using these utilities, two new test-suites are added for Python-using-Java GCP cross-language tests (one for Dataflow, one for Direct).
Adjusted Python SchemaTransform API for convenience when creating wrappers (continued in Use external config schema to construct Python SchemaTransform payload #26100)

Fixes #21961
Fixes #25669

ahmedabu98 · 2023-03-01T23:32:48Z

R: @chamikaramj

github-actions · 2023-03-01T23:33:52Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

codecov · 2023-03-02T00:17:50Z

Codecov Report

Merging #25685 (0999c68) into master (94e72e3) will decrease coverage by 0.02%.
The diff coverage is 65.09%.

@@            Coverage Diff             @@
##           master   #25685      +/-   ##
==========================================
- Coverage   71.43%   71.42%   -0.02%     
==========================================
  Files         779      779              
  Lines      102530   102629      +99     
==========================================
+ Hits        73245    73305      +60     
- Misses      27824    27864      +40     
+ Partials     1461     1460       -1

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/gcp/bigquery.py	`69.81% <31.91%> (-1.60%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery_tools.py	`74.61% <91.48%> (+1.32%)`	⬆️
sdks/python/apache_beam/transforms/external.py	`80.51% <91.66%> (+1.77%)`	⬆️

... and 14 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

chamikaramj

Thanks!

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

sdks/java/io/google-cloud-platform/expansion-service/build.gradle

...e/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProviderTest.java

sdks/python/apache_beam/io/gcp/bigquery.py

sdks/python/test-suites/xlang/build.gradle

chamikaramj · 2023-03-13T20:04:40Z

Seems like tests couldn't find the expansion service ?

https://ci-beam.apache.org/job/beam_PostCommit_Python_Xlang_Gcp_Dataflow_PR/21/testReport/junit/apache_beam.io.gcp.bigquery_write_it_test/BigQueryXlangStorageWriteIT/test_storage_write_beam_rows/

ahmedabu98 · 2023-03-13T20:13:17Z

Seems like tests couldn't find the expansion service ?

Yeah it's only seeing the default value (see debug println) that I set here. The setup task overwrites that value and launches an expansion service as expected, but the python task doesn't read the updated value. I'm not sure how scoping works between these tasks.

ahmedabu98 · 2023-03-14T23:28:10Z

R: @chamikaramj

Tests are passing now, ready for another review.

chamikaramj

Thanks. Almost there :)

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

sdks/python/apache_beam/io/gcp/bigquery.py

sdks/python/apache_beam/io/gcp/bigquery_tools.py

sdks/python/test-suites/xlang/build.gradle

chamikaramj

LGTM. Thanks.

.../java/org/apache/beam/sdk/expansion/service/ExpansionServiceSchemaTransformProviderTest.java

…or a specified expansion service

integrate with WriteToBigQuery use shadowJar move port acquisition closer to expansion service launch

…e_python_wrapper

ahmedabu98 · 2023-03-27T18:42:10Z

Run Python_Xlang_Gcp_Direct PostCommit

ahmedabu98 · 2023-03-27T19:20:20Z

Run Python_Xlang_Gcp_Dataflow PostCommit

chamikaramj

LGTM. Thanks.

github-actions bot added build gcp io java python infra labels Mar 1, 2023

ahmedabu98 marked this pull request as ready for review March 1, 2023 23:32

chamikaramj self-requested a review March 1, 2023 23:49

chamikaramj reviewed Mar 3, 2023

View reviewed changes

ahmedabu98 force-pushed the storage_write_python_wrapper branch from 7edafd1 to 4b476ad Compare March 10, 2023 22:39

ahmedabu98 closed this Mar 13, 2023

ahmedabu98 reopened this Mar 13, 2023

ahmedabu98 force-pushed the storage_write_python_wrapper branch from 34357d2 to 300dbbf Compare March 14, 2023 23:00

chamikaramj requested changes Mar 15, 2023

View reviewed changes

ahmedabu98 requested a review from chamikaramj March 23, 2023 17:57

chamikaramj approved these changes Mar 23, 2023

View reviewed changes

ahmedabu98 commented Mar 26, 2023

View reviewed changes

.../java/org/apache/beam/sdk/expansion/service/ExpansionServiceSchemaTransformProviderTest.java Show resolved Hide resolved

ahmedabu98 added 7 commits March 27, 2023 14:31

discover one config

493b249

creating dlq for the schematransform

79c00ce

bq storage write api wrapper

fa3eb4e

support creating easy Python-using-Java cross-language gradle tasks f…

18f5efe

…or a specified expansion service

add Python-using-Java jenkins tests for GCP expansion service

aea561b

Address comments:

db6f97f

integrate with WriteToBigQuery use shadowJar move port acquisition closer to expansion service launch

typo

6f71987

ahmedabu98 added 6 commits March 27, 2023 14:31

address comments and add tests for new bigquery tools

5d4a2c0

lint

b195d71

skip bq tools test if GCP dependencies not installed

704a7fb

sync with HEAD, fix encoding positions mismatch

ae2a03b

fix some tests

09ff611

return xvr job back to normal

52a6c4a

ahmedabu98 force-pushed the storage_write_python_wrapper branch from 9d3c870 to 52a6c4a Compare March 27, 2023 14:32

ahmedabu98 added 2 commits March 27, 2023 17:11

Merge branch 'master' of github.com:ahmedabu98/beam into storage_writ…

10bbcd1

…e_python_wrapper

missing import

0999c68

chamikaramj approved these changes Mar 27, 2023

View reviewed changes

chamikaramj merged commit fc7a240 into apache:master Mar 28, 2023

Abacn mentioned this pull request Mar 28, 2023

[Failing Test]: Python 3.7 PostCommit failing BigQueryXlangStorageWriteIT #26016

Closed

15 tasks

Abacn mentioned this pull request Apr 28, 2023

[Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding #25669

Closed

15 tasks

RyuSA mentioned this pull request May 15, 2023

[Task]: Missing Information about Storage Write API Support in Google BigQuery I/O connector for Python SDK #26693

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery Storage Write API wrapper for Python SDK #25685

BigQuery Storage Write API wrapper for Python SDK #25685

ahmedabu98 commented Mar 1, 2023 •

edited

Loading

ahmedabu98 commented Mar 1, 2023

github-actions bot commented Mar 1, 2023

codecov bot commented Mar 2, 2023 •

edited

Loading

chamikaramj left a comment

chamikaramj commented Mar 13, 2023

ahmedabu98 commented Mar 13, 2023 •

edited

Loading

ahmedabu98 commented Mar 14, 2023

chamikaramj left a comment

chamikaramj left a comment

ahmedabu98 commented Mar 27, 2023

ahmedabu98 commented Mar 27, 2023

chamikaramj left a comment

BigQuery Storage Write API wrapper for Python SDK #25685

BigQuery Storage Write API wrapper for Python SDK #25685

Conversation

ahmedabu98 commented Mar 1, 2023 • edited Loading

ahmedabu98 commented Mar 1, 2023

github-actions bot commented Mar 1, 2023

codecov bot commented Mar 2, 2023 • edited Loading

Codecov Report

chamikaramj left a comment

Choose a reason for hiding this comment

chamikaramj commented Mar 13, 2023

ahmedabu98 commented Mar 13, 2023 • edited Loading

ahmedabu98 commented Mar 14, 2023

chamikaramj left a comment

Choose a reason for hiding this comment

chamikaramj left a comment

Choose a reason for hiding this comment

ahmedabu98 commented Mar 27, 2023

ahmedabu98 commented Mar 27, 2023

chamikaramj left a comment

Choose a reason for hiding this comment

ahmedabu98 commented Mar 1, 2023 •

edited

Loading

codecov bot commented Mar 2, 2023 •

edited

Loading

ahmedabu98 commented Mar 13, 2023 •

edited

Loading