🎉 Source stripe - enhanced performance for streams which run substreams #10359

midavadim · 2022-02-15T19:03:34Z

Enhanced performance for stripe source

What

Tested on sync stripe -> bigquery (with airbyte creds):

Succeeded
40.75 MB | 18,385 records | 57m 31s | Sync

The problem was with a few streams:

invoice_line_items: 31m / 4985 items
2022-02-10 16: 23:40 �[44msource�[0m > Syncing stream: invoice_line_items
2022-02-10 16:30:19 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 5000
2022-02-10 16:37:07 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 6000
2022-02-10 16:43:50 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 7000
2022-02-10 16:49:47 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 8000
2022-02-10 16:54:20 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 9000
2022-02-10 16:54:23 �[44msource�[0m > Read 4985 records from invoice_line_items stream
2022-02-10 16:54:23 �[44msource�[0m > Finished syncing SourceStripe
2022-02-10 16:54:23 �[44msource�[0m > SourceStripe runtimes:
subscription_items: 10m / 1995 items
2022-02-10 16:58:30 �[44msource�[0m > Syncing stream: subscription_items
2022-02-10 17:00:22 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 15000
2022-02-10 17:05:57 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 16000
2022-02-10 17:08:44 �[44msource�[0m > Read 1995 records from subscription_items stream
2022-02-10 17:08:44 �[44msource�[0m > Finished syncing SourceStripe
2022-02-10 17:08:44 �[44msource�[0m > SourceStripe runtimes:

Reason:
invoice_line_items - stream runs 1 request for each of 4372 invoices (main stream)
subscription_items - stream runs 1 request for each of 1686 subscriptions (main stream)

How

Research shows that records from main streams already contain 1st page of needed items (invoice_line_items and subscription_items).
But In major cases, pagination requests are not performed because line items are fully reported in main streams' streams

invoice_line_items: 2:20m / 4988 items
2022-02-15 17:52:30 �[44msource�[0m > Syncing stream: invoice_line_items
2022-02-15 17:52:34 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 4000
2022-02-15 17:53:05 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 5000
2022-02-15 17:53:35 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 6000
2022-02-15 17:54:03 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 7000
2022-02-15 17:54:29 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 8000
2022-02-15 17:54:50 �[44msource�[0m > Read 4988 records from invoice_line_items stream
2022-02-15 17:54:50 �[44msource�[0m > Finished syncing SourceStripe
subscription_items: 50s / 1995 items
2022-02-15 17:58:53 �[44msource�[0m > Syncing stream: subscription_items
2022-02-15 17:59:03 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 15000
2022-02-15 17:59:31 �[32mINFO�[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):281 - Records read: 16000
2022-02-15 17:59:40 �[44msource�[0m > Read 1995 records from subscription_items stream
2022-02-15 17:59:40 �[44msource�[0m > Finished syncing SourceStripe

Results of manual tests:
Last: new version
Previous: Old version

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

…tem.

CLAassistant · 2022-02-15T19:03:45Z

All committers have signed the CLA.

codecov · 2022-02-15T19:07:22Z

Codecov Report

❗ No coverage uploaded for pull request base (master@342840d). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head c41d934 differs from pull request most recent head bfe9e81. Consider uploading reports for the commit bfe9e81 to get more accurate results

@@            Coverage Diff            @@
##             master   #10359   +/-   ##
=========================================
  Coverage          ?   70.15%           
=========================================
  Files             ?        3           
  Lines             ?      258           
  Branches          ?        0           
=========================================
  Hits              ?      181           
  Misses            ?       77           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 342840d...bfe9e81. Read the comment docs.

midavadim · 2022-02-15T19:41:55Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1849046238
❌ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1849046238
🐛 https://gradle.com/s/np7fmu4qgrcgq

midavadim · 2022-02-16T11:02:00Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1852384652
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1852384652
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/tests/test_incremental.py        69     38    45%
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/__init__.py       2      0   100%
source_stripe/source.py        22     11    50%
source_stripe/streams.py      237     90    62%
-----------------------------------------------
TOTAL                         261    101    61%

airbyte-integrations/connectors/source-stripe/source_stripe/schemas/invoice_line_items.json

bazarnov

Not the big deal, but I think we can optimise the code even more here, by reusing some parts, please read the comments bellow.

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py

antixar

Loading of the substreams was implemented without the normal iterable logic

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py

antixar

Could you add a simple integration test for this new logic? For example we can compare records for 2 flows:

your new logic
mock the lines property by an empty array.
Both lists should be same

bazarnov · 2022-02-21T13:37:22Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1876462955
❌ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1876462955
🐛 https://gradle.com/s/blbpleyvjzska

…pe-performance-improvement

midavadim · 2022-02-23T09:52:54Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1886541794
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1886541794
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_stripe/__init__.py       2      0   100%
source_stripe/streams.py      235     66    72%
source_stripe/source.py        22     11    50%
-----------------------------------------------
TOTAL                         259     77    70%

midavadim · 2022-02-28T13:06:23Z

Could you add a simple integration test for this new logic? For example we can compare records for 2 flows:

your new logic

mock the lines property by an empty array.
Both lists should be sam

It is not possible to do such test. The idea of this whole update is that then 'lines' is empty then we should not run any additional requests.
I will try to add some unit test for this update read_records method with mocked another methods

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py

ChristopheDuong · 2022-03-01T09:24:00Z

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py

+
+            # filter out 'bank_account' source items only
+            if self.filter:
+                items = [i for i in items if i.get(self.filter["attr"]) == self.filter["value"]]


What is this bank_account filter? where does it come from?

It seems like maybe the comment is not in sync with code, right?
(only used in customers.bank_account?)

can we add comments on what the generic filter is for? i am guessing bank_account is one example of such usage?

midavadim · 2022-03-10T15:15:28Z

@ChristopheDuong I saw you approved that review, but there were a few comments from you. I have replied to these comments. Could you please let me know if I can merge this PR into main.

midavadim · 2022-03-10T15:39:54Z

/publish connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1964024118
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/1964024118

Maksim is not able to finish review

enhanced performance for streams which run 1 requests for each main i…

cac4208

…tem.

github-actions bot added the area/connectors Connector related issues label Feb 15, 2022

This was linked to issues Feb 15, 2022

Source Stripe: sync freeze with 80k records #6417

Closed

Source Stripe: improve performance #9404

Closed

octavia-squidington-iii temporarily deployed to more-secrets February 15, 2022 19:44 Inactive

removed unused types

cdb80bf

midavadim temporarily deployed to more-secrets February 16, 2022 11:00 Inactive

octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 11:04 Inactive

midavadim requested review from grubberr, antixar and bazarnov February 16, 2022 11:12

midavadim commented Feb 16, 2022

View reviewed changes

airbyte-integrations/connectors/source-stripe/source_stripe/schemas/invoice_line_items.json Show resolved Hide resolved

bazarnov requested changes Feb 17, 2022

View reviewed changes

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py Show resolved Hide resolved

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py Outdated Show resolved Hide resolved

antixar previously requested changes Feb 18, 2022

View reviewed changes

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py Outdated Show resolved Hide resolved

moved common code to StripeSubStream

c5a56d4

midavadim temporarily deployed to more-secrets February 18, 2022 18:54 Inactive

midavadim requested review from antixar and bazarnov February 18, 2022 19:03

antixar reviewed Feb 21, 2022

View reviewed changes

octavia-squidington-iii temporarily deployed to more-secrets February 21, 2022 13:40 Inactive

Merge remote-tracking branch 'origin/master' into midavadim/9404-stri…

d205bc9

…pe-performance-improvement

midavadim temporarily deployed to more-secrets February 23, 2022 09:53 Inactive

octavia-squidington-iii temporarily deployed to more-secrets February 23, 2022 09:55 Inactive

bazarnov approved these changes Feb 23, 2022

View reviewed changes

midavadim requested a review from ChristopheDuong February 28, 2022 12:32

ChristopheDuong reviewed Mar 1, 2022

View reviewed changes

airbyte-integrations/connectors/source-stripe/source_stripe/streams.py Show resolved Hide resolved

ChristopheDuong reviewed Mar 1, 2022

View reviewed changes

ChristopheDuong approved these changes Mar 1, 2022

View reviewed changes

updated docs, updated docker version

c41d934

github-actions bot added the area/documentation Improvements or additions to documentation label Mar 10, 2022

midavadim temporarily deployed to more-secrets March 10, 2022 15:39 Inactive

updated connector version in source_specs.yaml

bfe9e81

midavadim temporarily deployed to more-secrets March 10, 2022 16:06 Inactive

midavadim requested a review from antixar March 10, 2022 16:08

midavadim merged commit a1a4bbc into master Mar 10, 2022

midavadim deleted the midavadim/9404-stripe-performance-improvement branch March 10, 2022 16:14

octavia-squidington-iii mentioned this pull request Mar 10, 2022

Bump Airbyte version from 0.35.49-alpha to 0.35.50-alpha #11050

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source stripe - enhanced performance for streams which run substreams #10359

🎉 Source stripe - enhanced performance for streams which run substreams #10359

midavadim commented Feb 15, 2022 •

edited

Loading

CLAassistant commented Feb 15, 2022 •

edited

Loading

codecov bot commented Feb 15, 2022 •

edited

Loading

midavadim commented Feb 15, 2022 •

edited by github-actions bot

Loading

midavadim commented Feb 16, 2022 •

edited by github-actions bot

Loading

bazarnov left a comment

antixar left a comment

antixar left a comment

bazarnov commented Feb 21, 2022 •

edited by github-actions bot

Loading

midavadim commented Feb 23, 2022 •

edited by github-actions bot

Loading

midavadim commented Feb 28, 2022

ChristopheDuong Mar 1, 2022 •

edited

Loading

midavadim commented Mar 10, 2022

midavadim commented Mar 10, 2022 •

edited by github-actions bot

Loading

🎉 Source stripe - enhanced performance for streams which run substreams #10359

🎉 Source stripe - enhanced performance for streams which run substreams #10359

Conversation

midavadim commented Feb 15, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Feb 15, 2022 • edited Loading

codecov bot commented Feb 15, 2022 • edited Loading

Codecov Report

midavadim commented Feb 15, 2022 • edited by github-actions bot Loading

midavadim commented Feb 16, 2022 • edited by github-actions bot Loading

bazarnov left a comment

Choose a reason for hiding this comment

antixar left a comment

Choose a reason for hiding this comment

antixar left a comment

Choose a reason for hiding this comment

bazarnov commented Feb 21, 2022 • edited by github-actions bot Loading

midavadim commented Feb 23, 2022 • edited by github-actions bot Loading

midavadim commented Feb 28, 2022

ChristopheDuong Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

midavadim commented Mar 10, 2022

midavadim commented Mar 10, 2022 • edited by github-actions bot Loading

midavadim commented Feb 15, 2022 •

edited

Loading

CLAassistant commented Feb 15, 2022 •

edited

Loading

codecov bot commented Feb 15, 2022 •

edited

Loading

midavadim commented Feb 15, 2022 •

edited by github-actions bot

Loading

midavadim commented Feb 16, 2022 •

edited by github-actions bot

Loading

bazarnov commented Feb 21, 2022 •

edited by github-actions bot

Loading

midavadim commented Feb 23, 2022 •

edited by github-actions bot

Loading

ChristopheDuong Mar 1, 2022 •

edited

Loading

midavadim commented Mar 10, 2022 •

edited by github-actions bot

Loading