🐛 Source Shopify: Fixed rate limits throttling #5470

bazarnov · 2021-08-17T17:25:24Z

What

#5441 - Shopify Connector Rate Limiting
#5416 - Source shopify: confusing error message when rate limit is hit

How

added decorator balance_rate_limit method to control sleep time between each API call, by leveraging the "X-Shopify-Shop-Api-Call-Limit" header information, as described in Shopify Documentation
added unit tests to cover the issue
added note in Performance considerations section of shopify.md about rate limits.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…te-limits-issue

bazarnov · 2021-08-17T17:27:32Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1140220772
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1140220772

…te-limits-issue

airbyte-integrations/connectors/source-shopify/source_shopify/source.py

airbyte-integrations/connectors/source-shopify/unit_tests/unit_test.py

…te-limits-issue

bazarnov · 2021-08-18T16:39:07Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1143924329
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1143924329

airbyte-integrations/connectors/source-shopify/unit_tests/test_control_rate_limit.py

…te-limits-issue

…ov/shopify-rate-limits-issue

…ields in schemas

bazarnov · 2021-08-19T12:28:58Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1146941752
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1146941752

bazarnov · 2021-08-20T13:21:40Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1150819644
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1150819644

…te-limits-issue

bazarnov · 2021-08-20T15:00:46Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1151119628
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1151119628

…te-limits-issue

bazarnov · 2021-08-23T11:35:09Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1158473499
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1158473499

Phlair

Like the decorator for throttling sleeps and thanks for adding unit tests.

Couple of comments and Qs :)

airbyte-integrations/connectors/source-shopify/source_shopify/utils.py

airbyte-integrations/connectors/source-shopify/unit_tests/test_control_rate_limit.py

docs/integrations/sources/shopify.md

…te-limits-issue

bazarnov · 2021-08-23T20:02:38Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1160053552
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1160053552

rparrapy · 2021-08-23T21:08:51Z

FWIW, I am eagerly waiting for this to be merged :)

Phlair

Thanks for making changes!
I'm approving now so we can get this released for users but for readability and structure sake, addressing the comment I left would be ideal. If not then at least making an issue to do so would be great (or let me know if you disagree!)

airbyte-integrations/connectors/source-shopify/source_shopify/utils.py

bazarnov · 2021-08-24T10:08:57Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1162135031
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1162135031

bazarnov · 2021-08-24T10:22:25Z

/publish connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1162176109
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/1162176109

…ts-issue

#5416 - source shopify: confusing error message when rate limit is hit #5441 - Shopify Connector Rate Limiting PR: #5470 - #5470 Co-authored-by: Oleksandr Bazarnov <[email protected]>

bazarnov added 2 commits August 17, 2021 20:11

fix for rate limits, added unit tests, bumped version

534deec

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

2866353

…te-limits-issue

bazarnov self-assigned this Aug 17, 2021

github-actions bot added the area/connectors Connector related issues label Aug 17, 2021

edited shopify.md

8296214

bazarnov requested review from antixar and annalvova05 August 17, 2021 17:26

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 17, 2021

This was linked to issues Aug 17, 2021

Shopify Connector Rate Limiting #5441

Closed

source shopify: confusing error message when rate limit is hit #5416

Closed

jrhizor temporarily deployed to more-secrets August 17, 2021 17:29 Inactive

bazarnov added 2 commits August 18, 2021 12:35

cleaned up code

721eb01

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

92d5bd5

…te-limits-issue

antixar reviewed Aug 18, 2021

View reviewed changes

annalvova05 approved these changes Aug 18, 2021

View reviewed changes

bazarnov added 2 commits August 18, 2021 19:37

updated due to review comments

016ab1c

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

e9b22bc

…te-limits-issue

jrhizor temporarily deployed to more-secrets August 18, 2021 16:41 Inactive

bazarnov requested a review from antixar August 18, 2021 16:54

antixar reviewed Aug 18, 2021

View reviewed changes

airbyte-integrations/connectors/source-shopify/unit_tests/test_control_rate_limit.py Outdated Show resolved Hide resolved

antixar self-requested a review August 18, 2021 20:52

updated due to review comments

d6abe44

antixar approved these changes Aug 19, 2021

View reviewed changes

bazarnov requested a review from Phlair August 19, 2021 08:02

bazarnov added 3 commits August 19, 2021 11:05

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

1c691c7

…te-limits-issue

Merge remote-tracking branch 'refs/remotes/origin/master' into bazarn…

1125aab

…ov/shopify-rate-limits-issue

changed Dockerfile regards new template, adeed format: date-time to f…

850ec4f

…ields in schemas

jrhizor temporarily deployed to more-secrets August 19, 2021 12:31 Inactive

jrhizor temporarily deployed to more-secrets August 20, 2021 13:24 Inactive

bazarnov added 2 commits August 20, 2021 17:50

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

90961fd

…te-limits-issue

fixed typos

c64d783

jrhizor temporarily deployed to more-secrets August 20, 2021 15:03 Inactive

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

f98c0cc

…te-limits-issue

jrhizor temporarily deployed to more-secrets August 23, 2021 11:37 Inactive

Phlair reviewed Aug 23, 2021

View reviewed changes

bazarnov added 2 commits August 23, 2021 19:36

Merge remote-tracking branch 'origin/master' into bazarnov/shopify-ra…

a4e417a

…te-limits-issue

updated after review

ee7b5fb

jrhizor temporarily deployed to more-secrets August 23, 2021 20:05 Inactive

Phlair approved these changes Aug 24, 2021

View reviewed changes

airbyte-integrations/connectors/source-shopify/source_shopify/utils.py Show resolved Hide resolved

bazarnov added 2 commits August 24, 2021 13:07

updated due to review comments'

fe3064f

formated

e3b8fc6

jrhizor temporarily deployed to more-secrets August 24, 2021 10:11 Inactive

jrhizor temporarily deployed to more-secrets August 24, 2021 10:25 Inactive

Merge remote-tracking branch 'origin' into bazarnov/shopify-rate-limi…

d4c6d0a

…ts-issue

bazarnov merged commit f76a135 into master Aug 24, 2021

bazarnov deleted the bazarnov/shopify-rate-limits-issue branch August 24, 2021 11:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Shopify: Fixed rate limits throttling #5470

🐛 Source Shopify: Fixed rate limits throttling #5470

bazarnov commented Aug 17, 2021 •

edited

Loading

bazarnov commented Aug 17, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 18, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 19, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 20, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 20, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 23, 2021 •

edited by github-actions bot

Loading

Phlair left a comment

bazarnov commented Aug 23, 2021 •

edited by github-actions bot

Loading

rparrapy commented Aug 23, 2021

Phlair left a comment

bazarnov commented Aug 24, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 24, 2021 •

edited by github-actions bot

Loading

🐛 Source Shopify: Fixed rate limits throttling #5470

🐛 Source Shopify: Fixed rate limits throttling #5470

Conversation

bazarnov commented Aug 17, 2021 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

bazarnov commented Aug 17, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 18, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 19, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 20, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 20, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 23, 2021 • edited by github-actions bot Loading

Phlair left a comment

Choose a reason for hiding this comment

bazarnov commented Aug 23, 2021 • edited by github-actions bot Loading

rparrapy commented Aug 23, 2021

Phlair left a comment

Choose a reason for hiding this comment

bazarnov commented Aug 24, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 24, 2021 • edited by github-actions bot Loading

bazarnov commented Aug 17, 2021 •

edited

Loading

bazarnov commented Aug 17, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 18, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 19, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 20, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 20, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 23, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 23, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 24, 2021 •

edited by github-actions bot

Loading

bazarnov commented Aug 24, 2021 •

edited by github-actions bot

Loading