Skip to content
This repository has been archived by the owner on Jan 24, 2025. It is now read-only.

add AirbyteConnection, AirbyteSync and run_connection_sync #44

Merged
merged 8 commits into from
Dec 28, 2022

Conversation

zzstoatzz
Copy link
Contributor

@zzstoatzz zzstoatzz commented Dec 28, 2022

Adds a AirbyteConnection(JobBlock) and AirbyteSync(JobRun) enabling the new flow run_connection_sync.

Adds a new client method get_job_info to allow inclusion of records_synced in the AirbyteSyncResult.

Closes #22

Example

from prefect import flow
from prefect_airbyte.server import AirbyteServer
from prefect_airbyte.connections import AirbyteConnection
from prefect_airbyte.flows import run_connection_sync

server = AirbyteServer()

connection = AirbyteConnection(
    airbyte_server=server,
    connection_id="7e78757d-fd7f-421b-80a5-f329dbdadf1b",
    status_updates=True,
)

@flow
def airbyte_syncs():
    # do some setup

    sync_result = run_connection_sync(
        airbyte_connection=connection,
    )

    # do some other things, like trigger DBT based on number of records synced
    print(f'{sync_result.records_synced}')

Screenshots

image
image

Checklist

  • References any related issue by including "Closes #" or "Closes ".
    • If no issue exists and your change is not a small fix, please create an issue first.
  • Includes tests or only affects documentation.
  • Passes pre-commit checks.
    • Run pre-commit install && pre-commit run --all locally for formatting and linting.
  • Includes screenshots of documentation updates.
    • Run mkdocs serve view documentation locally.
  • Summarizes PR's changes in CHANGELOG.md

@zzstoatzz zzstoatzz self-assigned this Dec 28, 2022
@zzstoatzz zzstoatzz marked this pull request as ready for review December 28, 2022 15:50
@zzstoatzz zzstoatzz requested a review from a team December 28, 2022 15:50
@zzstoatzz zzstoatzz changed the title add AirbyteConnection and run_airbyte_connection_sync add AirbyteConnection and run_connection_sync Dec 28, 2022
@zzstoatzz zzstoatzz changed the title add AirbyteConnection and run_connection_sync add AirbyteConnection, AirbyteSync and run_connection_sync Dec 28, 2022
README.md Outdated Show resolved Hide resolved
Copy link
Contributor

@ahuang11 ahuang11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good; just minor comments!

prefect_airbyte/connections.py Outdated Show resolved Hide resolved
prefect_airbyte/connections.py Show resolved Hide resolved
prefect_airbyte/connections.py Show resolved Hide resolved
prefect_airbyte/connections.py Outdated Show resolved Hide resolved
prefect_airbyte/connections.py Show resolved Hide resolved
prefect_airbyte/flows.py Show resolved Hide resolved
prefect_airbyte/client.py Outdated Show resolved Hide resolved
Copy link
Contributor

@ahuang11 ahuang11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more things I noticed.

prefect_airbyte/connections.py Outdated Show resolved Hide resolved
prefect_airbyte/connections.py Outdated Show resolved Hide resolved
prefect_airbyte/connections.py Outdated Show resolved Hide resolved
prefect_airbyte/connections.py Show resolved Hide resolved
@zzstoatzz zzstoatzz merged commit 66b2ded into main Dec 28, 2022
@zzstoatzz zzstoatzz deleted the job-block branch December 28, 2022 21:51
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create trigger_sync_and_wait_for_completion flow
2 participants