Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airflow-provider-vdk: Initial Airflow provider structure #772

Merged
merged 31 commits into from
Apr 2, 2022

Conversation

gabrielgeorgiev1
Copy link
Contributor

@gabrielgeorgiev1 gabrielgeorgiev1 commented Mar 18, 2022

Versatile Data Kit allows users to schedule job executions using
a cron-like interface, however there is no way to express job
dependencies through VDK. For example, a user might want to
run two different ingestion jobs, and when both pass successfully,
to run a transformation job on the ingested data.
Apache Airflow is an open-source workload scheduling framework
which allows exactly that sort of job dependency specification.
To take advantage of this, this PR sets the initial structure of the
VDK Airflow provider, which will integrate Airflow's scheduling
capabilities with VDK.

README is left short as much of the implementation and usage
is still somewhat up in the air AFAIK.
Interested in feedback on the way authentication is approached
in VDKHook.
Also interested in opinions on what would be good default values
for timeout and the retry params for VDKHook.

Testing done: package installs successfully

Signed-off-by: Gabriel Georgiev [email protected]

@gabrielgeorgiev1 gabrielgeorgiev1 marked this pull request as ready for review March 28, 2022 09:43
@gabrielgeorgiev1 gabrielgeorgiev1 changed the title versatile-data-kit: Initial Airflow provider structure airflow-provider-vdk: Initial Airflow provider structure Mar 28, 2022
Signed-off-by: Gabriel Georgiev <[email protected]>
…satile-data-kit into person/gageorgiev/aiflow-base
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
@antoniivanov
Copy link
Collaborator

Let's put it in the plugins directory. Technically it can be thought as of a plugin (extension) to vdk regardless if it uses vdk hooks or not.

Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
Signed-off-by: Gabriel Georgiev <[email protected]>
@gabrielgeorgiev1 gabrielgeorgiev1 merged commit d1a03da into main Apr 2, 2022
@gabrielgeorgiev1 gabrielgeorgiev1 deleted the person/gageorgiev/aiflow-base branch April 2, 2022 07:53
@gabrielgeorgiev1 gabrielgeorgiev1 restored the person/gageorgiev/aiflow-base branch April 2, 2022 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants