vdk-meta-jobs: Initial implementation #1249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
antoniivanov
force-pushed
the
person/aivanov/vdk-meta-jobs
branch
4 times, most recently
from
e7211e4 to
8084fe6
Compare
ivakoleva
reviewed
Oct 25, 2022
murphp15
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/meta_job_runner.py
Show resolved
Hide resolved
murphp15
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/remote_data_job.py
Show resolved
Hide resolved
murphp15
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/remote_data_job_executor.py
Show resolved
Hide resolved
murphp15
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/time_based_queue.py
Show resolved
Hide resolved
murphp15
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/time_based_queue.py
Show resolved
Hide resolved
dakodakov
reviewed
Oct 26, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/cached_data_job_executor.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/cached_data_job_executor.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/remote_data_job.py
Show resolved
Hide resolved
This was referenced Oct 28, 2022
ivakoleva
reviewed
Oct 31, 2022
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/api/meta_job.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/api/meta_job.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/remote_data_job.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/remote_data_job.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/meta_job_runner.py
Show resolved
Hide resolved
antoniivanov
force-pushed
the
person/aivanov/vdk-meta-jobs
branch
from
23de03d to
9851cb7
Compare
antoniivanov
force-pushed
the
person/aivanov/vdk-meta-jobs
branch
2 times, most recently
from
ae9a862 to
99cbae4
Compare
dakodakov
approved these changes
Nov 2, 2022
This is an pre-Alpha implementation of Meta Jobs functioanlity One crucial feature which has been missing from VDK is the ability to express workload (data job) interdependencies, meaning that a certain Data Job wil only trigger upon the successful completion (or other criteria) of one or more other Data Jobs. An integration with Apache Airflow was developed to address this, however this comes with its downsides: namely that teams would have to manage their own instance of Airflow, or pay for an externally managed service. The aim of this submission is to build a plugin for Versatile Data Kit which extends its API with an additional feature which allows users to trigger so called Meta Jobs - jobs which trigger one or more other jobs, wait for their completion, and then trigger another set of jobs until either the entire job pipeline has succeeded, or at least one of the jobs has failed. #1243 Signed-off-by: Antoni Ivanov <[email protected]>
Signed-off-by: Antoni Ivanov <[email protected]>
Signed-off-by: Antoni Ivanov <[email protected]>
The status calls were basically done in busy loop while jobs are running. This would be hammering the database which is not a good idea :) Signed-off-by: Antoni Ivanov <[email protected]>
Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
force-pushed
the
person/aivanov/vdk-meta-jobs
branch
from
99cbae4 to
05dee0a
Compare
doks5
reviewed
Nov 3, 2022
murphp15
approved these changes
Nov 3, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a pre-Alpha implementation of Meta Jobs functionality.
One crucial feature which has been missing from VDK is the ability to
express workload (data job) interdependencies, meaning that a certain
Data Job wil only trigger upon the successful completion (or other
criteria) of one or more other Data Jobs.
Integration with Apache Airflow was developed to address this,
however, this comes with its downsides: namely that teams would have to
manage their own instance of Airflow, or pay for an externally managed
service.
The aim of this submission is to build a plugin for Versatile Data Kit
which extends its API with an additional feature that allows users to
trigger so-called Meta Jobs - jobs that trigger one or more other jobs,
wait for their completion, and then trigger another set of jobs until
either the entire job pipeline has succeeded, or at least one of the
jobs has failed.
#1243
Testing Done: functional tests
Signed-off-by: Antoni Ivanov [email protected]