vdk-lineage: introducing POC (pre-alpha) implementation #783
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
antoniivanov
force-pushed
the
person/aivanov/vdk-lineage
branch
4 times, most recently
from
47fc17b to
1eec655
Compare
doks5
approved these changes
Apr 13, 2022
projects/vdk-plugins/vdk-lineage/src/vdk/plugin/lineage/plugin_lineage.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-lineage/src/vdk/plugin/lineage/sql_lineage_parser.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-lineage/src/vdk/plugin/marquez/marquez_server.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-lineage/tests/test_sql_lineage_parser.py
Outdated
Show resolved
Hide resolved
VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to pre-configured destination. At POC level currently (Pre-Alpha release). It collects lineage information for each job run and for each executed query. Query execution is currently before it's executed (so not query status is logged) as we lack the necessary hooks. There are plenty of TODOs in the code that need to be addressed before moving maturity level and add more tests. Testing Done: there are some tests and also manually: ``` vdk marquez-server --start vdk run some-job vdk marquez-server --stop ``` Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
force-pushed
the
person/aivanov/vdk-lineage
branch
from
1eec655 to
521082d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
VDK Lineage plugin provides lineage data (input data -> job -> output
data) information and send it to pre-configured
destination.
At POC level currently (Pre-Alpha release).
It collects lineage information for each job run and for each executed
query.
Query execution is currently before it's executed (so not query status
is logged) as we lack the necessary hooks.
There are plenty of TODOs in the code that need to be addressed before
moving maturity level and add more tests.
Testing Done: there are some tests and also manually:
and saw lineage in marquz ui
Signed-off-by: Antoni Ivanov [email protected]