vdk-impala: Refactor query classifier for data lineage #1239
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Why? Analysing the SQL query is not so trivial task, as there are number of variations of the DDL, DML and DDL with DML combinations. Current avoidance of single line comments does not work properly and skips queries that are eligible for lineage collection. # What? Added sqlparse library in order to externalise comment-cleaning task, now it leaves us only with the query and there isn't any need to analyse whole query. What we do now is we check if query starts with blacklisted keyword, if so - query is not eligible for lineage. Although some keywords blacklisted might contain lineage data, like CREATE statement, when table is created from another with SELECT statement. In this case blacklisted word needs to be whitelisted in some conditions. As sql statements are complex, we need to get which queries definitely has data lineage, instead of searching which query have for sure and for sure not data lineage. False positives queries for lineage, ie queries that does not have lineage information provided from the profile, but are marked as having one, will not break the logic at all, instead we optimize load on the impala server with a simple approach # How has this been tested? Unit and integration tests, also with production-grade queries # What type of change are you making? Bug-fixing Signed-off-by: Plamen Kostov <[email protected]>
for more information, see https://pre-commit.ci
projects/vdk-plugins/vdk-impala/src/vdk/plugin/impala/impala_lineage_plugin.py
Show resolved
Hide resolved
VladimirPetkov1
approved these changes
Oct 17, 2022
antoniivanov
approved these changes
Oct 17, 2022
# Why? Analysing the SQL query is not so trivial task, as there are number of variations of the DDL, DML and DDL with DML combinations. Current avoidance of single line comments does not work properly and skips queries that are eligible for lineage collection. # What? Added sqlparse library in order to externalise comment-cleaning task, now it leaves us only with the query and there isn't any need to analyse whole query. What we do now is we check if query starts with blacklisted keyword, if so - query is not eligible for lineage. Although some keywords blacklisted might contain lineage data, like CREATE statement, when table is created from another with SELECT statement. In this case blacklisted word needs to be whitelisted in some conditions. As sql statements are complex, we need to get which queries definitely has data lineage, instead of searching which query have for sure and for sure not data lineage. False positives queries for lineage, ie queries that does not have lineage information provided from the profile, but are marked as having one, will not break the logic at all, instead we optimize load on the impala server with a simple approach # How has this been tested? Unit and integration tests, also with production-grade queries # What type of change are you making? Bug-fixing Signed-off-by: Plamen Kostov <[email protected]>
kostoww
force-pushed
the
topic/pkostov/impala-refactor-lineage-detecting
branch
from
672cedc to
ce44501
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why?
Analysing the SQL query is not so trivial task, as there are number of variations of the DDL, DML and DDL with DML combinations. Current avoidance of single line comments does not work properly and skips queries that are eligible for lineage collection.
What?
Added sqlparse library in order to externalise comment-cleaning task, now it leaves us only with the query and there isn't any need to analyse whole query. What we do now is we check if query starts with blacklisted keyword, if so - query is not eligible for lineage. Although some keywords blacklisted might contain lineage data, like CREATE statement, when table is created from another with SELECT statement. In this case blacklisted word needs to be whitelisted in some conditions.
As sql statements are complex, we need to get which queries definitely has data lineage, instead of searching which query have for sure and for sure not data lineage. False positives queries for lineage, ie queries that does not have lineage information provided from the profile, but are marked as having one, will not break the logic at all, instead we optimize load on the impala server with a simple approach
How has this been tested?
Unit and integration tests, also with production-grade queries
What type of change are you making?
Bug-fixing
Signed-off-by: Plamen Kostov [email protected]