vdk-core: Verify payload after pre-processing it #777
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ivakoleva
approved these changes
Mar 28, 2022
antoniivanov
reviewed
Apr 1, 2022
projects/vdk-core/src/vdk/internal/builtin_plugins/ingestion/ingester_router.py
Outdated
Show resolved
Hide resolved
YanaZhivkova
force-pushed
the
person/yzhivkova/Verify_ingestion_payload_after_pre-process
branch
from
6d85dc4 to
ea20713
Compare
antoniivanov
approved these changes
Apr 14, 2022
Ingestion allows pre-processing payloads. If the payload is curated during pre-processing, currently, verification and subsequently job execution would fail. Verify payload after pre-processing it, since this pre-processing might be responsible for making it serializable. Add payload verification functional tests since it can no longer be tested in send_object_for_ingestion async function. Tested by: locally run ingestion job which curates json during pre-processing; functional test added. Signed-off-by: Yana Zhivkova <[email protected]>
YanaZhivkova
force-pushed
the
person/yzhivkova/Verify_ingestion_payload_after_pre-process
branch
from
ea20713 to
e65a8c6
Compare
ivakoleva
deleted the
person/yzhivkova/Verify_ingestion_payload_after_pre-process
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ingestion allows pre-processing payloads. If the payload
is curated during pre-processing, currently, verification
would fail.
Verify payload after pre-processing it, since this
pre-processing might be responsible for making it serializable.
Also, add VDK_ prefix to configs so that ingestion pre/post
processors are properly initialized.
Tested by: locally run ingestion job which curates json
during pre-processing
Signed-off-by: Yana Zhivkova [email protected]