vdk-core: pass vdk run arguments in standalone job mode as well #2601
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When running StandaloneDataJob we do not pass `vdk run` arguments to `vdk_start` hook, but some hook implementations rely on that to understand what command is being run without that it breaks their logic. JobConfigIni Configuraiton provider is one such hook implementation See https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/vdk_config.py#L189 As can be seen we infer the job paht from the command line and try to find config.ini there. This is hacky (there's unfixed TODO) and naturally causes such issues as hacky things do. But we have not come up with better way then that. This is causing issues with Jupyter integration where config.ini file is not being parsed and read! To fix that and for consistency since StandaloneDataJob simulates vdk run of a job we pass all the arguments to vdk_start hook as if vdk run command was run.
DeltaMichael
approved these changes
Aug 24, 2023
duyguHsnHsn
approved these changes
Aug 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running StandaloneDataJob we do not pass
vdk runarguments tovdk_starthook, but some hook implementations rely on that to understand what command is being run without that it breaks their logic.JobConfigIni Configuraiton provider is one such hook implementation See
https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-core/src/vdk/internal/builtin_plugins/config/vdk_config.py#L189
As can be seen we infer the job paht from the command line and try to find config.ini there.
This is hacky (there's unfixed TODO) and naturally causes such issues as hacky things do. But we have not come up with better way then that.
This is causing issues with Jupyter integration where config.ini file is not being parsed and read!
To fix that and for consistency since StandaloneDataJob simulates vdk run of a job we pass all the arguments to vdk_start hook as if vdk run command was run.