Pass parameters to task when run with --python-params and python_wheel_wrapper is true
#1037
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… and python_wheel_wrapper is true
pietern
approved these changes
Dec 1, 2023
| if python_params: | ||
| params = json.loads(python_params) | ||
| except Exception as e: | ||
| print(e) |
There was a problem hiding this comment.
This can mess with stdout; how about letting this bubble up?
There was a problem hiding this comment.
@pietern the reason I'm not re-raising this is that dbutils.widgets.get("__python_params") throws an exception when no python params passed, so we just log it. Since we're buffering Python wheel output below, it should not impact the stdout there.
There was a problem hiding this comment.
What kind of exception is thrown if the parameter doesn't exist? Could we make the exception we catch narrower?
There was a problem hiding this comment.
@pietern it's Py4JJavaError, not sure if it's specific enough to be helpful in this case
pietern
changed the title
--python-params and python_wheel_wrapper is true
pietern
changed the title
--python-params and python_wheel_wrapper is true--python-params and python_wheel_wrapper is true
andrewnester
added a commit
that referenced
this pull request
Dec 6, 2023
CLI: * Add documentation for positional args in commands generated from the Databricks OpenAPI specification ([#1033](#1033)). * Ask for host when .databrickscfg doesn't exist ([#1041](#1041)). * Add list of supported values for flags that represent an enum field ([#1036](#1036)). Bundles: * Fix panic when bundle auth resolution fails ([#1002](#1002)). * Add versioning for bundle templates ([#972](#972)). * Add support for conditional prompting in bundle init ([#971](#971)). * Pass parameters to task when run with `--python-params` and `python_wheel_wrapper` is true ([#1037](#1037)). * Change default_python template to auto-update version on each wheel build ([#1034](#1034)). Internal: * Rewrite the friendly log handler ([#1038](#1038)). * Move bundle schema update to an internal module ([#1012](#1012)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.26.0 to 0.26.1 ([#1040](#1040)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 6, 2023
CLI: * Add documentation for positional args in commands generated from the Databricks OpenAPI specification ([#1033](#1033)). * Ask for host when .databrickscfg doesn't exist ([#1041](#1041)). * Add list of supported values for flags that represent an enum field ([#1036](#1036)). Bundles: * Fix panic when bundle auth resolution fails ([#1002](#1002)). * Add versioning for bundle templates ([#972](#972)). * Add support for conditional prompting in bundle init ([#971](#971)). * Pass parameters to task when run with `--python-params` and `python_wheel_wrapper` is true ([#1037](#1037)). * Change default_python template to auto-update version on each wheel build ([#1034](#1034)). Internal: * Rewrite the friendly log handler ([#1038](#1038)). * Move bundle schema update to an internal module ([#1012](#1012)). Dependency updates: * Bump github.com/databricks/databricks-sdk-go from 0.26.0 to 0.26.1 ([#1040](#1040)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
It makes the behaviour consistent with or without
python_wheel_wrapperon when job is run with--python-paramsflag.In
python_wheel_wrappermode it converts dynamicpython_paramsin a dynamic specially namednotebook_paramand the wrapper reads them withdbutilsand pass tosys.argvFixes #1000
Tests
Added an integration test.
Integration tests pass.