[DAB] Add support for requirements libraries in Job Tasks #1543
Conversation
There was a problem hiding this comment.
Because we could reference the requirements.txt from local file system, we need to correctly overwrite the local to remote path, something like we do for wheels
https://github.com/databricks/cli/blob/main/bundle/artifacts/artifacts.go#L181-L184
or for any jobs paths https://github.com/databricks/cli/blob/main/bundle/config/mutator/translate_paths_jobs.go#L21-L54
|
@andrewnester Thanks for the feedback. I'll try to work on this in the coming days. |
|
Hey @andrewnester I updated the PR (also rebased onto latest main), please check if this goes into the right direction. I'm not very familiar with the codebase, so I used the existing source and test code as orientation. However, I'm not sure how to write a unit test for |
|
@andrewnester Thanks for the extensive feedback. I adapted the PR accordingly. I was also able to add a test in |
There was a problem hiding this comment.
Thanks for the contribution! I think there's a bit a confusion about approach we need to take, I guess it's better if I explain expectations a bit more.
The way this should work is customers specify the following section in their bundle config
libraries:
- requirements: ./local/path/to/requirements.txt
and nothing else.
What DABs should do is
- Upload requirements.txt to workspace file system - this is done automatically similarly to notebooks for example
- Rewrite local path to remote path (where requirements.txt was uploaded) in bundle config similarly to how this happens for notebooks now. - This is what needs to be implemented in this PR.
You don't need to use artifacts because artifacts is something that is being built, packaged and then uploaded by DABs, for requirements.txt is not needed.
I hope this clarifies what needs to be done
|
Hey @andrewnester Thank you very much! As mentioned, as I'm new to this codebase, I really appreciate your guidance! It's definitely a quite interesting task to get familiar with DAB. 😉 I reverted the changes regarding artifacts (as a requirements.txt is not seen as an artifact). Please check if it now makes sense to you |
|
Confirmed manually that requirements.txt is uploaded and used correctly on DBR 15.1+ |
|
Thanks for contribution @witi83 ! |
CLI: * Added filtering flags for cluster list commands ([#1703](#1703)). Bundles: * Remove reference to "dbt" in the default-sql template ([#1696](#1696)). * Pause continuous pipelines when 'mode: development' is used ([#1590](#1590)). * Add configurable presets for name prefixes, tags, etc. ([#1490](#1490)). * Report all empty resources present in error diagnostic ([#1685](#1685)). * Improves detection of PyPI package names in environment dependencies ([#1699](#1699)). * [DAB] Add support for requirements libraries in Job Tasks ([#1543](#1543)). * Add paths field to bundle sync configuration ([#1694](#1694)). Internal: * Add `import` option for PyDABs ([#1693](#1693)). * Make fileset take optional list of paths to list ([#1684](#1684)). * Pass through paths argument to libs/sync ([#1689](#1689)). * Correctly mark package names with versions as remote libraries ([#1697](#1697)). * Share test initializer in common helper function ([#1695](#1695)). * Make `pydabs/venv_path` optional ([#1687](#1687)). * Use API mocks for duplicate path errors in workspace files extensions client ([#1690](#1690)). * Fix prefix preset used for UC schemas ([#1704](#1704)).
CLI: * Added filtering flags for cluster list commands ([#1703](#1703)). Bundles: * Remove reference to "dbt" in the default-sql template ([#1696](#1696)). * Pause continuous pipelines when 'mode: development' is used ([#1590](#1590)). * Add configurable presets for name prefixes, tags, etc. ([#1490](#1490)). * Report all empty resources present in error diagnostic ([#1685](#1685)). * Improves detection of PyPI package names in environment dependencies ([#1699](#1699)). * [DAB] Add support for requirements libraries in Job Tasks ([#1543](#1543)). * Add paths field to bundle sync configuration ([#1694](#1694)). Internal: * Add `import` option for PyDABs ([#1693](#1693)). * Make fileset take optional list of paths to list ([#1684](#1684)). * Pass through paths argument to libs/sync ([#1689](#1689)). * Correctly mark package names with versions as remote libraries ([#1697](#1697)). * Share test initializer in common helper function ([#1695](#1695)). * Make `pydabs/venv_path` optional ([#1687](#1687)). * Use API mocks for duplicate path errors in workspace files extensions client ([#1690](#1690)). * Fix prefix preset used for UC schemas ([#1704](#1704)).
Changes
While experimenting with DAB I discovered that requirements libraries are being ignored.
One thing worth mentioning is that
bundle validateruns successfully, butbundle deployfails. This PR only covers the second part.Tests
Added a unit test