databricks bundle init template v1 #686

lennartkats-db · 2023-08-21T11:44:04Z

Changes

This adds a built-in "default-python" template to the CLI. This is based on the new default-template support of #685.

The goal here is to offer an experience where customers can simply type databricks bundle init to get a default template:

$ databricks bundle init
Template to use [default-python]: default-python
Unique name for this project [my_project]: my_project
✨ Successfully initialized template

The present template:

Works well with VS Code
Works well with the workspace
Works well with DB Connect
Uses minimal stubs rather than boiler-plate-heavy examples

I'll have a followup with tests + DLT support.

libs/template/templates/default-python/template/{{.project_name}}/res/job.yml.tmpl

...template/templates/default-python/template/{{.project_name}}/src/{{.project_name}}/common.py

...efault-python/template/{{.project_name}}/src/{{.project_name}}/tasks/sample_etl_task.py.tmpl

fjakobs

I have mostly naming questions and suggestions.

It's great for users migrating from dbx that the template is almost identical to the dbx one. However, I think there is room for an even simpler template that doesn't have quite as much boilerplate.

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl

libs/template/templates/default-python/template/{{.project_name}}/dev/exploration.py

...templates/default-python/template/{{.project_name}}/src/{{.project_name}}/notebook_sample.py

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl

…e}}/.vscode/__builtins__.pyi Co-authored-by: PaulCornellDB <[email protected]>

…e}}/databricks.yml.tmpl Co-authored-by: Pieter Noordhuis <[email protected]>

cmd/bundle/init.go

~(this should be changed to target `main`)~ This reveals the template from #686 in CLI prompts for once #686 and #708 are merged. --------- Co-authored-by: Andrew Nester <[email protected]> Co-authored-by: PaulCornellDB <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]>

This adds a default_python project based on the template of databricks/cli#686: ``` The 'default_python' project was generated by using the default-python template. ``` It also adds a LICENSE and removes the original stub example.

## Changes This follows up on #686. This PR makes our stubs optional + it adds DLT stubs: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project Include a stub (sample) notebook in 'my_project/src' [yes]: yes Include a stub (sample) DLT pipeline in 'my_project/src' [yes]: yes Include a stub (sample) Python package 'my_project/src' [yes]: yes ✨ Successfully initialized template ``` ## Tests Manual testing, matrix tests. --------- Co-authored-by: Andrew Nester <[email protected]> Co-authored-by: PaulCornellDB <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]>

This release includes permission related commands for a subset of workspace services where they apply. These complement the `permissions` command and do not require specification of the object type to work with, as that is implied by the command they are nested under. CLI: * Group permission related commands ([#730](#730)). Bundles: * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([#722](#722)). * Make resource and artifact paths in bundle config relative to config folder ([#708](#708)). * Add support for ordering of input prompts ([#662](#662)). * Fix IsServicePrincipal() only working for workspace admins ([#732](#732)). * databricks bundle init template v1 ([#686](#686)). * databricks bundle init template v2: optional stubs, DLT support ([#700](#700)). * Show 'databricks bundle init' template in CLI prompt ([#725](#725)). * Include $PATH in set of environment variables to pass along. ([#736](#736)). Internal: * Update Go SDK to v0.19.0 ([#729](#729)). * Replace API call to test configuration with dummy authenticate call ([#728](#728)). API Changes: * Changed `databricks account storage-credentials create` command to return . * Changed `databricks account storage-credentials get` command to return . * Changed `databricks account storage-credentials list` command to return . * Changed `databricks account storage-credentials update` command to return . * Changed `databricks connections create` command with new required argument order. * Changed `databricks connections update` command with new required argument order. * Changed `databricks volumes create` command with new required argument order. * Added `databricks artifact-allowlists` command group. * Added `databricks model-versions` command group. * Added `databricks registered-models` command group. * Added `databricks cluster-policies get-permission-levels` command. * Added `databricks cluster-policies get-permissions` command. * Added `databricks cluster-policies set-permissions` command. * Added `databricks cluster-policies update-permissions` command. * Added `databricks clusters get-permission-levels` command. * Added `databricks clusters get-permissions` command. * Added `databricks clusters set-permissions` command. * Added `databricks clusters update-permissions` command. * Added `databricks instance-pools get-permission-levels` command. * Added `databricks instance-pools get-permissions` command. * Added `databricks instance-pools set-permissions` command. * Added `databricks instance-pools update-permissions` command. * Added `databricks files` command group. * Changed `databricks permissions set` command to start returning . * Changed `databricks permissions update` command to start returning . * Added `databricks users get-permission-levels` command. * Added `databricks users get-permissions` command. * Added `databricks users set-permissions` command. * Added `databricks users update-permissions` command. * Added `databricks jobs get-permission-levels` command. * Added `databricks jobs get-permissions` command. * Added `databricks jobs set-permissions` command. * Added `databricks jobs update-permissions` command. * Changed `databricks experiments get-by-name` command to return . * Changed `databricks experiments get-experiment` command to return . * Added `databricks experiments delete-runs` command. * Added `databricks experiments get-permission-levels` command. * Added `databricks experiments get-permissions` command. * Added `databricks experiments restore-runs` command. * Added `databricks experiments set-permissions` command. * Added `databricks experiments update-permissions` command. * Added `databricks model-registry get-permission-levels` command. * Added `databricks model-registry get-permissions` command. * Added `databricks model-registry set-permissions` command. * Added `databricks model-registry update-permissions` command. * Added `databricks pipelines get-permission-levels` command. * Added `databricks pipelines get-permissions` command. * Added `databricks pipelines set-permissions` command. * Added `databricks pipelines update-permissions` command. * Added `databricks serving-endpoints get-permission-levels` command. * Added `databricks serving-endpoints get-permissions` command. * Added `databricks serving-endpoints set-permissions` command. * Added `databricks serving-endpoints update-permissions` command. * Added `databricks token-management get-permission-levels` command. * Added `databricks token-management get-permissions` command. * Added `databricks token-management set-permissions` command. * Added `databricks token-management update-permissions` command. * Changed `databricks dashboards create` command with new required argument order. * Added `databricks warehouses get-permission-levels` command. * Added `databricks warehouses get-permissions` command. * Added `databricks warehouses set-permissions` command. * Added `databricks warehouses update-permissions` command. * Added `databricks dashboard-widgets` command group. * Added `databricks query-visualizations` command group. * Added `databricks repos get-permission-levels` command. * Added `databricks repos get-permissions` command. * Added `databricks repos set-permissions` command. * Added `databricks repos update-permissions` command. * Added `databricks secrets get-secret` command. * Added `databricks workspace get-permission-levels` command. * Added `databricks workspace get-permissions` command. * Added `databricks workspace set-permissions` command. * Added `databricks workspace update-permissions` command. OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)

## Changes This pull request extends the templating support in preparation of a new, default template (WIP, databricks#686): * builtin templates that can be initialized using e.g. `databricks bundle init default-python` * builtin templates are embedded into the executable using go's `embed` functionality, making sure they're co-versioned with the CLI * new helpers to get the workspace name, current user name, etc. help craft a complete template * (not enabled yet) when the user types `databricks bundle init` they can interactively select the `default-python` template And makes two tangentially related changes: * IsServicePrincipal now uses the "users" API rather than the "principals" API, since the latter is too slow for our purposes. * mode: prod no longer requires the 'target.prod.git' setting. It's hard to set that from a template. (Pieter is planning an overhaul of warnings support; this would be one of the first warnings we show.) The actual `default-python` template is maintained in a separate PR: databricks#686 ## Tests Unit tests, manual testing Signed-off-by: Arpit Jasapara <[email protected]>

## Changes This adds a built-in "default-python" template to the CLI. This is based on the new default-template support of databricks#685. The goal here is to offer an experience where customers can simply type `databricks bundle init` to get a default template: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project ✨ Successfully initialized template ``` The present template: - [x] Works well with VS Code - [x] Works well with the workspace - [x] Works well with DB Connect - [x] Uses minimal stubs rather than boiler-plate-heavy examples I'll have a followup with tests + DLT support. --------- Co-authored-by: Andrew Nester <[email protected]> Co-authored-by: PaulCornellDB <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]> Signed-off-by: Arpit Jasapara <[email protected]>

~(this should be changed to target `main`)~ This reveals the template from databricks#686 in CLI prompts for once databricks#686 and databricks#708 are merged. --------- Co-authored-by: Andrew Nester <[email protected]> Co-authored-by: PaulCornellDB <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]> Signed-off-by: Arpit Jasapara <[email protected]>

…bricks#700) This follows up on databricks#686. This PR makes our stubs optional + it adds DLT stubs: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project Include a stub (sample) notebook in 'my_project/src' [yes]: yes Include a stub (sample) DLT pipeline in 'my_project/src' [yes]: yes Include a stub (sample) Python package 'my_project/src' [yes]: yes ✨ Successfully initialized template ``` Manual testing, matrix tests. --------- Co-authored-by: Andrew Nester <[email protected]> Co-authored-by: PaulCornellDB <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]>

This release includes permission related commands for a subset of workspace services where they apply. These complement the `permissions` command and do not require specification of the object type to work with, as that is implied by the command they are nested under. CLI: * Group permission related commands ([databricks#730](databricks#730)). Bundles: * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([databricks#722](databricks#722)). * Make resource and artifact paths in bundle config relative to config folder ([databricks#708](databricks#708)). * Add support for ordering of input prompts ([databricks#662](databricks#662)). * Fix IsServicePrincipal() only working for workspace admins ([databricks#732](databricks#732)). * databricks bundle init template v1 ([databricks#686](databricks#686)). * databricks bundle init template v2: optional stubs, DLT support ([databricks#700](databricks#700)). * Show 'databricks bundle init' template in CLI prompt ([databricks#725](databricks#725)). * Include $PATH in set of environment variables to pass along. ([databricks#736](databricks#736)). Internal: * Update Go SDK to v0.19.0 ([databricks#729](databricks#729)). * Replace API call to test configuration with dummy authenticate call ([databricks#728](databricks#728)). API Changes: * Changed `databricks account storage-credentials create` command to return . * Changed `databricks account storage-credentials get` command to return . * Changed `databricks account storage-credentials list` command to return . * Changed `databricks account storage-credentials update` command to return . * Changed `databricks connections create` command with new required argument order. * Changed `databricks connections update` command with new required argument order. * Changed `databricks volumes create` command with new required argument order. * Added `databricks artifact-allowlists` command group. * Added `databricks model-versions` command group. * Added `databricks registered-models` command group. * Added `databricks cluster-policies get-permission-levels` command. * Added `databricks cluster-policies get-permissions` command. * Added `databricks cluster-policies set-permissions` command. * Added `databricks cluster-policies update-permissions` command. * Added `databricks clusters get-permission-levels` command. * Added `databricks clusters get-permissions` command. * Added `databricks clusters set-permissions` command. * Added `databricks clusters update-permissions` command. * Added `databricks instance-pools get-permission-levels` command. * Added `databricks instance-pools get-permissions` command. * Added `databricks instance-pools set-permissions` command. * Added `databricks instance-pools update-permissions` command. * Added `databricks files` command group. * Changed `databricks permissions set` command to start returning . * Changed `databricks permissions update` command to start returning . * Added `databricks users get-permission-levels` command. * Added `databricks users get-permissions` command. * Added `databricks users set-permissions` command. * Added `databricks users update-permissions` command. * Added `databricks jobs get-permission-levels` command. * Added `databricks jobs get-permissions` command. * Added `databricks jobs set-permissions` command. * Added `databricks jobs update-permissions` command. * Changed `databricks experiments get-by-name` command to return . * Changed `databricks experiments get-experiment` command to return . * Added `databricks experiments delete-runs` command. * Added `databricks experiments get-permission-levels` command. * Added `databricks experiments get-permissions` command. * Added `databricks experiments restore-runs` command. * Added `databricks experiments set-permissions` command. * Added `databricks experiments update-permissions` command. * Added `databricks model-registry get-permission-levels` command. * Added `databricks model-registry get-permissions` command. * Added `databricks model-registry set-permissions` command. * Added `databricks model-registry update-permissions` command. * Added `databricks pipelines get-permission-levels` command. * Added `databricks pipelines get-permissions` command. * Added `databricks pipelines set-permissions` command. * Added `databricks pipelines update-permissions` command. * Added `databricks serving-endpoints get-permission-levels` command. * Added `databricks serving-endpoints get-permissions` command. * Added `databricks serving-endpoints set-permissions` command. * Added `databricks serving-endpoints update-permissions` command. * Added `databricks token-management get-permission-levels` command. * Added `databricks token-management get-permissions` command. * Added `databricks token-management set-permissions` command. * Added `databricks token-management update-permissions` command. * Changed `databricks dashboards create` command with new required argument order. * Added `databricks warehouses get-permission-levels` command. * Added `databricks warehouses get-permissions` command. * Added `databricks warehouses set-permissions` command. * Added `databricks warehouses update-permissions` command. * Added `databricks dashboard-widgets` command group. * Added `databricks query-visualizations` command group. * Added `databricks repos get-permission-levels` command. * Added `databricks repos get-permissions` command. * Added `databricks repos set-permissions` command. * Added `databricks repos update-permissions` command. * Added `databricks secrets get-secret` command. * Added `databricks workspace get-permission-levels` command. * Added `databricks workspace get-permissions` command. * Added `databricks workspace set-permissions` command. * Added `databricks workspace update-permissions` command. OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04) Signed-off-by: Arpit Jasapara <[email protected]>

lennartkats-db added 6 commits August 21, 2023 09:50

Add foundation for builtin templates

59c1c73

Avoid nil context

2b67262

Add missing file

737c612

Fix test in Windows

2955e84

Add initial default-python template

ea5666a

Avoid ipynb for now

a1c6b3b

lennartkats-db mentioned this pull request Aug 21, 2023

Add a foundation for built-in templates #685

Merged

lennartkats-db added 6 commits August 21, 2023 13:45

Fix typos

6ae0279

Remove the hard requirement of a target.prod.git setting

568afe0

Merge remote-tracking branch 'databricks/main' into template-foundation

37de1e7

Don't guess the main branch

a3c7e54

Remove old test

a7c6928

Fix typo

9435670

lennartkats-db commented Aug 21, 2023

View reviewed changes

libs/template/templates/default-python/template/{{.project_name}}/res/job.yml.tmpl Outdated Show resolved Hide resolved

lennartkats-db commented Aug 21, 2023

View reviewed changes

...template/templates/default-python/template/{{.project_name}}/src/{{.project_name}}/common.py Outdated Show resolved Hide resolved

lennartkats-db commented Aug 22, 2023

View reviewed changes

...efault-python/template/{{.project_name}}/src/{{.project_name}}/tasks/sample_etl_task.py.tmpl Outdated Show resolved Hide resolved

fjakobs reviewed Aug 23, 2023

View reviewed changes

andrewnester reviewed Aug 23, 2023

View reviewed changes

libs/template/templates/default-python/template/{{.project_name}}/databricks.yml.tmpl Outdated Show resolved Hide resolved

lennartkats-db added 5 commits August 23, 2023 22:04

Cleanup

cfa8a83

Add caching to is_service_principal

4340356

Fix tests

9d41ccc

Update template

581aae6

Merge branch 'template-foundation' into template

bc57e53

databricks deleted a comment from fjakobs Aug 24, 2023

lennartkats-db force-pushed the template branch from 1255004 to 870825e Compare August 25, 2023 08:54

Update template

79370f8

lennartkats-db force-pushed the template branch from 870825e to 79370f8 Compare August 25, 2023 08:56

lennartkats-db changed the title ~~[WIP] Builtin template~~ Builtin template Aug 25, 2023

lennartkats-db changed the base branch from template-foundation to main August 25, 2023 09:03

lennartkats-db changed the base branch from main to template-foundation August 25, 2023 09:04

lennartkats-db and others added 3 commits September 1, 2023 11:32

Update libs/template/templates/default-python/template/{{.project_nam…

a5068bb

…e}}/.vscode/__builtins__.pyi Co-authored-by: PaulCornellDB <[email protected]>

Update libs/template/templates/default-python/template/{{.project_nam…

aea403f

…e}}/databricks.yml.tmpl Co-authored-by: Pieter Noordhuis <[email protected]>

Don't show this template just yet

391ab24

lennartkats-db force-pushed the template branch from 32eaaed to 391ab24 Compare September 1, 2023 09:33

lennartkats-db mentioned this pull request Sep 1, 2023

Show 'databricks bundle init' template in CLI prompt #725

Merged

Merge remote-tracking branch 'databricks/main' into template

9063af7

lennartkats-db force-pushed the template branch from 73d6aea to 9063af7 Compare September 1, 2023 09:46

lennartkats-db added 3 commits September 4, 2023 11:21

Merge remote-tracking branch 'databricks/main' into template

2833db9

Remove artifacts section; no longer required

2b33682

Restore scratch directory

f650a3f

lennartkats-db requested a review from pietern September 4, 2023 14:07

lennartkats-db mentioned this pull request Sep 4, 2023

Add default_python example databricks/bundle-examples#3

Merged

Backport Databricks connect suggestion

db52765

lennartkats-db changed the title ~~Add a built-in template for 'databricks bundle init'~~ databricks bundle init template v1 Sep 5, 2023

pietern approved these changes Sep 5, 2023

View reviewed changes

cmd/bundle/init.go Show resolved Hide resolved

lennartkats-db added this pull request to the merge queue Sep 5, 2023

Merged via the queue into main with commit 8c2cc07 Sep 5, 2023

lennartkats-db deleted the template branch September 5, 2023 12:05

pietern mentioned this pull request Sep 6, 2023

Release v0.204.0 #738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

databricks bundle init template v1 #686

databricks bundle init template v1 #686

lennartkats-db commented Aug 21, 2023 •

edited by saadansari-db

Loading

fjakobs left a comment

databricks bundle init template v1 #686

databricks bundle init template v1 #686

Conversation

lennartkats-db commented Aug 21, 2023 • edited by saadansari-db Loading

Changes

fjakobs left a comment

Choose a reason for hiding this comment

lennartkats-db commented Aug 21, 2023 •

edited by saadansari-db

Loading