Added support for top-level permissions #928

andrewnester · 2023-10-27T13:24:21Z

Changes

Now it's possible to define top level permissions section in bundle configuration and permissions defined there will be applied to all resources defined in the bundle.

Supported top-level permission levels: CAN_MANAGE, CAN_VIEW, CAN_RUN.

Permissions are applied to: Jobs, DLT Pipelines, ML Models, ML Experiments and Model Service Endpoints

bundle:
  name: permissions

workspace:
  host: ***

permissions:
  - level: CAN_VIEW
    group_name: test-group
  - level: CAN_MANAGE
    user_name: [email protected]
  - level: CAN_RUN
    service_principal_name: 123456-abcdef

Tests

Added corresponding unit tests + ran bundle validate and bundle deploy manually

bundle/permissions/mutator_test.go

bundle/config/target.go

bundle/permissions/mutator.go

pietern · 2023-10-27T13:52:26Z

bundle/permissions/mutator.go

+func (m *topLevelPermissions) validate(b *bundle.Bundle) error {
+	for _, p := range b.Config.Permissions {
+		if !slices.Contains(allowedLevels, p.Level) {
+			return fmt.Errorf("invalid permission level: %s, allowed values: [%s]", p.Level, strings.Join(allowedLevels, ", "))


When the location annotations work we can figure out exactly where this was defined and emit a diagnostic.

pietern · 2023-10-27T13:54:57Z

bundle/permissions/utils.go

+	// If there is permission overlap, show a warning to the user
+	if isOverlap {
+		for _, d := range diagnostics {
+			cmdio.LogString(ctx, d.Summary)


Can you leave this one out for now? I'd like for us to collect all diagnostics at the top level and visualize jointly.

bundle/permissions/mutator.go

shreyas-goenka · 2023-10-30T19:18:42Z

bundle/permissions/utils.go

+		}
+
+		if notifyForPermissionOverlap(ctx, p, resourcePermissions, resourceName) {
+			continue


We should consider overriding permissions here if bundle permissions are larger in scope than the resource permissions.

Consider the case:

permissions: user_name: abc level: CAN_MANAGE jobs: foo: permissions: { user_name: "abc" level: CAN_VIEW },

In this case the correct behavior IMO might be for abc to have CAN_MANAGE access to the job. I guess the intention here is to provide users the ability to downgrade permission scope per resource?

@shreyas-goenka not really, as per discussuions with Pieter, Lennart and Fabian we agreed that at this momemnt if such overlap occurs we just want to notify customers about to avoid any unambiguity in permissions and potential errors in configurations.

alexott · 2023-11-02T14:58:23Z

I would recommend the ability to have permissions per environment - it's very typical that in prod environments engineers have only CAN_VIEW permission, whilst in dev environment - they may have CAN_MANAGE

pietern · 2023-11-03T16:58:14Z

@alexott That is possible with this PR. The permissions at the top level and the top level of the selected target are combined by appending the target level ones to the root ones.

bundle/permissions/mutator.go

bundle/phases/initialize.go

bundle/permissions/utils.go

bundle/tests/bundle_permissions/databricks.yml

This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects. CLI: * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)). * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)). * Make `databricks configure` save only explicit fields ([#973](#973)). * Add `databricks labs` command group ([#914](#914)). * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)). * Add `--configure-cluster` flag to configure command ([#1005](#1005)). * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)). Bundles: * Do not allow empty descriptions for bundle template inputs ([#967](#967)). * Added support for top-level permissions ([#928](#928)). * Allow jobs to be manually unpaused in development mode ([#885](#885)). * Fix template initialization from current working directory ([#976](#976)). * Add `--tag` and `--branch` options to bundle init command ([#975](#975)). * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)). * Enable `spark_jar_task` with local JAR libraries ([#993](#993)). * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)). * Improve error message when path is not a bundle template ([#985](#985)). * Correctly overwrite local state if remote state is newer ([#1008](#1008)). * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)). * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)). * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)). Internal: * Update CLI version in the VS Code extension during release ([#1014](#1014)). API Changes: * Changed `databricks functions create` command . New request type is . * Changed `databricks metastores create` command with new required argument order. * Removed `databricks metastores enable-optimization` command. * Removed `databricks account o-auth-enrollment` command group. * Removed `databricks apps delete` command. * Removed `databricks apps get` command. * Added `databricks apps delete-app` command. * Added `databricks apps get-app` command. * Added `databricks apps get-app-deployment-status` command. * Added `databricks apps get-apps` command. * Added `databricks apps get-events` command. * Added `databricks account network-connectivity` command group. OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23) Dependency updates: * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)). * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)). * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)). * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)). * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).

This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects. CLI: * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)). * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)). * Make `databricks configure` save only explicit fields ([#973](#973)). * Add `databricks labs` command group ([#914](#914)). * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)). * Add `--configure-cluster` flag to configure command ([#1005](#1005)). * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)). Bundles: * Do not allow empty descriptions for bundle template inputs ([#967](#967)). * Added support for top-level permissions ([#928](#928)). * Allow jobs to be manually unpaused in development mode ([#885](#885)). * Fix template initialization from current working directory ([#976](#976)). * Add `--tag` and `--branch` options to bundle init command ([#975](#975)). * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)). * Enable `spark_jar_task` with local JAR libraries ([#993](#993)). * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)). * Improve error message when path is not a bundle template ([#985](#985)). * Correctly overwrite local state if remote state is newer ([#1008](#1008)). * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)). * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)). * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)). Internal: * Update CLI version in the VS Code extension during release ([#1014](#1014)). API Changes: * Changed `databricks functions create` command. * Changed `databricks metastores create` command with new required argument order. * Removed `databricks metastores enable-optimization` command. * Removed `databricks account o-auth-enrollment` command group. * Removed `databricks apps delete` command. * Removed `databricks apps get` command. * Added `databricks apps delete-app` command. * Added `databricks apps get-app` command. * Added `databricks apps get-app-deployment-status` command. * Added `databricks apps get-apps` command. * Added `databricks apps get-events` command. * Added `databricks account network-connectivity` command group. OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23) Dependency updates: * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)). * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)). * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)). * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)). * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).

Added support for top-level permissions

e8dce9d

andrewnester requested review from pietern, lennartkats-db and shreyas-goenka October 27, 2023 13:24

pietern reviewed Oct 27, 2023

View reviewed changes

add write access for can_manage

1c3e18e

andrewnester requested a review from pietern October 30, 2023 12:38

remove file

aee9c5c

shreyas-goenka reviewed Oct 30, 2023

View reviewed changes

andrewnester requested a review from shreyas-goenka November 7, 2023 10:39

pietern reviewed Nov 8, 2023

View reviewed changes

bundle/permissions/mutator.go Outdated Show resolved Hide resolved

bundle/permissions/mutator.go Outdated Show resolved Hide resolved

bundle/phases/initialize.go Show resolved Hide resolved

add all level of permissions for workspace root

2ca9ce7

andrewnester requested a review from pietern November 10, 2023 10:53

pietern approved these changes Nov 10, 2023

View reviewed changes

bundle/permissions/utils.go Outdated Show resolved Hide resolved

bundle/tests/bundle_permissions/databricks.yml Show resolved Hide resolved

more tests

1897e4b

andrewnester enabled auto-merge November 13, 2023 11:08

andrewnester added this pull request to the merge queue Nov 13, 2023

Merged via the queue into main with commit f3db42e Nov 13, 2023

andrewnester deleted the top-level-permissions branch November 13, 2023 11:34

andrewnester mentioned this pull request Nov 17, 2023

[Question] permission switch between env #698

Closed

pietern mentioned this pull request Nov 29, 2023

Release v0.210.0 #1023

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for top-level permissions #928

Added support for top-level permissions #928

andrewnester commented Oct 27, 2023

pietern Oct 27, 2023

pietern Oct 27, 2023

shreyas-goenka Oct 30, 2023

andrewnester Nov 7, 2023

alexott commented Nov 2, 2023

pietern commented Nov 3, 2023

Added support for top-level permissions #928

Added support for top-level permissions #928

Conversation

andrewnester commented Oct 27, 2023

Changes

Tests

pietern Oct 27, 2023

Choose a reason for hiding this comment

pietern Oct 27, 2023

Choose a reason for hiding this comment

shreyas-goenka Oct 30, 2023

Choose a reason for hiding this comment

andrewnester Nov 7, 2023

Choose a reason for hiding this comment

alexott commented Nov 2, 2023

pietern commented Nov 3, 2023