Feature/new secure databricks (#76)

* initial commit of databricks changes * add nat * use nat and ip reosurces * paramatise nat timeout * add vars * add nat to both subnets * add pe subnet * add ability to create subnets * temp comment out count on outputs * update prefix vars to lists * update * add index * add index to nsg * add index to pip and nat * fix typo * add count to pe subnet * add index * fix typo * add index to outputs * update * update * make nat gateway single region * make pip single zone * add depends on * add service endpoints * add permissions for network and dns * add index * add permissions to depends on * add databricks user * update * add peering * remove reference to data lookup * add index to vnet * add permissions to db * comment out perms * add rg * add depends on * use name instead of id * remove depends on * update id * remove db peer * update var for public access * add route table * fix typo * add prefix/ * remove rt * add config for lb * update lb name to local * add index * update * remove param * temp comment out all params * comment out rules temp * temp commetn out pe * temp commetn out pe * temp commetn out pe * add configurable managed vnet * update priority * update vnet id * update index * temp comment out nsg * remove reference to nsg ass * temp fix subnet names * add nsg ass to custom param * update nsg rule * temp comment out count * update index * update" * temp comment out count on subnets * revert stuff * comment out pe * comment out data * comment out data * comment out rule * comment out service endpoints * create pr subnet and pe * update subnet range * update prefix to list * configure pip * update networking * updates * update name * variable for pe prefix * add datalookup for pe subnet * update pe subnet name to var * split var for creating pe subnet * update condition * update condition * update names * update condition * update name * update readme and tidy up * add nsg rule to allow databricks into vnet * add nsg rules * add depends on * add depends on * add auth pe * comment out data * update depends on * update * add depends on to db resources * make string * update * remove whitespace * update dbfs to string * temp remove dbfs explore * add dbfs explorer * temp remove dbfs * update * update dns zone name * run fmt * add permissions * fix name * temp comment out * remove perm * add db back in * remove depends on * comment out db * update pe * comment out db * rename dns zone * add users and conf * add dns update * update default value * update var name * add condition * add condition * update rg name * add condition to cname * tidy up comments * tidy up comments * updated for new * feat: added dns for adls * updated * updated * removed databrick provider code * updated for pe --------- Co-authored-by: Rhys Bushnell <[email protected]> Co-authored-by: Satenderrathee <[email protected]>
Ensono · Jul 20, 2023 · 9000259 · 9000259
1 parent 22ab206
commit 9000259
Show file tree

Hide file tree

Showing 9 changed files with 502 additions and 60 deletions.
diff --git a/azurerm/modules/azurerm-adb/README.md b/azurerm/modules/azurerm-adb/README.md
diff --git a/azurerm/modules/azurerm-adb/constraints.tf b/azurerm/modules/azurerm-adb/constraints.tf
@@ -5,9 +5,5 @@ terraform {
       source  = "hashicorp/azurerm"
       version = "~> 3.0"
     }
-    #TODO: note this is just added right now without any use, can be used In future for databricks provider
-    databricks = {
-      source = "databricks/databricks"
-    }
   }
 }
diff --git a/azurerm/modules/azurerm-adb/data.tf b/azurerm/modules/azurerm-adb/data.tf
@@ -0,0 +1,45 @@
+data "azurerm_client_config" "current" {}
+
+# data "databricks_current_user" "db" {
+#   count      = var.enable_private_network ? 1 : 0
+#   depends_on = [azurerm_databricks_workspace.example]
+# }
+
+data "azurerm_resource_group" "vnet_rg" {
+  count = var.enable_private_network ? 1 : 0
+  name  = var.vnet_resource_group
+}
+
+data "azurerm_virtual_network" "vnet" {
+  count               = var.enable_private_network ? 1 : 0
+  name                = var.vnet_name
+  resource_group_name = var.vnet_resource_group
+}
+
+data "azurerm_subnet" "public_subnet" {
+  count                = var.enable_private_network == true && var.create_subnets == false && var.managed_vnet == false ? 1 : 0
+  name                 = var.public_subnet_name
+  virtual_network_name = var.vnet_name
+  resource_group_name  = var.vnet_resource_group
+}
+
+data "azurerm_subnet" "private_subnet" {
+  count                = var.enable_private_network == true && var.create_subnets == false && var.managed_vnet == false ? 1 : 0
+  name                 = var.private_subnet_name
+  virtual_network_name = var.vnet_name
+  resource_group_name  = var.vnet_resource_group
+}
+
+data "azurerm_subnet" "pe_subnet" {
+  count = var.enable_private_network == true && var.create_pe_subnet == false && var.managed_vnet == false ? 1 : 0
+
+  name                 = var.pe_subnet_name
+  resource_group_name  = var.vnet_resource_group
+  virtual_network_name = var.vnet_name
+}
+
+data "azurerm_private_dns_zone" "adb_pvt_dns" {
+  count               = var.enable_private_network ? 1 : 0
+  name                = var.private_dns_zone_name
+  resource_group_name = var.dns_resource_group_name
+}
diff --git a/azurerm/modules/azurerm-adb/load-balancer.tf b/azurerm/modules/azurerm-adb/load-balancer.tf
@@ -0,0 +1,39 @@
+
+resource "azurerm_lb" "lb" {
+  count = var.enable_private_network && var.create_lb && var.managed_vnet == false ? 1 : 0
+
+  name                = local.lb_name
+  location            = var.resource_group_location
+  resource_group_name = var.resource_group_name
+
+  sku = "Standard"
+
+  frontend_ip_configuration {
+    name                 = "Databricks-PIP"
+    public_ip_address_id = azurerm_public_ip.pip[0].id
+  }
+}
+
+resource "azurerm_lb_outbound_rule" "lb_rule" {
+  count = var.enable_private_network && var.create_lb && var.managed_vnet == false ? 1 : 0
+
+  name = "Databricks-LB-Outbound-Rules"
+
+  loadbalancer_id          = azurerm_lb.lb[0].id
+  protocol                 = "All"
+  enable_tcp_reset         = true
+  allocated_outbound_ports = 1024
+  idle_timeout_in_minutes  = 4
+
+  backend_address_pool_id = azurerm_lb_backend_address_pool.lb_be_pool[0].id
+
+  frontend_ip_configuration {
+    name = azurerm_lb.lb[0].frontend_ip_configuration[0].name
+  }
+}
+
+resource "azurerm_lb_backend_address_pool" "lb_be_pool" {
+  count           = var.enable_private_network && var.create_lb && var.managed_vnet == false ? 1 : 0
+  loadbalancer_id = azurerm_lb.lb[0].id
+  name            = "Databricks-BE"
+}
diff --git a/azurerm/modules/azurerm-adb/local.tf b/azurerm/modules/azurerm-adb/local.tf
@@ -0,0 +1,5 @@
+locals {
+  public_ip_name   = "${var.resource_namer}-pip"
+  nat_gateway_name = "${var.resource_namer}-nat-gw"
+  lb_name          = "${var.resource_namer}-lb"
+}
diff --git a/azurerm/modules/azurerm-adb/main.tf b/azurerm/modules/azurerm-adb/main.tf
@@ -1,9 +1,28 @@
 
 resource "azurerm_databricks_workspace" "example" {
-  name                = var.resource_namer
-  location            = var.resource_group_location
-  resource_group_name = var.resource_group_name
-  sku                 = var.databricks_sku
+  name                                  = var.resource_namer
+  location                              = var.resource_group_location
+  resource_group_name                   = var.resource_group_name
+  sku                                   = var.databricks_sku
+  public_network_access_enabled         = var.public_network_access_enabled
+  network_security_group_rules_required = var.managed_vnet ? null : var.network_security_group_rules_required
+  managed_resource_group_name           = "databricks-rg-${var.resource_group_name}"
+  load_balancer_backend_address_pool_id = var.create_lb ? azurerm_lb_backend_address_pool.lb_be_pool[0].id : null
+
+  dynamic "custom_parameters" {
+    for_each = var.enable_private_network == false ? toset([]) : toset([1])
+    content {
+      no_public_ip                                         = true
+      public_subnet_name                                   = var.managed_vnet ? null : (var.create_subnets ? azurerm_subnet.public_subnet[0].name : data.azurerm_subnet.public_subnet[0].name)
+      private_subnet_name                                  = var.managed_vnet ? null : (var.create_subnets ? azurerm_subnet.private_subnet[0].name : data.azurerm_subnet.private_subnet[0].name)
+      virtual_network_id                                   = var.managed_vnet ? null : data.azurerm_virtual_network.vnet[0].id
+      vnet_address_prefix                                  = var.managed_vnet ? null : (var.vnet_address_prefix == "" ? null : var.vnet_address_prefix)
+      public_subnet_network_security_group_association_id  = var.managed_vnet ? null : azurerm_subnet_network_security_group_association.public[0].id
+      private_subnet_network_security_group_association_id = var.managed_vnet ? null : azurerm_subnet_network_security_group_association.private[0].id
+      nat_gateway_name                                     = var.managed_vnet ? null : (var.create_nat ? azurerm_nat_gateway.nat[0].name : null)
+      public_ip_name                                       = var.managed_vnet ? null : (var.create_nat ? azurerm_public_ip.pip[0].name : null)
+    }
+  }
 
 
   tags = var.resource_tags
@@ -12,6 +31,7 @@ resource "azurerm_databricks_workspace" "example" {
       tags,
     ]
   }
+  depends_on = [azurerm_subnet.public_subnet, azurerm_subnet.private_subnet, data.azurerm_subnet.public_subnet, data.azurerm_subnet.private_subnet]
 }
 
 
@@ -55,32 +75,3 @@ resource "azurerm_monitor_diagnostic_setting" "databricks_log_analytics" {
   }
 }
 
-resource "databricks_workspace_conf" "this" {
-  count = var.enable_enableDbfsFileBrowser ? 1 : 0
-  custom_config = {
-
-    "enableDbfsFileBrowser" : true
-
-  }
-}
-
-
-resource "databricks_user" "rbac_users" {
-  for_each     = var.add_rbac_users ? var.rbac_databricks_users : {}
-  display_name = each.value.display_name
-  user_name    = each.value.user_name
-  active       = each.value.active
-}
-
-resource "databricks_group" "project_users" {
-  count                 = var.add_rbac_users ? 1 : 0
-  display_name          = var.databricks_group_display_name
-  workspace_access      = var.enable_workspace_access
-  databricks_sql_access = var.enable_sql_access
-}
-
-resource "databricks_group_member" "project_users" {
-  for_each  = var.add_rbac_users ? databricks_user.rbac_users : {}
-  group_id  = databricks_group.project_users[0].id
-  member_id = each.value.id
-}
diff --git a/azurerm/modules/azurerm-adb/nat.tf b/azurerm/modules/azurerm-adb/nat.tf
@@ -0,0 +1,31 @@
+############################################
+# NAT GATEWAY
+############################################
+
+resource "azurerm_nat_gateway" "nat" {
+  count                   = var.enable_private_network && var.create_nat && var.managed_vnet == false ? 1 : 0
+  name                    = local.nat_gateway_name
+  location                = var.resource_group_location
+  resource_group_name     = var.resource_group_name
+  sku_name                = "Standard"
+  idle_timeout_in_minutes = var.nat_idle_timeout
+  zones                   = ["1"]
+}
+
+resource "azurerm_nat_gateway_public_ip_association" "nat_ip" {
+  count                = var.enable_private_network && var.create_nat && var.managed_vnet == false ? 1 : 0
+  nat_gateway_id       = azurerm_nat_gateway.nat[0].id
+  public_ip_address_id = azurerm_public_ip.pip[0].id
+}
+
+resource "azurerm_subnet_nat_gateway_association" "public_subnet_nat" {
+  count          = var.enable_private_network && var.create_nat && var.managed_vnet == false ? 1 : 0
+  subnet_id      = var.create_subnets ? azurerm_subnet.public_subnet[0].id : data.azurerm_subnet.public_subnet[0].id
+  nat_gateway_id = azurerm_nat_gateway.nat[0].id
+}
+
+resource "azurerm_subnet_nat_gateway_association" "private_subnet_nat" {
+  count          = var.enable_private_network && var.create_nat && var.managed_vnet == false ? 1 : 0
+  subnet_id      = var.create_subnets ? azurerm_subnet.private_subnet[0].id : data.azurerm_subnet.private_subnet[0].id
+  nat_gateway_id = azurerm_nat_gateway.nat[0].id
+}
Name	Type
azurerm_databricks_workspace.example	resource
azurerm_monitor_diagnostic_setting.databricks_log_analytics	resource
databricks_group.project_users	resource
databricks_group_member.project_users	resource
databricks_user.rbac_users	resource
databricks_workspace_conf.this	resource
azurerm_monitor_diagnostic_categories.adb_log_analytics_categories	data source
Name	Type
azurerm_databricks_workspace.example	resource
azurerm_lb.lb	resource
azurerm_lb_backend_address_pool.lb_be_pool	resource
azurerm_lb_outbound_rule.lb_rule	resource
azurerm_monitor_diagnostic_setting.databricks_log_analytics	resource
azurerm_nat_gateway.nat	resource
azurerm_nat_gateway_public_ip_association.nat_ip	resource
azurerm_network_security_group.nsg	resource
azurerm_private_dns_cname_record.cname	resource
azurerm_private_dns_zone.dns	resource
azurerm_private_dns_zone_virtual_network_link.db_dns_vnet_link	resource
azurerm_private_endpoint.databricks	resource
azurerm_public_ip.pip	resource
azurerm_subnet.pe_subnet	resource
azurerm_subnet.private_subnet	resource
azurerm_subnet.public_subnet	resource
azurerm_subnet_nat_gateway_association.private_subnet_nat	resource
azurerm_subnet_nat_gateway_association.public_subnet_nat	resource
azurerm_subnet_network_security_group_association.private	resource
azurerm_subnet_network_security_group_association.public	resource
databricks_group.project_users	resource
databricks_group_member.project_users	resource
databricks_user.rbac_users	resource
databricks_workspace_conf.this	resource
azurerm_client_config.current	data source
azurerm_monitor_diagnostic_categories.adb_log_analytics_categories	data source
azurerm_resource_group.vnet_rg	data source
azurerm_subnet.pe_subnet	data source
azurerm_subnet.private_subnet	data source
azurerm_subnet.public_subnet	data source
azurerm_virtual_network.vnet	data source
databricks_current_user.db	data source
Name	Description	Type	Default	Required
add_rbac_users	If set to true, the module will create databricks users and group named 'project_users' with the specified users as members, and grant workspace and SQL access to this group. Default is false.	`bool`	`true`	no
data_platform_log_analytics_workspace_id	The Log Analytics Workspace used for the whole Data Platform.	`string`	`null`	no
databricks_group_display_name	If 'add_rbac_users' set to true then specifies databricks group display name	`string`	`"project_users"`	no
databricks_sku	The SKU to use for the databricks instance	`string`	`"premium"`	no
databricksws_diagnostic_setting_name	The Databricks workspace diagnostic setting name.	`string`	`"Databricks to Log Analytics"`	no
enable_databricksws_diagnostic	Whether to enable diagnostic settings for the Azure Databricks workspace	`bool`	`false`	no
enable_enableDbfsFileBrowser	Whether to enable Dbfs File browser for the Azure Databricks workspace	`bool`	`false`	no
enable_sql_access	Whether to enable sql access for the databricks group	`bool`	`true`	no
enable_workspace_access	Whether to enable workspace access for the databricks group	`bool`	`true`	no
rbac_databricks_users	If 'add_rbac_users' set to true then specifies RBAC Databricks users	map(object({ display_name = string user_name = string active = bool }))	`null`	no
resource_group_location	Location of Resource group	`string`	`"uksouth"`	no
resource_group_name	Name of resource group	`string`	n/a	yes
resource_namer	User defined naming convention applied to all resources created as part of this module	`string`	n/a	yes
resource_tags	Map of tags to be applied to all resources created as part of this module	`map(string)`	`{}`	no
Name	Description
adb_databricks_id	n/a
databricks_hosturl	Azure Databricks HostUrl