azurerm_key_vault not setting purge protection and error when attempting to purge keys.

[jwshive]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

I am using azurerm provider 2.41.0 and terraform binary 0.14.3

Affected Resource(s)

• azurerm_key_vault

Terraform Configuration Files

resource "azurerm_key_vault" "key_vault" {
  location = var.resource-location
  name = "${var.agency-name}-${var.department-name}-${var.application-name}-${var.environment}-kv"
  resource_group_name = "zus1-${var.agency-name}-${var.application-name}-${var.environment}-v1-rg"
  sku_name = var.key-vault-sku-name
  tenant_id = data.azurerm_client_config.current_client_config.tenant_id
  soft_delete_enabled = false
  purge_protection_enabled = false
  tags = merge(local.common_tags, map("type", "key-vault"))
}

Debug Output

Error: purging Secret "sql-1-password" (Key Vault "https://my-key-vault.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=400 --
Original Error: autorest/azure: Service returned an error. Status=400 Code="NotSupported" Message="Operation "purge" is not enabled for this vault."

Panic Output

Expected Behaviour

I expect the key vault to be created with enablePurgeProtection: false and a terraform destroy to not error out when removing a key vault.

Actual Behaviour

The key vault is created, but purge protection is null and not false.
...
"enablePurgeProtection": null,
"enableRbacAuthorization": false,
"enableSoftDelete": false,
"enabledForDeployment": false,
"enabledForDiskEncryption": false,
"enabledForTemplateDeployment": false,
....

A terraform destroy results in the error in debug output.

Steps to Reproduce

1. terraform destroy

Important Factoids

This was working last week as of Thursday or Friday.

References

This may be related to

• key vault soft-delete causing failure when updating secrets or certificates  #5659

[esciara]

Same here. When creating a key vault using the defaults, creating a secret, and then destroying the secret, the above mentioned error occurs.

with azurerm version 2.41.0.

[ruandersMSFT]

I'll have to run a test to know if it is a permission problem or is in relation to the Key Vault itself, but for consideration be aware a breaking change was recently enforced in which Soft Delete is now required to be enabled.

From the FAQ, have you granted Purge permissions to the necessary identity?

#9988 opened, as soft_delete_enabled should no longer be allowed to set to false

[jwshive]

I gave my service principal the purge permission as documented here and I still get the error posted in the opening notes.

My key vault sp permissions creation code is now

resource "azurerm_key_vault_access_policy" "sp-key_vault_access_policy" {
  key_vault_id = azurerm_key_vault.key_vault.id
  object_id = data.azurerm_client_config.current_client_config.object_id
  tenant_id = data.azurerm_client_config.current_client_config.tenant_id
  key_permissions = ["get", "list", "create", "encrypt", "delete", "purge"]
  secret_permissions = ["get", "list", "set", "delete", "purge"]
  depends_on = [azurerm_key_vault.key_vault]
}

Also, soft delete is enabled.

    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": false,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": false,
    "enabledForTemplateDeployment": false,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",

Edit: When I posted the above I had forgot to enable soft delete. I enabled that in the key vault and when running terraform destroy I now see...

Error: purging Secret "sql-1-password" (Key Vault "https://my-key-vault.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=409 -- 
Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" Message="Secret is currently being deleted." InnerError={"code":"ObjectIsBeingDeleted"}

Which seems like it would be obvious during a destroy that the secret is being deleted.

[Abhishekqwerty]

Using hashicorp/azurerm v2.41.0
I am also facing the same issue when destroying secrets for the first time.
Getting the below error:
keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NotSupported" Message="Operation "purge" is not enabled for this vault."
This deletes secrets but does not delete KeyVault.

When running terraform destroy again after the first attempt, it destroys the key vault, but gives the following error:
Message="The user, group or application does not have secrets delete permission on key vault."
This deletes the resources but gives the error message.

[jackofallops]

Hi all - just looking at a possibly related issue and thought I should chip in with a couple things:

purge_protection_enabled - should only be set to true, the API doesn't accept false, so we don't send it (so will always be null / nil if not true). If Purge Protection is not desired, please omit this property completely (note: Once set to true, it cannot be removed or set to false)

soft_delete_enabled can now only be true as of December 31st 2020, and defaults to this value regardless of what is set in Terraform - we're looking at either accepting 9988, or deprecating the property altogether (and ignoring any value set) - depending on the behaviours of the Azure environments other than Public (it's high on our list to look at).

The behaviour for controlling whether a Key Vault item (Key / Secret etc) is purged when deleted is controlled by the features block in the provider config. e.g.

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy = false
    }
  }
}

This setting defaults to true, so if you do not wish for deleted items to be purged, this can be set to false, which will allow the recovery of these items.

The 409 Conflict is of interest, @jwshive is that consistent? We explicitly check the item is deleted before attempting to purge it, so this may indicate a bug or inconsistency between the read and purge operations.

[peteneville]

Same for me on an apply:
Error: purging Certificate "91-cert" (Key Vault "https://91-uks.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedCertificate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NotSupported" Message="Operation "purge" is not enabled for this vault."

Using azurem 2.41.0 and not setting purge_protection_enabled (so should have default of false).

Cert was removed though in above run (not unexpected due to code change) and subsequent attempt re-created it.

[U-bit-solo]

My KeyVault has SoftDelete and Purge Protection enabled. But I am getting this error, using a Service Principal after updating to version terraform-provider-azurerm: 2.43.0. The config. file was working fine before
Error: purging Key "keystxxxxx" (Key Vault "https://...vault.azure.net/"): keyvault.BaseClient#PurgeDeletedKey: Failure responding to request: StatusCode=403 Code="Forbidden" Message="The user, group or application 'appid=5xxxxxxx;oid=xxxxxxx-aa5cxxxx;numgroups=1;iss=https://sts.windows.net/xxxxxb7e-28xxxxb/' does not have keys purge permission on key vault 'akvteu2xxxxx;location=eastus2'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"ForbiddenByPolicy"}

[tommydejong]

I am getting the same error on azurerm 2.43.0. I'm using a module to deploy virtual machines in Azure using GitLab CI and a service principal. I used to store the storage account access key in the module, but I've removed that feature. When I try to run the module again it attempts to delete the secret in the key vault (not the key vault itself), I get the following:

Error: purging Secret "sa-access-key" (Key Vault "https://...vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid=xxx;oid=xxx;iss=https://sts.windows.net/xxx/' does not have secrets purge permission on key vault '...;location=westeurope'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"ForbiddenByPolicy"}

This is even though I am certain the service principal has the right access to the key vault:

I'm using the following config:

resource "azurerm_key_vault" "vre-kv" {
  name                = "${local.rootname}-kv"
  location            = var.location
  resource_group_name = azurerm_resource_group.vre-rg.name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  enabled_for_disk_encryption     = true
  enabled_for_template_deployment = true
  enabled_for_deployment          = true
  enable_rbac_authorization       = false

  soft_delete_retention_days = 7
  purge_protection_enabled   = false
  sku_name                   = "standard"

  network_acls {
    default_action = "Allow"
    bypass         = "AzureServices"
  }
  tags = local.tags
}

resource "azurerm_key_vault_access_policy" "vre-kv-ap" {
  key_vault_id       = azurerm_key_vault.vre-kv.id
  tenant_id          = data.azurerm_client_config.current.tenant_id
  object_id          = data.azurerm_client_config.current.object_id
  key_permissions    = ["create", "get", "list", "delete", "purge", ]
  secret_permissions = ["set", "get", "delete", "list", "purge", ]

  depends_on = [
    azurerm_key_vault.vre-kv
  ]
}

And for the provider configuration:

provider "azurerm" {
  subscription_id = "xxx"

  features {
    virtual_machine {
      delete_os_disk_on_deletion = true
    }
    key_vault {
      recover_soft_deleted_key_vaults = false
      purge_soft_delete_on_destroy    = true
    }
  }
}

[omerfsen]

+1 I get it too with terraform 0.14.6 and azure provider 2.46.1. On first run of destroy i get this error when running destroy again i dont get this error

Update: Actually with 2.46.1 i have given Purge permissions (which was missing before) and it worked fine. I think this is a new requirement with 2.41.0 ( https://github.com/terraform-providers/terraform-provider-azurerm/releases/tag/v2.41.0 )

[jason-johnson]

The 409 Conflict is of interest, @jwshive is that consistent? We explicitly check the item is deleted before attempting to purge it, so this may indicate a bug or inconsistency between the read and purge operations.

@jackofallops

For us this is very consistent since this morning at least. Create environment, then destroy and we get this error. If we simply run destroy again we get a permission error (presumably the policy access was deleted successfully) and running a 3rd and all consecutive times reports the key vault cannot be found. Rerun create again and then destroy and the 409 error comes back.

So to summarise, the first destroy we run always gets a 409 on the key vault secrets now. We are running azurerm 2.47.0 and terraform 0.14.7. But I bumped the versions hoping it would fix it. This morning we were on azurerm 2.44.0 and terraform 0.14.4 with the exact same problem.

[tommydejong]

My previous problem seems to be resolved after upgrading to terraform 0.14.7 and azurerm 2.48.0, but I'm experiencing the 409 error now as well:

Error: purging Secret "admin-password" (Key Vault "https://...-kv.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" Message="Secret is currently being deleted." InnerError={"code":"ObjectIsBeingDeleted"}

My provider settings are the following:

key_vault {
      recover_soft_deleted_key_vaults = false
      purge_soft_delete_on_destroy    = true
}

Setting purge_soft_delete_on_destroy to false solves my issue but this doesn't solve my issue. It seems @jackofallops is on the right track with the comment about a potential bug or inconsistency between the read and purge operations

[schlbra]

Our pipelines are dead in the water due to this bug and our heavy reliance on azure key vault. :(

[davenicoll]

I just ran into this issue too (azurerm 2.50, terraform 0.13.6). My keyvaults have soft delete enabled. Running a destroy successfully deletes the secrets, but throws a 403 Forbidden when trying to purge them and ultimately fails. If I re-run terraform, instead of creating the secret with the values of the applied plan, the deleted secrets are restored in the keyvault with their previous values. The permission model used is vault access policy.

[mattduguid]

Using terraform 0.14.6 and azurerm v2.5.0.

On recent attempts to destroy we received following error when it tries to delete a keyvault, the weird bit was that it did manage to delete the keys just not the keyvault itself,

Error: Error purging of Secret "<SECRET_NAME>" (Key Vault "https://<KEYVAULT_NAME>.vault.azure.net/") : keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NotSupported" Message="Operation "purge" is not enabled for this vault."

Similar to @tommydejong we added following and our http/400 issue is now resolved,

provider "azurerm" {
  ...
  features {
   key_vault {
    purge_soft_delete_on_destroy = false # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#purge_soft_delete_on_destroy
    recover_soft_deleted_key_vaults = false # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#recover_soft_deleted_key_vaults
   }
  }
  ...
}

We did initially think issue #10707 was related but its now looking like a separate one that still need resolution

[daguito81]

Im getting this error as well. But left purge protection from the resource blockfor they keyvault which would default to "no purge protection" But then I start getting errors destroying the environment and I see sometimes the KeyVault with Purge protection enabled. But it's been inconsistent. Trying to reproduce

This is my provider:

provider "azurerm" { skip_provider_registration = false disable_terraform_partner_id = true subscription_id = var.SUB_ID tenant_id = var.TENANT_ID features { key_vault { recover_soft_deleted_key_vaults = true purge_soft_delete_on_destroy = false } } client_id = var.DEVOPS_CLIENT_ID client_secret = var.DEVOPS_CLIENT_SECRET }

and then my key vault:
resource "azurerm_key_vault" "key" { location = azurerm_resource_group.rg.location name = "${var.prefix}k2" resource_group_name = azurerm_resource_group.rg.name sku_name = "standard" tenant_id = data.azurerm_client_config.current.tenant_id }

So kept the purge protection off as stated on another comment in this thread.

After doing tf plan and tf apply. It's created, but it has purge protection active.

EDIT:
After a bit more testing, it wasn't just the keyvault being deployed that triggered the Purge protection to be enabled. It happened when an Azure ML Workspace was deployed that linked this keyvault as its keyvault. I don't know why the AzureML workspace deploying would activate purge protection on a keyvault

[mattduguid]

@daguito81 have you tried with both set to false in the provider and false in the resource from a brand new environment,

provider "azurerm" {
...
features {
key_vault {
recover_soft_deleted_key_vaults = false
purge_soft_delete_on_destroy = false
}
}
...
}

resource "azurerm_key_vault" "azure" {
...
purge_protection_enabled = false # once enabled cannot be unset, if enabled and you delete keyvault then it remains soft deleted for value of soft_delete_retention_days before name can be reused which causes problems for terraform if you delete and create an environment quickly as name cannot be reused until keyvault manually purged from azure portal
soft_delete_retention_days = 7 # set lowest possible just in case parameter purge_protection_enabled is set to true
...
}

also this article has some information on soft delete/purge protection for azure keyvaults -> https://docs.microsoft.com/en-us/azure/key-vault/general/soft-delete-change

[daguito81]

@mattduguid Yeah, i tested it with both and the behaviour was still funy. I did some tests just now. And terraform is deploying the KeyVault the right way. The problem was basically that attaching a KeyVault to an Azure Machine Learning workspace, automatically activates Purge Protection on that Keyvault. So Terraform deploys without Purge, then AzureML turns on the purge and then terraform tries to turn purge protection from true to nil, and it can't do that because purge protection can't be deactivated.

Then when destroying the environment, if you have the option "purge deleted keyvaults" set to true. Then AzureML (and maybe other attached services have this behaviour as well) turns on purge protection, terraform tries to purge on delete, and everything fails.

I'm trying to figure out why AzureML is activating purge protection on the attached key vault.

[bitsofinfo]

same problem: using 2.54

The KV was created with purge_protection_enabled = false/null (or omit it), then on another run. It says purge protection is enabled AND like @daguito81 notes, this KV is being attached to a ML workspace.

Error: updating Vault: (Name "my-kv" / Resource Group "my-rg"): once Purge Protection has been Enabled it's not possible to disable it

terraform {
  required_version = "= 0.14.9"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 2.54"
    }
  }
}


data "azurerm_client_config" "current" {}


resource "azurerm_key_vault" "kv" {
  name                = "${var.name_prefix}-kv"
  location            = var.location
  resource_group_name = var.resource_group_name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  sku_name                  = "standard"
  enable_rbac_authorization = true

  enabled_for_disk_encryption     = false
  enabled_for_template_deployment = false
  enabled_for_deployment          = false
  
  purge_protection_enabled = false
  soft_delete_retention_days = 30


  network_acls {
    bypass                     = "AzureServices"
    default_action             = "Deny"
    ip_rules                   = var.network_acl_ip_rules
    virtual_network_subnet_ids = var.network_acl_virtual_network_subnet_ids
  }

  tags = var.tags

}

[PhilippeVandeVyverInetumRealdolmen]

Has someone found a solution for this?

[schonfinkel]

@mattduguid Yeah, i tested it with both and the behaviour was still funy. I did some tests just now. And terraform is deploying the KeyVault the right way. The problem was basically that attaching a KeyVault to an Azure Machine Learning workspace, automatically activates Purge Protection on that Keyvault. So Terraform deploys without Purge, then AzureML turns on the purge and then terraform tries to turn purge protection from true to nil, and it can't do that because purge protection can't be deactivated.

Then when destroying the environment, if you have the option "purge deleted keyvaults" set to true. Then AzureML (and maybe other attached services have this behaviour as well) turns on purge protection, terraform tries to purge on delete, and everything fails.

I'm trying to figure out why AzureML is activating purge protection on the attached key vault.

Can confirm this behavior when deploying Azure ML workspaces, they broke our current CI pipelines.

[daguito81]

Yeah, I don't think this a terraform problem. This is an Azure problem as they auto activate purge protection on Key Vaults when attached to a Machine Learning Workspace. If you do it "by hand" the result is the same.

I changed my architecture to basically keep the AML Key Vault so assuming it's purge protected from the beginning. So I enable the recovery of purged key vaults in terraform instead. Not perfect, but it works and if I need to have a different key vault I just switch the name and purge them once the 90 days are done.

[wahyuen]

For those following this issue, we ran into the same problem (not in Terraform, but using Bicep). After a few back and forth tickets, it appears Microsoft have now rolled out a change so that linking a KV to an ML workspace via ARM does NOT auto activate purge protection.

I have tested this in australiaeast region, YMMV depending what region you are doing this to.

From the support ticket we raised

We are on our way to make purge protection not required when creating ML workspace.

Hopefully this helps some of you with your own use cases!

[tombuildsstuff]

👋

Since this has been fixed in the upstream API I'm going to close this issue out - #10273 is tracking support for more granular purge protection for resources, as such if your interested in that would you mind subscribing to that issue for updates?

Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.