kube->kms s2s Policy is Being Deleted Too Early

[stephaniegalang]

Upon cluster deletion, tf deletes the kms s2s policy before (or too soon after?) cluster deletion. This breaks the kms key that was associated with the deleted cluster, since if the kms s2s policy is deleted prior to IKS removing the cluster's association with the kms key, IKS cannot execute the removal and the key becomes un-deletable.

This appears to be happening after the #645 fix, where users are now able to provision clusters with kms keys.

The kms s2s policy MUST exist during cluster deletion. IKS may need a few seconds minutes after cluster deletion to execute the removal of the association with the kms key using the kms s2s policy.

Some fix suggestions:

• Add a sleep to delay s2s policy deletion similar to:

  terraform-ibm-landing-zone/vpe.tf

  Line 39 in c7d443d

  resource "time_sleep" "wait_30_seconds" {

  • https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep
• Make the kms s2s policy depends_on kms key deletion

Note: I am a representative of the Key Protect service. I do not have any logs associated with this bug. It is possible that this bug produces no error in tf

Affected modules

Terraform CLI and Terraform provider versions

• Terraform version:
• Provider version:

Terraform output

Debug output

Expected behavior

When deleting an IKS cluster associated with a kms key, a s2s policy between the cluster and kms should be in place to allow IKS to remove the association with the kms key, allowing the key to be deleted.

Actual behavior

When deleting IKS cluster associated with a kms key, the s2s policy between IKS and kms is removed. Without the s2s policy, IKS cannot remove the cluster's association with the kms key. The kms key cannot be deleted unless this association is removed

Steps to reproduce (including links and screen captures)

1. Use tf to create an IKS cluster associated with a kms key
2. Use tf to delete IKS cluster from step 1
3. In UI/API/CLI, attempt to delete kms key from step 1. Fails with HTTP 409 "The key cannot be deleted because it's protecting a cloud resource that has a retention policy"

Anything else

---

By submitting this issue, you agree to follow our Code of Conduct

[stephaniegalang]

@Ak-sky @ocofaigh can you help with this?

[ryan-cradick]

Is the deletion of the service authorization policy required here? The policy may have existed prior to TF attempting to create it. Looking at a few logs, it will likely take at least 5 minutes from the cluster deletion request before the key association is removed.

[ocofaigh]

@ryan-cradick We are actually explicitly creating a more locked down s2s auth policy here, so its not going to delete any one that was auto created by kube. We found we had to do this because kube only auto creates the auth policy for cluster data encryption, however for boot volume encryption, the policy needs to exist before cluster creation hence we need to explicitly create it.

[tyao117]

This is under the assumption that the cluster is the vpc cluster.
Some suggestions:

• Would a prevent_destroy condition help here? .

• implementing a https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep with the attribute destroy_duration should work as proposed above

• You make the create_before_destroy block on the service-to-service authorization(this attribute should invert the dependency chain during destroy operations).

• The other possible way to mitigate this behavior would be changing the Destroy func here where it creates the s2s authorization during the destroy: https://github.com/IBM-Cloud/terraform-provider-ibm/blob/50e6a844c6e74151a62233bd3fad30a4970c3141/ibm/service/kubernetes/resource_ibm_container_vpc_cluster.go#L1032

I'm shooting some suggestions, feel free to counter with alternatives if needed.

[tyao117]

Talking to @ocofaigh about all the proposed fixes. A depends_on will be used on the vpc iks cluster resource block

[ocofaigh]

@stephaniegalang A fix has been included in https://github.com/terraform-ibm-modules/terraform-ibm-landing-zone/releases/tag/v5.19.3 if you want to try it out? If any further issues please feel free to re-open