MSK Configuration: ConflictException: A resource with this name already exists

[Ezward]

I attempted to change delete.topic.enable from true to false in an aws_msk_configuration. It marked it for replacement. The apply then says it destroyed it, but the create of the replacement fails because the prior configuration exists.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

terraform -v
Terraform v0.12.20
+ provider.aws v2.56.0
+ provider.http v1.2.0
+ provider.local v1.4.0
+ provider.null v2.1.2
+ provider.random v2.2.1
+ provider.template v2.1.2

Your version of Terraform is out of date! The latest version
is 0.12.24. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

• aws_msk_configuration

Terraform Configuration Files

resource "aws_msk_configuration" "logs_configuration" {
  kafka_versions = [var.kafka_infra["kafka_version"]]
  name           = "logs-configuration"

  server_properties = <<PROPERTIES
auto.create.topics.enable = false
delete.topic.enable = true
default.replication.factor = ${var.kafka_infra["replication_factor"]}
num.partitions = ${var.kafka_infra["num_partitions"]}
log.retention.hours	= ${var.kafka_infra["retention_hours"]}
PROPERTIES
}

Debug Output

  # aws_msk_configuration.logs_configuration must be replaced
-/+ resource "aws_msk_configuration" "logs_configuration" {
      ~ arn               = "arn:aws:kafka:us-west-2:XXXXXXXXXXXX:configuration/logs-configuration/32608117-0101-44ee-9869-2dedbf22681d-3" -> (known after apply)
      ~ id                = "arn:aws:kafka:us-west-2:XXXXXXXXXXXX:configuration/logs-configuration/32608117-0101-44ee-9869-2dedbf22681d-3" -> (known after apply)
        kafka_versions    = [
            "2.3.1",
        ]
      ~ latest_revision   = 1 -> (known after apply)
        name              = "logs-configuration"
      ~ server_properties = <<~EOT # forces replacement
          - auto.create.topics.enable = true
          + auto.create.topics.enable = false
            delete.topic.enable = true
            default.replication.factor = 2
            num.partitions = 3
            log.retention.hours	= 2160
        EOT
    }

---
aws_msk_configuration.logs_configuration: Destroying... [id=arn:aws:kafka:us-west-2:XXXXXXXXXXXX:configuration/logs-configuration/32608117-0101-44ee-9869-2dedbf22681d-3]
aws_msk_configuration.logs_configuration: Destruction complete after 0s
aws_msk_configuration.logs_configuration: Creating...

Error: error creating MSK Configuration: ConflictException: A resource with this name already exists.
{
  InvalidParameter: "name",
  Message_: "A resource with this name already exists."
}

  on kafka.tf line 103, in resource "aws_msk_configuration" "logs_configuration":
 103: resource "aws_msk_configuration" "logs_configuration" {

Expected Behavior

I expect that the configuration's revision would be incremented and the configuration would be updated.

Actual Behavior

• attempts to delete the configuration (which it does not happen, but does not error)
• attempts to create a new configuration with the same name, which fails because the current configuration exists.

[bflad]

Hi @Ezward 👋 Thank you for submitting this and sorry you ran into trouble. There's a few factors contributing to this unexpected behavior.

The MSK API does not currently support deleting Configurations. We try to note this on the aws_msk_configuration resource documentation, but that can understandably be missed if not looking there. We do have some future plans to introduce warnings into the Terraform AWS Provider as part of some major work being done upstream in the next version of the Terraform Plugin SDK, which will align with version 3.0.0 of the Terraform AWS Provider as it will require removing Terraform CLI 0.11 support. Theoretically in these situations we can raise a warning in the resource logic that will bubble up to the Terraform CLI user interface (plan/apply output).

The MSK API unfortunately does not currently support updating Configurations either (the UpdateClusterConfiguration API call is for updating the Configuration a Cluster is using). While there are pieces of the API that appear ready for multiple versions of configurations (such as the latest revision response value), without an API call for updating the Configuration itself, the Terraform resource cannot support that functionality.

This leaves us with the situation that occurs now. If you have a MSK Configuration that needs updating, it must be recreated with a new name. The aws_msk_configuration resource does not make this easy at the moment. A potential workaround for this situation is using another resource that can maintain a rotating ID that changes when the configuration also changes like using the random_id resource as shown in its example documentation.

We can however implement name generation support into the resource as well, e.g. supporting a name_prefix argument and allowing operators to choose one of name/name_prefix or omit both arguments for a fully generated name. When Terraform needs to recreate the resource, the name would automatically be re-generated. Information about how to implement this type of functionality into Terraform AWS Provider resources can be found in the Adding Resource Name Generation Support section of the Contributing Guide. This would be our preferred solution in this case.

For now, I'm going to mark this as an enhancement to implement the name generation support for this resource since other changes required above are longer term or need to wait until the API is changed, which is outside the Terraform AWS Provider maintainers control.

Hope this helps.

[Ezward]

Thank you. I did end up generating a new name to solve the problem.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!