Issues with Terraform AWS Kinesis Firehose Module for Cross-Account Setup

[alexfdsl]

Hi,

I encountered several issues while using this Terraform module to set up Kinesis Firehose for consuming data from an MSK topic and delivering it to an S3 bucket in a cross-account setup:

1. Trust Policy Issue for Cross-Account Setup

The trust policy attached to the role created by this module causes problems in cross-account setups. Specifically, the condition in the trust policy only allows the account in which Kinesis Firehose is created to assume the role. This restriction makes it impossible to assume the role from a different account.

Reference: iam.tf#L36

2. Incorrect ARN in MSK Policy

The MSK policy generated by the module contains incorrect ARNs for topic and consumer group permissions. Instead of using the topic and group ARNs, it uses the cluster ARN, which results in invalid permissions.

References:
iam.tf#L131
iam.tf#L141

3. Inconsistent Logic for create_role and is_msk_source Variables

When attempting to use a custom role and policy by setting create_role = false, the module fails due to a logic issue. Specifically:
If create_role = false, the module tries to look for the policy ARN, but this fails because the is_msk_source variable is always true when integrating with MSK.

Reference: iam.tf#L150

The dependency for creating the policy when is evaluated differently that the code that create the policy, causing further inconsistencies.

Reference: iam.tf#L110

[fdmsantos]

Thank you for detailed issue. I will work on this

[fdmsantos]

Hi @alexfdsl ,

I just release a new version (v3.8.1). This version fixes the point 3 (Inconsistent Logic for create_role and is_msk_source Variables
). I think this version will allow you use this module to create kinesis delivery stream. For MSK Cross Account setup, for now, you need create role outside the module create_role = false as you mentioned. I'm planning fix this limitation (allow create Kinesis role for cross account MSK setup via module) in next versions.

Do you have working example with firehose and MSK cross account setup working? Could you share it?
Are you using Private or Public MSK?

Thanks

[alexfdsl]

Hi, yes.

I ended up creating this module, to make it clear, Firehose and MSK are in the same account, the destination, s3 in this case, is a different account. I am using private MSK

resource "aws_kinesis_firehose_delivery_stream" "firehose" {
  name        = var.firehose_name
  destination = "extended_s3"

  msk_source_configuration {
    msk_cluster_arn = var.msk_source_cluster_arn
    topic_name      = var.msk_source_topic_name

    authentication_configuration {
      connectivity = var.msk_source_connectivity_type
      role_arn     = var.firehose_role
    }
  }

  extended_s3_configuration {
    role_arn           = var.firehose_role
    bucket_arn         = var.s3_bucket_arn
    prefix             = var.s3_prefix
    error_output_prefix = var.s3_error_prefix
    compression_format = var.compression_format

    cloudwatch_logging_options {
      enabled         = var.enable_cloudwatch_logs
      log_group_name  = coalesce(var.cloudwatch_log_group, var.firehose_name)
      log_stream_name = coalesce(var.cloudwatch_log_stream, var.firehose_name)
    }
  }
}

This is how the trust policy for the firehose's role needs to look like

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "firehose.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

I remember the role for firehose created by your code, added a condition to it, making the integration to fail.

[fdmsantos]

Hi @alexfdsl ,

Thanks for your input.

Cross Account S3 is supported by the module. You need configure s3_cross_account variable equals to true.

You can check this example DirectPut-To-CrossAccount-S3

About using MSK in same account, for now it's necessary creating the role outside the module (This is fixed in new version), I need to fix the Point 2 that on mentioned to the module became fully working with MSK in Same Account with Firehose. I hope find some time to fix this week (Testing MSK is very annoying, the MSK takes several minutes to create :) )

I will continue update the issue until everyting becames fixed.