RDS DBCluster provisioning with composite KMS Key fails with 'The specified KMS key [...] does not exist'

[robertgates55]

I'm using the Composition below to create a DBCluster. I'd like to create a KMS key per DBCluster.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: mysqlclusters.aws.database.du.co
spec:
  compositeTypeRef:
    apiVersion: database.du.co/v1alpha1
    kind: XMySQLCluster
  resources:
  - name: kmskey
    base:
      apiVersion: kms.aws.crossplane.io/v1alpha1
      kind: Key
      spec:
        forProvider:
          region: eu-west-1
  - name: dbcluster
    base:
      apiVersion: rds.aws.crossplane.io/v1alpha1
      kind: DBCluster
      spec:
        forProvider:
          ...
          region: eu-west-1
          kmsKeyIDSelector:
            matchControllerRef: true
          ...

This successfully creates the kms key:

➜  kubectl get xmysqlcluster
NAME                    READY   COMPOSITION                        AGE
rg-test-cluster-5bp8r   False   mysqlclusters.aws.database.du.co   2d17h

➜  kubectl get keys.kms.aws.crossplane.io -l crossplane.io/composite=rg-test-cluster-5bp8r
NAME                          READY   SYNCED   EXTERNAL-NAME
rg-test-cluster-5bp8r-zzp65   True    True     85c59e3b-9b0f-437d-b721-a8d094c56d4f

But the DBCluster fails to provision:

➜   kubectl get dbcluster.rds -l crossplane.io/composite=rg-test-cluster-5bp8r
NAME                          READY   SYNCED   EXTERNAL-NAME
rg-test-cluster-5bp8r-88lq4   False   False    rg-test-cluster-5bp8r-88lq4

with this error:

  Warning  CannotCreateExternalResource     2s                    managed/dbcluster.rds.aws.crossplane.io  cannot create DBCluster in AWS: KMSKeyNotAccessibleFault: The specified KMS key [rg-test-cluster-5bp8r-zzp65] does not exist, is not enabled or you do not have permissions to access it.
           status code: 400, request id: 9ffbdf72-077d-4729-8287-efa720a8b3fc

`
The key does exist in the AWS UI.

What permissions do I need to give it?

Should it be looking up using the name rg-test-cluster-5bp8r-zzp65 or does it actually need to be using the externalName?

Any ideas where I should look to find more?

[robertgates55]

Looking in Cloudtrail, I can see the following errorMessage. That KMS key does indeed not exist - that's just the crossplane 'internal' (?) name the externalName resource: 85c59e3b-9b0f-437d-b721-a8d094c56d4f

Is this the issue? Should it be looking up the externalName and passing that through?

{
...
    "errorMessage": "The specified KMS key [rg-test-cluster-5bp8r-zzp65] does not exist, is not enabled or you do not have permissions to access it.",
    "requestParameters": {
        "dBClusterIdentifier": "rg-test-cluster-5bp8r-88lq4",
...
        "kmsKeyId": "rg-test-cluster-5bp8r-zzp65",
        "enableIAMDatabaseAuthentication": false,
        "engineMode": "provisioned",
        "deletionProtection": false,
        "copyTagsToSnapshot": false
    },
...

[robertgates55]

Confirming a further few things:

• I've added a policy to the kms key to allow the crossplane user to see it. I don't think this was necessary, but it felt like the next thing to try.

• I updated the composition to use the crossplane-provisioned key directly using the actual externalName (eg kmsKeyID: 85c59e3b-9b0f-437d-b721-a8d094c56d4f) and it created fine

I guess the kmsKeyIDSelector should actually be a kmsKeyIDRefSelector?

[robertgates55]

A kubectl get dbclusters.rds rg-test-cluster-4bfzf-z8dg2 -o yaml shows that crossplane has added both a kmsKeyID and a kmsKeyIDRef:

...
    engine: aurora-mysql
    engineMode: provisioned
    engineVersion: 5.7.mysql_aurora.2.10.0
    kmsKeyID: rg-test-cluster-4bfzf-6xg6f
    kmsKeyIDRef:
      name: rg-test-cluster-4bfzf-6xg6f
    kmsKeyIDSelector:
      matchControllerRef: true
    masterUserPasswordSecretRef:
...

That KeyID is obviously incorrect - that's a crossplane resource name, not an AWS-known one.

I guess it's then using the KeyID as it's the most precise?

[robertgates55]

And if I kubectl edit dbclusters.rds rg-test-cluster-4bfzf-z8dg2 and delete the kmsKeyID: rg-test-cluster-4bfzf-6xg6f, then run kubectl get dbclusters.rds rg-test-cluster-4bfzf-z8dg2 -o yaml again, it's then replaced with the correct ref:

...
    engine: aurora-mysql
    engineMode: provisioned
    engineVersion: 5.7.mysql_aurora.2.10.0
    kmsKeyID: 5dbbec3c-2a11-4185-b467-b8a59d947689  <<<< UPDATED
    kmsKeyIDRef:
      name: rg-test-cluster-4bfzf-6xg6f
    kmsKeyIDSelector:
      matchControllerRef: true
    masterUserPasswordSecretRef:
...