DBInstance: Late Initialize all fields that cannot be set when connecting to a DBCluster

[eloo]

What happened?

Late init causes an update on not allowed fields of DBInstances which are member of a DBCluster.

The result is that all DBInstances in a DBCluster are going out of sync and not be synced again.

A part of this issue is already fixed in #895
But only a subset of disallowed values was fixed here so the bug is still present and prevents DBInstances from being Healthy.

Thanks to @iAnomaly he found the proper resource of which fields are disallowed in DBCluster instances and was so kind to provide a PR. (https://github.com/crossplane/provider-aws/pull/1285/files)

You can visit the AWS CLI docs for modify-db-instance and search for the string "managed by the DB cluster" to find all fields that cannot be set when associating a DBInstance with a DBCluster.

Together with the last fix of #1197 we are currently running into heavy issues as it seems that the password in the connection is not going to match anymore to the password set in AWS as the sync is not possible right now because of this bug.

How can we reproduce it?

Setup a DBCluster with DBInstances member and try to update the DBInstance afterwards

What environment did it happen in?

Crossplane version: 0.26.1

[eloo]

@haarchri this issue is still not fixed.
not sure why this issue is closed without being tests :(

please either wait till a fix really confirmed as fixed or test it on your side.

this is really a pitty that this issue is open for more than a half year and not properly fixed yet.
further it seems that the test coverage or test suite is not really satisfying.

please reopen this issue as this bug is still open

[eloo]

the current issue is here:

 message: "update failed: cannot update DBInstance in AWS: InvalidParameterCombination:
      The specified DB Instance is a member of a cluster. Modify deletion protection
      for the DB Cluster using the ModifyDbCluster API\n\tstatus code: 400, request
      id: "

[haarchri]

we test on our side this issue in real environment with existing databases and with complete new databases - for exisiting database we removed all wrong init fields with a script

following fields:
https://github.com/crossplane/provider-aws/pull/1285/files#diff-66c455e871b6c0c85f9cbb0b878f062942153e47df0a90fda16530a38369557bR208-R214

if this issue is still valid in your environment please add your manifest file that we can check which field leads to the issue

re-checked:

kubectl get dbinstance.rds.aws.crossplane.io/production-postgres-instance-1-t6kc4-dfr95
NAME                                    READY   SYNCED   EXTERNAL-NAME
production-postgres-instance-1-t6kc4-dfr95   True    True     production-postgres-instance-1

[haarchri]

So looks like deletionProtection is one more - interesting that we can't see this in our environment - possible to share your manifest ?

[iAnomaly]

Frustrating. AWS does not document anywhere I can find that DeletionProtection is a cluster level field.

[eloo]

yep thats indeed frustrating :(
@haarchri which manifest do you need?
from the created MR, the claim or the composition?

and if you can provide test build to fix this issue i will to test them as soon as possible so can make this work together soon 👍

[haarchri]

From the MR is enough thanks

[eloo]

@haarchri

apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBInstance
metadata:
  annotations:
    crossplane.io/external-create-failed: "2022-04-11T15:22:40Z"
    crossplane.io/external-create-pending: "2022-04-11T15:22:41Z"
    crossplane.io/external-create-succeeded: "2022-04-11T15:22:42Z"
    crossplane.io/external-name: test
  creationTimestamp: "2022-05-11T10:05:11Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: test-sv8vk-
  generation: 11
  labels:
    test.cloud.test/managed-by: test
    crossplane.io/claim-name: test
    crossplane.io/claim-namespace: flo
    crossplane.io/composite: test-sv8vk
    velero.io/backup-name: lovejoy-2022-05-11-1652259396
    velero.io/restore-name: lovejoy-2022-05-11-1652259396-20220511120324
  name: test-sv8vk
  ownerReferences:
  - apiVersion: test.cloud.test/v1
    controller: true
    kind: XRdsInstance
    name: test-sv8vk
    uid: 180906f3-cdd2-485c-bf62-9eb2d7407d60
  resourceVersion: "25307888"
  uid: 2857b623-036c-474c-ba11-bc92fb95bbd4
spec:
  deletionPolicy: Orphan
  forProvider:
    autoMinorVersionUpgrade: true
    availabilityZone: eu-central-1a
    copyTagsToSnapshot: false
    dbClusterIdentifier: trial-service-flo-rds-cluster
    dbInstanceClass: db.t3.small
    dbName: trial_service_flo
    dbParameterGroupName: elysium-aurora-mysql-5-7
    dbSubnetGroupName: test.elysium.test.cloud.test
    deletionProtection: false
    enablePerformanceInsights: false
    engine: aurora-mysql
    engineVersion: 5.7.mysql_aurora.2.10.2
    kmsKeyID: arn:aws:kms:eu-central-1:test:key/da8c1243-ed59-4de0-81ef-d18129ce901f
    licenseModel: general-public-license
    masterUsername: master
    monitoringInterval: 0
    multiAZ: false
    port: 3306
    preferredMaintenanceWindow: fri:22:24-fri:22:54
    promotionTier: 1
    publiclyAccessible: false
    region: eu-central-1
    storageEncrypted: true
    storageType: aurora
    tags:
    - key: claim-namespace
      value: flo
    - key: claim-name
      value: test
    - key: managed-resource-name
      value: test-sv8vk
    vpcSecurityGroupIDs:
    - sg-test
  providerConfigRef:
    name: aws-provider-config
status:
  atProvider:
    caCertificateIdentifier: rds-ca-2019
    customerOwnedIPEnabled: false
    dbInstanceARN: arn:aws:rds:eu-central-1:test:db:test
    dbInstanceIdentifier: test
    dbInstancePort: 0
    dbInstanceStatus: available
    dbParameterGroups:
    - dbParameterGroupName: elysium-aurora-mysql-5-7
      parameterApplyStatus: in-sync
    dbSubnetGroup:
      dbSubnetGroupDescription: Database subnet group for test.elysium.test.cloud.test
      dbSubnetGroupName: test.elysium.test.cloud.test
      subnetGroupStatus: Complete
      subnets:
      - subnetAvailabilityZone:
          name: eu-central-1c
        subnetIdentifier: subnet-test
        subnetOutpost: {}
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: eu-central-1b
        subnetIdentifier: subnet-test
        subnetOutpost: {}
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: eu-central-1a
        subnetIdentifier: subnet-test
        subnetOutpost: {}
        subnetStatus: Active
      vpcID: vpc-test
    dbiResourceID: db-test
    endpoint:
      address: test.test.eu-central-1.rds.amazonaws.com
      hostedZoneID: test
      port: 3306
    iamDatabaseAuthenticationEnabled: false
    instanceCreateTime: "2022-04-11T15:29:02Z"
    optionGroupMemberships:
    - optionGroupName: default:aurora-mysql-5-7
      status: in-sync
    pendingModifiedValues: {}
    performanceInsightsEnabled: false
    tagList:
    - key: managed-resource-name
      value: test-sv8vk
    - key: claim-namespace
      value: flo
    - key: claim-name
      value: test
    vpcSecurityGroups:
    - status: active
      vpcSecurityGroupID: sg-test
  conditions:
  - lastTransitionTime: "2022-05-23T14:05:29Z"
    message: "update failed: cannot update DBInstance in AWS: InvalidParameterCombination:
      The specified DB Instance is a member of a cluster. Modify deletion protection
      for the DB Cluster using the ModifyDbCluster API\n\tstatus code: 400, request
      id: "
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2022-05-11T11:00:42Z"
    reason: Available
    status: "True"
    type: Ready

[eloo-abi]

i have created a PR which fixes the late init issue in our cluster