IAM and Route53 resources fail to Reconcile in non aws partitions

[smcavallo]

What happened?

In non-aws partitions (such as gov cloud and cn) iam and route53 resources fail to reconcile.
While using aws credentials the exception is InvalidClientTokenId: The security token included in the request is invalid.
While using pod-iam it is not able to find the OIDC provider.

How can we reproduce it?

Attempt to create any iam or route53 resource in gov cloud or cn or non-aws based partition.

What environment did it happen in?

Crossplane version: 0.17.0 (all)

The issue is that these clients have a global region hardcoded as aws-global which is only valid for the aws partition.
https://github.com/crossplane/provider-aws/blob/35d9474782ee0f85a3820ea8175e5d59c2c3e4b8/pkg/clients/aws.go#L59

I was able to successfully create iam resources in gov cloud by changing the GlobalRegion to aws-us-gov-global per the below.

aws partition
https://github.com/aws/aws-sdk-go-v2/blob/59774b18837c670a2c5327ae244c664459462697/codegen/smithy-aws-go-codegen/src/main/resources/software/amazon/smithy/aws/go/codegen/endpoints.json#L3048-L3062

gov partition
https://github.com/aws/aws-sdk-go-v2/blob/59774b18837c670a2c5327ae244c664459462697/codegen/smithy-aws-go-codegen/src/main/resources/software/amazon/smithy/aws/go/codegen/endpoints.json#L8103-L8120

Supported partitions according to the aws-sdk

const GlobalRegion = "aws-global"
const GlobalRegionGov = "aws-us-gov-global"
const GlobalRegionCn = "aws-cn-global"
const GlobalRegionIso = "aws-iso-global"
const GlobalRegionIsoB = "aws-iso-b-global"

[hasheddan]

@smcavallo Thanks for identifying this! Are credentials for non-aws-global global regions scoped only to their single global region? If so, this feels like a ProviderConfig setting.

[smcavallo]

@smcavallo Thanks for identifying this! Are credentials for non-aws-global global regions scoped only to their single global region? If so, this feels like a ProviderConfig setting.

Yes - if I'm understanding what you're asking - you can't share credentials or use role trust across aws partitions, and as such anything scoped for a global region would be limited to that single global region. Ex. setting a global region to aws-us-gov-global would limit scope to aws-us-gov partition (and aws-us-gov-global)

Alternatively - if the actual region is passed through the aws sdk is smart enough to use the correct endpoint.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html
https://docs.aws.amazon.com/general/latest/gr/iam-service.html

I was able to verify that using a region of us-gov-west-1 in gov cloud works as expected, without needing to use the global region. Ex. an actual (non-global) region could be configured at the ProviderConfig level

[muvaf]

The issue is that these clients have a global region hardcoded as aws-global which is only valid for the aws partition.

Note that this constant is used only in IAM because it is region-less. In other resources, we use only the region parameter given in spec.

[smcavallo]

Note that this constant is used only in IAM because it is region-less. In other resources, we use only the region parameter given in spec.

Yes - 100%. Though when provided with a regional endpoint the default endpoint resolvers should find the right endpoint to send requests to. Ex. if you set a region of us-east-1 the endpoint resolvers will make sure your request is sent to the right place. There shouldn't be a need to hard-code/override the region to aws-global

This issue can be worked around when using some of the identity providers by setting an annotation on the IAMRole object, in which case the iam requests will be sent to https://iam.us-gov.amazonaws.com regardless of the region set in any of the configs since the custom resolver will override the default resolver and send the requests to that valid endpoint.

  annotations:
    aws.alpha.crossplane.io/endpointServiceID: iam
    aws.alpha.crossplane.io/endpointURL: https://iam.us-gov.amazonaws.com
    aws.alpha.crossplane.io/endpointSigningRegion: us-gov-west-1

The above workaround won't work when using pod-iam though.
https://github.com/crossplane/provider-aws/blob/master/pkg/clients/aws.go#L233-L234
because the sts request would still use the hard-coded global region.
The custom endpoint resolver is not set until after the sts request is made to retrieve credentials, which is why in non aws partitions it is unable to find the identity provider.

[negz]

Could we add optional region fields to these types (despite that being a bit weird given that that they're "regionless"), that would default to aws-global? This seems like it would be in line with our current approach, where most resources have a spec.forProvider.region field.

[muvaf]

@negz haha this is exactly what we have decided on in our DM before the community meeting.

[haarchri]

@smcavallo any chance to check if #1329 fixes your issue ? Is merged in master

[nabuskey]

With the PR mentioned above, what are the expectations for users in the non global partitions? Are users in non global partitions expected to use endpoint configuration for all of their ProviderConfig? With the PR, I can get IAM to reconcile correctly in China regions with IRSA but requires endpoint configuration like this:

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  assumeRole:
    roleARN: arn:aws-cn:iam::123456:role/crossplane-provider-aws
  credentials:
    source: InjectedIdentity
  endpoint:
    partitionId: aws-cn
    signingRegion: cn-north-1
    url:
      type: Dynamic
      dynamic:
        protocol: https
        host: amazonaws.com.cn

If users would like to managed IAM resources in non global partitions, are they expected to use a particular ProviderConfig with endpoint configuration? Or does it make sense to handle non global partitions in the provider?

[haarchri]

New Implementation with #1364