Allow owners to arbitrary ARM IDs

[Sbou]

Describe the current behavior
Currently, the type of the owner field is ArbitraryOwnerReference. It's only support another resource defined in the cluster.

Describe the improvement
It would be great if I could define an armId to be able to add a role to an existing resource outside of the cluster.

Thanks

[matthchr]

Is the resource you're attempting to add the RoleAssignment to of a kind that ASO doesn't support? If ASO does support it, you can import it into ASO and have ASO just shadow it by annotating that resource with the skip reconcile-policy as a workaround.

[Sbou]

Hello,

yes it’s a resource supported by ASO. So I can import it.
But for some others resources, it could interesting

thank you

[matthchr]

We're still considering this but haven't decided to do it yet.

[ombre9]

We are facing the same problem, currently creating role assignments on secret level within a key vault is not possible.

[ombre9]

Any update on this, what is the plan here? This is really blocking us to adapt ASO V2

[matthchr]

Sorry I missed this in Feb @ombre9. For future reference if we're slow responding to a question/issue you can also ping us on the k8s slack in the azure-service-operator channel.

We've been waiting for enough customer use-cases to accumulate that it makes sense to do this. We're probably at or near that point now, which means this item needs to get a design done for it and then assuming that design is accepted we need to implement it.

Given how close we are to our GA release of 2.0.0 this would likely come in 2.1.0. We don't have concrete timelines for each release, it depends on what features we triage in, but we try to do one at least every 1-2 months, which would probably put 2.1.0 somewhere around May/June 2023 timeframe.

Is RoleAssignments for KeyVault secrets your primary use-case for this feature or are there others?

[ombre9]

@matthchr atm yes!
Potentially in the future we have to create role assignments on a resource we manage but not created within kubernetes. As a workaround we could do what you suggested shadowing the resource with a skip reconcile annotation but it's not ideal.

[tyler-gs]

+1 to this feature - assigning arbitrary ARM IDs would be ideal for the workflow I'm intending: creating a cluster outside of ASO using something like Terraform, and then creating the application-specific nodepools using ASO by referencing that ARM ID.

[matthchr]

Until we get this feature, I wanted to document the current workaround:

Imagine you want to create a resource B inside of an existing resource A, such that A owns B. A concrete example of this would be creating a StorageAccount inside an existing ResourceGroup.

1. Create the owner resource (A) in ASO. When creating this resource, specify serviceoperator.azure.com/reconcile-policy: skip. This will "import" the resource A into ASO but not have ASO manage it. ASO will just report the status of the resource.
   • When "creating" the resource A in ASO, you may specify only the minimal set of properties required for Kubernetes to accept the resource (all of the required properties in the CRD). You need not actually make the resource match exactly what is already in Azure, as serviceoperator.azure.com/reconcile-policy: skip set means that ASO will never apply that resource to Azure anyway.
2. Create the "owned" resource (B) in ASO, referring to A. This resource should be created as you normally would (without reconcile-policy specified).

If your hierarchy is longer (i.e. A -> B -> C -> D, and you just want to create D) you have to create A, B and C following step 1 above.

[skinny]

Thanks for the workaround @matthchr !

a big upvote from me as we have several resources already in place and managed by other teams but we still like to use them with ASO managed resources without worrying about them expect pointing to them directly

[ombre9]

@matthchr thanks for documenting the workaround. Unfortunately it doesn't work for our use case. I cannot shadow a secret in a keyvault. We are really waiting for the feature, cause we want to adopt ASO v2 as soon as possible.

[matthchr]

One note (mostly to myself) about this: When implementing this we need to be a bit careful because there is some code right now that relies on being able to retrieve its owner in Kubernetes.

I think that this code only exists in handcrafted resources such as mysql user.

There are a couple of options for how we can handle this:

1. We can require that handcrafted resources (right now only SQL users) are created only on their Kubernetes owners and not the raw ARM IDs
2. We can allow Users to be created on direct ARM IDs by adding an optional fqdn field which must be specified if we cannot find the Kubernetes owner.

[AndreiAkhrymenko]

+1 to this feature

[ombre9]

@matthchr Hey, I see this feature is being pushed over and over again to a new milestone and we've been waiting for this to finally decommission ASO v1. This becomes a problem for us because of an internal compliance issue with ASO v1 (as mentioned above suggested workaround doesn't work for our use case). Can you provide an update? thanks

[matthchr]

Sorry @ombre9. You're totally right, this has been in the backlog for a long time and hasn't been prioritized because we've been working on other things. That's wrong though, clearly there's strong desire for this feature.

I've bumped it up to high-priority and moved it into 2.3.0. There's no guarantee it makes it into 2.3.0 but we'll try our best to get it in.