[BUG] ManagedIdentityCredential authentication failed: Status: 410 (Gone)

[zhiweiv]

Library name and version

Azure.Identity 1.6.0

Describe the bug

Sometimes managed identity authentication failed due to 410 error from IMDS.

Expected behavior

Per https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service?tabs=linux#frequently-asked-questions

Why am I getting the error 500 Internal Server Error or 410 Resource Gone?

Retry your request. For more information, see Transient fault handling. If the problem persists, create a support issue in the Azure portal for the VM.

410 is a retriable error, Azure.Identity should retry for better stability. I guess it is probably here:

azure-sdk-for-net/sdk/identity/Azure.Identity/src/ManagedIdentitySource.cs

Lines 151 to 162 in 068b3de

	private class ManagedIdentityResponseClassifier : ResponseClassifier
	{
	public override bool IsRetriableResponse(HttpMessage message)
	{
	return message.Response.Status switch
	{
	404 => true,
	502 => false,
	_ => base.IsRetriableResponse(message)
	};
	}
	}

Environment

.net 6.0 console app in AKS Linux container

[jsquire]

//cc: @christothes

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[SKUD201]

Hello,

We had the exact same issue in the same environment. Being on version 1.5.0, it triggerred an exception similar to #24158.

We will update to 1.6.0 in the meantime.
Thanks.

[semornas]

Hello,

We had the exact same issue in the same environment. Being on version 1.5.0, it triggerred an exception similar to #24158.

We will update to 1.6.0 in the meantime. Thanks.

Hi there,
Any news regarding this issue? We have a bug in our backlog that is blocked because of this :(
Any help would be appreciated!

[devlie]

@christothes is there any plan to fix this?

IMDS can return HTTP 410 by design and their documentation requires caller to retry up to 70 secs, but ManagedIdentityCredential does not retry even once. Worse, because we don't retrieve the token ourselves (Storage/Cosmos clients do that automatically), we can't really retry the error ourselves without polluting our business layer which has no business of knowing which auth is used.

[christothes]

Just out of curiosity - how often are you seeing failures due to this error code (410)? I would think that it's rare.

[github-actions]

Hi @zhiweiv. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[devlie]

Just out of curiosity - how often are you seeing failures due to this error code (410)? I would think that it's rare.

@christothes, not super high rate, but enough to cause us troubles. It caused our services to have four sev3 incidents in the past 2 weeks alone.

[christothes]

Just out of curiosity - how often are you seeing failures due to this error code (410)? I would think that it's rare.

@christothes, not super high rate, but enough to cause us troubles. It caused our services to have four sev3 incidents in the past 2 weeks alone.

That's way more frequently than I would have guessed! We can add status 410 to the retry statuses, however we wouldn't want to extend the default retry timeout to accommodate the 70 second delay. The main reason being that in a local development scenario, we need the endpoint failure to fail fast.

There is a workaround, however. You could supply a custom retry policy to the credentialOptions and customize the behavior for your specific scenario. See these docs for details

[github-actions]

Hi @zhiweiv, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[tjrobinson]

@christothes Thanks for getting this change in. Do you know if there's a target date/release for this yet?

[christothes]

It should be released on the next monthly release, which is early next month.