[BUG] Azure.Monitor.OpenTelemetry.AspNetCore 1.0.0-beta.6 does not work

[lmolkova]

Library name and version

Azure.Monitor.OpenTelemetry.AspNetCore 1.0.0-beta.6

Describe the bug

When using Azure.Monitor.OpenTelemetry.AspNetCore 1.0.0-beta.6 in .NET 7 minimal API ASP.NET Core app running on localhost, my application does not start within ~30 sec.

It works perfectly fine with Azure.Monitor.OpenTelemetry.AspNetCore 1.0.0-beta.5.

If I debug the application, I can see a bunch of HttpRequestException potentially trying to resolve resources

Expected behavior

Application starts and reports telemetry

Actual behavior

Application does not start

Reproduction Steps

app:

using Azure.Monitor.OpenTelemetry.AspNetCore;
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor(o => o.ConnectionString = "...");

var app = builder.Build();
app.MapGet("/ping", async () =>
{
    await new HttpClient().GetStringAsync("https://microsoft.com");
    return new { message = "hello"};
});

app.Run();

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.0.0-beta.6" />
  </ItemGroup>

</Project>

Environment

No response

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @reyang @rajkumar-rangaraj @TimothyMothra @vishweshbankwar.

[cijothomas]

Is the app not starting at all (i.e crashes with exception), or it starts and works fine, but takes much longer time?

[lmolkova]

it does not crash, I assume it'll eventually start after it exhausted all retries and timeouts trying to get IMDS from VM, but I did not wait long enough.

Either way, it seems VM resource detector is blocking and needs way smaller timeout and no retries.

[cijothomas]

Since Resource must be added before building the Providers, this is going to be an issue. Any Detector taking time will add to the startup time.... (Azure/AWS all should have same challenges..)
Don't think there is any retry logic, so it must be purely the timeout waiting for the response.

Some possible (not perfect) mitigations:

It might be possible for the distro to detect that its is running in appservice (env var check), and then do not add VM detectors.

Or modify the AzureVM detector itself to quickly bail out, similar to what legacy sdks do.

[vishweshbankwar]

open-telemetry/opentelemetry-dotnet-contrib#1307
This change is not released yet. This should help but we are still looking for ways to further reduce the start up time.

[lmolkova]

it seems there is no timeout set on that client https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.ResourceDetectors.Azure/AzureVmMetaDataRequestor.cs#L31 and the default value is 100 sec!

the IMDS endpoint is local, so it's really reliable. 1 sec would probably be enough.

It's something that would block onboarding and work in dev environment and has to be fixed.

[TimothyMothra]

open-telemetry/opentelemetry-dotnet-contrib#1307 This change is not released yet. This should help but we are still looking for ways to further reduce the start up time.

@vishweshbankwar That PR is from a month ago. Do you know when the next release will be?

[TimothyMothra]

it seems there is no timeout set on that client

Thanks for investigating @lmolkova, would you be able to submit a PR for the quick fix?

[lmolkova]

Not this week (and I'm OOf on the next one). The fix is short and simple, but the key part here is testing it - I wonder why it was not detected via automation.

[vishweshbankwar]

it seems there is no timeout set on that client

Thanks for investigating @lmolkova, would you be able to submit a PR for the quick fix?

@TimothyMothra We can get the fix done and request for release next week.

[vishweshbankwar]

@lmolkova - Distro should still work with delayed start up time, do you see different behavior?

I wonder why it was not detected via automation

Are you referring to tests?

[lmolkova]

@lmolkova - Distro should still work with delayed start up time, do you see different behavior?

I was not patient enough to wait and I assume it's unreasonable to expect production applications (not running on app serivce of AzureVM) start time to take 100 sec.

Are you referring to tests?

Yep, do we have integration tests for distro? If so, why we didn't detect it?

[TimothyMothra]

Yep, do we have integration tests for distro? If so, why we didn't detect it?

The integration tests run in an Azure VM and don't have the delay.

[TimothyMothra]

I believe this issue can be closed. The root cause was related to Timeouts for Azure VM Metadata service. This has been fixed in both the Exporter and ResourceDetector.