Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture errors with startup hook auto instrumentation #1298

Merged
merged 1 commit into from
May 27, 2021

Conversation

russcam
Copy link
Contributor

@russcam russcam commented May 6, 2021

This commit updates the startup hook auto instrumentation feature to capture thrown exceptions through the subscribed
AspNetCoreDiagnosticListener.

When an unhandled exception is thrown, ASP.NET Core's DeveloperExceptionPageMiddleware will raise a diagnostic
event with key "Microsoft.AspNetCore.Diagnostics.UnhandledException" if there is a listener listening to the event. If no listener is subscribed, the event is not captured.

Update how assemblies are loaded in the startup hook auto instrumentation. Register an event handler to the AssemblyLoadContext.Default.Resolving event to load assemblies that exist in the loader directory with matching version and public key. For assemblies starting with Elastic.Apm, that may contain DiagnosticListeners, load into AssemblyLoadContext.Default to allow listeners to register via DiagnosticListener.AllListeners.Subscribe(). For any other assemblies, load them into a separate AssemblyLoadContext to avoid version conflicts with assemblies that the application may reference. An example of the problem this avoids is System.Reflection.Metadata; The APM agent's source included version of Ben.Demystifier depends on System.Reflection.Metadata 5.0.0, but an application may reference a different version, such as ASP.NET Core 3.0, which references System.Reflection.Metadata 1.4.4.0 by default.

Add integration tests for capturing errors from startup hooks for netcoreapp3.0, netcoreapp3.1 and net5.0.

Update documentation for startup hooks to indicate that this feature is supported in .NET Core 3.0 and newer. .NET Core 2.2 is End of Life (EOL) and no longer supported by Microsoft. In testing startup hooks with .NET Core 2.2, Diagnostic Listeners do not subscribe to ASP.NET Core diagnostic events. On initial investigation, the System.Diagnostics.DiagnosticSource assembly loaded in the startup hooks is version 4.0.3.1. When listeners come to subscribe, System.Diagnostics.DiagnosticSource 4.0.4.0 that listeners are compiled against is loaded into the separate AssemblyLoadContext, which might be why listeners do not end up subscribing. Considering .NET Core 2.2 is EOL however, no further effort has been put into making this work.

Fixes #1233

This commit updates the startup hook auto instrumentation
feature to capture thrown exceptions through the subscribed
AspNetCoreDiagnosticListener.

When an unhandled exception is thrown, ASP.NET Core's
DeveloperExceptionPageMiddleware will raise a diagnostic
event with key "Microsoft.AspNetCore.Diagnostics.UnhandledException"
if there is a listener listening to the event. If no listener is
subscribed, the event is not captured.

Update how assemblies are loaded in the startup hook
auto instrumentation. Register an event handler to the
AssemblyLoadContext.Default.Resolve event to load assemblies
that exist in the loader directory with matching version and public key.
For assemblies starting with Elastic.Apm, that may contain DiagnosticListeners,
load into AssemblyLoadContext.Default to allow
listeners to register via DiagnosticListener.AllListeners.Subscribe(). For
any other assemblies, load them into a separate AssemblyLoadContext to avoid
version conflicts with assemblies that the application may reference. An example
of the problem this avoids is System.Reflection.Metadata; The APM agent's
source included version of Ben.Demystifier depends on System.Reflection.Metadata
5.0.0, but an application may reference a different version, such as ASP.NET Core 3.0,
which references System.Reflection.Metadata 1.4.4.0 by default.

Add integration tests for capturing errors from startup hooks for netcoreapp3.0,
netcoreapp3.1 and net5.0.

Update documentation for startup hooks to indicate that this feature is supported
in .NET Core 3.0 and newer. .NET Core 2.2 is End of Life (EOL) and no longer supported
by Microsoft. In testing startup hooks with .NET Core 2.2, Diagnostic Listeners do
not subscribe to ASP.NET Core diagnostic events. On initial investigation, the
System.Diagnostics.DiagnosticSource assembly loaded in the startup hooks is
version 4.0.3.1. When listeners come to subscribe, System.Diagnostics.DiagnosticSource
4.0.4.0 that listeners are compiled against is loaded into the separate AssemblyLoadContext,
which might be why listeners do not end up subscribing. Considering .NET Core 2.2 is EOL
however, no further effort has been put into making this work.

Fixes elastic#1233
@russcam russcam added bug Something isn't working v1.10.0 labels May 6, 2021
@russcam russcam requested a review from gregkalapos May 6, 2021 09:22
@apmmachine
Copy link
Contributor

apmmachine commented May 6, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Branch indexing

  • Start Time: 2021-05-27T02:20:23.321+0000

  • Duration: 46 min 56 sec

  • Commit: 5ba3052

Test stats 🧪

Test Results
Failed 14
Passed 17596
Skipped 76
Total 17686

Trends 🧪

Image of Build Times

Image of Tests

Test errors 14

Expand to view the tests failures

> Show only the first 10 test failures

Initializing / Parallel / Linux / Test / CentralConfigFetcherTests.Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_IgnoreMessageQueues_Configuration – Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests
    Expand to view the error details

     Expected collection not to be empty. 
    

    Expand to view the stacktrace

     Expected collection not to be empty.
    Stack Trace:
       at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
       at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
       at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
       at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
       at FluentAssertions.Collections.CollectionAssertions`2.NotBeEmpty(String because, Object[] becauseArgs)
       at Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_IgnoreMessageQueues_Configuration()
    Standard Output:
        Elastic APM .NET Tests> 292> Xunit> [2021-05-27 02:42:59.776 +00:00][Info] - {LoggingTestBase} Starting test: Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_IgnoreMessageQueues_Configuration...
        Elastic APM .NET Tests> 292> Xunit> [2021-05-27 02:42:59.847 +00:00][Error] - {CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `http://localhost:8200/config/v1/agents?service.name=xunit_execution_dotnet'. Apm Server base URL: `http://localhost:8200/'. WaitInterval: 5m. dbgIterationsCount: 1.
        Elastic APM .NET Tests> 292> Xunit> +-> Request:
        Elastic APM .NET Tests> 292> Xunit>     Method: GET, RequestUri: 'http://localhost:8200/config/v1/agents?service.name=xunit_execution_dotnet', Version: 1.1, Content: <null>, Headers:
        Elastic APM .NET Tests> 292> Xunit>     {
        Elastic APM .NET Tests> 292> Xunit>       User-Agent: elasticapm-dotnet/1.9.0+9cf869a228c3e5f5a50a468f5e3e25583b3c5c54
        Elastic APM .NET Tests> 292> Xunit>       User-Agent: System.Net.Http/4.700.19.56404
        Elastic APM .NET Tests> 292> Xunit>       User-Agent: .NET_Core/3.1.0
        Elastic APM .NET Tests> 292> Xunit>     }
        Elastic APM .NET Tests> 292> Xunit> +-> Response:
        Elastic APM .NET Tests> 292> Xunit>     StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StringContent, Headers:
        Elastic APM .NET Tests> 292> Xunit>     {
        Elastic APM .NET Tests> 292> Xunit>       ETag: "etag"
        Elastic APM .NET Tests> 292> Xunit>       Content-Type: text/plain; charset=utf-8
        Elastic APM .NET Tests> 292> Xunit>     }
        Elastic APM .NET Tests> 292> Xunit> +-> Response body [length: 34]:
        Elastic APM .NET Tests> 292> Xunit>     { "ignore_message_queues": "foo" }
        Elastic APM .NET Tests> 292> Xunit> +-> Exception: System.NullReferenceException: Object reference not set to an instance of an object.
        Elastic APM .NET Tests> 292> Xunit>    at Elastic.Apm.BackendComm.CentralConfig.CentralConfigFetcher.WorkLoopIteration() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/src/Elastic.Apm/BackendComm/CentralConfig/CentralConfigFetcher.cs:line 165
        Elastic APM .NET Tests> 292> Xunit> [2021-05-27 02:44:00.026 +00:00][Info] - {LoggingTestBase} Finished test: Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_IgnoreMessageQueues_Configuration 
    

Initializing / Parallel / Linux / Test / CentralConfigFetcherTests.Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_Logger_That_Is_ILogLevelSwitchable – Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests
    Expand to view the error details

     Expected testLogger.LogLevelSwitch.Level to be Error, but found Trace. 
    

    Expand to view the stacktrace

     Expected testLogger.LogLevelSwitch.Level to be Error, but found Trace.
    Stack Trace:
       at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
       at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
       at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
       at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
       at FluentAssertions.Primitives.ObjectAssertions.Be(Object expected, String because, Object[] becauseArgs)
       at Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_Logger_That_Is_ILogLevelSwitchable() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.Tests/BackendCommTests/CentralConfig/CentralConfigFetcherTests.cs:line 78
    Standard Output:
        Elastic APM .NET Tests> 249> Xunit> [2021-05-27 02:46:21.380 +00:00][Info] - {LoggingTestBase} Starting test: Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_Logger_That_Is_ILogLevelSwitchable...
        Elastic APM .NET Tests> 249> Xunit> [2021-05-27 02:47:21.548 +00:00][Info] - {LoggingTestBase} Finished test: Elastic.Apm.Tests.BackendCommTests.CentralConfig.CentralConfigFetcherTests.Should_Update_Logger_That_Is_ILogLevelSwitchable 
    

Initializing / Parallel / Windows .NET Core / Test / ElasticsearchTests.Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests.Elasticsearch_Span_Does_Not_Have_Http_Child_Span – Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests
    Expand to view the error details

     System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
    ---- System.TimeoutException : The operation has timed out. 
    

    Expand to view the stacktrace

     System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
    ---- System.TimeoutException : The operation has timed out.
    Stack Trace:
       at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
       at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
       at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
       at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
       at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
       at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
       at Elastic.Apm.Elasticsearch.Tests.ElasticsearchFixture.InitializeAsync() in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1298\apm-agent-dotnet\test\Elastic.Apm.Elasticsearch.Tests\ElasticsearchFixture.cs:line 30
    ----- Inner Stack Trace -----
       at System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32 timeout, CancellationToken cancellationToken, Int32 startTime)
       at System.IO.Pipes.NamedPipeClientStream.<>c__DisplayClass20_0.<ConnectAsync>b__0()
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
    --- End of stack trace from previous location where exception was thrown ---
       at Docker.DotNet.DockerClient.<>c__DisplayClass6_0.<<-ctor>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
       at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
       at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
       at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
       at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
       at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
       at DotNet.Testcontainers.Client.DockerApiClient..cctor() 
    

Initializing / Parallel / Linux / Test / ElasticsearchTests.Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests.Elasticsearch_Span_Does_Not_Have_Http_Child_Span – Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests
    Expand to view the error details

     Expected searchResponse.Success to be true, but found False. 
    

    Expand to view the stacktrace

     Expected searchResponse.Success to be true, but found False.
    Stack Trace:
       at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message) in C:\projects\fluentassertions-vf06b\Src\FluentAssertions\Execution\XUnit2TestFramework.cs:line 32
       at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc) in C:\projects\fluentassertions-vf06b\Src\FluentAssertions\Execution\AssertionScope.cs:line 181
       at FluentAssertions.Primitives.BooleanAssertions.BeTrue(String because, Object[] becauseArgs) in C:\projects\fluentassertions-vf06b\Src\FluentAssertions\Primitives\BooleanAssertions.cs:line 59
       at Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests.Elasticsearch_Span_Does_Not_Have_Http_Child_Span() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.Elasticsearch.Tests/ElasticsearchTests.cs:line 39
    --- End of stack trace from previous location where exception was thrown --- 
    

Initializing / Parallel / Linux / Test / EfCoreWithMsSqlTests.Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests.BothEfCoreAndSqlClientCapturingActive – Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests
    Expand to view the error details

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnection.cs:line 615
       at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 1183
       at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 373
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1741
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1408
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1311
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 439
       at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlConnectionFactory.cs:line 135
       at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Common\src\Microsoft\Data\ProviderBase\DbConnectionFactory.cs:line 111
       at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 695
       at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1643
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1102
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1069
       at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionFactory.cs:line 154
       at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Common\src\Microsoft\Data\ProviderBase\DbConnectionInternal.cs:line 328
       at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlConnection.cs:line 1467
       at Microsoft.Data.SqlClient.SqlConnection.Open() in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlConnection.cs:line 981
       at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
       at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerDatabaseCreator.<>c__DisplayClass18_0.<Exists>b__0(DateTime giveUp)
       at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c__DisplayClass12_0`2.<Execute>b__0(DbContext c, TState s)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
       at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation, Func`2 verifySucceeded)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerDatabaseCreator.Exists(Boolean retryOnNotExists)
       at Microsoft.EntityFrameworkCore.Storage.RelationalDatabaseCreator.EnsureCreated()
       at Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests.BothEfCoreAndSqlClientCapturingActive() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/EfCoreWithMsSqlTests.cs:line 53
    Standard Output:
        [2021-05-27 02:37:11.927 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:37:11.955 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:37:11.969 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:37:11.970 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted. 
    

Initializing / Parallel / Linux / Test / SqlClientListenerTests.Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan – Elastic.Apm.SqlClient.Tests.SqlClientListenerTests
    Expand to view the error details

     System.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     System.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)
       at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.<>c__DisplayClass8_0.<<SqlClientDiagnosticListener_ShouldCaptureSpan>b__0>d.MoveNext() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 75
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan(String providerName, Func`2 connectionCreator) in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 85
    --- End of stack trace from previous location where exception was thrown ---
    Standard Output:
        [2021-05-27 02:37:29.127 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:37:29.128 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:37:29.129 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:37:29.129 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted.
        System.Data.SqlClient 
    

Initializing / Parallel / Linux / Test / SqlClientListenerTests.Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan – Elastic.Apm.SqlClient.Tests.SqlClientListenerTests
    Expand to view the error details

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnection.cs:line 615
       at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 1183
       at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 373
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1741
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1408
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1311
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 439
       at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlConnectionFactory.cs:line 135
       at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Common\src\Microsoft\Data\ProviderBase\DbConnectionFactory.cs:line 111
       at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 695
       at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1643
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1102
       at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen() in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1011
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.<>c__DisplayClass8_0.<<SqlClientDiagnosticListener_ShouldCaptureSpan>b__0>d.MoveNext() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 75
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan(String providerName, Func`2 connectionCreator) in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 85
    --- End of stack trace from previous location where exception was thrown ---
    Standard Output:
        [2021-05-27 02:37:44.116 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:37:44.116 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:37:44.117 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:37:44.117 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted.
        Microsoft.Data.SqlClient 
    

Initializing / Parallel / Linux / Test / SqlClientListenerTests.Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient – Elastic.Apm.SqlClient.Tests.SqlClientListenerTests
    Expand to view the error details

     System.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     System.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)
       at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.<>c__DisplayClass9_0.<<SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient>b__0>d.MoveNext() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 130
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient(String providerName, Func`2 connectionCreator) in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 147
    --- End of stack trace from previous location where exception was thrown ---
    Standard Output:
        [2021-05-27 02:37:58.707 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:37:58.720 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:37:58.720 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:37:58.720 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted.
        System.Data.SqlClient 
    

Initializing / Parallel / Linux / Test / SqlClientListenerTests.Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient – Elastic.Apm.SqlClient.Tests.SqlClientListenerTests
    Expand to view the error details

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnection.cs:line 615
       at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 1183
       at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\TdsParser.cs:line 373
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1741
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1408
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 1311
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, SqlAuthenticationProviderManager sqlAuthProviderManager) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlInternalConnectionTds.cs:line 439
       at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\SqlClient\SqlConnectionFactory.cs:line 135
       at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Common\src\Microsoft\Data\ProviderBase\DbConnectionFactory.cs:line 111
       at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 695
       at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1643
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1102
       at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen() in E:\SqlClientInternal\agent-1\_work\5\s\src\Microsoft.Data.SqlClient\netcore\src\Microsoft\Data\ProviderBase\DbConnectionPool.cs:line 1011
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.<>c__DisplayClass9_0.<<SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient>b__0>d.MoveNext() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 130
    --- End of stack trace from previous location where exception was thrown ---
       at Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient(String providerName, Func`2 connectionCreator) in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlClientListenerTests.cs:line 147
    --- End of stack trace from previous location where exception was thrown ---
    Standard Output:
        [2021-05-27 02:38:13.386 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:38:13.387 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:38:13.387 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:38:13.387 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted.
        Microsoft.Data.SqlClient 
    

Initializing / Parallel / Linux / Test / EfCoreWithMsSqlTests.Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests.BothEfCoreAndSqlClientCapturingActive – Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests
    Expand to view the error details

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server) 
    

    Expand to view the stacktrace

     Microsoft.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
    Stack Trace:
       at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType, SqlAuthenticationProviderManager sqlAuthProviderManager)
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
       at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, SqlAuthenticationProviderManager sqlAuthProviderManager)
       at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
       at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
       at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
       at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
       at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
       at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
       at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
       at Microsoft.Data.SqlClient.SqlConnection.Open()
       at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
       at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerDatabaseCreator.<>c__DisplayClass18_0.<Exists>b__0(DateTime giveUp)
       at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.<>c__DisplayClass12_0`2.<Execute>b__0(DbContext c, TState s)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
       at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation, Func`2 verifySucceeded)
       at Microsoft.EntityFrameworkCore.ExecutionStrategyExtensions.Execute[TState,TResult](IExecutionStrategy strategy, TState state, Func`2 operation)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerDatabaseCreator.Exists(Boolean retryOnNotExists)
       at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerDatabaseCreator.Exists()
       at Microsoft.EntityFrameworkCore.Storage.RelationalDatabaseCreator.EnsureCreated()
       at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.EnsureCreated()
       at Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests.BothEfCoreAndSqlClientCapturingActive() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/EfCoreWithMsSqlTests.cs:line 52
    Standard Output:
        [2021-05-27 02:39:01.138 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The service name will be automatically discovered.
        [2021-05-27 02:39:01.220 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service name. The automatically discovered service name is xunit_execution_dotnet
        [2021-05-27 02:39:01.226 +00:00][Info] - {AbstractConfigurationReader (MockConfigSnapshot)} The agent was started without a service version. The service version will be automatically discovered.
        [2021-05-27 02:39:01.234 +00:00][Warning] - {AbstractConfigurationReader (MockConfigSnapshot)} Failed to discover service version, the service version will be omitted. 
    

Steps errors 5

Expand to view the steps failures

Windows Batch Script
  • Took 0 min 4 sec . View more details on here
  • Description: .ci/windows/msbuild.bat
Archive the artifacts
  • Took 0 min 1 sec . View more details on here
  • Description: [2021-05-27T02:37:13.419Z] Archiving artifacts script returned exit code 1
Test & coverage
  • Took 14 min 35 sec . View more details on here
  • Description: .ci/windows/test.bat
Archive the artifacts
  • Took 0 min 0 sec . View more details on here
  • Description: [2021-05-27T02:53:32.475Z] Archiving artifacts withAzureCredentials: error hudson.AbortException: s
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: withAzureCredentials: error hudson.AbortException: script returned exit code 1

Log output

Expand to view the last 100 lines of log output

[2021-05-27T03:05:19.014Z] Starting copy source file Elastic.Apm/Libraries/Newtonsoft.Json/Converters/StringEnumConverter.cs. 
[2021-05-27T03:05:19.091Z] Copied Elastic.Apm/Libraries/Newtonsoft.Json/Converters/StringEnumConverter.cs. 
[2021-05-27T03:05:19.091Z] Starting copy source file Elastic.Apm/Libraries/Newtonsoft.Json/Utilities/DynamicProxyMetaObject.cs. 
[2021-05-27T03:05:19.171Z] Copied Elastic.Apm/Libraries/Newtonsoft.Json/Utilities/DynamicProxyMetaObject.cs. 
[2021-05-27T03:05:19.171Z] Starting copy source file Libraries/Newtonsoft.Json/Converters/StringEnumConverter.cs. 
[2021-05-27T03:05:19.265Z] Copied Libraries/Newtonsoft.Json/Converters/StringEnumConverter.cs. 
[2021-05-27T03:05:19.265Z] Starting copy source file Libraries/Newtonsoft.Json/Serialization/JsonPrimitiveContract.cs. 
[2021-05-27T03:05:19.338Z] Copied Libraries/Newtonsoft.Json/Serialization/JsonPrimitiveContract.cs. 
[2021-05-27T03:05:19.338Z] Starting copy source file Elastic.Apm/Config/AbstractConfigurationReader.cs. 
[2021-05-27T03:05:19.430Z] Copied Elastic.Apm/Config/AbstractConfigurationReader.cs. 
[2021-05-27T03:05:19.430Z] Starting copy source file Elastic.Apm/Filters/SpanStackTraceCapturingFilter.cs. 
[2021-05-27T03:05:19.511Z] Copied Elastic.Apm/Filters/SpanStackTraceCapturingFilter.cs. 
[2021-05-27T03:05:19.511Z] Starting copy source file Elastic.Apm/Helpers/ContractExtensions.cs. 
[2021-05-27T03:05:19.578Z] Copied Elastic.Apm/Helpers/ContractExtensions.cs. 
[2021-05-27T03:05:19.579Z] Starting copy source file Libraries/Newtonsoft.Json/Schema/JsonSchemaBuilder.cs. 
[2021-05-27T03:05:20.239Z] Copied Libraries/Newtonsoft.Json/Schema/JsonSchemaBuilder.cs. 
[2021-05-27T03:05:20.239Z] Starting copy source file Report/Serialization/MetricSetConverter.cs. 
[2021-05-27T03:05:20.309Z] Copied Report/Serialization/MetricSetConverter.cs. 
[2021-05-27T03:05:20.309Z] Starting copy source file Libraries/Newtonsoft.Json/Utilities/DynamicProxy.cs. 
[2021-05-27T03:05:20.379Z] Copied Libraries/Newtonsoft.Json/Utilities/DynamicProxy.cs. 
[2021-05-27T03:05:20.379Z] Starting copy source file Cloud/AzureCloudMetadataProvider.cs. 
[2021-05-27T03:05:20.443Z] Copied Cloud/AzureCloudMetadataProvider.cs. 
[2021-05-27T03:05:20.443Z] Starting copy source file Api/Context.cs. 
[2021-05-27T03:05:20.509Z] Copied Api/Context.cs. 
[2021-05-27T03:05:20.509Z] Starting copy source file Elastic.Apm/Libraries/Newtonsoft.Json/Serialization/DefaultContractResolver.cs. 
[2021-05-27T03:05:20.607Z] Copied Elastic.Apm/Libraries/Newtonsoft.Json/Serialization/DefaultContractResolver.cs. 
[2021-05-27T03:05:20.607Z] Starting copy source file Elastic.Apm/Logging/LogValuesFormatter.cs. 
[2021-05-27T03:05:20.674Z] Copied Elastic.Apm/Logging/LogValuesFormatter.cs. 
[2021-05-27T03:05:20.674Z] Starting copy source file Config/AbstractConfigurationReader.cs. 
[2021-05-27T03:05:20.753Z] Copied Config/AbstractConfigurationReader.cs. 
[2021-05-27T03:06:14.255Z] [Checks API] No suitable checks publisher found.
[2021-05-27T03:06:14.641Z] [INFO] getVaultSecret: Getting secrets
[2021-05-27T03:06:14.699Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-27T03:06:15.186Z] Running in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/apm-agent-dotnet
[2021-05-27T03:06:15.213Z] [INFO] Codecov: Getting branch ref...
[2021-05-27T03:06:15.505Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-05-27T03:06:16.072Z] [INFO] Codecov: Sending data...
[2021-05-27T03:06:16.634Z] + echo curl -sSLo codecov.sh https://codecov.io/bash
[2021-05-27T03:06:16.635Z] curl -sSLo codecov.sh https://codecov.io/bash
[2021-05-27T03:06:16.993Z] + echo bash codecov.sh
[2021-05-27T03:06:16.993Z] bash codecov.sh
[2021-05-27T03:06:17.369Z] Archiving artifacts
[2021-05-27T03:06:17.435Z] java.lang.InterruptedException: no matches found within 10000
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:3069)
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2978)
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2929)
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3317)
[2021-05-27T03:06:17.435Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to apm-ci-immutable-ubuntu-1804-1622082030932593985
[2021-05-27T03:06:17.435Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2021-05-27T03:06:17.435Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2021-05-27T03:06:17.435Z] 		at hudson.remoting.Channel.call(Channel.java:1001)
[2021-05-27T03:06:17.435Z] 		at hudson.FilePath.act(FilePath.java:1159)
[2021-05-27T03:06:17.435Z] 		at hudson.FilePath.act(FilePath.java:1148)
[2021-05-27T03:06:17.435Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2927)
[2021-05-27T03:06:17.435Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
[2021-05-27T03:06:17.435Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:99)
[2021-05-27T03:06:17.435Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:69)
[2021-05-27T03:06:17.435Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2021-05-27T03:06:17.435Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2021-05-27T03:06:17.435Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-05-27T03:06:17.435Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-05-27T03:06:17.435Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-05-27T03:06:17.435Z] Caused: hudson.FilePath$TunneledInterruptedException
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3319)
[2021-05-27T03:06:17.435Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2021-05-27T03:06:17.435Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2021-05-27T03:06:17.435Z] 	at hudson.remoting.Request$2.run(Request.java:375)
[2021-05-27T03:06:17.435Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-05-27T03:06:17.435Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:118)
[2021-05-27T03:06:17.435Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath.act(FilePath.java:1161)
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath.act(FilePath.java:1148)
[2021-05-27T03:06:17.435Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2927)
[2021-05-27T03:06:17.435Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
[2021-05-27T03:06:17.435Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:99)
[2021-05-27T03:06:17.435Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:69)
[2021-05-27T03:06:17.435Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-05-27T03:06:17.435Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-05-27T03:06:17.435Z] 	at java.lang.Thread.run(Thread.java:748)
[2021-05-27T03:06:17.435Z] No artifacts found that match the file pattern "/var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298/**/MSBuild_*.failure.txt". Configuration error?
[2021-05-27T03:06:17.697Z] Stage "Release to feedz.io" skipped due to earlier failure(s)
[2021-05-27T03:06:17.757Z] Stage "Release" skipped due to earlier failure(s)
[2021-05-27T03:06:17.787Z] Stage "Release" skipped due to earlier failure(s)
[2021-05-27T03:06:17.844Z] Stage "Release" skipped due to earlier failure(s)
[2021-05-27T03:06:17.948Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2021-05-27T03:06:17.978Z] Stage "AfterRelease" skipped due to earlier failure(s)
[2021-05-27T03:06:18.483Z] Running on Jenkins in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1298
[2021-05-27T03:06:18.544Z] [INFO] getVaultSecret: Getting secrets
[2021-05-27T03:06:18.598Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-27T03:06:19.678Z] + chmod 755 generate-build-data.sh
[2021-05-27T03:06:19.678Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1298/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1298/runs/6 FAILURE 2756089
[2021-05-27T03:06:19.678Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1298/runs/6/steps/?limit=10000 -o steps-info.json
[2021-05-27T03:06:21.022Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1298/runs/6/tests/?status=FAILED -o tests-errors.json
[2021-05-27T03:06:24.487Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1298/runs/6/log/ -o pipeline-log.txt

Copy link
Contributor

@gregkalapos gregkalapos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure this fixes #1233 - or changes anything around that.

I did some tests to reproduce the original problem (no success so far) and documented my findings here: #1233 (comment).

If you add the new test Auto_Instrument_With_StartupHook_Should_Capture_Error (with throw new Exception("..."); to Elastic.Apm.StartupHook.Sample) on master, it'll be green. With my testing today and this fact I think the original problem is more like an edge case.

I think the main change here is that we load dependencies into an extra AssemblyLoadContext - I think that does not change how the ASP.NET Core diagnostic source events are sent once we already subscribed (and subscribing worked before as well).

I added some comments regarding this change below.

@@ -37,23 +37,9 @@ private static string AssemblyDirectory
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static string AssemblyDirectory can be removed as well

@@ -20,7 +20,7 @@
namespace Elastic.Apm.StartupHook.Loader
{
/// <summary>
/// Loads the agent assemblies, its dependent assemblies and starts it
/// Starts the agent
/// </summary>
internal class Loader
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change the name of the type and the whole project isn't very accurate - it's not really loading things anymore, it only subscribes. It's not a big deal, and I don't have super strong opinion, but wanted to note this. Maybe we could consider renaming.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading the loader assembly and invoking the Initialize() method causes the Elastic.Apm.* assemblies it references to be loaded, though this is implicit now rather than being explicitly loaded. Thoughts on a better name?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on a better name?

Not really - and you have a good point, it still triggers the agent dlls to be loaded. Let's leave it for now.

{
// load Elastic.Apm assemblies with the default assembly load context, to allow DiagnosticListeners to subscribe.
// For all other dependencies, load with a separate load context to not conflict with application dependencies.
return name.Name.StartsWith("Elastic.Apm", StringComparison.Ordinal)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we totally sure about this? 🤔

This means if we have let's say Elstic.Apm.X which depends on assembly Y, we load Elastic.Apm.X into the default context, but load Y into the custom context - in this case Elastic.Apm.X would have missing typerefs in the default load context.

This seems to run and with our current references it does not seem to have any problems, but this looks a bit strange to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two problems that this is attempting to address:

  1. The Elastic.Apm assemblies that contain Diagnostic Listeners must be loaded into the default load context to be able to subscribe to DiagnosticListeners.AllListeners and observe diagnostic events - subscription will not occur if loaded into a different load context.
  2. Dependencies of Elastic.Apm assemblies cannot version conflict with dependencies of the application. In testing and debugging, the dependencies of the internalized Ben.Demystifier, namely, System.Reflection.Metadata, can very likely be a different version to the one referenced by the application, which was true for ASP.NET Core 3.0 if I recall.

Are there additional tests that can be written to provide more assurance in this approach?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there additional tests that can be written to provide more assurance in this approach?

I did some manual testing and so far everything worked. Maybe later we could add more tests where:

  • We make sure we also trigger stack trace capturing (e.g. making an outgoing HTTP call in the sample app)
  • Where we have agent assemblies with additional dependencies (e.g. using elastisearch-net and make the app depend on a different version than the agent)

I think those will trigger code paths that are critical. I did those tests manually and those worked for me.

private static void LoadAssembliesFromLoaderDirectory(string loaderDirectory)
{
var context = new ElasticApmAssemblyLoadContext();
AssemblyLoadContext.Default.Resolving += (_, name) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously when we had Newtonsoft.Json and Ben.Demystifier as normal dependencies, often I ran into stackoverflow exceptions with this. Seems working now, just noting it here.

@russcam
Copy link
Contributor Author

russcam commented May 26, 2021

I'm not really sure this fixes #1233 - or changes anything around that.

When I first started looking at this, there were a couple of runs with ASP.NET Core 3.0 where an error was not captured. I don't recall being able to replicate, but I did find an issue with a TypeLoadException thrown when attempting to load the agent referenced version of System.Reflection.Metadata, to demystify a stack trace, which led to the loading changes in this PR.

@gregkalapos
Copy link
Contributor

We discussed this in a meeting - we agreed to merge this as it is.

The TypeLoadException related to Ben.Demystifier is a real issue which we should address. We (or rather I :) ) still don't fully see if and how this changes #1233 - but we can follow up on that.

Merging this in now.

@gregkalapos gregkalapos merged commit 8dd60c4 into elastic:master May 27, 2021
v1v added a commit to v1v/apm-agent-dotnet that referenced this pull request Jun 8, 2021
…u-20

* upstream/master: (21 commits)
  Prefer W3C traceparent over elastic-apm-traceparent (elastic#1302)
  fix spacing and cross references in docs (elastic#1328)
  Update README (elastic#1325)
  Mark MicrosoftAzureBlobStorageTracer internal (elastic#1326)
  Update docs (elastic#1327)
  Update context.destination.address (elastic#1324)
  synchronize json schema specs (elastic#1320)
  Don't package Elastic.Apm.Specification (elastic#1316)
  Update setup.asciidoc (elastic#1318)
  Prepare release v.1.10.0 (elastic#1314)
  Fix nullref in Elastic.Apm.Extensions.Logging (elastic#1311)
  Capture errors with startup hook auto instrumentation (elastic#1298)
  Use Logger to log exception in AgentComponents initialization (elastic#1305)
  fix: use .NET native SDK for build and test (elastic#1301)
  Skip running Elasticsearch docker test when docker not available (elastic#1312)
  Use TraceLogger as default logger in ASP.NET Full Framework (elastic#1288)
  Create receive messaging span when inside transaction (elastic#1308)
  Fix SanitizeFieldNamesTests (elastic#1299)
  Do not capture HTTP child spans for Elasticsearch (elastic#1306)
  Use storage account in destination.service.resource (elastic#1284)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agent-dotnet bug Something isn't working v1.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Startup hook setup + ASP.NET Core: HTTP request with unhandled exception is not captured
3 participants