Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Settings_MaxConcurrentStreamsGet_Server test failure #27379

Closed
BrennanConroy opened this issue Oct 30, 2020 · 4 comments · Fixed by #32504
Closed

Settings_MaxConcurrentStreamsGet_Server test failure #27379

BrennanConroy opened this issue Oct 30, 2020 · 4 comments · Fixed by #32504
Labels
affected-few This issue impacts only small number of customers area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions severity-major This label is used by an internal tool task test-failure
Milestone
Test failures

Comments

@BrennanConroy
Copy link
Member 

System.Net.Http.HttpRequestException : The request was aborted.
---- System.Net.Http.Http2StreamException : The HTTP/2 server reset the stream. HTTP/2 error code 'REFUSED_STREAM' (0x7).
   at System.Net.Http.Http2Connection.ThrowRetry(String message, Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, CancellationTokenSource cts, Boolean disposeCts, Int64 timeoutTime, CancellationToken originalCancellationToken)
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Testing/src/TaskExtensions.cs:line 31
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Testing/src/TaskExtensions.cs:line 31
   at Interop.FunctionalTests.HttpClientHttp2InteropTests.Settings_MaxConcurrentStreamsGet_Server(String scheme) in /_/src/Servers/Kestrel/test/Interop.FunctionalTests/HttpClientHttp2InteropTests.cs:line 1234
--- End of stack trace from previous location ---
----- Inner Stack Trace -----

[0.001s] [TestLifetime] [Information] Starting test Settings_MaxConcurrentStreamsGet_Server-https at 2020-10-11T10:03:17
[0.002s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting starting
[0.165s] [Microsoft.Hosting.Lifetime] [Information] Now listening on: https://127.0.0.1:37301
[0.166s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Loaded hosting startup assembly Interop.FunctionalTests
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Application started. Press Ctrl+C to shut down.
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Hosting environment: Production
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Content root path: /datadisks/disk1/work/A9620907/w/B2D909A0/e/
[0.166s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting started
[0.167s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" accepted.
[0.169s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" started.
[0.230s] [Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware] [Debug] Connection "0HM3DPJFMBLMP" established using the following protocol: Tls12
[0.236s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received SETTINGS frame for stream ID 0 with length 6 and flags NONE
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 1 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 3 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 5 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 7 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 9 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 11 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.245s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 3 with length 45 and flags END_STREAM, END_HEADERS
[0.245s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 11 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream() in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 1037
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 973
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 618
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 463
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 242
[0.246s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 2.5278ms
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending RST_STREAM frame for stream ID 11 with length 4 and flags 0x0
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 7 with length 7 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 13 with length 20 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 2.4357ms
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 15 with length 20 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 13 with length 7 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 17 with length 20 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.4306ms
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 19 with length 20 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 15 with length 7 and flags END_STREAM, END_HEADERS
[0.248s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.3089ms
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 19 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream() in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 1037
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 973
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 618
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 463
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 242
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending RST_STREAM frame for stream ID 19 with length 4 and flags 0x0
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received SETTINGS frame for stream ID 0 with length 0 and flags ACK
[0.248s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.249s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 17 with length 7 and flags END_STREAM, END_HEADERS
[0.249s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 1.5675ms
[0.250s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 5 with length 7 and flags END_STREAM, END_HEADERS
[0.250s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 6.7389ms
[0.251s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 9 with length 7 and flags END_STREAM, END_HEADERS
[0.252s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 21 with length 20 and flags END_STREAM, END_HEADERS
[0.252s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.253s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 8.2426ms
[0.253s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 21 with length 7 and flags END_STREAM, END_HEADERS
[0.253s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.6910ms
[0.253s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 1 with length 7 and flags END_STREAM, END_HEADERS
[0.254s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 10.6481ms
[0.271s] [Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets] [Debug] Connection id "0HM3DPJFMBLMP" received FIN.
[0.272s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" is closed. The last processed stream ID was 21.
[0.272s] [Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets] [Debug] Connection id "0HM3DPJFMBLMP" sending FIN because: "The client closed the connection."
[0.278s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" stopped.
[0.281s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Some connections failed to close gracefully during server shutdown.
[0.286s] [TestLifetime] [Information] Finished test Settings_MaxConcurrentStreamsGet_Server-https in 0.2845283s
@BrennanConroy BrennanConroy added this to the Next sprint planning milestone Oct 30, 2020
@ghost
Copy link

ghost commented Oct 30, 2020

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

@halter73
Copy link
Member

halter73 commented Oct 31, 2020 •
edited
Loading

One thing that might help make these HttpClientHttp2InteropTests more reliable (but probably won't fix this issue) is to use SocketHttpHandler's new ConnectCallback to wire it up with Kestrel's in-memory test transport like Http2EndToEndTests does.

aspnetcore/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2EndToEndTests.cs

Lines 47 to 61 in 540680a

using var connection = server.CreateConnection();
using var socketsHandler = new SocketsHttpHandler()
{
ConnectCallback = (_, _) =>
{
if (connectionCount != 0)
{
throw new InvalidOperationException();
}
connectionCount++;
return new ValueTask<Stream>(connection.Stream);
},
};

If we do that, we should move it into InMemory.FunctionalTests.

@BrennanConroy BrennanConroy added affected-few This issue impacts only small number of customers severity-major This label is used by an internal tool task labels Nov 6, 2020 — with ASP.NET Core Issue Ranking
@BrennanConroy BrennanConroy mentioned this issue Nov 13, 2020
Closed
@BrennanConroy
Copy link
Member Author

We'll see what happens with dotnet/runtime#44669 before deciding what to do with this test.

@BrennanConroy BrennanConroy added the blocked The work on this issue is blocked due to some dependency label Nov 13, 2020
@BrennanConroy BrennanConroy removed the blocked The work on this issue is blocked due to some dependency label May 7, 2021
@BrennanConroy
Copy link
Member Author

Runtime fixed the issue, we should be able to unquarantine.

BrennanConroy added a commit that referenced this issue May 7, 2021
@BrennanConroy
Update HttpClientHttp2InteropTests.cs
ac6a89e 
Passing for past 30 days and Runtime issue that was causing it has been fixed.
#27379
@BrennanConroy BrennanConroy mentioned this issue May 7, 2021
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Jun 7, 2021
@amcasey amcasey added area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions and removed area-runtime labels Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
affected-few This issue impacts only small number of customers area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions severity-major This label is used by an internal tool task test-failure
Projects
None yet
Milestone
Test failures
Development

Successfully merging a pull request may close this issue.

Unquarantine Settings_MaxConcurrentStreamsGet_Server dotnet/aspnetcore
3 participants
@halter73 @BrennanConroy @amcasey