Pinned Heap Constantly Grows and Never Goes Down when Buffering Feature is Enabled For The Streaming Responses

[Kharlap-Sergey]

Problem definition

Pinned Heap Constantly Grows and Never Goes Down when Buffering Feature is Enabled For The Streaming Responses

Application Used

For the test I'm using Simple Grpc Servier
My Grpc Server has two methods SimpleStreaming and SimpleStreamingWithBuffer

 public override async Task SimpleStreaming(
        SimpleRequest request,
        IServerStreamWriter<SimpleResponse> responseStream,
        ServerCallContext context)
    {
        await foreach (var response in GetResponses(request))
        {
            await responseStream.WriteAsync(response);
        }
    }

    public override async Task SimpleStreamingWithBuffer(
        SimpleRequest request,
        IServerStreamWriter<SimpleResponse> responseStream,
        ServerCallContext context)
    {
        responseStream.WriteOptions = new WriteOptions(WriteFlags.BufferHint);

        await foreach (var response in GetResponses(request).WithCancellation(context.CancellationToken))
        {
            await responseStream.WriteAsync(response);
        }
    }

Test 1 (simulate jobs) Client Used

The first test simulates a job which starts periodically.

Run client to call plain method

> dotnet GrpcDebug.ClientTest.dll --Runs=5 --RepeatsInRun=1

Run client to call method with enabled buffering

> dotnet GrpcDebug.ClientTest.dll --Runs=5 --RepeatsInRun=1 --Type=buf

Comparision

As we can see, every new job run allocates more and more on POH. The memory is not released even after 30 minutes, and new runs allocate more and more.

Test2 Client Used

The same client as for Test1, however instead of creating new connections every time we repeat with the same connection.

Run client to call plain method

> 'dotnet GrpcDebug.ClientTest.dll --Runs=1 --RepeatsInRun=5'

Run client to call method with enabled buffering

> 'dotnet GrpcDebug.ClientTest.dll --Runs=1 --RepeatsInRun=5 --Type=buf'

Comparision

As we can see, reusing the existing connection partially solves the problem, but the buffered solution still allocates a lot.
Why? It is unclear to me, to be honest.

The problem I see here, is that bad clients can break my server, and I have no options to protect against it.

If you know the reason for these awful allocations, or if there are some configurations I'm missing, you are more than welcome to share.

[Kharlap-Sergey]

Test 3 Client Used

I also tested with the client using DI DI-registered GRPC service

localBuilder.Services.AddGrpcClient<SimpleService.SimpleServiceClient>(o =>
    {
        o.Address = new Uri("http://localhost:5172");
    });

As I expected it reuses the connection, however sometime you can see a double allocation at the beginning.
> 'dotnet GrpcDebug.ClientTest.dll --Runs=1 --RepeatsInRun=5 --Type=buf'

[JamesNK]

WriteFlags.BufferHint tells the server not to send messages to the client and buffer them. They're buffered in memory. That means memory grows.

What are you expecting to happen?

[Kharlap-Sergey]

@JamesNK, Why is it constantly growing? The client app starts, does some work (server allocates X pinned memory) and shuts down. This X Pinned Memory is not released. If it is preserved for later usage, then why when In an hour another client app starts to do the same work (so the same buffer size should be enough) the server allocates the same X memory again (so now it is 2X) ? Repeat it a few more times and you used up all resources

[JamesNK]

Hmm, ok. When the client shuts down, is the server request ending? It looks like you're using the cancellation to stop the request on the server side, but it's worth double checking.

I'll try out your code in a couple of days when I have time if you haven't figured it out.

[Kharlap-Sergey]

@JamesNK, it will be great. Thanks.
I waited for 1h after the client shutdown. Tried to play with different server' and kestrel' configurations, but without any successful results.

[JamesNK]

This is an issue in Kestrel. See dotnet/aspnetcore#27394 and dotnet/aspnetcore#55490

The fix is to flush between each message. That means either not using buffer hint or periodically flushing the response yourself.