x/net/http2: frameScratchBuffer can consume lots of memory for idle conns

[zoic21]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build656777711=/tmp/go-build -gno-record-gcc-switches"

What did you do?

To begin I'am new in goland community so maybe it's an issue in my side... I use this project https://github.com/mmatczuk/go-http-tunnel to make a http2 reverse proxy. I test with about 700 connected clients (but with 0 I get same issue). Every request grow up memory, after some analyse with pprof I got :

(pprof) top
Showing nodes accounting for 449.13MB, 98.24% of 457.15MB total
Dropped 42 nodes (cum <= 2.29MB)
Showing top 10 nodes out of 14
      flat  flat%   sum%        cum   cum%
  446.12MB 97.59% 97.59%   446.12MB 97.59%  golang.org/x/net/http2.(*ClientConn).frameScratchBuffer
    2.51MB  0.55% 98.14%     2.51MB  0.55%  bufio.NewWriterSize (inline)
    0.50MB  0.11% 98.24%     3.01MB  0.66%  golang.org/x/net/http2.(*Transport).newClientConn
         0     0% 98.24%     2.51MB  0.55%  bufio.NewWriter (inline)
         0     0% 98.24%     5.01MB  1.10%  crypto/tls.(*Conn).Handshake
         0     0% 98.24%     5.01MB  1.10%  crypto/tls.(*Conn).serverHandshake
         0     0% 98.24%     5.01MB  1.10%  crypto/tls.(*serverHandshakeStateTLS13).handshake

And deeper :

(pprof) list frameScratchBuffer
Total: 457.15MB
ROUTINE ======================== golang.org/x/net/http2.(*ClientConn).frameScratchBuffer in /root/go/src/golang.org/x/net/http2/transport.go
  446.12MB   446.12MB (flat, cum) 97.59% of Total
         .          .    884:                   cc.mu.Unlock()
         .          .    885:                   return buf[:size]
         .          .    886:           }
         .          .    887:   }
         .          .    888:   cc.mu.Unlock()
  446.12MB   446.12MB    889:   return make([]byte, size)
         .          .    890:}
         .          .    891:
         .          .    892:func (cc *ClientConn) putFrameScratchBuffer(buf []byte) {
         .          .    893:   cc.mu.Lock()
         .          .    894:   defer cc.mu.Unlock()

Same request few minute later show 517.31MB (I always request the http2 server)

What did you expect to see?

Stabilization of memory occupancy

What did you see instead?

Memory grows without stopping

I discover goland univers and I hope that it's a bad analysis on my part but if someone cloud help me it's with pleasure.

Thank in advance

[fraenkel]

Every new connection has its own set of scratch buffers which can be at most 4 x 512KB.

[zoic21]

Ok so If I understand max memory should be arround 2Mo*number client ? I will make some test to see that.

It's possible to change this buffer to 2 * 512KB for exemple ?

[fraenkel]

Sorry, its hard coded.
https://github.com/golang/net/blob/master/http2/transport.go#L895

[andybons]

Will await test from @zoic21

[zoic21]

After 1 day test with 700 client with 5 requetes in parrallele for 10 hours in continous, memory do that :

I think it's ok. We can see that the memory grows very quickly at the beginning (surely the time that the buffers fill up) and that thereafter it calms down until stabilizing around 1.1go.

But if someday someone add I option to change maxBuffer I'll take it, or maybe a system to `remove old client buffer. For information what I have now on go pprof :

Showing nodes accounting for 1205.87MB, 97.59% of 1235.60MB total
Dropped 81 nodes (cum <= 6.18MB)
Showing top 10 nodes out of 27
      flat  flat%   sum%        cum   cum%
 1198.35MB 96.98% 96.98%  1198.35MB 96.98%  golang.org/x/net/http2.(*ClientConn).frameScratchBuffer
    7.03MB  0.57% 97.55%     7.03MB  0.57%  golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry
    0.50MB  0.04% 97.59%    10.54MB  0.85%  golang.org/x/net/http2.(*Transport).newClientConn
         0     0% 97.59%     7.01MB  0.57%  crypto/tls.(*Conn).Handshake
         0     0% 97.59%     7.01MB  0.57%  crypto/tls.(*Conn).serverHandshake
         0     0% 97.59%     7.01MB  0.57%  crypto/tls.(*serverHandshakeStateTLS13).handshake
         0     0% 97.59%     7.17MB  0.58%  github.com/jeedom/gotunnel.(*Server).ServeHTTP
         0     0% 97.59%    18.04MB  1.46%  github.com/jeedom/gotunnel.(*Server).handleClient
         0     0% 97.59%    10.54MB  0.85%  github.com/jeedom/gotunnel.(*connPool).AddConn
         0     0% 97.59%     6.67MB  0.54%  github.com/jeedom/gotunnel.transfer
(pprof) list frameScratchBuffer
Total: 1.21GB
ROUTINE ======================== golang.org/x/net/http2.(*ClientConn).frameScratchBuffer in /root/go/src/golang.org/x/net/http2/transport.go
    1.17GB     1.17GB (flat, cum) 96.98% of Total
         .          .    884:                   cc.mu.Unlock()
         .          .    885:                   return buf[:size]
         .          .    886:           }
         .          .    887:   }
         .          .    888:   cc.mu.Unlock()
    1.17GB     1.17GB    889:   return make([]byte, size)
         .          .    890:}
         .          .    891:
         .          .    892:func (cc *ClientConn) putFrameScratchBuffer(buf []byte) {
         .          .    893:   cc.mu.Lock()
         .          .    894:   defer cc.mu.Unlock()

So for me it's ok after understand how it's work, we can close the issue (sorry for the inconvenience) if everybody is ok.

[fraenkel]

Given go-http-tunnel has vendored the http2 code, you could experiment and adjust the value. I would however be curious how it affects throughput as well as memory.

[zoic21]

I already change that by last version of net/http2, my go.mod :

module github.com/jeedom/gotunnel

go 1.14

require (
	github.com/calmh/luhn v2.0.0+incompatible
	github.com/felixge/tcpkeepalive v0.0.0-20160804073959-5bb0b2dea91e
	github.com/go-sql-driver/mysql v1.5.0
	github.com/inconshreveable/go-vhost v0.0.0-20160627193104-06d84117953b
	golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
	golang.org/x/text v0.1.1-0.20171102192421-88f656faf3f3 // indirect
	gopkg.in/yaml.v2 v2.2.2
	github.com/golang/mock v1.2.0
	github.com/cenkalti/backoff v2.1.1+incompatible
	)

Same result

[neild]

Every http2.ClientConn keeps a cache of up to four scratch buffers, with each scratch buffer being up to 512KiB in size. Each in-flight request requires a scratch buffer. Scratch buffers are allocated as needed, so the steady-state per-ClientConn memory usage is a function of the maximum number of simultaneous requests made.

The graph in #38049 (comment) looks like what I'd expect from this implementation: Initial growth as buffers are allocated and retained, eventually leveling off once every conn has allocated all its buffers.

So I don't believe there is a memory leak here. (If you leak ClientConns, then you're potentially leaking a lot of memory in scratch buffers, though, so definitely don't do that.)

The per-ClientConn buffer freelist will waste a lot of memory on a server with many connections that are usually idle but occasionally bursty. Perhaps we should have a global sync.Pool instead, although this gets tricky because the maximum buffer size varies by ClientConn.

The variable buffer size is something else that might be worth looking at. The buffer is sized to min(peer's max frame size, 512KiB), which means a ClientConn to a peer with the spec default of 16KiB will consume vastly less memory than one to a peer with a default of >= 512KiB.

There are a few different approaches we could take and I don't have a good sense yet for which one I favor, but the current state where idle ClientConns consume 64-2048KiB of memory doesn't seem like a good one.

[DeepAQ]

In current http/2 server implementation, the default value of MaxReadFrameSize is 1MiB, and the server will send this value as MAX_FRAME_SIZE to clients in setting frames.

As a result, the frame buffer size is always 512KiB at the client side, which means the total buffer size per client connection can reach 2MiB (there can be 4 cached buffers). This is not very friendly to devices with lower size of RAM.

Can we make the two constants (maxAllocFrameSize and maxBufs) adjustable for http/2 clients?

[neild]

I think we should just have reasonable behavior here without requiring users to fiddle around with tuning knobs to avoid wasting a bunch of idle memory.

This probably means having a single shared buffer pool rather than maintaining a per-ClientConn pool. We can't safely just use a sync.Pool, however, because buffer sizes vary. Something along the lines of @dsnet's proposal for a pool of heterogenously-sized buffers seems like it would work.

[gopherbot]

Change https://golang.org/cl/325169 mentions this issue: http2: remove the frameScratchBuffer

[adranwit]

I believe there is somewhere memory leak on the HTTP2, client, my staging boxes is out of memory after 2 hours, where frameScratchBuffer can amass up to 4GB+ memory, tried various think and failed to see what's going on,
once switch to HTTP 1.0 application does not exhibit any memory issue.
Could you please reopen the issue and help:
For reference:
client implementation:
https://github.com/viant/mly/tree/main/shared/client
server implementation:
https://github.com/viant/mly/tree/main/service/endpoint