Performance: Add the option to disable huffman encoding for HTTP2 in certain scenarios.

[KBaichoo]

Description:

Describe the desired behavior, what scenario it enables and how it
would be used.

Huffman encoding is a tradeoff between CPU and bandwidth / memory. While it makes sense in edge use cases, it is not as helpful with local traffic (on same machine via loopback) as in mesh usecases as there is a high bandwidth low latency channel available such that the CPU spent is not worth it.

I patched nghttp2 to disable huffman encoding and ran the following experiment on my local machine.

Setup:

Nighthawk (10 workers) <-> Envoy (1 worker) <-> 10 Go servers responding hello world

The connection from NH to Envoy is HTTP2 and likewise between Envoy and the Go servers.
Huffman encoding is thus off from Envoy -> Nighthawk and from Envoy -> Go servers.

The payload used is 100 headers totaling ~5kb for the request and 50 headers totaling 6kb for the response.

Results

Here is a graph of the latencys NH records:
We can see we hit a knee point at around 4k RPS and 4.5k RPS for huffman, no huffman respectively when queueing becomes significant and the latencies rise linearly.

Looking before the points of saturation <= 4K RPS we see that generally removing huffman encoding saves anywhere from 100-300us across various latency percentiles.

cc @jmarantz @birenroy @yanavlasov @howardjohn

[optional Relevant Links:]

Any extra documentation required to understand the issue.

Related issue: #19103

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.