Ingester data received has increased several fold since update to 2.7

[remi-hackle]

I noticed something strange after updating tempo to 2.7.0 this time.
After updating to that version, the amount of data received by ingester increased several times.
I didn't see any major changes in the release notes for this,
so is there an option change or special configuration that I didn't notice?

Updated to tempo 2.7.0 around 12:00 on January 20, 2025
7 days worth of metrics

Rolled back to 2.6 at around 10:50 today (1/24/2024)
metric numbers since rolling back to 2.6.

[mapno]

Hi @remi-hackle. gRPC compression was disabled by default in #4429, which landed in v2.7. We found that Tempo performed better without compression between components.

[remi-hackle]

@mapno Ah, I see, thanks for the answer.

[mdisibio]

You can restore compression between distributors and ingesters with this config:

distributor:
  ingester_client:
    grpc_client_config:
      grpc_compression: "snappy"

[edgarkz]

I think the change should be highlighted ask it has high impact on cluster cost
We had a massive increase of network cost between tempo components due to 2.7 release.

[joe-elliott]

It's mentioned here:

https://grafana.com/docs/tempo/latest/release-notes/v2-7/#grpc-compression-disabled

Do you think it should be more prominent? I think the read path change is important but less impactful. Perhaps we should revert the write path change?

[edgarkz]

Hi Joe,

I believe it’s worth noting, as mentioned above, that this comes with increased network costs in large clusters. An alternative approach could be enabling compression by default, while advising that it can be turned off to reduce CPU usage—though this would impact network traffic.

In any case, for oss setups, cost allocations are crucial, and these changes need to be better communicated along with their potential consequences.

[knylander-grafana]

I'll be happy to update the docs to clarify.

[knylander-grafana]

I've added doc for gRPC compression to the Configuration doc to make it more visible: #4626