Memory leak introduced in 0.6.9 or 0.6.10

[zbjornson]

Since google-fluentd updated from 1.5.22-1 to 1.5.23, we're seeing a ton of retained memory (820 MB retained for ~30 MB of logs over ~7 hours). The only change in google-fluentd between those versions is upgrading fluent-plugin-google-cloud from 0.6.8 to 0.6.10.

Looking at the commit log here, it looks like this might be from another dependency (google-api-client, google-cloud-logging or gRPC?)...

(Someone else said they saw this also with 1.5.23 and not 1.5.20 in the Google Cloud Platform Slack community.)

[igorpeshansky]

@qingling128 Mind taking a look?

[qingling128]

Sure. We've noticed this memory leak issue during the recent benchmarking tests as well. Will take a look. I've also created an internal ticket to track it.

[bivald]

I'm seeing this as well on 0.6.10 where google-fluentd (1.5.23) easily consumes 1GB of memory for a low log usage server. All logs are flushed to stackdriver (visible in the log viewer)

Perhaps more troubling for us then most other customers, since we run 50+ instances with low ram (g1-small etc)

enplore@admin:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          1704       1626         77         16         46        239

enplore@admin:~$ sudo systemctl stop google-fluentd

enplore@admin:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          1704        618       1086         16         46        239

Let me know if I can help with any testing or debugging

[jrun]

To pile on here, we are seeing the same behavior. It's giving us a headache in our test environment because the instances don't have the extra memory to spare which leads to OOM errors.

We have instances in production where google-fluentd 1.5.23-1 is using 2GB+ RAM.

[qingling128]

Sorry for the inconvenience. We are actively looking into this. Will see if we can quickly locate the root cause or at least a safe dependency version to roll back.

[qingling128]

Some initial investigation and benchmarking revealed that the memory leak seems to be isolated to gRPC only (not the REST path). Note that we turned on gRPC by default in v1.5.19.

Might worth trying setting use_grpc false in the fluentd config before we find the root cause.

$ cat /etc/google-fluentd/google-fluentd.conf
...
<match **>
  @type google_cloud
  ...
  use_grpc false
  ...
</match>

[qingling128]

Update:
We were able to reproduce the fast memory leak and located the root cause being a dependency upgrade of gRPC. We've rolled back the version in #196 and ran benchmark against the new gem. The memory usage looks stable.

Will release a new gem and new package soon. Thanks for the patience.

[qingling128]

fluent-plugin-google-cloud v0.6.11 is released. Releasing google-fluentd package next.

[zbjornson]

Thanks. v1.5.24-1 looks good.