-
Notifications
You must be signed in to change notification settings - Fork 258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Server terminates with "out of memory" periodically #762
Comments
@vlidholt Have you had some time to look at this maybe? This issue is very critical for us as it prevents the server from scaling in any way, these "out of memory" crashes happen with the stable memory usage. Please let me know if there's anything I can do, I'd be happy to provide any more details about this issue. |
Hi @bselwe! I tried to reproduce this issue, but couldn't. Do you have a minimal test case? That would help immensely. I have a Pixorama server which has been running for months without a restart, and the memory profile on that looks very stable. There are serval thousands of connections made to the server. |
@bselwe a thought. Have you tried running the server with another value for |
Possibly related? See especially this comment: |
I did some investigation on this issue, there seem to be some weird interplay between docker and dart with websockets. I'm not able to reproduce this outside of a Docker container. |
This problem has now been resolved in the dart SDK, meaning next dart version should resolve this bug. Closing this issue as resolved with that fix. |
Hey @vlidholt, we're using Serverpod with around 200 daily users. Each user is connected to Serverpod through the WebSocket connection and we are running periodic future calls (every 1 minute per each user). We've been experiencing "Out of memory" crashes every few hours, consistently. The server is running in a Docker container on AWS, with Dart 2.19.2, Serverpod 0.9.18, and
--old_gen_heap_size=0
to encourage garbage collection more aggresively.One thing that we managed to reproduce locally is the increasing memory usage of the long-running server. We found out that it was due to a large number of allocations of
_ZLibInflateFilter
and_ZLibDeflateFilter
instances, which are related to compressions of the WebSocket messages. This happens when reconnecting WebSocket clients frequently. See below:Incresing memory usage due to WebSocket compressions
By disabling the compression of WebSocket messages, we were able to fix the above issue and get stable memory usage of the server. It's just a workaround though, ideally, the server's memory usage should be stable with the compression enabled. See below:
Memory usage graph with WebSocket compression enabled
Memory usage graph with WebSocket compression disabled (fix)
With the WebSocket compression enabled, the server crashed periodically at around 25-30% of the total available memory. With the disabled WebSocket compression we are getting stable memory usage, but the "Out of memory" failures still seem to occur periodically, every 2-3 hours. These failures happen in the main isolate and we aren't able to reproduce them locally. The issue is most likely not related to WebSocket compressions as the "Out of memory" exception occurs in both cases. Even though the server is crashing with "Out of memory", the DevTools are not showing any increased memory usage, as seen from the graphs.
Our issue seems to be very similar to dart-lang/sdk#50642 where the memory usage is normal and the "Out of memory" issue is thrown.
It is quite difficult to provide any reproduction steps for this issue. Any tips or insights on how to troubleshoot this issue would be very helpful and appreciated.
Below are one of the error details that we are getting.
Error output and stack trace
cc @rafaelortizzableh
The text was updated successfully, but these errors were encountered: