Memory Leak when handling WebSocket at server side #27414

jimmyshiau · 2016-09-22T03:53:06Z

Simple test case

import "dart:io";
import "dart:async";

void main() {
  HttpServer
  .bind(InternetAddress.ANY_IP_V4, 8081)
  .then((server) {
    server.autoCompress = true;
    server.listen((HttpRequest request) async {
      if (request.uri.path == "/q") {
        await socket(request);
      } else {
        request
        ..response.headers.contentType = ContentType.HTML
        ..response.write('<html><body>Hello, world!'
          '<script type="text/javascript">'
          'var ws = new WebSocket("ws://"+window.location.host+"/q");'
          'ws.onopen = function() {console.log("open");};'
          '</script>'
          '</body></html>');
      }
      request.response.close();
    });
  });
}

Future socket(HttpRequest request) async {
  return WebSocketTransformer.upgrade(request)
  .then(handleWebSocket);
}


void handleWebSocket(WebSocket socket){
  print('handleWebSocket');
  socket.listen(_onSocketData, onDone: _onSocketDone);
}

void _onSocketData(data) {print(data);}
void _onSocketDone() {print('_onSocketDone');}

Server RES

66080
133060
147224
193972
193932
224692
245348
256536
278660
259632
261072
281472
261316
263444
266912
266896
287424
287484
288996
304852
345812
347388
358788

At each run, I opened 100 browser tabs and then close them all. Also, before checking memory use, I ran GC first.

The text was updated successfully, but these errors were encountered:

jimmyshiau · 2016-09-22T03:54:36Z

Dart Observatory

Uint8List Instances
23
1282
1798

johnmccutchan · 2016-09-22T15:51:27Z

@zanderso

zanderso · 2016-09-22T16:03:50Z

@jimmyshiau Could you say what version of the VM and what OS you're running on? Thanks!

tomyeh · 2016-09-22T16:21:49Z

Dart 1.19.1 and Linux (Jimmy is out of the office)

zanderso · 2016-09-23T17:00:40Z

I've repro'd on 1.19.1, but this doesn't repro at top of tree. It also doesn't repro in 1.19.1 with tcmalloc patched in. After ~10 runs RSS is reasonably stable at about 180MB, whereas before it would grow ~10-15MB per run as in the report.

@jimmyshiau @tomyeh Can you try on your end with a recent 1.20 dev? E.g.: https://storage.googleapis.com/dart-archive/channels/dev/release/1.20.0-dev.7.0/sdk/dartsdk-linux-x64-release.zip. Thanks!

jimmyshiau · 2016-09-26T03:05:25Z

Hi @zanderso

1.20.0-dev.7.0 is better, but it still grew up about 1MB per run: 100 > 101 > 102

Dart VM version: 1.20.0-dev.7.0 (Tue Sep 20 14:27:56 2016) on "linux_x64"

http://screencast.com/t/RHy2EGpO
http://screencast.com/t/BliLRjpmA

RES

67652
71780
86164
95600
80064
93920
95140
95596
96196
96712
97516
97748
98556
99776
100296
101068
102068
102640
103416
103660
104436
105220

Dart Observatory

Uint8List Instances
853 > 1859
Stackmap Instances
455 > 1093

tomyeh · 2016-10-11T10:43:39Z

Hi, any update? Can you reproduce it? Or, we mis-interpreted something here?

zanderso · 2016-10-12T06:25:49Z

When I tried at top-of-tree a couple of weeks ago, RSS stabilized eventually, but I can take another look. If there is any script you are using to drive the browser, that may help me carry out the exact same steps that you are following.

jimmyshiau · 2016-10-12T06:39:18Z

You can save your page link as a bookmark, then put it in a bookmark folder and make 100 copies

right click on the folder and select open all bookmarks

http://screencast.com/t/RHy2EGpO

tomyeh · 2016-12-30T02:05:36Z

Some background information:
In our production environment, the memory usage of one VM grows from 200M to over 1GB within a week (on Dart 1.20/1.21 -- much better than 1.19). And, we can observe a similar pattern in our test environment. If we turned off WebSocket and used periodic Ajax polling instead (in our test environ), the problem was gone (to some extent). It is why this tiny program was created.

Please tell us know if anything we can help to clarify the issue. Thanks.

mraleph · 2018-04-24T21:25:29Z

I am assuming this issue is stale. Please reopen if more work need to happen in this area.

tomyeh · 2018-04-25T01:29:09Z

No, the problem is still there. We're forced to restart the server if exceeding a limit. Please look at it. Thanks.

mraleph · 2018-04-25T04:02:12Z

@zanderso is this something you might have free cycles to look into?

@tomyeh is the program from the first comment representative of the memory leak you are seeing?

tomyeh · 2018-04-25T04:29:21Z

The program in the first comment is what we tried to reproduce with a simple case. The real program is of course much more complicated. Here is what we have done. Hope it helps.

We found memory leak in the production environment
We reproduced it with a stress test locally
Once tracking down, we suspected it was caused by WebSocket, so we ran the stress test again by turning off WebSocket (i.e., we use Ajax to pull periodically). There was no memory leak found in this test.
So, we created a very simple case as shown in the first comment to reproduce the issue. And, it did.

tomyeh · 2018-06-20T10:35:00Z

I made another simplified test code as follows. From server's log, you'll find RSS keeps growing. It seems the more concurrent connections, the faster memory use grows.

//server
import "dart:io";

void main() {
  HttpServer
  .bind(InternetAddress.ANY_IP_V4, 8081)
  .then((server) {
    server.listen((request) {
      int count = 0;
      WebSocketTransformer.upgrade(request)
      .then((socket){
        socket.listen((data) {
          count += data.length;
        }, onDone: () {print('Done: $count; rss=${ProcessInfo.currentRss~/(1024*1024)}');});
        return socket.done;
      });
    });
  });
}

and

//client
import "dart:io";
import "dart:async";

main() async {
  for (int i = 0; i++ < 10;) {
    final ops = <Future>[];
    for (int j = 0; j++ < 100;) {
      ops.add(WebSocket.connect("ws://localhost:8081")
        .then((socket) {
          for (int k = 0; k++ < 100;)
            socket.add("data: $i,$j,$k".codeUnits);
          socket.close();
        }));
    }
    await Future.wait(ops);
   }
}

tomyeh · 2018-07-20T01:15:42Z

I found a workaround: cancel subscription when the WebSocket connection is closing.

var subpt
subpt = socket.listen(..., onDone: subpt.cancel);

Not sure if memory consumption is stabilised (I just watched it a day), but at least the growth of memory consumption is reduced from 900M to 400M after online 20 hours.

It seems be cyclic reference issue. Hope this information helps.

sgehrman · 2020-06-04T11:51:39Z

I'm on the latest flutter beta and latest dart.

Flutter (Channel beta, 1.18.0-11.1.pre, on Linux, locale en_US.UTF-8)
• Flutter version 1.18.0-11.1.pre at /home/steve/google/flutter
• Framework revision 2738a1148b (3 weeks ago), 2020-05-13 15:24:36 -0700
• Engine revision ef9215ceb2
• Dart version 2.9.0 (build 2.9.0-8.2.beta)

I'm seeing a massive memory leak using websockets. I'm sending a 1gb file to my shelf server and I dart observatory shows 3gb of _Uint8Lists leaking. If I close the server, a ton of ram is released, but the server runs all the time, I can't close it everytime someone uploads a file. What can I do?

My server runs in it's own isolate if that matters. The server is very simple. It receives the UInt8List data and writes it to a file.

tomyeh · 2020-06-04T14:24:27Z

@sgehrman Did you try Dart 2.8 at server side? Based on our experiences, the memory usage is much better than older versions. Our server can run days under heavy load without significant memory surge. However, we don't use Dart 2.9 in production system yet, so no sure if any difference.

zanderso · 2020-06-04T15:05:28Z

@sgehrman can you share a small code example for the situation you describe that causes that much memory to be retained. (for example like the one in #27414 (comment))

@zichangg @a-siva

sgehrman · 2020-06-04T23:19:59Z

@tomyeh The server is running in my flutter mobile app. I have a feature where my app hosts a web page locally and uses can view and drag and drop files from their browser on the desktop. The files are send with websockets sendBlob. So, I don't know how easy it is to downgrade dart. I read Dart version is tied to the Flutter SDK?

@zanderso I'm using all these packages like shelf, shelf_web_socket, web_socket_channel. It's not super easy to make a small code example. The code is very simplistic.

The browser side:

final html.WebSocket webSocket =
       html.WebSocket('ws://${hostUri.host}:8764');
  
    // after socket opens
   _transferSocket.sendBlob(File(path));

The server side:

import 'package:shelf/shelf_io.dart' as io;
import 'package:web_socket_channel/web_socket_channel.dart';

Future<void> _serve() async {
    final wsh = webSocketHandler(_webSocketHandler);

    final cascade = shelf.Cascade().add(wsh);

    _server = await io.serve(cascade.handler, InternetAddress.anyIPv4, port);
  }

  void _webSocketHandler(WebSocketChannel webSocket) {
    webSocket.stream.listen((dynamic message) {
      if (message is Uint8List) {
       File(path).writeAsBytesSync(message);
    }
}

If I posted an apk would that be enough to debug it on your end?

I could also add you as a github collaborator if you wanted to get the code and run the app. The code is private. Hoping to release it in a few weeks.

zichangg · 2020-06-08T05:12:46Z

@sgehrman Hi, I'd like to help!
Unfortunately, I don't have a mobile device to test with my WFH environment. What platform your app is running? And can you also try not to write into a file? It is possible that file.writeAsBytesSync() is leaking.

If you can also try your code with Http? Package_http is very easy to use. I just wanna make sure the websocket is indeed the culprit.

Isakdl · 2024-11-12T16:00:54Z

I have similar problems as reported here, after extensive testing, I found no issues running locally while monitoring with Dart DevTools. However, in a Docker container, memory issues arise if the client does not explicitly call socket.close() before disconnecting. In such cases, memory usage on the server increases to about 50% of the allocated 1 GB. Performance degrades with the server handling requests very slowly, ultimately leading to an out-of-memory (OOM) kill by Docker, despite reported memory usage remaining at ~50%.

Note that it also seems like the docker container keeps the reported memory usage relatively constant after new allocations. E.g. I'm not seeing big reductions in memory usage. But when running locally both in JIT and AOT mode I observe that the process gets a big reduction of memory usage now and then.

Another peculiar fact is that the size of the data sent on the socket also seems to impact how fast I can reach the problems.

Dart version: 3.5

I used the code by @sgehrman for my testing, and the official docker example (https://hub.docker.com/_/dart):
Server code

import "dart:async";
import "dart:io";

void main() {
  HttpServer.bind(InternetAddress.anyIPv4, 8081).then((server) {
    server.listen((request) {
      int count = 0;
      late StreamSubscription subscription;
      WebSocketTransformer.upgrade(request).then((socket) {
        socket.pingInterval = const Duration(seconds: 1);
        subscription = socket.listen(
          (dynamic data) {
            count += (data as List).length;
          },
          onDone: () async {
            print(
              'Done: $count; rss=${ProcessInfo.currentRss ~/ (1024 * 1024)}',
            );
            await subscription.cancel(); //This solution does not help
          },
        );
        return socket.done;
      });
    });
  });
}

Client code

import "dart:io";
import "dart:async";

final String data = '<a long string>';

main() async {
  for (int i = 0; i++ < 100;) {
    final ops = <Future>[];
    for (int j = 0; j++ < 100;) {
      ops.add(WebSocket.connect("ws://localhost:8081").then((socket) {
        for (int k = 0; k++ < 100;) {
          socket.add(data.codeUnits);
        }
        //socket.close(); // If this call is not done we get memory problems
      }));
    }
    print('awaiting ops');
    await Future.wait(ops);
  }

  exit(0);
}

Dockerfile

FROM dart:stable AS build

WORKDIR /app
COPY pubspec.* ./
RUN dart pub get

COPY . .
RUN dart pub get --offline
RUN dart compile exe lib/websocket_server.dart -o bin/server

FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/

EXPOSE 8081
CMD ["/app/bin/server"]

mraleph · 2024-11-18T13:07:50Z

I took a look at this today using the code provided by @Isakdl above. It clearly was leaking for me inside and outside of the Docker container: what I saw is _NativeSocket objects leaking.

This seems to be happening because something is wrong with shutdown sequence: we shutdown read direction on the socket first, and then we shutdown write direction - but then we don't actually recognize that both directions are now shutdown and we need to close and destroy the socket. And the socket is left lingering in the zombie state.

The code in shutdownWrite says:

   void shutdownWrite() {
    if (!isClosing && !isClosed) {
      if (closedReadEventSent) {
        close();
      } else {
        sendToEventHandler(1 << shutdownWriteCommand);
      }
      isClosedWrite = true;
    }
  }

We arrive here with isClosedRead but !closedReadEventSent - which I think happens because we have also disabled read events by that time. ~~Because readEventsEnabled is false we will never set closedReadEventSent to true and as a result we will never fully close the socket in this case. This seems wrong.~~

UPDATE: My initial analysis was incorrect here. It is by design that shutting down read direction does not simply discard the data and instead expects the reader to drain the socket before dispatching closedRead event. The actual problem was that the socket had no outstanding incoming data when its read direction is closed. This means that there are we are not going to schedule any read events to be delivered. The fix is to issueReadEvent when we close read direction.

mraleph · 2024-11-20T13:30:44Z

This CL should fix the leak of _NativeSocket objects. My previous analysis was slightly incorrect, I have amended the comment above for posterity.

While the leak seems to be fixed with my CL, I still observe somewhat suboptimal behavior from the GC - we don't seem to trigger mark sweep GC often enough which means we keep accumulating some external memory (IO buffers?) which causes RSS to grow.

@rmacnak-google could you take a look if we could tweak our GC heuristics here? You would need to apply my CL to fix the leak and then just run the code from the comment above (no Docker is needed to reproduce this).

- Don't treat heap_growth_max as a lower bound when using the ratio heuristic. For small heaps this can cause us to exceed the working set size by more than 10x. - Stick with the ratio heuristic when we're staying under the desired time fraction. - Avoid tiny initial thresholds by growing at least by one new-space. TEST=golem Bug: #27414 Change-Id: I64b2994ea9a32ce8a8bbec15cd89d8b46a01b1bb Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/397460 Reviewed-by: Slava Egorov <[email protected]> Commit-Queue: Ryan Macnak <[email protected]>

Isakdl · 2024-11-29T16:01:12Z

@mraleph Good job! Thank you so much for this! 😀 👍 👏

kasperl added the area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. label Sep 22, 2016

johnmccutchan assigned zanderso Sep 22, 2016

zanderso added the type-bug Incorrect behavior (everything from a crash to more subtle misbehavior) label Nov 6, 2016

tomyeh mentioned this issue Mar 2, 2017

API to know the memory use of Dart VM #28951

Closed

mraleph closed this as completed Apr 24, 2018

mraleph added the closed-obsolete Closed as the reported issue is no longer relevant label Apr 24, 2018

mraleph removed the closed-obsolete Closed as the reported issue is no longer relevant label Apr 25, 2018

mraleph reopened this Apr 25, 2018

jumperchen added a commit to rikulo/socket.io-dart that referenced this issue Jul 20, 2018

add a workaround for dart-lang/sdk#27414

53b0774

zanderso removed their assignment Jun 4, 2020

zanderso added the library-io label Jun 4, 2020

wytrych mentioned this issue Nov 2, 2020

Memory leak in Dart VM? #44009

Open

Isakdl mentioned this issue Nov 12, 2024

Server terminates with "out of memory" periodically serverpod/serverpod#762

Closed

copybara-service bot closed this as completed in e2c8c23 Nov 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory Leak when handling WebSocket at server side #27414

Memory Leak when handling WebSocket at server side #27414

jimmyshiau commented Sep 22, 2016 •

edited

Loading

jimmyshiau commented Sep 22, 2016

johnmccutchan commented Sep 22, 2016

zanderso commented Sep 22, 2016

tomyeh commented Sep 22, 2016

zanderso commented Sep 23, 2016 •

edited

Loading

jimmyshiau commented Sep 26, 2016 •

edited

Loading

tomyeh commented Oct 11, 2016

zanderso commented Oct 12, 2016

jimmyshiau commented Oct 12, 2016

tomyeh commented Dec 30, 2016 •

edited

Loading

mraleph commented Apr 24, 2018

tomyeh commented Apr 25, 2018

mraleph commented Apr 25, 2018

tomyeh commented Apr 25, 2018

tomyeh commented Jun 20, 2018 •

edited

Loading

tomyeh commented Jul 20, 2018

sgehrman commented Jun 4, 2020 •

edited

Loading

tomyeh commented Jun 4, 2020 •

edited

Loading

zanderso commented Jun 4, 2020

sgehrman commented Jun 4, 2020 •

edited

Loading

zichangg commented Jun 8, 2020

Isakdl commented Nov 12, 2024

mraleph commented Nov 18, 2024 •

edited

Loading

mraleph commented Nov 20, 2024

Isakdl commented Nov 29, 2024

Memory Leak when handling WebSocket at server side #27414

Memory Leak when handling WebSocket at server side #27414

Comments

jimmyshiau commented Sep 22, 2016 • edited Loading

Simple test case

Server RES

jimmyshiau commented Sep 22, 2016

Dart Observatory

johnmccutchan commented Sep 22, 2016

zanderso commented Sep 22, 2016

tomyeh commented Sep 22, 2016

zanderso commented Sep 23, 2016 • edited Loading

jimmyshiau commented Sep 26, 2016 • edited Loading

RES

Dart Observatory

tomyeh commented Oct 11, 2016

zanderso commented Oct 12, 2016

jimmyshiau commented Oct 12, 2016

tomyeh commented Dec 30, 2016 • edited Loading

mraleph commented Apr 24, 2018

tomyeh commented Apr 25, 2018

mraleph commented Apr 25, 2018

tomyeh commented Apr 25, 2018

tomyeh commented Jun 20, 2018 • edited Loading

tomyeh commented Jul 20, 2018

sgehrman commented Jun 4, 2020 • edited Loading

tomyeh commented Jun 4, 2020 • edited Loading

zanderso commented Jun 4, 2020

sgehrman commented Jun 4, 2020 • edited Loading

zichangg commented Jun 8, 2020

Isakdl commented Nov 12, 2024

mraleph commented Nov 18, 2024 • edited Loading

mraleph commented Nov 20, 2024

Isakdl commented Nov 29, 2024

jimmyshiau commented Sep 22, 2016 •

edited

Loading

zanderso commented Sep 23, 2016 •

edited

Loading

jimmyshiau commented Sep 26, 2016 •

edited

Loading

tomyeh commented Dec 30, 2016 •

edited

Loading

tomyeh commented Jun 20, 2018 •

edited

Loading

sgehrman commented Jun 4, 2020 •

edited

Loading

tomyeh commented Jun 4, 2020 •

edited

Loading

sgehrman commented Jun 4, 2020 •

edited

Loading

mraleph commented Nov 18, 2024 •

edited

Loading