Possible Memory Leak in HTTP

[nickydonna]

We've being experiencing an steady memory consumption in on our servers:

The drops in memory are when we deploy a new version (the server restarts) or we manually restart the servers.

We started to debug, and came to the conclusion that the issue was in express, when we created the issue in express, we tried the same test on a barebone node server and the we had the memory leak.

(There is a lot of information in the express issue)

Platform:

• Version: 7.6.0, 8.4.0 and 9.5.0
• Platform: Darwin Kernel Version 16.7.0
• Subsystem: HTTP

Here is the code:

// [START app]
const http = require('http');

const server = http.createServer(app);
const heapdump = require('heapdump');

heapdump.writeSnapshot('./' + Date.now() + '.heapsnapshot');

// Just a function to log the memory consumption after autocannon
function generateHeapDumpAndStats() {
  //1. Force garbage collection every time this function is called
  try {
    global.gc();
  } catch (e) {
    console.log("You must run program with 'node --expose-gc'");
    process.exit();
  }

  //2. Output Heap stats
  var heapUsed = process.memoryUsage().heapUsed;
  console.log('Program is using ' + heapUsed + ' bytes of Heap.');
}

function app(req, res) {
  res.statusCode = 200;
  res.end('Hello, world!');
}

// Start the server
const PORT = process.env.PORT || 9000;
server.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
});

setInterval(generateHeapDumpAndStats, 3000);
// [END app]

run with node --expose-gc server.js

autocannon command autocannon -a100000 -c5 http://localhost:9000/

I have tested the code for the memory logging and it doesn't leak.

I've attached the heapdump

Archive.zip

The first snapshot is after the server starts and the memory has been stabilized.
The second is after an autocannon run and the memory has been stabilized.
The third is after a second autocannon run and the memory has been stabilized.

Could you help us find out if the issue is really with node?

[jasnell]

/cc @mcollina

[mcollina]

I cannot reproduce with Node 8.9.4 using:

const http = require('http');

const server = http.createServer(app);

function app(req, res) {
  res.statusCode = 200;
  res.end('Hello, world!');
}

// Start the server
const PORT = process.env.PORT || 9000;
server.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
});

(nor I can with heapdump enabled)

(diagram generated with http://npmjs.com/clinic)

My suggestion is that the problem might be in your application, and not in Node core or Express.

In the diagram you posted, you show that the process grows up to 2.5gb. Can you reproduce that kind of growth?

[nickydonna]

@mcollina we are constantly restarting the node servers, because every other day the memory reaches those levels. The thing that I don't understand is that we've tried it in 3 differents dev Mac and the servers, but neither you nor the express people can reproduce it.

[nickydonna]

Do you see anything weird in the heapdumps I submitted???

[mcollina]

A heapdump of that size is unlikely to be representative of anything.

[nickydonna]

@mcollina that is after some 100.000 requests, I can upload heapdumps with more requests if that would help

[mcollina]

I have run that basic node core server with a 1 billion request, and saw no appreciable difference in the Heap size.

Let me ask again: in the diagram you posted, you show that the process grows up to 2.5gb. Can you reproduce that kind of growth? That looks like a memory leak. This example don't, so it seems that it is your application that is leaking.
As a general rule, any type of growth registered when the process is under 100-150 MB is too inaccurate to track.

[bnoordhuis]

No follow-up from OP. I'll close this out so we're below 700 open issues again. :-)

[nickydonna]

We haven't had anything new to provide as evidence, so go ahead an close this. Thanks