Closed connections stay in "FIN_WAIT2" state

[bachwehbi]

I noticed that some closed connections (on bad mobile networks) stay in FIN_WAIT2 state.

This is the case for permanent connections where the server half closes the connection but the client (boggy or due to bad network) does not close the connection (See this node issue: nodejs/node-v0.x-archive#3613).

May I ask if you or any Mosca user have encountered such problem?

I'm investigating using socket.destroy() instead of socket.end() in:

https://github.com/mcollina/mosca/blob/master/lib/client.js#L317
https://github.com/mcollina/mosca/blob/master/lib/client.js#L326
https://github.com/mcollina/mosca/blob/master/lib/client.js#L569

[mcollina]

I fear you cannot do much in node-land. Using destroy will probably matter a lot.

A good measure is to reduce the FIN timeout: https://drupal.star.bnl.gov/STAR/blog-entry/jeromel/2009/feb/18/tcp-parameters-linux-kernel.

Also I have seen this change a lot between cloud providers.

[bachwehbi]

Thanks @mcollina

Yeah, this is what I was expecting! I only encounter this problem on Internet facing Mosca servers (dev environment), putting them behind a loadbalancer makes the problem disappear.

I'm closing this issue!

[eladnava]

Unfortunately, lowering the net.ipv4.tcp_fin_timeout won't change anything.

I noticed that if keepalive is set, it takes tcp_keepalive_time instead of tcp_fin_timeout when in FIN_WAIT2. So by default it is 7200 seconds (2 hours) even if tcp_fin_timeout is set to the default 60 seconds. If however keepalive is NOT set, then the socket goes into FIN_WAIT2 infinitely when the socket is not closed properly from the client side. This happens when node does first shutdown, then close. In my opinion this is a bug in node which can be exploited easily by a DOS attack.
nodejs/node-v0.x-archive#3613 (comment)

So, for anyone who experiences lots of FIN_WAIT_2 / FIN_WAIT_1 "zombie" connections, a workaround is to lower the net.ipv4.tcp_keepalive_time variable, as explained here, since by default, these kinds of connections will linger for 2 whole hours before being terminated by the OS.

[behrad]

👍 @eladnava

[eladnava]

@mcollina I suffered from too many FIN_WAIT1 / FIN_WAIT2 sockets again because all of my MQTT clients are mobile phones with unstable connections, so lots of them don't send the final FIN acknowledgement due to unstable connectivity -- so I modified mosca to use destory() instead of end() and the FIN_WAIT sockets pretty much vanished!

Client connects to node server, client is physically disconnected from the connection, node server tries to disconnect with socket.end(). This results in a FIN_WAIT1.
nodejs/node-v0.x-archive#3613 (comment)

Can I submit a PR to replace that.connection.stream.end() with that.connection.stream.destroySoon() at client.js#L611?

[mcollina]

I'm ok with destroySoon, if it's present. The reason why end was used is to support websocket-stream, which does not have destroySoon.

[eladnava]

Excellent, I'll submit that PR! =) Thanks for the quick answer!