Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) #10331

TwizzyDizzy · 2018-04-04T14:55:16Z

Description:

This is happening when running Rocket.Chat 0.61.2 as well as 0.63.0 on NodeJS 8.11.1. Both versions don't exhibit this behaviour when run on NodeJS 8.9.4. I had NodeJS 8.11.1 and Rocket.Chat running for a while on my testing instance, which didn't exhibit this behaviour. This leads me to think that the suspect is NodeJS 8.11.1 in combination with:

either the load of the Rocket.Chat server
or the data in MongoDB

Server Setup Information:

Version of Rocket.Chat Server: 0.63.0 & 0.61.2 (this may affect other versions)
Operating System: Oracle Linux 7
Deployment Method(snap/docker/tar/etc): tar
Number of Running Instances: 1
DB Replicaset Oplog: -
Node Version: 8.11.1
mongoDB Version: 2.6.12

Steps to Reproduce:

I can only guess here:

run a decently sized Rocket.Chat (in terms of amount of users) server on NodeJS 8.11.1

Expected behavior:

No SEGV

Actual behavior:

SEGV. Restart (due to systemd unit definition) of Rocket.Chat at random intervals

Relevant logs:

strace of the NodeJS process is available, but I will only share it as a last resort with one of the Rocket.Chat developers, as it possibly contains private/sensitive information.

Last lines in strace before SEGV:

read(12, "\27\3\3\0\265\252\244\276\253\262\345\32\335\230b\255\311H\331p\2200\10\245\222.\26\313\2035\210\327"..., 16384) = 9462
rt_sigprocmask(SIG_SETMASK, [], [], 8)  = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
+++ killed by SIGSEGV +++

auditd-log of failing node processes:

ANOM_ABEND: Triggered when a processes ends abnormally (with a signal that could cause a core dump, if enabled).

root@chat01 [/var/log] # ausearch --comm node
----
time->Wed Apr  4 14:35:17 2018
type=ANOM_ABEND msg=audit(1522845317.790:75): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=1043 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:47:00 2018
type=ANOM_ABEND msg=audit(1522846020.846:1164): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=4595 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:49:03 2018
type=ANOM_ABEND msg=audit(1522846143.096:1227): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=5458 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:50:34 2018
type=ANOM_ABEND msg=audit(1522846234.113:1269): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=5562 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:57:58 2018
type=ANOM_ABEND msg=audit(1522846678.970:1448): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=5643 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:58:05 2018
type=ANOM_ABEND msg=audit(1522846685.473:1460): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=5878 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 14:59:29 2018
type=ANOM_ABEND msg=audit(1522846769.954:1477): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=5929 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 15:02:32 2018
type=ANOM_ABEND msg=audit(1522846952.269:1538): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=6007 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 15:23:04 2018
type=ANOM_ABEND msg=audit(1522848184.122:2496): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=9055 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 15:35:35 2018
type=ANOM_ABEND msg=audit(1522848935.572:3405): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=11501 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 15:40:33 2018
type=ANOM_ABEND msg=audit(1522849233.904:3470): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=11899 comm="node" reason="memory violation" sig=11
----
time->Wed Apr  4 15:43:02 2018
type=ANOM_ABEND msg=audit(1522849382.823:3519): auid=4294967295 uid=990 gid=987 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=12040 comm="node"

/var/log/messages (notice, that the times are identical to the auditd-logs above)

root@chat01 [~] # grep SEGV /var/log/messages
Apr  4 14:35:17 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:47:00 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:49:03 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:50:34 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:57:59 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:58:05 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 14:59:29 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 15:02:32 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 15:23:04 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 15:35:35 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 15:40:33 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV
Apr  4 15:43:02 chat01 systemd: rocketchat.service: main process exited, code=killed, status=11/SEGV

The text was updated successfully, but these errors were encountered:

geekgonecrazy · 2018-04-04T15:46:06Z

@TwizzyDizzy weird.. 0.63.0 in combo with Node.js 8.11.1 should have actually solved this seg fault

https://github.com/meteor/meteor/blob/devel/History.md#v1611-2018-04-02

According to the meteor release which we updated to for the 0.63.0 release... Node.js 8.11.1 actually solved the seg fault.

So for sure 0.63.0 in combo with Node.js 8.11.1 gave the seg fault and not some other combo?

TwizzyDizzy · 2018-04-04T16:25:36Z

Hi Aaron,

yes, I just replicated this: Rocket.Chat 0.63.0 vs. NodeJS 8.11.1.

Upgrade to NodeJS 8.11.1
do a clean (except for data in MongoDB) install via ansible
Server gets killed after some (not always the same) time.

... downgrade to NodeJS 8.9.4: no such behaviour anymore.

Cheers
Thomas

ghost · 2018-04-04T16:46:05Z

Same here. Still crashing. Rocketchat 0.63.0 / NodeJS 8.11.1. Downgrading NodeJS to 8.9.4 solves it.

cheers
t.

geekgonecrazy · 2018-04-04T18:47:03Z

@rodrigok @sampaiodiego thoughts? This seems to be doing the complete opposite of what upgrading to 8.11.1 was supposed to give us

ghost · 2018-04-04T20:41:38Z

Well they said the patch which should solve the problem should be in 8.11.1. Maybe it's not? Here the nodejs issue for reference. nodejs/node#19274

thanks and cheers

geekgonecrazy · 2018-04-04T22:01:55Z

meteor/meteor#9783 (comment) yup looks like they didn't include the segfault in 8.11.1 instead it might be in 8.11.2 🙄

trstn70 · 2018-04-05T09:11:56Z

How do I downgrade the NodeJs Version within the Rocketchat server (snap)? (for dummies?) ..

sampaiodiego · 2018-04-05T12:15:55Z

I don't think you can @trstn70 .. but @geekgonecrazy released a fix yesterday, please try running sudo snap refresh rocketchat-server

graywolf336 · 2018-04-07T06:01:50Z

~~Closing this~~ Due to the merging of #10351 and the release of Rocket.Chat v0.63.1 :) @geekgonecrazy informs me that a snaps release will follow suit in a day or so. :D

gu1ll0me · 2018-04-07T14:12:01Z

Still crashing with 0.63.1 and Node 8.11.1 here. Please re-open.

Revert to 8.9.4 solve the problem.

We are not using the Snap release.

geekgonecrazy · 2018-04-07T15:05:23Z

If you are using 8.11.1 please downgrade node version to 8.9.4. Unfortunately until node.js releases another hot fix... We have no other choice. In snap installs we just downgraded to keep people from being effected. Docker images are already downgraded. It's only manual installs left that you have to downgrade nodejs if you did upgrade

geekgonecrazy · 2018-04-07T15:10:11Z

Also updated release notes with this note
https://forums.rocket.chat/t/rocket-chat-0-63-0-released-updated-for-0-63-1/479

TwizzyDizzy · 2018-05-16T10:56:13Z

NodeJS 8.11.2 is out. I've just upgraded my production instance and the behaviour described in this issue does not occur anymore. This is why I am closing this issue.

Cheers
Thomas

graywolf336 assigned rodrigok Apr 4, 2018

geekgonecrazy mentioned this issue Apr 5, 2018

[FIX] Snaps crashing due to Node v8.11.1 Segfault #10351

Merged

This was referenced Apr 23, 2018

RocketChat periodically crashes #10538

Closed

Crash on startup after upgrade to 0.62.2 #10114

Closed

TwizzyDizzy changed the title ~~Rocket.Chat vs. NodeJS 8.11.1: Random SEGV (segmentation violation)~~ Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) Apr 24, 2018

TwizzyDizzy mentioned this issue May 3, 2018

chain crash after update to version 0.63.3 #10644

Closed

TwizzyDizzy closed this as completed May 16, 2018

TwizzyDizzy mentioned this issue May 17, 2018

Exception when freshly installed from snap #9580

Closed

immanuelfodor mentioned this issue Sep 2, 2018

Everyday the service drops #9864

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) #10331

Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) #10331

TwizzyDizzy commented Apr 4, 2018 •

edited

Loading

geekgonecrazy commented Apr 4, 2018

TwizzyDizzy commented Apr 4, 2018 •

edited

Loading

ghost commented Apr 4, 2018

geekgonecrazy commented Apr 4, 2018

ghost commented Apr 4, 2018

geekgonecrazy commented Apr 4, 2018

trstn70 commented Apr 5, 2018 •

edited

Loading

sampaiodiego commented Apr 5, 2018

graywolf336 commented Apr 7, 2018 •

edited

Loading

gu1ll0me commented Apr 7, 2018

geekgonecrazy commented Apr 7, 2018

geekgonecrazy commented Apr 7, 2018 •

edited

Loading

TwizzyDizzy commented May 16, 2018

Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) #10331

Rocket.Chat vs. NodeJS 8.11.1 (or rather > 8.9.4): Random SEGV (segmentation violation) #10331

Comments

TwizzyDizzy commented Apr 4, 2018 • edited Loading

Description:

Server Setup Information:

Steps to Reproduce:

Expected behavior:

Actual behavior:

Relevant logs:

geekgonecrazy commented Apr 4, 2018

TwizzyDizzy commented Apr 4, 2018 • edited Loading

ghost commented Apr 4, 2018

geekgonecrazy commented Apr 4, 2018

ghost commented Apr 4, 2018

geekgonecrazy commented Apr 4, 2018

trstn70 commented Apr 5, 2018 • edited Loading

sampaiodiego commented Apr 5, 2018

graywolf336 commented Apr 7, 2018 • edited Loading

gu1ll0me commented Apr 7, 2018

geekgonecrazy commented Apr 7, 2018

geekgonecrazy commented Apr 7, 2018 • edited Loading

TwizzyDizzy commented May 16, 2018

TwizzyDizzy commented Apr 4, 2018 •

edited

Loading

TwizzyDizzy commented Apr 4, 2018 •

edited

Loading

trstn70 commented Apr 5, 2018 •

edited

Loading

graywolf336 commented Apr 7, 2018 •

edited

Loading

geekgonecrazy commented Apr 7, 2018 •

edited

Loading