Hard Crash starting daemon CentOS 7

[necrogami]

I'm also testing with a centos server and whenever i try to run the daemon i get this hard crash sent to the accounts mailbox

https://gist.github.com/necrogami/d793eb9f0744bd46b97dfb391a6c24fb

uname -a

Linux WyldLife.dedi.creeperhost.net 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

node -v

v6.9.1

docker info

Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-253:0-135142182-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 361.7 MB
Data Space Total: 107.4 GB
Data Space Available: 50.27 GB
Metadata Space Used: 1.29 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: journald
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 31.22 GiB
Name: WyldLife.dedi.creeperhost.net
ID: QT5P:7TGB:HH3X:GYKH:JTS2:J6JW:PTOU:SYCN:VYWT:S255:OPCG:FGR6
WARNING: bridge-nf-call-ip6tables is disabled
Registries: docker.io (secure)

[DaneEveritt]

Can you update docker to version 1.12.3 for me. I highly doubt that is the cause of the crash, but I'd like to make sure as few things are out of date as possible.

Can you also provide the logs from the daemon, they're in /srv/daemon/logs.

[necrogami]

https://gist.github.com/necrogami/869afed51583819b225d13b64cb01c03 info.log (all other logs are empty)

More details for you. So it starts the sftp docker container.
It opens 8080
it stays up until i goto my node list and it tries to access the daemon and crashes.

[necrogami]

install/download were https but base wasn't. Updated base to https. ran npm start and as soon as i refreshed the node list page it crashed again.

[DaneEveritt]

Alright, thanks, It'll take some time to dig into this, thanks for the information so far though.

[DaneEveritt]

This is gonna sound strange, but can you disable SSL on the daemon and see if it crashes still?

[necrogami]

It doesn't crash, but it doesn't connect either.

02:44:08.253Z  WARN wings: + -------------------- +
02:44:08.261Z  INFO wings: Initialization Successful!
02:44:08.264Z  INFO wings: Pterodactyl Daemon is now listening for insecure connections on 0.0.0.0:8080```

[parkervcp]

The daemon doesn't say it connects to anything, It just listens.

[DaneEveritt]

So, we talked on Discord a bit about this, and its looking like your system is at fault, specifically libssl:

:Dec 02 08:25:18 WyldLife.dedi.creeperhost.net kernel: node[1149]: segfault at 9 ip 00007f865df7f459 sp 00007fffe9669098 error 4 in libcrypto.so.1.0.1e[7f865de85000+1bd000]
:Dec 02 08:25:29 WyldLife.dedi.creeperhost.net kernel: node[1884]: segfault at 9 ip 00007f0e1f962459 sp 00007ffe8008f718 error 4 in libcrypto.so.1.0.1e[7f0e1f868000+1bd000]

If the daemon isn't crashing with SSL disabled, then this is most definitely the issue. Most everything you're running appears to be pretty out of date, docker should be 1.12.3 by now to address security issues, as well as implement new features that we use. The 3.10 kernel is fine, but the absolute minimum that docker supports, and not necessarily the most up-to-date.

[necrogami]

It's default centos 7, I just today upgraded all the packages but, I did install docker from their repo so it's now up to date.

…

On Fri, Dec 2, 2016, 9:52 PM Dane Everitt ***@***.***> wrote: So, we talked on Discord a bit about this, and its looking like your system is at fault, specifically libssl: :Dec 02 08:25:18 WyldLife.dedi.creeperhost.net kernel: node[1149]: segfault at 9 ip 00007f865df7f459 sp 00007fffe9669098 error 4 in libcrypto.so.1.0.1e[7f865de85000+1bd000] :Dec 02 08:25:29 WyldLife.dedi.creeperhost.net kernel: node[1884]: segfault at 9 ip 00007f0e1f962459 sp 00007ffe8008f718 error 4 in libcrypto.so.1.0.1e[7f0e1f868000+1bd000] If the daemon isn't crashing with SSL disabled, then this is most definitely the issue. Most everything you're running appears to be pretty out of date, docker should be 1.12.3 by now to address security issues, as well as implement new features that we use. The 3.10 kernel is fine, but the absolute minimum that docker supports, and not necessarily the most up-to-date. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#196 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACA_CjsuA-iSFqHK1Z9-k0WbMQ3_puhks5rENl4gaJpZM4LC-tu> .

[necrogami]

And in terms of connecting, when i open the node list page in the panel it connects to each of the daemons for a heartbeat check, This doesn't happen and doesn't show connected from the panel.

After updating to the latest packages available for centos 7 (core only) and newest docker, re-enabling ssl this is still crashing. You're currently listing centos 7 as officially supported. However out of the box it's not.

Here's the current data after updates: insecure (with crtl+c), secure ending in crash, yum update, mail crash log, and updated docker info

https://gist.github.com/necrogami/762289ca176f498fe5d0197ad247a250

[DaneEveritt]

Yeah, we've done preliminary testing and it was working fine on the systems we tested with. I'm still thinking it might be libssl causing issues, because you said above it doesn't crash when SSL is disabled. I'll have time in the next few days to spin up a test system and try to match what you have going on here.

The bottom of your crash log shows a segfault with libcrypto.so.1.0.1e, which is leading me to believe there is an issue there.

:Dec 02 08:18:31 WyldLife.dedi.creeperhost.net yum[31564]: Installed: 1:nodejs-6.9.1-4.el7.x86_64
:Dec 02 08:25:18 WyldLife.dedi.creeperhost.net kernel: node[1149]: segfault at 9 ip 00007f865df7f459 sp 00007fffe9669098
 error 4 in libcrypto.so.1.0.1e[7f865de85000+1bd000]
:Dec 02 08:25:29 WyldLife.dedi.creeperhost.net kernel: node[1884]: segfault at 9 ip 00007f0e1f962459 sp 00007ffe8008f718
 error 4 in libcrypto.so.1.0.1e[7f0e1f868000+1bd000]
:Dec 02 08:25:29 WyldLife.dedi.creeperhost.net abrt-hook-ccpp[1909]: Not saving repeating crash in '/usr/bin/node'
:[User Logs]:
:Dec 02 08:18:31 WyldLife.dedi.creeperhost.net yum[31564]: Installed: 1:nodejs-6.9.1-4.el7.x86_64
:Dec 02 08:25:29 WyldLife.dedi.creeperhost.net abrt-hook-ccpp[1909]: Not saving repeating crash in '/usr/bin/node'

In fact, there is a related issue to it here which suggests that a different version of libcrypto fixes it.

Theres an issue related on Github as well for Saltstack on Centos.

This issue suggests it might be a kernel related issue.

[necrogami]

So i've solved this issue with an odd work around. This appears to be a bug related to the repo-installed version of nodejs on default centos servers. If i download the pre-compiled software from nodejs site and move that to /opt then symlink node/npm to /usr/bin this problem goes away.

I have confirmed that this is now working via Node js 6.9.1 and centos 7 an annoying work around but it works.

[sgallagher]

We are working with the maintainers of OpenSSL in RHEL/CentOS to resolve the issue, however it's worth noting this thread: nodejs/node#9551

Essentially, the problem is that OpenSSL parses certain oddly-arranged certificate files incorrectly. See nodejs/node#9551 (comment) for a workaround that doesn't require reinstalling Node.js.