Prebuilt gRPC doesn't work with the Ubuntu 18.04 nodejs runtime.

[przemek-instapage]

While trying to use Google PubSub Node.js library that uses grpc-node lib I get an error:
node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node: undefined symbol: SSL_library_init

I have tried to reinstall this module but that does not help.

[nicolasnoble]

Please give us more details on this. What's your version of nodejs, operating system, environnement, and any other detail that might be helpful.

[przemek-instapage]

Here you are:

$ node -v
v8.10.0

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"

$ npm -v
3.5.2

$ apt list --installed | grep ssl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libgnutls-openssl27/bionic,now 3.5.18-1ubuntu1 amd64 [installed]
libio-socket-ssl-perl/bionic,bionic,now 2.056-1 all [installed]
libnet-smtp-ssl-perl/bionic,bionic,now 1.04-1 all [installed]
libnet-ssleay-perl/bionic,now 1.84-1build1 amd64 [installed]
libssl-dev/bionic,now 1.1.0g-2ubuntu4 amd64 [installed]
libssl-doc/bionic,bionic,now 1.1.0g-2ubuntu4 all [installed]
libssl1.0.0/bionic,now 1.0.2n-1ubuntu5 amd64 [installed]
libssl1.1/bionic,now 1.1.0g-2ubuntu4 amd64 [installed,automatic]
openssl/bionic,now 1.1.0g-2ubuntu4 amd64 [installed]
perl-openssl-defaults/bionic,now 3build1 amd64 [installed,automatic]
python-openssl/bionic,bionic,now 17.5.0-1ubuntu1 all [installed,automatic]
ssl-cert/bionic,bionic,now 1.0.39 all [installed]

Do you need anything more?

[nicolasnoble]

Is your version of node installed via nvm or apt ?

[przemek-instapage]

Via apt nodejs/bionic,now 8.10.0~dfsg-2 amd64 [installed]

[nicolasnoble]

Great, thanks. I can then make a bug report to Ubuntu, because I can confirm their nodejs package is broken. That's the rootcause issue here.

[nicolasnoble]

In the meantime, you can make this work for your machine by doing an npm rebuild --build-from-source. I just confirmed this works.

[nicolasnoble]

For reference, the Ubuntu package is breaking the node-gyp spec explained here: https://github.com/nodejs/node-gyp/wiki/Linking-to-OpenSSL

[przemek-instapage]

@nicolasnoble I have got the same error message, even after using npm rebuild --build-from-source command.

[nicolasnoble]

Hmmm, that's odd. I guess I am missing something. Could you try reproducing everything inside of docker? I may have been too quick on my conclusion.

$ docker run -t -i ubuntu:18.04 bash -l
# apt update
# apt install nodejs npm
# npm install ...

[nicolasnoble]

Or more specifically, if you could give us a reproduction case that exhibits the issue within a ubuntu:18.04 docker image, that'd be great.

[przemek-instapage]

@nicolasnoble while using ubuntu docker image it worked without any problems at all. I guess there is an issue with my environment only...

[nicolasnoble]

So that's disturbing. The nodejs runtime is supposed to expose its SSL symbols to any extension. For some reasons, on your environment only, this doesn't seem to work fine. There might be interference caused by another package or daemon that would impose this. My gut feeling would be apparmor, but it's a long shot.

Could you check the contents of your syslog or dmesg buffer as you try to load the extension to see if something somewhere is denying the load?

Given it's on your environment only, I would be tempted to close this one as unreproducible, but I'd still love to get to the bottom of this.

How did you install your environment? Is it desktop? Did you upgrade from a former version of ubuntu? Did you install any custom package? Maybe give me the full output of dpkg -l, and we can try reproducing your issue in a virtualbox desktop, if you did install a desktop version.

[przemek-instapage]

There is nothing in syslog/dmesg that seems related to this issue. This is the desktop installation, the OS was upgraded from Ubuntu 16.04. I have attached the full output of dpkg -l

Thanks for your help, I understand that's a problem with my environment only, so I don't mind if you close this issue. Anyway I will try to solve it on my own in my free time.
dpkg_list.txt

[nicolasnoble]

Thanks, I'll try reproducing it. My point is that if you have troubles, other users might as well, so I want to make sure that it's either a non-widespread issue, or that there's a known workaround of something.

[nicolasnoble]

I spent some time trying to reproduce this one, without success. I have installed the same set of packages you have, or a very close approximation; it seems you have some fairly old packages there still installed that aren't supported by Ubuntu since a while, so I wasn't able to install those, but this shouldn't have been the problem, given they are mostly old development libraries.

I am going to close it for now, but you will still be able to update it, in case you ever find the rootcause for it. But I might actually suggest to reinstall the OS from a fresh 18.04 image, since this seems to be working fine from where I stand.

[przemek-instapage]

Just FYI, to work around my problem, I have installed node v10.1.0 through nodesource repository and that way everything works just fine.

[nicolasnoble]

I am not surprised, yes.

[gregd72002]

Same problem here. Stock Ubuntu 18.04, nodejs: 8.10.0, npm: 3.5.2. I will try using nodesource instead

Update: using now nodesource and all works (nodejs: 8.11.1, npm: 5.8.0)

[nicolasnoble]

How did you install the distribution, @gregd72002? Was it an upgrade or a fresh 18.04 install?

[gregd72002]

It was a fresh install. Not sure if this helps but it was a brand new Azure VM: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/Canonical.UbuntuServer1804LTS?tab=Overview

[nicolasnoble]

It helps a lot, yes, thanks.

[nicolasnoble]

@gregd72002 alright, I'd love a bit more explicit reproduction case, because I'm at a loss. I've created an Azure VM following the link you gave, clicked all the next button selecting all the defaults. Eventually, I ended up on an Ubuntu VM in which I did npm install nodejs npm to install node.

Afterward, I did npm install @google-cloud/pubsub which transitively picked [email protected]. Then, I managed to safely do require('grpc') and require('@google-cloud/pubsub'), meaning I can't reproduce it on Azure either.

[fit-aditya]

I can confirm that installing nodejs v10.4.1 from nodesource and running npm rebuild --build-from-source solves this problem.
I am on Ubuntu 18.04 upgraded from 16.04

[nicolasnoble]

I can't stress this enough: we are 100% unable to reproduce this, no matter how we tried. If anyone can provide reproducible steps that creates an environment where this happens, I would be really glad.

[hi5san]

@nicolasnoble I have a ubuntu 18.04 LTS vagrant box provisioned configuration that reproduces the issue (the npm rebuild does not help), but unsure if it would help as its not configured to be a clean/simple testcase. Will you be interested in it?

[nicolasnoble]

If you're willing to either image it and give the image to me, or give me access to it, yes.

[hi5san]

@nicolasnoble I appreciate your accepting the offer despite it might not be what you're looking for. Thank you. I've prepared both Vagrantfile and a box image. Could I send the information to your email address or other direct method?

[nicolasnoble]

I just sent you an e-mail from my work address.

[hi5san]

@nicolasnoble Thanks, I've sent you details. Hope it helps any.

[nicolasnoble]

It reproduces alright, and this is gnarly.

[nicolasnoble]

Okay, I know exactly what's going on now, thanks a lot for the image and the repro case, @hi5san. Unfortunately, this isn't fixable, at all. I managed to find at least one another npm package (uws) that is plagued with the same exact problem, so this isn't on gRPC.

I am going to try creating a small reproduction case to submit a bug in Ubuntu and/or Debian - because that's an actual issue with the package itself.

The node binary in Ubuntu's package is just not conforming to the node extension ABI, and as such, will never be able to load any precompiled package out there that uses SSL. But this non-conformance is due to Debian's rules around OpenSSL, therefore I have little hopes this will ever get fixed, since fixing it would make the package non-conformant.

Your best bet in using gRPC at the moment under Ubuntu 18.04 is to:

• install nodejs using nvm
  or
• install gRPC by compiling from source: npm install --build-from-source=grpc.

I will leave the issue opened for the time being, as this is clearly a problem, and we know its cause, but don't expect any traction from it from our end beside the provided workarounds above.

[nicolasnoble]

Also the reason it took me so long to reproduce is because we're lazily loading OpenSSL's symbols. We don't get into trouble until creating an actual secure connection.

[Mambix]

fyi

Building gRPC from source did not work for me. I did

npm unistall grpc
npm install --build-from-source=grpc

So I went the nvm way:

sudo apt purge nodejs*
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
nvm install node

After this I removed node_modules folder and ran npm i just in case.
And now it's working 👍

[hi5san]

@nicolasnoble Thanks for your expertise and prompt action.

fyi -
@Mambix Thanks for your tip. It was same for me. Here's the nodesource way that I used to workaround it fyi 👍 .

curl -sL https://deb.nodesource.com/setup_8.x | sudo bash -
apt-get install -y nodejs

[nicolasnoble]

Filed a detailed Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1779863 - please follow along there instead.

[ikrestov]

I am experiencing same issue on Arch Linux, but rebuilding from source on node v8.10.x and v10.x does not help, as since node v8.10 arch packages use openssl 1.1.

So far the only thing that worked for me was downgrading to v8.9.0 (last node carbon using openssl 1.0) or building node from source and linking to openssl-1.0.

It is not a bug on arch per se, as it's supposed to be rolling release.

Maybe, until grpc supports openssl 1.1, grpc-node could check openssl version and show more helpful error message?

[murgatroid99]

Have you tried using the official Node.js Linux binary distribution that is distributed on the Node website?

[ikrestov]

I forgot about those. Tested 8.11.3, it works.

[nicolasnoble]

Also, it's not an issue of gRPC supporting OpenSSL 1.1 or not. It's an issue that there are ABI breakages between OpenSSL 1.0 and 1.1, and that the official nodejs 8 distribution is built against OpenSSL 1.0, so our prebuilt binaries are picking up OpenSSL 1.0's ABI. Loading a binary module that expects OpenSSL 1.0's ABI into a runtime that exposes OpenSSL 1.1's ABI will never work.

[nicolasnoble]

In other words, if you are building NodeJS 8 against OpenSSL 1.1, you are breaking NodeJS' ABI, and your runtime will never be able to load prebuilt binary extensions that use OpenSSL.

[nicolasnoble]

So, if arch linux does the same as Ubuntu, then it's also an arch linux bug, as this breaks the official NodeJS ABI.

[nicolasnoble]

Added an arch linux bug: https://bugs.archlinux.org/task/59394

[kwek20]

@hi5san and @Mambix together solved it for me! Thanks guys.
Also thanks @nicolasnoble for trying to fix it!

[nicolasnoble]

Note that Ubuntu and arch have agreed to fix this.

[ahmadalibaloch]

used nvm to install node 10.10, set it on work and then run npm rebuild in the directory and then it worked.

[nicolasnoble]

So, the nodejs Ubuntu package has been fixed and published as of a few hours ago, and therefore I can close this issue. This was an interesting journey.