Modules fail on "npm start"

[doccaz]

Description

While trying out the self-hosting instructions from README, some modules will fail to start after the "npm start".

Steps to reproduce

After installing the system dependencies (docker, rust, etc):

# git clone https://github.com/mozilla/fxa.git
# cd fxa
# npm install
# npm start

The command "./pm2 status shows that the modules "auth-server key server PORT 9000", "event-broker", "payments server PORT 3031" and "support admin panel PORT 7100" errored. All other modules start fine.

Expected result

All modules should start up, or at least give a meaningful error message.

Actual result

The output of "pm2 logs" for all failed modules are similar to this:

There are no other logs that I could find. It seems to be interpreting the /usr/bin/node shell script as a javascript file(???)

Environment

SUSE Linux Enterprise Server 15 SP1, NodeJS 10, Docker 19.03.1.

┆Issue is synchronized with this Jira Task
┆Issue Number: FXA-6

[LZoog]

Hey @doccaz, our master branch just recently updated to Node 12 and we need to update our docs. Will you try deleting your node_modules and npm i'ing while using Node 12 and verify Docker is running and other dependencies are installed? Try also restarting those servers, or just running ./pm2 restart. We'll need more log details to help much further since I haven't seen this issue before, but it's likely a local problem somewhere.

[doccaz]

Hi @LZoog , I updated to node 12, deleted not only node_modules but also the .npm, .cache, .local, .cargo, .pm2 directories and pulled the fxa repo again to start from scratch, with the same results. Here are my PM2 logs and the output of "docker ps -a".

pm2logs.zip
docker-ps.txt

I did find some references on the net about these being caused by missing "shebangs" on .js files. But the modules in question appear to have them.

[doccaz]

If I try to go into packages/fxa-event-broker and packages/fxa-auth-server and run "npm run start", I get these outputs:
2019-12-19T21_16_50_696Z-debug.log
2019-12-19T21_16_26_424Z-debug.log

[jaredhirsch]

(Self-assigning, since I started the self-hosting discussion recently on dev-fxacct. Thanks for jumping in, @LZoog! Feel free to add suggestions as we go along here.)

@doccaz Sorry it's not working for you. FxA doesn't have any SUSE folks on the team right now, so I'm not sure how nodejs differs (if at all) in your environment, but I think we can do some more debugging. How did you install node on your system? Package manager or via the website?

For event-broker, it looks like you should try a slightly different command, npm run start-dev. Maybe that will work?

For auth server, I've run into some annoying issues with it complaining that the pushbox.key config value must be set in production. Try setting NODE_ENV=dev when you npm run start, and see if that helps.

If you need to upload logs, probably best to pastebin them and add a comment here with the link.

I know that some folks on the team do their development in a docker image; if you're interested in that, I can ask around and get them to drop some hints (and maybe we'll add them to the self-hosting docs, too).

[doccaz]

@doccaz Sorry it's not working for you. FxA doesn't have any SUSE folks on the team right now, so I'm not sure how nodejs differs (if at all) in your environment, but I think we can do some more debugging. How did you install node on your system? Package manager or via the website?

Hi there! I installed node via the RPM package maintained at SUSE's OBS (Open Build Service):
https://build.opensuse.org/project/show/devel:languages:nodejs

We use the update-alternatives method for maintaining multiple versions installed. Maybe that is what's causing the confusion? I'll try removing our packages and using the one from the site. We sure would like it to be as compatible as possible, feel free to suggest any changes to our packaging. Maybe we could even try to package Sync on OBS in the future? 😄

For event-broker, it looks like you should try a slightly different command, npm run start-dev. Maybe that will work?

I'll try that, thanks.

For auth server, I've run into some annoying issues with it complaining that the pushbox.key config value must be set in production. Try setting NODE_ENV=dev when you npm run start, and see if that helps.

Will try that too and report back, thanks!

If you need to upload logs, probably best to pastebin them and add a comment here with the link.

Noted.

I know that some folks on the team do their development in a docker image; if you're interested in that, I can ask around and get them to drop some hints (and maybe we'll add them to the self-hosting docs, too).

That would be fantastic! I'm currently reviewing the Firefox Sync suite for a customer, which might use it for (quite a few) workstations. This image would be great to compare my environment.

[doccaz]

Follow-up: I was able to start the missing services manually using NODE_ENV=dev. Later on I removed all installed native nodejs/npm packages and tried the pre-compiled binaries from nodejs.org. The services started fine with PM2 after a git pull, so there is definitely something wrong in the call chain from PM2 to /usr/bin/node that causes the problem with the bash script being called as JS. Will investigate and open a proper issue/pull request in the future. Thanks for the help so far! Will close this issue.

Just one last question: when I access the sign-up page on port 3030, I get either a "system unavailable" or "Loading chunk fxaClient failed" error. I noticed some errors about fxa-geodb on the content server, and found a couple of issues mentioning that this error occurs when using an invalid/internal IP, as it cannot find out the geographical location. Can this be bypassed? I intend to use this in a completely private network, with invalid addresses (10.x.x.x).

[jaredhirsch]

Hey @doccaz, sorry about the delay getting back to you. Glad you've gotten the stack working locally. Am I correct in understanding that you had to use the version of node from nodejs.org, and not the version installed by the SUSE Enterprise package manager? I don't know if update-alternatives would cause any problems, but I kind of doubt it. I think our code just assumes node is on the user's path, but I'll poke around.

As far as other errors you're encountering when running locally, I think it's probably best to open new issues for individual error messages as you encounter them. That'll give more experienced people on the team a chance to comment, and it'll leave some clues for the next person who runs into those errors

(Edit: realized you already mentioned which RPM repository you used in your previous comment)

[doccaz]

Hey @doccaz, sorry about the delay getting back to you. Glad you've gotten the stack working locally. Am I correct in understanding that you had to use the version of node from nodejs.org, and not the version installed by the SUSE Enterprise package manager? I don't know if update-alternatives would cause any problems, but I kind of doubt it. I think our code just assumes node is on the user's path, but I'll poke around.

Yes, that's correct. I uninstalled the distro's packages for nodejs and used the pre-compiled binaries from nodejs.org, and the errors went away. I think somewhere along the call chain from the first "npm start" it wrongly assumes that the /usr/bin/node stub (which is a bash script) it a .js file.

I found quite a few reports about this problem with many other nodeJS-based projects (but mostly on Windows), like this, this, this and this.

The first issue mentions this:

In fact, SyntaxError: missing ) after argument list is reported by node.js and is not reported by shell. So this does not necessarily mean something goes wrong with the shim shell script.
With some reading of the code of istanbul and testing, I think this is caused by running the shim shell script via node.

The last issue mentions that:

It seems like the solution is to replace the string ./node_modules/.bin/webpack in the package.json command with ./node_modules/webpack/bin/webpack.js

so it appears like the system finds a file with no extension that has a shebang, which calls a new node instance... which is a bash script, and somehow is wrongly interpreted as a JS.

[vbudhram]

Thank you for filing this issue. This is not currently on our roadmap, and in an effort to focus our work, we are closing old issues that we are unlikely to be closed in the future. Thanks again.