The Chromium Renderer process just crashed - 'sad face' Error

[JimLynchCodes]

Hi, I am running m cypress script in headless mode on a remote ubuntu 18.03 machine. The strange thing is that it works fine on my local macbook, but I'm getting an error that the Chromium renderer is crashing when running on the ubuntu server. Here is the output:

We detected that the Chromium Renderer process just crashed.
[1] 
[1] This is the equivalent to seeing the 'sad face' when Chrome dies.
[1] 
[1] This can happen for a number of different reasons:
[1] 
[1] - You wrote an endless loop and you must fix your own code
[1] - There is a memory leak in Cypress (unlikely but possible)
[1] - You are running Docker (there is an easy fix for this: see link below)
[1] - You are running lots of tests on a memory intense application
[1] - You are running in a memory starved VM environment
[1] - There are problems with your GPU / GPU drivers
[1] - There are browser bugs in Chromium
[1] 
[1] You can learn more including how to fix Docker here:
[1] 
[1] https://on.cypress.io/renderer-process-crashed
[1] TypeError: onError is not a function
[1]     at BrowserWindow.onCrashed (/home/ubuntu/.cache/Cypress/4.4.0/Cypress/resources/app/packages/server/lib/modes/run.js:538:7)
[1]     at WebContents.<anonymous> (/home/ubuntu/.cache/Cypress/4.4.0/Cypress/resources/app/packages/server/lib/gui/windows.js:181:34)
[1]     at WebContents.emit (events.js:215:7)
[1] 
[1] TypeError: onError is not a function
[1]     at BrowserWindow.onCrashed (/home/ubuntu/.cache/Cypress/4.4.0/Cypress/resources/app/packages/server/lib/modes/run.js:538:7)
[1]     at WebContents.<anonymous> (/home/ubuntu/.cache/Cypress/4.4.0/Cypress/resources/app/packages/server/lib/gui/windows.js:181:34)
[1]     at WebContents.emit (events.js:215:7)
[1] 
[1] The Test Runner unexpectedly exited via a exit event with signal SIGILL
[1] 
[1] Please search Cypress documentation for possible solutions:
[1] 
[1] https://on.cypress.io
[1] 
[1] Check if there is a GitHub issue describing this crash:
[1] 
[1] https://github.com/cypress-io/cypress/issues
[1] 
[1] Consider opening a new issue.
[1] 
[1] ----------
[1] 
[1] Platform: linux (Ubuntu Linux - 18.04)
[1] Cypress Version: 4.4.0

Is there any way for me to see why it is crashing? 🤔

[JimLynchCodes]

hmm it seems that there is some memory leak with Cypress. After running it locally and using Activity Monitor to profile the "Google Chrome Helper (Renderer)" process I can see that it starts at about 400mb memory and slowly climbs up to 4gb as the test runs... 🤔

[JimLynchCodes]

I added the profiling environment variable DEBUG=cypress:server:util:process_profiler cypress run, but I'm still not really sure what's causing the issue.

My output looks like this image below. It seems that with every print the first like, "Electron", is taking more and more memory usage. Everything else seems to be pretty stable.

[zac11]

I am facing the same issue when trying to run Cypress on local Jenkins. See this Issue-7200

[JimLynchCodes]

hmm... how are you getting the exact memory usage displayed when it crashes? that's pretty cool. haha

I am running it on a 1gb digital ocean linux server, but still I wouldn't expect it to just continually increase in memory.

@zac11 are you switching pages a lot? I am ,and it seems like every time I switch pages memory goes up like 150mb. I am also creating big objects on each page (but I was hoping they would be freed once the leaving the page).

[jennifer-shehane]

The CPU usage is based on a number of things. The Cypress app itself is running, Chromium is running and the application under test is running. If you're doing very memory intensive things within testing the app, then yes, this will require more CPU on the machine to handle. If it's maxing out, we recommend increasing the machine resources.

[JimLynchCodes]

Thanks @jennifer-shehane, but I don't think I am doing very memory-intensive things.

I am going to a few different pages (and on each pages I would think the memory for the previous pages could be freed, which it seems that it is not).

I am building up an object of data, but after every 4 pages I reassign that object to {}. I was thinking that would allow the memory it was taking up to be freed, but it seems not.

The memory never goes down at all-just continually goes up. 🤔

[zac11]

@JimLynchCodes Did you try the same tests in Chrome headless?

I disabled the shm usage as per the suggestion from @jennifer-shehane here #7200 and even then Electron would crash and the Chromium renderer error would be thrown.

I changed the run command to have --headless --browser chrome and now the tests are running fine. Can you check if your tests fail on Chrome too?

[zac11]

Also how did you do this. I want to have a comparison between Electron and Chrome to see the difference

[JimLynchCodes]

@zac11 hmm I did not disable shm usage... I will try that. Btw does anyone know what "shm" stands for? lol

In node you can set arbitrary PATH variables before you execute the script / command. It seems like "cypress:server:util:process_profiler" is some special value for the DEBUG variable that cypress looks for.

So if you normally run cypress run then you can run it with the extra profiling output like this:

DEBUG=cypress:server:util:process_profiler cypress run

[probil]

I have the same issue while running cypress in headless mode in Gitlab CI. We have only 5 tests

Version 4.4.1 - works fine
Version 4.5+ - crashes almost every time

We are using docker image provided by cypress: cypress/base:12.18.0 which should be fine but for some reason it started to fail.

For now we stick to version 4.4.1 as it seems to be stable enough. We are considering firefox as it tend to use less memory but we haven't gave it a shot just yet

[probil]

We setup a custom gitlab runner as described here
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2168#note_190617588
It works fine now

[jennifer-shehane]

Unfortunately we'll have to close this issue if there is not enough information to reproduce the problem. This does not mean that your issue is not happening - it just means that we do not have a path to move forward.

There's a lot of things that could contribute to the browser crashing, including issues from the underlying application under test that don't have to do with Cypress. Please comment in this issue with a reproducible example so we can assess that this is an issue specific to Cypress.

[JimLynchCodes]

Sorry I haven't commented on this in a while.

To be fair I was trying to run this on the 1GB cheapest digital ocean server. 😅

Currently I have it running via travisCI which is free since the code is on a public repo (those boxes seem to be ~7gb, and the tests finish successfully after about 4 minutes).

I am doing some relatively intense stuff (like scraping a bunch of text from the page, storing it in a giant object, and running a little backend server on the side which saves the object to a db...)

It would be really interesting though to get some kind of memory profile (over time?) that shows how much mem is being consumed by cypress and by each of these other things. 🤔

[CoryDanielson]

I had a similar issue to what you're describing. Running tests in parallel will mostly mitigate the issue, but a big spec file could still cause the process to crash. I was able to address the memory issue by exposing the garbage collector and running it between tests.

See #8525 and #350 (comment)

Garbage collection is an expensive operation. Javascript's automatic garbage collector will try to stay out of your way, and only run when there is downtime in the code. The issue with this is that even very basic Cypress tests are quite intensive programs that do not have much downtime. When one test finishes, another test runs and so on until there are no tests remaining. The garbage collector will sometimes step in to free up memory, but not often enough to prevent out of memory crashes.

When testing a complex page with lots of requests and events, this issue becomes more noticeable. As Electron/Chrome approaches their max MB threshold (2GB by default), the JS runtime does not seem to get more aggressive with garbage collection. In my case, I saw 1.8GB used, and no decrease -- the next test to kick off would put it over 2GB and the app would crash.

[jennifer-shehane]

@JimLynchCodes You can log the memory and CPU usage of Cypress by following these instructions: https://on.cypress.io/troubleshooting#Log-memory-and-CPU-usage This will help determine if Cypress is running out of resources and whether the machine requires more.

[jennifer-shehane]

Closing as resolved. Running Cypress does require some resources, so running it on an underpowered machine can cause Chrome to crash, especially when testing pages with lots of requests and DOM interactions.

@CoryDanielson We'll have to take a look at the window.gc() suggections.

If you're experiencing a bug similar to this in Cypress, please open a new issue with a fully reproducible example that we can run. There may be a specific edge case with the issue that we need more detail to fix.

[trainoasis]

This fixed it for me: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2168#note_190617588

[garyburgmann]

a combination of:

using image:

image: cypress/browsers:node14.16.0-chrome89-ff86

adding --disable-dev-shm-usage for chrome:

module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // ignore examples/
  config.ignoreTestFiles = "**/examples/*.spec.js";

  // https://github.com/cypress-io/cypress/issues/349
  // add --disable-dev-shm-usage chrome flag
  on('before:browser:launch', (browser, launchOptions) => {
    if (browser.family === 'chromium') {
      console.log('Adding Chrome flag: --disable-dev-shm-usage');
      launchOptions.args.push('--disable-dev-shm-usage');
    }
    return launchOptions;
  });

  return config;
}

and explicitly running as chrome:

$ npx cypress run --browser chrome

got this working for me