Build hangs on CI (Traivs) #1333
Comments
|
Doesn't really make sense because there are lots of other travis builds / runs / etc that do complete correctly. Not seeing this in any other CI provider either :/ |
|
Travis doesn't offer a way to SSH in, so it's impossible to iterate in their environment. |
|
@brian-mann Seeing this error on jenkins 2.105 also. |
|
I can look through the server code and likely guess where this is failing. After it displays the Before doing that, it will end the video capture stream on ffmpeg. If this promise does not resolve... it will hang indefinitely. Likely the root issue is something with the @kartikupreti does this happen every time for you? Nothing has actually changed related to this behavior in |
|
Reading through the code some more... it's actually possible that its hanging on closing the browser. It actually does that first before attempting to end the What we can do is add better debug logs for both of these tasks so we can definitely see where it's hanging. |
|
@kartikupreti After investigated #674, this error is completely and totally unrelated to it. In this error, the tests are completing... Cypress just isn't completing the processes that come after all of that. |
|
@brian-mann Yes, apologies I just re-read #674, this is not related. |
|
@brian-mann It doesn't seem to be related to me setting the videoRecording option. However, I upgraded to cypress 2.0.2 and ran a few times and have pasted the debug log below if it helps. For a successful quit. and a 'hung' run - was hung for 15 mins printing the logs from timers component till a SIGINT was sent from jenkins. |
|
Can you update to 3.0.2 and see if this issue is resolved? We updated ffmpeg to 4.x.x there and added many more debugging logs. Also please be sure that you use DEBUG=cypress:* cypress run when running so that we can get the debug logs to understand what is happening. |
|
@brian-mann I'm seeing tests hang occasionally on 3.0.2, running with After that it hangs indefinitely. Let me know if any other logs would be helpful. |
|
Yup everyone is having the same issue. We've added more debug logs in It honestly makes no sense because it appears to be hanging on synchronous code randomly. With additional debug logs we should have a clearer picture but regardless we'll figure it out. To workaround this todayJust switch to using Chrome + our Docker containers (or install them yourself). I believe Chrome is already installed by default in Travis. That will "just work" as this is an Electron only problem. |
|
@brian-mann This works. Thank you. However, now our team must decide either to lose confidence in our tests (occasionally they hang) or we don't test our Stripe payment flows. Please please please fix soon! |
|
We have spent a considerable amount of time tracking this down and looking into it. The problem is that it fails randomly (and not in every CI provider). Because of that, it's extremely difficult to understand the root cause. It's a high up item on our list and we'll get it fixed soon. We prioritize problems in CI. |
brian-mann
added
the
stage: ready for work
|
@brian-mann I'm seeing similar behavior with Chrome inside the Docker container. I'm on version After about half a dozen feature files the test freeze up while switching from one test to another. The Should this be a separate issue or likely the same cause? |
|
Mine might be more fitting under #1912 |
|
I experience this issue today on CI (specs below). I tried running with Chrome (also inside Docker container) but experienced the same issue as using electron. I'm on Cypress 3.0.2 and running on CircleCI. As a workaround, I wrapped the Cypress runtime in a Node script and that allowed my test process to finish. Sharing the approach here in case it helps anyone else. (uses the death package for process cleanup) const ON_DEATH = require('death')
const { spawn } = require('child_process');
const cy = spawn('node', ['./node_modules/.bin/cypress', 'run', '--record'], {
cwd: __dirname,
env: process.env
});
let out = ''
const end = () => {
process.kill(cy.pid);
// looks for X of Y failed in the test summary, if finds, exits with 1
// otherwise exits with 0.
process.exit(/\d+\s+of\s+\d+\s+failed/.test(out) ? 1 : 0)
}
cy.stdout.on('data', (data) => {
const str = data.toString();
out = out + str;
// a simple test to listen for the end of the run
// wait 20secs to exit on its own
if (/Run Finished/i.test(str)) {
setTimeout(end, 20 * 1000);
}
console.log(str)
});
cy.stderr.on('data', (data) => {
console.error(data.toString());
});
cy.on('close', (code) => {
process.exit(code);
});
ON_DEATH(() => {
process.kill(cy.pid);
});Then, in my bash script, I call Thanks for the debugging/fixes from the Cypress team. Look forward to getting this one solved! |
|
@sjones6 You may find our Module API helpful for doing what you're doing. https://on.cypress.io/module-api |
|
@jennifer-shehane : Thanks for pointer. That does look like a cleaner way to implement that. I'd just need to confirm that the module API doesn't have the same problem hanging that the CLI has on CI. I'll give it a try next week and report back if so. |
|
@sjones6 did you end up leveraging the module-api to accomplish the same thing. Our team is having the same issue and updating to chrome + docker did not help us out. |
|
Environment: CircleCI-2.1 (docker image: cypress/base:8) As our CI/CD workflow requires 0 manual intervention, we wanted to reduce any manual re-runs so we borrowed from @sjones6, we are making use of the following script. It's not foolproof but it is reducing the amount of manual intervention necessary. I will file a separate ticket with my debug logs but for context we are running over 100 tests across 16 containers and this happens about 1/4 times on the same 2 containers. #!/usr/bin/env node
/* eslint-env node */
const ON_DEATH = require('death')
const {spawn} = require('child_process')
const path = require('path');
const cypress = require('cypress');
const parseArgs = require('minimist');
const fs = require('fs-extra');
const console = require('console');
const getSpecs = () => {
const argv = parseArgs(process.argv.slice(2));
if (argv && argv._ && argv._.length > 0) {
return argv._.join(',') ;
}
throw new Error('--specs are required');
};
const specs = getSpecs();
var lastMessage = new Date();
var pids = [];
var spawnCy = () => {
var localCy = spawn('node', [
'./node_modules/.bin/cypress',
'run',
'--spec', specs,
'--record'], {
env: process.env
});
localCy.stdout.on('data', (data) => {
lastMessage = new Date();
process.stdout.write(data.toString())
});
localCy.stderr.on('data', (data) => {
lastMessage = new Date();
process.stderr.write(data.toString());
});
localCy.on('close', (code) => {
process.exit(code);
});
return localCy;
}
var cy = spawnCy(specs);
pids.push(cy.pid);
var numRetries = 0;
const end = () => {
let pid = cy.pid
process.kill(pid, 'SIGKILL');
console.error(`Killed pid ${pid}`)
// looks for X of Y failed in the test summary, if finds, exits with 1
// otherwise exits with 0.
// process.exit(/\d+\s+of\s+\d+\s+failed/.test(out) ? 1 : 0)
}
const TIMEOUT_FROM_LAST_MESSAGE_IN_SEC = 3 * 60;
const INTERVAL_OF_CHECKS_IN_MS = 30 * 1000;
var checkLastMessage = () => {
var diffInSeconds = (new Date() - lastMessage)/1000;
console.log(`Last time we saw a message was ${diffInSeconds} seconds ago`)
if (diffInSeconds > TIMEOUT_FROM_LAST_MESSAGE_IN_SEC) {
console.log("The last time we saw a message was greater than 3 minutes ago, killing");
end(cy.pid);
console.info(`Retries: ${numRetries}`)
if (numRetries < 4)
{
cy = spawnCy(specs);
pids.push(cy.pid);
numRetries++;
}
else
{
console.error("Retried 4 times to re-run cypress and still couldnt get it to not get stuck");
process.exit(1);
}
}
setTimeout(checkLastMessage, 30000);
}
setTimeout(checkLastMessage, 30000);
ON_DEATH(() => {
pids.forEach(pid => {
try {
console.info(`On death cleaning up if we got stuck along the way. ${pid}`)
process.kill(pid);
} catch (err) {
console.error(`Caught error with killing pid ${pid}. Probably already killed.`)
}
});
});our package.json {
"name": "cypress_build",
"version": "0.1",
"engines": {
"node": ">=8.9"
},
"dependencies": {
"cypress": "3.1.0",
"death": "1.1.0",
"fs-extra": "5.0.0",
"minimist": "1.2.0",
"mocha": "^3.0.1",
"mocha-junit-reporter": "^1.13.0",
"mocha-multi-reporters": "^1.1.4"
},
"scripts": {
"cypress:run": "./scripts/run_cypress.js",
....We then call |
|
@sjones6 switch to Cypress |
|
I'm going to close this issue as I believe it's been resolved by the primary cause of the specs hanging in |
jennifer-shehane
removed
stage: ready for work
jennifer-shehane
added
CI
Seeing this error for Cypress 2.0.0 and 2.0.1 on Travis CI. The build has finished successfully, but keeps hanging.
Example https://travis-ci.org/cypress-io/cypress-example-recipes/jobs/342663254
The text was updated successfully, but these errors were encountered: