Random test failures in travis

[ErisDS]

Not had one of these in a while, but I think it would be a good idea to have one again. I'm noticing random test failures have picked up a bit again, and that usually means there's something not-so-random about them - a timing error or race condition.

If you restart a failed build, please log the error in a comment!

Example

Job: https://travis-ci.org/TryGhost/Ghost/jobs/163983522
Error:

  1) Users API "before each" hook: setup for "Author CANNOT edit all roles":
     Error: timeout of 30000ms exceeded. Ensure the done() callback is being called in this test.

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/164456493
Errors: in a gist as there were 14!! All of them are "before all" hook errors: https://gist.github.com/ErisDS/831629d4761f1684196dc85bd3a2672d

Same build, different job
Job: https://travis-ci.org/TryGhost/Ghost/jobs/164456494
Errors: (13 this time) https://gist.github.com/ErisDS/db3ede879db2ab5d05e63501d9c785f1

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/164463453
Errors (50!): https://gist.github.com/ErisDS/2f5a19aac7fd13edf424c16ac1164b5b

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/164621966
Errors (4): https://gist.github.com/ErisDS/cab3e57f8232c53da1273d44b6968d7d

Most of them were "Maximum call stack exceeded", which is interesting? e.g.

 1) Module Setup should resolve with a ghost-server instance:
     Unable to populate database: Maximum call stack size exceeded
  Error
      at Error.InternalServerError (core/server/errors/internal-server-error.js:6:18)
      at populateDatabaseError (core/server/data/migration/populate.js:52:31)
      at tryCatcher (node_modules/bluebird/js/release/util.js:16:23)

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/164982523
Errors:

1) Slug API "before all" hook:
     Error: timeout of 30000ms exceeded. Ensure the done() callback is being called in this test.

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/165498753
Errors: https://gist.github.com/ErisDS/a63a989d7c46dfa487a3f8819729bb66
This was the maximum call stack exceeded error again, only now with better error handling we can SEE the error:

 errors.txt
  Admin Routing
Ghost Error:  GhostError {
  statusCode: 500,
  errorType: 'InternalServerError',
  level: 'normal',
  context: 'Unable to populate database!',
  help: undefined,
  name: 'InternalServerError',
  errorDetails: undefined,
  property: undefined,
  value: undefined,
  message: 'Maximum call stack size exceeded',
  hideStack: undefined }
RangeError: Maximum call stack size exceeded
    at ColumnBuilder.(anonymous function) [as notNullable] (/home/travis/build/TryGhost/Ghost/node_modules/knex/lib/schema/columnbuilder.js:48:47
...

The ColumnBuilder error repeated infinitely

Then, same build, different job, different errors!
Job: https://travis-ci.org/TryGhost/Ghost/jobs/165498760
Errors (50, but this time, we can see WHAT THEY ARE) https://gist.github.com/ErisDS/3db717bde77e31ab649f025d2762458c

[kirrg001]

knex/knex#1725

[kevinansfield]

Job: https://travis-ci.org/TryGhost/Ghost/builds/166367951
Errors:

  1) Frontend Routing Subdirectory (no slash) "before all" hook:
     Error: Timed out waiting for child process
      at Socket.<anonymous> (core/test/utils/fork.js:107:40)
      at net.js:459:14
  2) Frontend Routing Subdirectory (no slash) "after all" hook:
     Error: No forked ghost process exists, test setup must have failed.
      at Context.<anonymous> (core/test/functional/routes/frontend_spec.js:488:22)
  3) Frontend Routing Subdirectory (with slash) "before all" hook:
     Error: Child process exit code: 0
      at ChildProcess.<anonymous> (core/test/utils/fork.js:121:36)
      at Process.ChildProcess._handle.onexit (child_process.js:1078:12)
  4) Frontend Routing Subdirectory (with slash) "after all" hook:
     Error: No forked ghost process exists, test setup must have failed.
      at Context.<anonymous> (core/test/functional/routes/frontend_spec.js:564:22)
  5) Frontend Routing HTTPS "before all" hook:
     Error: Child process exit code: 0
      at ChildProcess.<anonymous> (core/test/utils/fork.js:121:36)
      at Process.ChildProcess._handle.onexit (child_process.js:1078:12)
  6) Frontend Routing HTTPS "after all" hook:
     Error: No forked ghost process exists, test setup must have failed.
      at Context.<anonymous> (core/test/functional/routes/frontend_spec.js:659:22)

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/166646096 (Node 4, SQLITE)
Errors: Not seen these before:

TypeError: Attempted to wrap emit which is already wrapped

https://gist.github.com/ErisDS/d888895e0e5d794d752eb99ff3292549

[kirrg001]

https://travis-ci.org/TryGhost/Ghost/jobs/174214440

  1) Notifications API "before all" hook:
     Error: select * from "migrations" - SQLITE_IOERR: disk I/O error
      at Error (native)

[ErisDS]

Job: https://travis-ci.org/TryGhost/Ghost/jobs/175630488
Errors:

1) Channel Routes Tag Paged "after all" hook: teardown:
     Error: timeout of 30000ms exceeded. Ensure the done() callback is being called in this test.

  2) Channel Routes Tag "after all" hook: teardown:
     Error: done() invoked with non-Error: {"statusCode":500,"name":"KnexMigrateError","errorType":"KnexMigrateError","id":0,"message":"DROP DATABASE ghost_testing; - ER_DB_DROP_EXISTS: Can't drop database 'ghost_testing'; database doesn't exist","code":"ER_DB_DROP_EXISTS","errno":1008,"sqlState":"HY000"}

• 2 more errors, see gist for full log

https://gist.github.com/ErisDS/239fd0c3b2c23e0c91786cadc91a1123

[kirrg001]

We experience less random test failures, especially on master. Over the time we added some improvements to our functional tests - maybe that helped 🤖 🚀

[ErisDS]

If you want to close this and reopen if the issue occurs again that seems sane? 😁

[kirrg001]

Totally sane 😎

[ErisDS]

Got a new one: https://gist.github.com/ErisDS/96ab4781c16c12b083c80910d18389bc
Happened once on travis, happened a couple of times for me locally over the last week.

[kirrg001]

👍 Thanks

[kirrg001]

Got a new one: https://gist.github.com/ErisDS/96ab4781c16c12b083c80910d18389bc

Not seeing this one anymore with knex-migrator 2.0.9.

Discovered a new/old/different one:

Ghost Error:  { InternalServerError: (EADDRINUSE) Cannot start Ghost.
  at new GhostError (/home/travis/build/TryGhost/Ghost/core/server/errors.js:9:26)
  at Server.<anonymous> (/home/travis/build/TryGhost/Ghost/core/server/ghost-server.js:82:30)

https://gist.github.com/kirrg001/5aa476c011fcb876ba78e84b19e96be4

[kirrg001]

https://gist.github.com/kevinansfield/bb56229a4c78975f280c433fb5a24d50

I am not sure that these errors are fixable, because they are caused by a timeout (might be travis being slow and then a number of other tests start failing).

---

I would like to pause looking at random failures for now and wait for more recent reports. Otherwise i am searching in the dark.

@AileenCGN @ErisDS @kevinansfield
If you see any tests failing, please report here again. Thanks :)

[kirrg001]

I am closing this issue, haven't seen any random failure in the last 3 days. We can always re-open :)

[kirrg001]

We are seeing new random timeouts. Mostly for routing tests as far as i can tell.
This could be caused by Travis but we have to investigate. Travis sometimes doesn't even start a build.

@ErisDS had to rebuild her PR 4 times - very annoying.

[kirrg001]

We are seeing new random timeouts. Mostly for routing tests as far as i can tell.
This could be caused by Travis but we have to investigate. Travis sometimes doesn't even start a build.

Have not seen this in the last week. I assume it's only happening if Travis is very busy? Not really sure what to improve 🤔

[kirrg001]

I really have the feeling that the tests (mostly) fail, when travis is very busy.
e.g. in the evening (UTC+2) the spike of builds increases a lot (https://www.traviscistatus.com/).

[ghost]

I've currently got this PR open which fixes a couple of the responsive layout issues in the Ghost Admin. I cannot get it to pass the Travis checks and I'm 99% sure the "not OK" issues are nothing to do with my code changes... I could be wrong though:

not ok 177 Chrome 65.0 - Acceptance: Signup can signup successfully
not ok 178 Chrome 65.0 - Acceptance: Signup "after each" hook for "can signup successfully"
not ok 170 Chrome 65.0 - Acceptance: Setup with a new blog has a successful happy path

These failures could be entirely my fault, but I'm not sure. When I run ember test locally, these tests actually pass, which is even more confusing 😢

https://travis-ci.org/TryGhost/Ghost-Admin/builds/360482091

[kirrg001]

Travis currently always fails randomly on

Node v6, MySQL

1. Spam Prevention API Ensure reset works: password grant type:
   Error: Timeout of 30000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

Would be great to figure out the reason. Any help is very appreciated 👍

[kirrg001]

And it happens again.

  1) Spam Prevention API Ensure reset works: password grant type:
      AssertionError: expected 2 to equal 1
      + expected - actual
      -2
      +1
      
      at Assertion.fail (node_modules/should/cjs/should.js:275:17)
      at Assertion.value (node_modules/should/cjs/should.js:356:19)
      at core/test/functional/api/v0.1/spam_prevention_spec.js:95:44

I am getting this random failure now all the time.

[kirrg001]

Closed via #10281