ARM test timeouts

[silverwind]

A few test currently fail on ARM because they have too high expectations for processing time for these rather slow platforms. I'll work in fixing this for v6 and v7 which I can test on both RPi 1 and 2.

@rvagg you mention in #283 (comment) a very-slow flag for tests. Could you point me how I'd add something like that to the test framework? Maybe you have an branch I could base my work off?

[silverwind]

Failing tests are:

ARMv6:

parallel/test-child-process-fork-net2
parallel/test-fs-empty-readStream (may not be timeout related)
parallel/test-debug-signal-cluster
parallel/test-http-end-throw-socket-handling
parallel/test-repl-timeout-throw (may not be timeout related)
parallel/test-stringbytes-external
parallel/test-tls-fast-writing
parallel/test-tls-wrap-timeout
sequential/test-force-repl
sequential/test-net-GH-5504
sequential/test-pipe

ARMv7

parallel/test-child-process-fork-net2
parallel/test-tls-wrap-timeout
sequential/test-force-repl

[silverwind]

Looks like my overclocked RPi2 already passes all tests as-is. It barely makes the repl test in like 900ms of the 1000ms available. Standard clocked RPi2 still needs like 1300ms though :P

WIP branch: https://github.com/silverwind/io.js/tree/arm-test-timeout

[rvagg]

@silverwind tbh I was just imagining something in common.js that did a lazy check of the arch when requested and if it returned armv6 then it would be identified as a "slow" platform and the tests that run too slow would internally adjust their timeout length. I haven't looked in detail at how the armv7 builds are going but we may need to have a semi-slow in there as well, or else adjust the timeouts for all platforms to make them work.

Currently in the CI mix we have:

• iojs-armv7-ubuntu140 - an Odroid XU3 which is a bit of a beast and doesn't seem to have trouble with the tests
• iojs-armv7-wheezy - a couple of armv7 servers provided by Scaleway that run Debian Wheezy armhf (i.e. not a Raspbian variant or anything fancy). I haven't been watching test results for these and I don't know much about the hardware these are running on but I'm betting they aren't as powerful as the XU3 machine.
• iojs-armv8-ubuntu1404 - armv8 / aarch64 provided by Linaro on their cluster, this is on an APM X-Gene Mustang board, I doubt we're at the stage of looking at timeouts on this
• iojs-raspbian-wheezy-pi1 - an armv6 rotation of 2 x Pi v1 and 1 x Pi v1plus, this is where we have most of the timeout problems and a lot of it seems to be related to how long it takes to start a process
• iojs-raspbian-wheezy-pi2 - one Pi v2 which is armv7, I haven't looked in detail at the state of timeouts and how this compares with the two other armv7 configurations.

/cc @iojs/platform-arm @iojs/build

[Fishrock123]

iojs-armv7-ubuntu140 - an Odroid XU3 which is a bit of a beast and doesn't seem to have trouble with the tests

Samsung Exynos5422 Cortex™-A15 2.0Ghz quad core and Cortex™-A7 quad core CPUs [1]

Dang. :P

iojs-armv7-wheezy - a couple of armv7 servers provided by Scaleway that run Debian Wheezy armhf (i.e. not a Raspbian variant or anything fancy). I haven't been watching test results for these and I don't know much about the hardware these are running on but I'm betting they aren't as powerful as the XU3 machine.

I did some digging about this the other day, best I could find is https://www.scaleway.com/faq/server, which doesn't have too much perf / processor info. I do suspect these have higher perf than an rpi2 however.

[rvagg]

yeah, the XU3 is a beautiful piece of hardware, even has a tiny little fan on it that spins up occasionally, I'd like a second one for the cluster so I might put out a call some time to get the funds for one.

The Scaleway machines have 3 of these:

processor   : 3
model name  : ARMv7 Processor rev 2 (v7l)
Features    : half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae
CPU implementer : 0x56
CPU architecture: 7
CPU variant : 0x2
CPU part    : 0x584
CPU revision    : 2

Reporting:

Hardware    : Marvell Armada 370/XP (Device Tree)
Revision    : 0000
Serial      : 0000000000000000

[Fishrock123]

Only 3 cores eh? the scaleway deployed ones are supposed to be 4-cores. Hmm.

[rvagg]

sorry, I meant 4

[silverwind]

Here's what I currently have:

https://github.com/silverwind/io.js/blob/2774b24050464e031d2824f0cfbaa4b3325e90b3/test/common.js#L182-L200

I think the best approach would probably be to benchmark process startup time once in common.js and calculate a "slowness factor" from that. Could probably do that for all platforms or just ARM.

[rvagg]

Maybe make it lazy like opensslCli, inFreeBSDJail and localhostIPv4 so the user of the variable doesn't need to care about how it works and it doesn't need to be cached within tests.

At one stage I had the impression that some timeouts needs a 10x increase on the Pi1 machines. Perhaps you could just wire up the test cases that need this and tune from there though.

[silverwind]

Yeah, the armv6 value is arbitrary right now. I only ran armv7 so far, which seems to need something like 1.3-1.4 on a standard RPi2.

[jbergstroem]

How about moving test we know are slow to a slow folder? That way we can run them as a separate job (tools/test.py --timeout $foo slow)

[silverwind]

@jbergstroem Yeah we could bunch them together so we know the problematic ones. Would you suggest creating subfolders like slow/parallel and slow/sequential?

Also, not sure about that arg. I think with a quick benchmark in common.js in place, we don't need to to fiddle with timeout values.

[jbergstroem]

@silverwind my suggestion would be to just have one level depth. Half of the problem with timeouts is that the test runner might be busy doing other stuff. Running them sequentially will reduce time spent in each test (but probably longer overall).

[bnoordhuis]

We have test/pummel for slow tests. I filed PR #1357 for fudging the per-test timeout factor.

[silverwind]

Maybe make it lazy like opensslCli, inFreeBSDJail and localhostIPv4

Those are getters which can't have arguments, so not really possible through a getter at least.

[silverwind]

Fixed by 7049d7b

[rvagg]

whoa, all green (blue), great work everyone!

[silverwind]

Now we just need to squash the few intermittent failures that come up on CI randomly.