Several OSUOSL AIX boxes offline

[sxa]

Some of the older OSUOSL boxes are currently offline. My understanding is that there has been a power issue which will hopefully be resolved soon. EDIT(@sam-github): note this affects ONLY the 3 {test,release}-osuosl-aix61-* hosts, not centos/ubuntu ppc64 (I guess they are on different power).

[sam-github]

Hopefully it won't take too long, or CI will backup. If it becomes a big issue, I'll look into how to remove node-test-commit-aix from the job deps. If there is a lightweight way to pull it out temporarily I will.

But lets see when power comes back up.

[richardlau]

Hopefully it won't take too long, or CI will backup. If it becomes a big issue, I'll look into how to remove node-test-commit-aix from the job deps. If there is a lightweight way to pull it out temporarily I will.

But lets see when power comes back up.

@sam-github Just disabling the node-test-commit-aix project should be sufficient.

or

[sam-github]

@richardlau Thanks! I'll keep an eye on https://ci.nodejs.org/job/node-test-commit-aix/, if I see the backlog grow, I'll disable it.

[sam-github]

Update: Just saw in internal chat that a manual reboot is required, and is being done now.

[sam-github]

machines are online and ramdisks are mounted:

% parallel-ssh -h hosts/aix61-ppc64_be -i "df | grep ram && mount | grep ram"                                                      
                                       [1] 11:43:41 [SUCCESS] test-osuosl-aix61-ppc64_be-3
/dev/ramdisk0   10000000   9956864    1%        4     1% /ramdisk0
/dev/ramdisk1   23000000  22905728    1%        4     1% /home/iojs/build
         /dev/ramdisk0    /ramdisk0        jfs2   Dec 19 11:38 rw,log=/dev/ramdisk0
         /dev/ramdisk1    /home/iojs/build jfs2   Dec 19 11:44 rw,log=/dev/ramdisk1
[2] 11:43:41 [SUCCESS] release-osuosl-aix61-ppc64_be-1
/dev/ramdisk0   10000000   9956864    1%        4     1% /ramdisk0
/dev/ramdisk1   23000000  22905728    1%        4     1% /home/iojs/build
         /dev/ramdisk0    /ramdisk0        jfs2   Dec 19 13:39 rw,log=/dev/ramdisk0
         /dev/ramdisk1    /home/iojs/build jfs2   Dec 19 13:44 rw,log=/dev/ramdisk1
[3] 11:43:41 [SUCCESS] test-osuosl-aix61-ppc64_be-1
/dev/ramdisk0   10000000   9956864    1%        4     1% /ramdisk0
/dev/ramdisk1   23000000  22883168    1%      118     1% /home/iojs/build
         /dev/ramdisk0    /ramdisk0        jfs2   Dec 19 13:38 rw,log=/dev/ramdisk0
         /dev/ramdisk1    /home/iojs/build jfs2   Dec 19 13:44 rw,log=/dev/ramdisk1
[4] 11:43:41 [SUCCESS] test-osuosl-aix61-ppc64_be-2
/dev/ramdisk0   10000000   9956864    1%        4     1% /ramdisk0
/dev/ramdisk1   23000000  22905728    1%        4     1% /home/iojs/build
         /dev/ramdisk0    /ramdisk0        jfs2   Dec 19 13:38 rw,log=/dev/ramdisk0
         /dev/ramdisk1    /home/iojs/build jfs2   Dec 19 13:44 rw,log=/dev/ramdisk1

[richardlau]

I resumed a build and it failed: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/27414/console

14:51:35 ccache: error: Failed to create directory /home/iojs/.ccache/tmp: No such file or directory

[richardlau]

I resumed a build and it failed: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/27414/console

14:51:35 ccache: error: Failed to create directory /home/iojs/.ccache/tmp: No such file or directory

@sam-github is the above related to moving the ccache onto the ramdisk (#1623 (comment))?

[sam-github]

It is. https://ci.nodejs.org/job/node-test-commit-aix/27416/nodes=aix61-ppc64/console is in progress.

[sam-github]

It might be slow, since ccache was destroyed on machine reboot.

[sam-github]

Build passed: https://ci.nodejs.org/job/node-test-commit-aix/27416/

Looks to be up and alive again.