Most orka hosted macs offline in Jenkins

[richardlau]

Probably linked to a series of service emails/notifications we've had from MacStadium today.

I've tried restarting the Jenkins agents via ansible.nodejs.org but that doesn't seem to have brought the agents back online (although the script in AWX succeeded). I am able to ssh into the machines, so they're at least reachable.

[richardlau]

/Users/iojs/jenkins_err.log on both test-orka-macos11-x64-1 and test-orka-macos11-x64-2 is full of

The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

messages.

[richardlau]

I'm rerunning the Ansible playbooks against the orka machines to see if that puts Java back on them.

[mhdawson]

It looks like all of -1 -2, -3 test-orka-macos10.14-x64-1 are trying to connect as the same machine. It's possible they need to be re-ansibled?

[richardlau]

Looks like issues running Ansible against the majority of the machines -- I don't have time to look at those tonight.

PLAY RECAP *************************************************************************************************************************************************
release-orka-macos10.15-x64-1 : ok=7    changed=0    unreachable=1    failed=0    skipped=2    rescued=0    ignored=0
test-orka-macos10.14-x64-1 : ok=10   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
test-orka-macos10.14-x64-2 : ok=10   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
test-orka-macos10.14-x64-3 : ok=10   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
test-orka-macos10.15-x64-1 : ok=10   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
test-orka-macos10.15-x64-2 : ok=10   changed=1    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
test-orka-macos11-x64-1    : ok=39   changed=12   unreachable=0    failed=0    skipped=128  rescued=0    ignored=1
test-orka-macos11-x64-2    : ok=39   changed=12   unreachable=0    failed=0    skipped=128  rescued=0    ignored=1

the two machines that did successfully have Ansible run against them,
test-orka-macos11-x64-1 and test-orka-macos11-x64-2, now look to be online again and have picked up jobs.

[mhdawson]

Running ansible on test-orka-macos10.14-x64-2 to see if I have any better luck.

[mhdawson]

I don't know if it is stuck or just taking a long time, but it's still running the "Upgrade installed pacakges" step

[mhdawson]

Still running. So Cancelled and tried again.

[mhdawson]

That seemed to then run through quickly

[mhdawson]

But it seems to have failed with

ASK [package-upgrade : Update Casks] *******************************************************************************************************************************************
fatal: [test-orka-macos10.14-x64-2]: FAILED! => {"changed": false, "msg": "Error: brew cask is no longer a brew command. Use brew <command> --cask instead."}

[richardlau]

I'm unable to ssh into release-orka-macos10.15-x64-1 with the release ssh key but can with the test key and it looks like the VM is test-orka-macos10.15-x64-1.

I'm trying to figure out if the NAT translation has broken.

[richardlau]

I think the NAT is okay -- if I touch a file on release-orka-macos10.15-x64-1 it doesn't show up on test-orka-macos10.15-x64-1. I think the VMs have reset to some snapshotted state. For example, the script to start the Jenkins agent has SECRET as the agent secret instead of an actual secret corresponding to the Jenkins agent:

$ ssh test-orka-macos10.15-x64-1
Last login: Tue Jun 14 04:54:53 2022 from 172.16.44.16
administrator@test-orka-macos10 ~ % cat /Users/iojs/start.sh
#!/bin/bash
export HOME=/Users/iojs
export NODE_TEST_DIR="$HOME/tmp"
export JOBS="4"

export OSTYPE=osx
export ARCH=x64
export DESTCPU=x64

PATH="/usr/local/opt/ccache/libexec:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" java -Xmx128m \
    -jar /Users/iojs/slave.jar -secret SECRET \
    -jnlpUrl https://ci.nodejs.org/computer/test-orka-macos10.15-x64-1/slave-agent.jnlp
administrator@test-orka-macos10 ~ %

I'm going to try updating community.general (ansible-galaxy collection install -U community.general) to see if that fixes the Error: brew cask is no longer a brew command. Use brew <command> --cask instead. issue with the homebrew_cask task and allows the Ansible scripts to complete.

[AshCripps]

FYI the base images in ORKA all have the hostname set as test- cause I used a test image as the base so they likely have been turned off and reset to their base state so will need ansibling again.

[richardlau]

I've been able to get the two macOS 10.15 x64 test VMs reansibled and online. The 10.15 release machine and two of the 10.14 machines continue to have issues with the package upgrade task running brew.

[richardlau]

I got the release machine back online but I'm fairly certain if it has been reset to base state we'll need to manually get the full xcode and signing certs on there.