Errors running Ansible across current cluster

[rvagg]

I decided to just run our ansible worker create script across our cluster today in light of the sudo security flaw that was announced. In the process I've encountered a bunch of errors that we should probably look in to. Recording here as a list so we can tick them off. I'd appreciate help in dealing with these.

I didn't run any updates on:

• Windows - shouldn't need it, I think they all do auto updates and regular restarts anyway
• IBM platforms (including ibmi, aix, rhel-s390x, zos) I'll let @nodejs/build IBM folks deal with that
• macOS - @AshCripps would you mind doing these? I'm a little afraid to just run the scripts against the existing infra.

Errors:

Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)

• test-digitalocean-debian9-x64-1
• test-digitalocean-centos5-x86-1 Removed
• test-softlayer-centos5-x64-2 Removed
• test-softlayer-centos5-x64-1 Removed
• release-digitalocean-centos5-x64-1 Removed
• release-digitalocean-centos6-x86-1 Removed ansible: add RHEL 8 x64 instances #2886
• release-softlayer-centos5-x86-1 Removed
• test-digitalocean-ubuntu1404-x64-1
• test-digitalocean-ubuntu1404-x86-1 Removed ansible: add RHEL 8 x64 instances #2886
• test-digitalocean-ubuntu1604-x86-2
• test-digitalocean-ubuntu1804-x64-1
• test-rackspace-ubuntu1204-x64-1 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu-1 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu-2 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu3-1 (should remove)
• release-scaleway-ubuntu1604-armv7l-1 (should remove)
• release-scaleway-ubuntu1604-armv7l-2 (should remove)

Unreachable

• test-joyent-smartos18-x64-1 Removed
• test-joyent-smartos18-x64-2 Removed
• test-digitalocean-freebsd10-x64-1 (should remove?)
• test-joyent-freebsd10-x64-1 Removed
• test-joyent-freebsd10-x64-2 Removed
• test-rackspace-freebsd10-x64-1 (should remove?)
• test-digitalocean-ubuntu1204-x64-1 (should remove)
• test-digitalocean-ubuntu1204-x64-2 (should remove)
• test-scaleway-ubuntu1804-armv7l-2 (should remove)
• test-scaleway-ubuntu1804-armv7l-3 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-1 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-2 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-3 (should remove)

Failed on 'baselayout : run ccache installer'

• test-joyent-smartos15-x64-3 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos15-x64-4 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos16-x64-3 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos16-x64-4 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos17-x64-3 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
• test-joyent-smartos17-x64-4 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
• release-joyent-smartos15-x64-2 (gone: Replacing machines from Joyent data center #2552)
• release-joyent-smartos17-x64-2 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533

Failed update

• test-digitalocean-debian8-x64-1 (jessie-backports 404)
• test-softlayer-centos7-x64-1 (?) ansible: fix centos7 x64 ansible errors #2651
• test-digitalocean-freebsd11-x64-2 ("pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version 2") Errors running Ansible across current cluster #2531 (comment)
• test-nearform_intel-ubuntu1604-x64-1 Errors running Ansible across current cluster #2531 (comment)

Failed 'bootstrap : install libselinux-python bindings'

• test-digitalocean-fedora30-x64-1 ansible: fix multiple register vars on Fedora #2654
• test-digitalocean-fedora30-x64-2 ansible: fix multiple register vars on Fedora #2654
• release-softlayer-centos6-x64-1 removed ansible: add RHEL 8 x64 instances #2886
• test-softlayer-centos6-x64-1 removed ansible: add RHEL 8 x64 instances #2886
• test-softlayer-centos6-x64-2 removed ansible: add RHEL 8 x64 instances #2886

Failed 'set hostname'

(We should remove these entirely)

• test-softlayer-ubuntu1404-x64-1 (remove?)
• test-softlayer-ubuntu1404-x86-1 (remove?)

Failed 'baselayout : centos7 | install ius'

"Name or service not known", is IUS still a thing?

• test-rackspace-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651
• release-digitalocean-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651

Failed 'jenkins-worker : install tap2junit'

• test-digitalocean-freebsd11-x64-1 ansible: fixes for freebsd #2652

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'pkg_resources'
{"changed": false, "msg": "Failed to import the required Python library (setuptools) on test-digitalocean-freebsd11-x64-1's Python /usr/local/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

Failed uninstall node

• test-digitalocean-ubuntu1604-x86-1 Errors running Ansible across current cluster #2531 (comment)

FAILED! => {"changed": false, "cmd": "node -v", "delta": "0:00:01.462308", "end": "2021-01-27 20:56:35.373427", "failed_when_result": true, "rc": 0, "start": "2021-01-27 20:56:33.911119", "stderr": "", "stderr_lines": [], "stdout": "v9.11.2", "stdout_lines": ["v9.11.2"]}

No "containers" config

(these probably need to be removed, they were part of my next-gen containerisation experiment iirc)

• test-digitalocean-ubuntu2004_docker-x64-1
• test-digitalocean-ubuntu2004_docker-x64-2

[AshCripps]

Running into some ansible issues myself currently with this error

* Failed to parse /Users/ash/github/build/ansible/plugins/inventory/nodejs_yaml.py with ini plugin: /Users/ash/github/build/ansible/plugins/inventory/nodejs_yaml.py:25: Expected key=value host variable assignment, got: __future__

But once i get it working ill get on to ansibling the macs.

[richardlau]

FTR IBM i and z/OS do not have sudo available/installed.

[richardlau]

Ran the ansible scripts against our LinuxONE (rhel-s390x) hosts and they run into an error in the baselayout : run ccache installer task. I've opened #2533 to cover separately. The scripts did get past the package update task so sudo has been updated on all of the LinuxONE hosts (to sudo-1.8.23-10.el7_9.1.s390x)

[richardlau]

I've updated the AIX hosts. All IBM platforms that have sudo have been updated to patch the advisory.

[AshCripps]

Ive run against all the macs only error I faced is this on release-nearform-macos10.15-x64-1 which im currently investigating:

TASK [jenkins-worker : download slave.jar] ********************************************************************************************************************************************************************************************
task path: /Users/ash/github/build/ansible/roles/jenkins-worker/tasks/main.yml:165
fatal: [release-nearform-macos10.15-x64-1]: FAILED! => {"changed": false, "dest": "/Users/iojs/slave.jar", "elapsed": 0, "gid": 0, "group": "wheel", "mode": "0644", "msg": "Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>", "owner": "root", "size": 877037, "state": "file", "uid": 0, "url": "https://ci.nodejs.org/jnlpJars/slave.jar"}

[richardlau]

I'm looking at test-nearform_intel-ubuntu1604-x64-1.

Manually running `apt get update` fails with an expired gpg key:

# apt-get update
Hit:1 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:3 http://ie.archive.ubuntu.com/ubuntu xenial InRelease
Hit:4 http://ie.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:5 http://ie.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:6 https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease [3,607 B]
Err:6 https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease
  The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
Fetched 113 kB in 0s (124 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease: The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
W: Failed to fetch https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu/xenial/InRelease  The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
W: Some index files failed to download. They have been ignored, or old ones used instead.
#

[richardlau]

I'm looking at test-nearform_intel-ubuntu1604-x64-1.

Manually running apt get update fails with an expired gpg key:

root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key list | grep "expired: "
pub   2048R/E084DAB9 2010-10-19 [expired: 2020-10-16]
root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key adv --keyserver keyserver.ubuntu.com --refresh-keys
Executing: /tmp/tmp.mckX5xKLZr/gpg.1.sh --keyserver
keyserver.ubuntu.com
--refresh-keys
gpg: refreshing 7 keys from hkp://keyserver.ubuntu.com
gpg: requesting key 437D05B5 from hkp server keyserver.ubuntu.com
gpg: requesting key C0B21F32 from hkp server keyserver.ubuntu.com
gpg: requesting key EFE21092 from hkp server keyserver.ubuntu.com
gpg: requesting key FBB75451 from hkp server keyserver.ubuntu.com
gpg: requesting key E084DAB9 from hkp server keyserver.ubuntu.com
gpg: requesting key AF4F7421 from hkp server keyserver.ubuntu.com
gpg: requesting key BA9EF27F from hkp server keyserver.ubuntu.com
gpg: key 437D05B5: "Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>" 43 new signatures
gpg: key C0B21F32: "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" 17 new signatures
gpg: key EFE21092: "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" 65 new signatures
gpg: key FBB75451: "Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>" 93 new signatures
gpg: key E084DAB9: "Michael Rutter <marutter@gmail.com>" 2 new signatures
gpg: key AF4F7421: "Sylvestre Ledru - Debian LLVM packages <sylvestre@debian.org>" 1 new signature
gpg: key BA9EF27F: "Launchpad Toolchain builds" not changed
gpg: Total number processed: 7
gpg:              unchanged: 1
gpg:         new signatures: 221
root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key list | grep "expired: "
root@test-nearform--intel-ubuntu1604-x64-1:~#

Refreshing keys has allowed the create worker ansible playbook to successfully run on test-nearform_intel-ubuntu1604-x64-1.

[richardlau]

Removed the smartos 15/16 hosts from the list in the description as they've been removed: #2552

[richardlau]

Failed update

• test-digitalocean-freebsd11-x64-2 ("pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version 2")

Fixed by running

$ pkg --version
1.10.5
$ sudo pkg bootstrap -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.16.3...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1104001
- running kernel: 1102000
Ignore the mismatch and continue? [y/N]:
package pkg is already installed, forced install
Extracting pkg-1.16.3: 100%
$ pkg --version
1.16.3
$

1.16.3 matches the version of pkg on test-digitalocean-freebsd11-x64-1.

Running the playbook on test-digitalocean-freebsd11-x64-2 now errors at the same point as test-digitalocean-freebsd11-x64-1

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'pkg_resources'
fatal: [test-digitalocean-freebsd11-x64-2]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (setuptools) on test-digitalocean-freebsd11-x64-2's Python /usr/local/bin/python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

[richardlau]

Failed 'baselayout : centos7 | install ius'

"Name or service not known", is IUS still a thing?

• test-rackspace-centos7-x64-1
• release-digitalocean-centos7-x64-1

build/ansible/roles/baselayout/tasks/partials/repo/centos7.yml

Lines 8 to 12 in 722fdc1

	- name: centos7 | install ius
	when: "arch != 'arm64' and arch != 'ppc64'"
	yum:
	name: "https://centos{{ ansible_distribution_major_version }}.iuscommunity.org/ius-release.rpm"
	state: present

I'm not sure what we're installing from there, but the "centos*.iuscommunity.org" URLs were retired in 2020: iusrepo/announce#18

[richardlau]

Re. "Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)" -- Having double checked we no longer have them in neither public nor release Jenkins, I've removed all the centos5 entries from secrets/inventory.yml in addition to the ones marked "should remove", and added the missing secrets for the others in the list.

Edit: Centos 5 was removed from Jenkins in #1984

[richardlau]

Failed uninstall node

• test-digitalocean-ubuntu1604-x86-1

FAILED! => {"changed": false, "cmd": "node -v", "delta": "0:00:01.462308", "end": "2021-01-27 20:56:35.373427", "failed_when_result": true, "rc": 0, "start": "2021-01-27 20:56:33.911119", "stderr": "", "stderr_lines": [], "stdout": "v9.11.2", "stdout_lines": ["v9.11.2"]}

This was a snap install of Node.js.

root@test-digitalocean-ubuntu1604-x86-1:~# which node
/snap/bin/node
root@test-digitalocean-ubuntu1604-x86-1:~#

Removed via snap remove node.