CI/CD checks fails on agent-flood.

[nsednev]

During basic CI/CD checks I see that Packit-as-a-Service / testing-farm:centos-stream-9-x86_64:e2e-ffi fails on /tests/ffi/agent-flood.
Here goes some Testing Farm artifacts and
Dashboard.

WARN[0010] StopSignal SIGTERM failed to stop container ffi-qm in 10 seconds, resorting to SIGKILL
Deleted: c2febf90cc5baa7cfbdcecc1e485300cd110c0c0bb9b39fd3f39564d8916e330
[ INFO ] PASS: Command podman exec -it qm /bin/bash -c "podman rmi -f --all" successful
[ INFO ] PASS: Command rm -rf /var/qm/lib/containers/registry/tools-ffi:latest/b8e9f060dd20af7fdd2e577300d219b18aefc753538f18b603885a6d0945cb29 successful
[ INFO ] PASS: Command rm -rf /etc/containers/systemd/qm.container.d/ successful
[ INFO ] PASS: Command systemctl daemon-reload successful
[ INFO ] PASS: Command systemctl restart qm successful
[ INFO ] PASS: Command rm -f successful
[ INFO ] PASS: Command mkdir -p /etc/containers/systemd/qm.container.d/ successful
[ INFO ] PASS: Command systemctl daemon-reload successful
[ INFO ] PASS: Command systemctl restart qm successful
The system is not running under OSTree.
Trying to pull quay.io/centos-sig-automotive/ffi-tools:latest...
Getting image source signatures
Copying blob sha256:972d4b520f080b2d880670a56c847c9e42ad7a01207b43066f965c69a04a6d91
Copying blob sha256:376ee6a8ddb9979b92f63edbb9a11f5f26ef64580d37a54cd9edb27dc5842f98
Copying config sha256:c2febf90cc5baa7cfbdcecc1e485300cd110c0c0bb9b39fd3f39564d8916e330
Writing manifest to image destination
c2febf90cc5baa7cfbdcecc1e485300cd110c0c0bb9b39fd3f39564d8916e330
[ INFO ] PASS: Command podman pull quay.io/centos-sig-automotive/ffi-tools:latest successful
[ INFO ] PASS: Command mkdir -p /var/qm/lib/containers/registry successful
Getting image source signatures
Copying blob sha256:b8e9f060dd20af7fdd2e577300d219b18aefc753538f18b603885a6d0945cb29
Copying blob sha256:67a169f476d08ca9c72c80655f347ed334f08cebcff1f5d9935d150c0c3ed048
Copying config sha256:c2febf90cc5baa7cfbdcecc1e485300cd110c0c0bb9b39fd3f39564d8916e330
Writing manifest to image destination
[ INFO ] PASS: Command podman push c2febf90cc5b dir:/var/qm/lib/containers/registry/tools-ffi:latest successful
Untagged: quay.io/centos-sig-automotive/ffi-tools:latest
Deleted: c2febf90cc5baa7cfbdcecc1e485300cd110c0c0bb9b39fd3f39564d8916e330
[ INFO ] PASS: Command podman rmi -f c2febf90cc5b successful
[ INFO ] PASS: Command podman exec -it qm /bin/bash -c "systemctl stop bluechi-agent" successful
[ INFO ] setup_test_containers_in_qm(): prepare quadlet files for bluechi-tester-1.container
[ INFO ] setup_test_containers_in_qm(): updating AllowedNodeNames in /etc/bluechi/controller.conf
[ INFO ] setup_test_containers_in_qm(): bluechi-controller reload & restart
[ INFO ] PASS: Command systemctl daemon-reload successful
[ INFO ] PASS: Command systemctl restart bluechi-controller successful
[ INFO ] setup_test_containers_in_qm(): prepare quadlet files for bluechi-tester-2.container
[ INFO ] setup_test_containers_in_qm(): updating AllowedNodeNames in /etc/bluechi/controller.conf
[ INFO ] setup_test_containers_in_qm(): bluechi-controller reload & restart
[ INFO ] PASS: Command systemctl daemon-reload successful
[ INFO ] PASS: Command systemctl restart bluechi-controller successful
[ INFO ] PASS: Command systemctl restart bluechi-agent successful
[ INFO ] run_test_containers(): bluechi-tester-1 reload & restart
[ INFO ] PASS: Command podman exec qm systemctl daemon-reload successful
Job for bluechi-tester-1.service failed because a timeout was exceeded.
See "systemctl status bluechi-tester-1.service" and "journalctl -xeu bluechi-tester-1.service" for details.
[ FAILED ] Error: Command podman exec qm systemctl restart bluechi-tester-1 failed with exit code: 1
Terminated
Shared connection to 3.14.29.231 closed.

From my brief look it seems like the error message indicates that the bluechi-tester-1 service failed to start due to a timeout. This service is being restarted in the run_test_containers function, which is defined in tests/ffi/agent-flood/test.sh.

The error is likely caused by the systemctl restart command timing out while trying to restart the bluechi-tester-1 service.

Afew days before the check was passing just fine.