Utilise containers in Azure dynamic VMs for AQA runs

[sxa]

Initial implementation of #5684 based on last year's work in #5683

This PR is designed to allow dynamic containers, provisioned with Azure, to be used for AQA jobs. It spins up a system with docker installed (defined by the Azure VM agents plugin config) and then pulls down a docker image (currently UBI10) and runs a job on it. This behaviour is triggered by setting the parameter CLOUD_PROVIDER=azure in the Test jobs.

This also makes a small modification to the weston support in EL10 distributions to allow a dynamic XDG_RUNTIME_DIR to be used (within the jenkins workspace) so that it does not haver to be in a fixed, reusable location on the machine. FYI @llxia @AdamBrousseau as it would be good to have your confirmation that this does not break anything on your side for EL10.

In the future it is envisioned that this will allow support for different distributions (at the moment we have ubi10, ubuntu2204 and ubuntu2404 on the ghcr.io registry. It is also likely that we will change the address from ghcr.io/adoptium/test-containers to something else in the future to avoid confusion with other products of a similar name :-)

[sxa]

Conflicts resolved. Pre-conflict-resultion branch has been tested again at with ci.agent.dynamic as the label: https://ci.adoptium.net/job/Grinder/15435/ (That's a bad idea - it should be ADDITIONAL LABEL otherwise it ignores the platform - testing that (with conflicts resolved) at https://ci.adoptium.net/job/Grinder/15486/

Marking ready for review as an initial pass that can be tested.

[sxa]

Hmm in fact using ci.agent.dynamic as an ADDITIONAL_LABEL doesn't work as the ci.agent.dynamic machines get spun up with the labels: ubuntux86-64x64ci.agent.dynamicsw.os.linuxhw.arch.x86ubuntu2204 and not ci.role.test. Tested with LABEL=ci.role.dynamic again at https://ci.adoptium.net/job/Grinder/15487/ (The previous Grinder one from the last comment got stuck)

[sxa]

Made a temporary change to https://ci.adoptium.net/manage/cloud/Azure/template/test-linux-x64/ to add ci.role.test to the list of labels on the dynamic machines and it worked properly, although the "stuck" Grinder 15486 didn't progress and was still stuck.
I have now reverted that change just in case it causes any side effects that I'm not aware of.

[smlambert]

Try different test targets and jdk versions

• https://ci.adoptium.net/job/Grinder/15489/ - sanity.functional
• https://ci.adoptium.net/job/Grinder/15490/ - jdk_net (as those tests would be most likely to fail if config of agent differs from static agents), a few failures
  • com/sun/net/httpserver/ServerStopTerminationTest.java.ServerStopTerminationTest
  • java/net/CookieStoreTest.java.CookieStoreTest
  • java/net/MulticastSocket/NoLoopbackPackets.java.NoLoopbackPackets
  • java/net/MulticastSocket/Test.java.Test

14:45:20  [ JUnit Containers: found 4, started 4, succeeded 4, failed 0, aborted 0, skipped 0]
14:45:20  [ JUnit Tests: found 5, started 5, succeeded 4, failed 1, aborted 0, skipped 0]
14:45:20  
14:45:20  java.lang.Exception: JUnit test failure
14:45:20  	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:149)
14:45:20  	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
14:45:20  	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:61)
14:45:20  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
14:45:20  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
14:45:20  	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
14:45:20  	at java.base/java.lang.Thread.run(Thread.java:1583)

• https://ci.adoptium.net/job/Grinder/15491/ - run sanity.openjdk in parallel across 6 nodes (to check it sends to all dynamic nodes) - looks good, 1 testcase failure out of 1657, java/lang/ProcessHandle/OnExitTest.java

[sxa]

* https://ci.adoptium.net/job/Grinder/15489/ - sanity.functional

PASS!

* https://ci.adoptium.net/job/Grinder/15490/ - jdk_net (as those tests would be most likely to fail if config of agent differs from static agents)

Hmmm 4 test case failures there, 2 of which are in multicast tests.

* https://ci.adoptium.net/job/Grinder/15491/ - run sanity.openjdk in parallel across 6 nodes (to check it sends to all dynamic nodes)

Timeout in OnExitTest in testList_4 - regrind at https://ci.adoptium.net/job/Grinder/15520

[smlambert]

Yes, multicast and I believe some testcases may be trying to launch containers, given the output text of failed testcases is saying

14:45:20  [ JUnit Containers: found 4, started 4, succeeded 4, failed 0, aborted 0, skipped 0]
14:45:20  [ JUnit Tests: found 5, started 5, succeeded 4, failed 1, aborted 0, skipped 0]
14:45:20  
14:45:20  java.lang.Exception: JUnit test failure
14:45:20  	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:149)

https://github.com/openjdk/jtreg/blob/master/src/share/classes/com/sun/javatest/regtest/agent/JUnitRunner.java#L149

was going to check if those run 'fine' on a static docker container, if we have one for x64 Linux.

[sxa]

was going to check if those run 'fine' on a static docker container, if we have one for x64 Linux.

100 iteration grinders 15515, 15516 and 15517 on x64 static docker systems all seem to fail.

[smlambert]

100 iteration grinders 15515, 15516 and 15517 on x64 static docker systems all seem to fail.

I suspected as much, we may want to figure out how to exclude or skip those testcases in containerized environments.

[sxa]

100 iteration grinders 15515, 15516 and 15517 on x64 static docker systems all seem to fail.

I suspected as much, we may want to figure out how to exclude or skip those testcases in containerized environments.

The 100 iteration failures are from running against the release builds. java_net_0 nightlies (See Grinders 15642 through 15647 excluding 15646) seem ok so failures were likely a test material mismatch.

[smlambert]

@sophia-guo - if you are okay with it, I suggest we merge this PR and come in with a 2nd set of changes that:

• randomly cycles through the selection of images
• pulls from the directory that is not named test-container

[smlambert]

Once this is merged, we will make an update in ci-jenkins-pipeline code to set CLOUD_PROVIDER=azure for x64 Linux so that when we have exhausted all static nodes, this logic will kick in and spin up dynamic containers.