Plugin is not honoring the configured Deployment timeout setting for windows.

[pradeepbagul]

Jenkins and plugins versions report

Jenkins: 2.516.1
OS: Linux - 5.14.0-570.24.1.el9_6.x86_64
Java: 21.0.8 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Parameterized-Remote-Trigger:3.2.2
analysis-model-api:13.8.0-902.v26f80296f743
ant:518.v8d8dc7945eca_
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-166.v870a_96374f91
artifactory:4.0.8
asm-api:9.8-163.vb_2a_96d3f9c3c
audit-trail:436.vc0d1e79fc5a_3
authentication-tokens:1.144.v5ff4a_5ec5c33
authorize-project:2.0.0
aws-credentials:254.v978a_5e206a_d7
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.31.63-54.vcf1e5a_c56c49
aws-java-sdk2-ec2:2.31.63-54.vcf1e5a_c56c49
azure-credentials:357.v6447d38fb_007
azure-sdk:243.v4a_0d23a_c71a_7
azure-vm-agents:1043.v6c34a_871003d
blueocean:1.27.21
blueocean-bitbucket-pipeline:1.27.21
blueocean-commons:1.27.21
blueocean-config:1.27.21
blueocean-core-js:1.27.21
blueocean-dashboard:1.27.21
blueocean-display-url:2.4.4
blueocean-events:1.27.21
blueocean-git-pipeline:1.27.21
blueocean-github-pipeline:1.27.21
blueocean-i18n:1.27.21
blueocean-jwt:1.27.21
blueocean-personalization:1.27.21
blueocean-pipeline-api-impl:1.27.21
blueocean-pipeline-editor:1.27.21
blueocean-pipeline-scm-api:1.27.21
blueocean-rest:1.27.21
blueocean-rest-impl:1.27.21
blueocean-web:1.27.21
bootstrap5-api:5.3.7-860.v1251c115c90c
bouncycastle-api:2.30.1.81-264.v95c79c0e772c
branch-api:2.1244.vf95c81f1641c
build-name-setter:2.5.1
build-timeout:1.38
build-timestamp:1.1.0
caffeine-api:3.2.2-178.v353b_8428ed56
checks-api:373.vfe7645102093
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:936.4.4
cloudbees-folder:6.1037.v4cb_8573b_72a_a_
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-1
commons-lang3-api:3.18.0-98.v3a_674c06072d
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
config-file-provider:994.v3d4a_5fa_f353a_
configuration-as-code:1985.vdda_32d0c4ea_b_
coverage:2.2912.v3dea_6a_4a_73f7
credentials:1419.v2337d1ceceef
credentials-binding:702.vfe613e537e88
data-tables-api:2.3.3-1383.va_5607a_a_3f3c2
dependency-check-jenkins-plugin:5.6.1
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:621.va_73f881d9232
dtkit-api:3.0.3
durable-task:595.ve87b_f1318d67
echarts-api:6.0.0-1146.v5c8f3b_8f0573
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1925.v1598902b_58dd
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.237.v79163ca_8b_892
font-awesome-api:7.0.0-851.vd1feb_218a_a_63
forensics-api:3.1754.v2a_6613b_77002
git:5.7.0
git-changelog:3.45
git-client:6.3.3
git-server:137.ve0060b_432302
github:1.44.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1848.v42f74f7f4500
gitlab-api:5.6.0-100.v83f8f4b_f1129
gitlab-branch-source:723.v090f23d3cc49
gitlab-plugin:1.9.9
gradle:2.16.1149.v711b_998b_0532
gson-api:2.13.1-153.vb_3d0c48a_a_b_4a_
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
hashicorp-vault-plugin:371.v884a_4dd60fb_6
htmlpublisher:427
http_request:1.20
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.19.2-408.v18248a_324cfe
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-3
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.21
jersey2-api:2.47-165.ve7809a_3e87e0
jira:3.19
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
jobConfigHistory:1356.ve360da_6c523a_
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery:1.12.4-3
jquery3-api:3.7.1-594.vb_3864f326cf0
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.9.0-190.veefca_05d5477
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1354.v5b_348580de7f
ldap:780.vcb_33c9a_e4332
lockable-resources:1412.v3f305a_fb_a_117
mailer:522.va_995fa_cfb_8b_d
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-project:856.v4c352b_3a_b_23e
maven-plugin:3.27
metrics:4.2.33-484.v2fcd689980d1
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
multibranch-action-triggers:1.8.10
netty-api:4.1.118.Final-9.v776038d601a_7
nunit:593.v76f7a_5f959c1
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:392.v27a_482d90083
pam-auth:1.12
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:241.vc3d48fb_b_2582
pipeline-graph-view:628.va_6f6a_1d12848
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:534.v352f0a_e98918
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2265.v140e610fe9d5
pipeline-model-definition:2.2265.v140e610fe9d5
pipeline-model-extensions:2.2265.v140e610fe9d5
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2265.v140e610fe9d5
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1167.v022176c7e0ca_
prism-api:1.30.0-609.vf0a_df102d9a_f
pubsub-light:1.19
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
role-strategy:799.v5b_e7b_ecc231e
scm-api:707.v749f968369d4
script-security:1378.vf25626395f49
slack:795.v4b_9705b_e6d47
slave-setup:129.vee7cd5a_0e204
snakeyaml-api:2.3-125.v4d77857a_b_402
sse-gateway:1.28
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1071.v0d059c7b_c555
sshd:3.374.v19b_d59ce6610
structs:353.v261ea_40a_80fb_
support-core:1763.ve4ce0d10a_3f0
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
versioncolumn:320.v6b_b_814ca_01f7
warnings-ng:12.9783.ve1cb_9f060738
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4183.v94b_6fd39da_c1
workflow-durable-task-step:1452.v0ee719c104a_7
workflow-job:1540.v295eccc9778f
workflow-multibranch:811.vcd33d074c2a_0
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:706.v518c5dcb_24c0
workflow-support:976.vb_d9493c2eb_09
ws-cleanup:0.49

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins: Version 2.516.1
azure-vm-agents:1043.v6c34a_871003d
controller - AlmaLinux9

Reproduction steps

• Configure an Azure VM agent plugin to provision a new virtual machine.
• Set the "Deployment timeout" parameter to a value greater than 10 minutes (e.g., 30 minutes, 60 minutes).
• Initiate the VM provisioning process.
• Wait for the deployment to run.

Observed Behavior:

• The deployment fails at approximately the 10-minute mark.
• The error message indicates a timeout, but it is not aligned with the "Deployment timeout" value we configured.
• The partially created virtual machine and its associated resources are automatically and permanently deleted from the Azure environment.
• Enabling boot diagnostics provided little to no help, as the VM is deleted too quickly to capture useful information.

Expected Results

The VM provisioning process should respect the "Deployment timeout" value we configured. It should only fail after the specified duration, allowing for longer deployments to complete successfully

Actual Results

The plugin is not honoring the configured "Deployment timeout" setting.

Anything else?

No response

Are you interested in contributing a fix?

No response

[pradeepbagul]

Any update on this?

[timja]

No I haven't had time to look at it. Why does your VM take that long to deploy though?

[pradeepbagul]

It does not happen every time, but when it happens, we observed the above flow that I pointed out in the defect.
It would be beneficial to understand why this happens.

[pradeepbagul]

Please have a look and help us figure out how we can avoid this.

[tibco-postman]

No I haven't had time to look at it. Why does your VM take that long to deploy though?

We're observing large fluctuations in deploy time for windows. Sometimes, it takes less than a minute, sometimes several minutes and sometimes it times out... See #712

[tibco-postman]

@timja We've been in contact with Microsoft on timeouts. Message from Microsoft support is that if there is a what is called PPS (pre-provisioned VMs) available, then provisioning is fast. In essence, PPS VMs are cached VMs kept available to reduce deployment time.
If there is no PPS VMs, Azure performs a full operating system initialization, including agent setup, extension registration, and other first-boot configuration steps.
For Windows VMs, this process naturally takes longer time.

Consequently the, seemingly hard-coded, timeout of 10 minutes need to be configurable.

[timja]

Searching the plugin for 10 I found this:

azure-vm-agents-plugin/src/main/java/com/microsoft/azure/vmagent/ProvisionStrategy.java

Line 9 in 82f824b

private static final long MAX_INTERVAL = 10 * 60 * 1000; // 10 minutes

Are you able to try build the plugin with a much higher max and see if that resolves the issue for you?

[pradeepbagul]

Is it possible for you to increase the timeout to 20 minutes, and provide us build or executable plugin so that we can try it out?

[timja]

You can try this:
https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/azure-vm-agents/1067.v0fa_a_b_61e22f9/azure-vm-agents-1067.v0fa_a_b_61e22f9.hpi

From: #724

[pradeepbagul]

Thanks I will try and update you

[pradeepbagul]

Nopes.. it does not create a machine and on Azure console i see its creating even after wait for 10 min.
is size that you provided is correct (I see its size 1.63MB)

I also see below message in system log.

{"az.sdk.message":"The following Netty versions were found on the classpath and have a mismatch with the versions used by azure-core-http-netty. If your application runs without issue this message can be ignored, otherwise please align the Netty versions used in your application. For more information, see https://aka.ms/azsdk/java/dependency/troubleshoot.","azure-netty-version":"4.1.127.Final","azure-netty-native-version":"2.0.74.Final","classpath-netty-version-io.netty:netty-common":"4.1.118.Final","classpath-netty-version-io.netty:netty-handler":"4.1.118.Final","classpath-netty-version-io.netty:netty-handler-proxy":"4.1.118.Final","classpath-netty-version-io.netty:netty-buffer":"4.1.118.Final","classpath-netty-version-io.netty:netty-codec":"4.1.118.Final","classpath-netty-version-io.netty:netty-codec-http":"4.1.118.Final","classpath-netty-version-io.netty:netty-codec-http2":"4.1.118.Final","classpath-netty-version-io.netty:netty-transport-native-unix-common":"4.1.118.Final","classpath-netty-version-io.netty:netty-transport-native-epoll":"4.1.118.Final","classpath-netty-version-io.netty:netty-transport-native-kqueue":"4.1.118.Final"}

[timja]

yes its a small plugin. that's normal, just ignore that.

Do you mean that the Azure plugin is giving up at 10 min? Have you changed the timeout setting? As the change I did only increases the maximum not the configured setting.

[pradeepbagul]

What I see is that the machine is still showing in the creating stage at Azure. portal even after 10 min,

Below error in the Azure log

Failure creating provisioned agent 'bawd691b0'
com.microsoft.azure.vmagent.exceptions.AzureCloudException: Deployment Failed: Microsoft.Compute/virtualMachines:bawd691b0 - Conflict - OS Provisioning for VM 'bawd691b0' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).
 * Instructions for Windows: https://learn.microsoft.com/azure/virtual-machines/windows/prepare-for-upload-vhd-image
 * Instructions for Linux: https://learn.microsoft.com/azure/virtual-machines/linux/create-upload-generic
 * If you are deploying more than 20 Virtual Machines concurrently, consider moving your custom image to shared image gallery. Please refer to https://aka.ms/movetosig for the same.
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:37)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud.createProvisionedAgent(AzureVMCloud.java:584)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:866)
	at PluginClassLoader for azure-vm-agents//com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:843)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Deployment timeout is set to (seconds)--1200