Allow full clone volumes with thin provisioning #11177

JoaoJandre · 2025-07-10T12:09:49Z

Description

This PR adds a configuration called create.full.clone to the agent.properties file. When set to true, all QCOW2 volumes created will be full-clone. If false (default), the current behavior remains, where only FAT and SPARSE volumes are full-clone and THIN volumes are linked-clone.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Without setting the property, I deployed a VM, which used linked-clone to create the volume.

Afterward, I added the create.full.clone property to agent.properties, restarted the agent service, and created another VM. This time, the volume was full-clone.

How did you try to break this feature and the system with this change?

codecov · 2025-07-10T12:17:37Z

Codecov Report

❌ Patch coverage is 11.11111% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.57%. Comparing base (8e4fe1c) to head (b942c0d).
⚠️ Report is 125 commits behind head on main.

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11177      +/-   ##
============================================
- Coverage     16.57%   16.57%   -0.01%     
  Complexity    13988    13988              
============================================
  Files          5745     5745              
  Lines        510847   510853       +6     
  Branches      62140    62142       +2     
============================================
- Hits          84696    84682      -14     
- Misses       416677   416705      +28     
+ Partials       9474     9466       -8

Flag	Coverage Δ
uitests	`3.91% <ø> (ø)`
unittests	`17.47% <11.11%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sureshanaparti · 2025-07-14T11:22:42Z

@blueorangutan package

Copilot

Pull Request Overview

This PR adds a new create.full.clone configuration to allow creating THIN-provisioned QCOW2 volumes as full clones instead of linked clones.

Introduces a boolean agent property CREATE_FULL_CLONE (default false).
Updates createDiskFromTemplate to branch on the new property for THIN provisioning.
Adds default entry in agent.properties and Javadoc in AgentProperties.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java	Conditional full-clone logic added for THIN volumes based on new property
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java	Defined `CREATE_FULL_CLONE` property with Javadoc and default value
agent/conf/agent.properties	Added commented default for `create.full.clone`

Comments suppressed due to low confidence (1)

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java:1327

There are no unit or integration tests verifying the new full-clone path; adding tests to cover both true and false scenarios for CREATE_FULL_CLONE would ensure the behavior remains correct.

                        if (createFullClone) {

Copilot · 2025-07-14T11:23:29Z

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

@@ -1315,14 +1317,22 @@ public KVMPhysicalDisk createDiskFromTemplate(KVMPhysicalDisk template,
                        passphraseObjects.add(QemuObject.prepareSecretForQemuImg(format, QemuObject.EncryptFormat.LUKS, keyFile.toString(), "sec0", options));
                        disk.setQemuEncryptFormat(QemuObject.EncryptFormat.LUKS);
                    }
+
+                    QemuImgFile srcFile = new QemuImgFile(template.getPath(), template.getFormat());
+                    Boolean createFullClone = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE);


Reading the agent properties on each disk creation may incur unnecessary overhead; consider caching the CREATE_FULL_CLONE value or loading it once at adaptor initialization.

Suggested change

Boolean createFullClone = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE);

Copilot · 2025-07-14T11:23:29Z

...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

@@ -1315,14 +1317,22 @@ public KVMPhysicalDisk createDiskFromTemplate(KVMPhysicalDisk template,
                        passphraseObjects.add(QemuObject.prepareSecretForQemuImg(format, QemuObject.EncryptFormat.LUKS, keyFile.toString(), "sec0", options));
                        disk.setQemuEncryptFormat(QemuObject.EncryptFormat.LUKS);
                    }
+
+                    QemuImgFile srcFile = new QemuImgFile(template.getPath(), template.getFormat());
+                    Boolean createFullClone = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE);


[nitpick] Prefer using a primitive boolean instead of the Boolean wrapper to avoid potential null unboxing issues and reduce overhead.

Suggested change

Boolean createFullClone = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE);

boolean createFullClone = Boolean.TRUE.equals(AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE));

DaanHoogland

took me a while of looking at this to have the penny drop. Code looks good @JoaoJandre , but what is the use case for this? I’d say one would want thin to be thin and fat to be fat. It seems now thin still makes a full clone given this setting and effectively becomes fat.

JoaoJandre · 2025-07-14T14:16:26Z

took me a while of looking at this to have the penny drop. Code looks good @JoaoJandre , but what is the use case for this? I’d say one would want thin to be thin and fat to be fat. It seems now thin still makes a full clone given this setting and effectively becomes fat.

@DaanHoogland THIN, FAT and SPARSE are translated to preallocation methods in qemu-img. THIN being OFF, SPARSE being METADATA and FAT being FULL.

When using preallocation=full, qemu-img will write the full disk size to storage (filling it with zeros). When using preallocation=off, only the actual necessary data is written.

So, if you convert an image with virtual size as 50 GB, but real size 5 GB, using FULL, you'll get an image with 50 GB of real size. Using OFF, you'll end up with an image of 5GB of real size.

This is the use case of THIN and full-clone. You copy the template but do not preallocate the full disk size.

DaanHoogland · 2025-07-14T14:39:27Z

This is the use case of THIN and full-clone. You copy the template but do not preallocate the full disk size.

ok, so this is not a thick provisioned copy but just a byte by byte copy.

slavkap · 2025-07-15T10:38:50Z

For me, the code looks good, but my only concern here is that the configuration create.full.clone should come from the management service as an API parameter or a property of a volume/VM, or template (I'm not sure exactly where). This is in case the administrators want some volumes to be full cloned and others not. In that case, they should change the value of the configuration every time from the agent.properties file, and restart the agent service

weizhouapache · 2025-07-15T11:38:17Z

For me, the code looks good, but my only concern here is that the configuration create.full.clone should come from the management service as an API parameter or a property of a volume/VM, or template (I'm not sure exactly where). This is in case the administrators want some volumes to be full cloned and others not. In that case, they should change the value of the configuration every time from the agent.properties file, and restart the agent service

makes sense

rosi-shapeblue · 2025-07-28T06:25:40Z

@blueorangutan package

blueorangutan · 2025-07-28T06:28:03Z

@rosi-shapeblue a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-28T08:35:25Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14409

JoaoJandre · 2025-07-29T12:00:51Z

@slavkap @weizhouapache I understand your point of having more flexibility by being able to choose whether individual volumes will be full or linked clone. However, in the use cases I've seen and discussed with users, they either wanted all the volumes to be linked or all of them to be full clone, that is why it was done this way.

Could we continue with the PR as is and create an issue to extend this so that operators can choose if they want full or linked clone with more granularity?

DaanHoogland · 2025-07-29T13:11:49Z

@slavkap @weizhouapache I understand your point of having more flexibility by being able to choose whether individual volumes will be full or linked clone. However, in the use cases I've seen and discussed with users, they either wanted all the volumes to be linked or all of them to be full clone, that is why it was done this way.

Could we continue with the PR as is and create an issue to extend this so that operators can choose if they want full or linked clone with more granularity?

I agree we can make this a follow-up issue. et tu, @slavkap ?

slavkap · 2025-07-29T14:32:37Z

@DaanHoogland, I just expressed my concerns, and opening an issue is fine with me. I'm also okay with the way the solution has been presented, but I’ll probably be able to review it in more detail tomorrow.

rosi-shapeblue

LGTM

Tested create.full.clone functionality on KVM hosts. Works as expected.

Testing Results:

Default/missing property: THIN volumes use linked clones (backing files created)
create.full.clone=true: THIN volumes use full clones (no backing files created)
create.full.clone=false: THIN volumes revert to linked clones (backing files created)
FAT/SPARSE volumes: Unaffected, remain full clones (tested with create.full.clone=true / create.full.clone=false)

Edge Cases Tested:

Missing property line: Defaults correctly to linked clone behavior
Property changes without restart: Requires agent restart (expected behavior)

Verified with qemu-img info - linked clones show backing files, full clones don't. No regressions observed.

sureshanaparti · 2025-07-30T09:53:45Z

@slavkap are you good with the PR, any concerns?

sureshanaparti · 2025-07-30T09:59:40Z

@JoaoJandre Will there be any issue in migration when one host has create.full.clone=false,and the other has it true. Is it marked for the volume any where in the volumes/other table? Also, Can we leverage the provisioning type in the disk offering for this?

JoaoJandre · 2025-07-30T16:30:36Z

@JoaoJandre Will there be any issue in migration when one host has create.full.clone=false,and the other has it true. Is it marked for the volume any where in the volumes/other table? Also, Can we leverage the provisioning type in the disk offering for this?

@sureshanaparti I have tested the migration and found no issues between hosts with different configurations.

It is not marked in the database currently, do you see a reason to do this?

I have explained here why the provisioning type cannot be the only parameter for this.

sureshanaparti · 2025-07-30T17:15:58Z

It is not marked in the database currently, do you see a reason to do this?

@JoaoJandre in any case, if the user or operator wants to know if the volume is a full clone or not (can get it through list volumes response?), and later point of time, if any operation on the volume depends on full clone factor.

slavkap

code LGTM, I didn't test it

JoaoJandre · 2025-07-31T12:44:11Z

It is not marked in the database currently, do you see a reason to do this?

@JoaoJandre in any case, if the user or operator wants to know if the volume is a full clone or not (can get it through list volumes response?), and later point of time, if any operation on the volume depends on full clone factor.

Currently no operation is affected by having a full clone or linked clone. There is a possible optimization to be done where we save whether a volume is full or linked clone and, during migration, check this information to know if we must have the template on the destination or not. But the process works 100% as it does now even without this optimization.

In any case, I would like to point out that if we take a snapshot of a linked clone volume and restore it, the volume will be "full clone" (it will not depend on the template anymore), and everything works as normal.

We could map an issue to show whether a volume depends on a template or not, although I don't see much need for it. And it would have to contemplate cases like the one I pointed regarding snapshot restoration.

create-full-clone-thin

b942c0d

JoaoJandre added this to the 4.21.0 milestone Jul 10, 2025

boring-cyborg bot added component:agent component:kvm labels Jul 10, 2025

sureshanaparti added this to Apache CloudStack 4.21.0 Jul 14, 2025

sureshanaparti moved this to In Progress in Apache CloudStack 4.21.0 Jul 14, 2025

sureshanaparti requested review from slavkap, DaanHoogland, weizhouapache, winterhazel, bernardodemarco and Copilot July 14, 2025 11:21

sureshanaparti added the status:needs-testing label Jul 14, 2025

Copilot AI reviewed Jul 14, 2025

View reviewed changes

DaanHoogland reviewed Jul 14, 2025

View reviewed changes

rosi-shapeblue self-assigned this Jul 22, 2025

rosi-shapeblue approved these changes Jul 30, 2025

View reviewed changes

rosi-shapeblue removed the status:needs-testing label Jul 30, 2025

slavkap approved these changes Jul 31, 2025

View reviewed changes

sureshanaparti merged commit 5ea1ada into apache:main Jul 31, 2025
22 of 25 checks passed

github-project-automation bot moved this from In Progress to Done in Apache CloudStack 4.21.0 Jul 31, 2025

	Boolean createFullClone = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE);
	boolean createFullClone = Boolean.TRUE.equals(AgentPropertiesFileHandler.getPropertyValue(AgentProperties.CREATE_FULL_CLONE));

Allow full clone volumes with thin provisioning #11177

Allow full clone volumes with thin provisioning #11177

Conversation

JoaoJandre commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sureshanaparti commented Jul 14, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

JoaoJandre commented Jul 14, 2025

Uh oh!

DaanHoogland commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slavkap commented Jul 15, 2025

Uh oh!

weizhouapache commented Jul 15, 2025

Uh oh!

rosi-shapeblue commented Jul 28, 2025

Uh oh!

blueorangutan commented Jul 28, 2025

Uh oh!

blueorangutan commented Jul 28, 2025

Uh oh!

JoaoJandre commented Jul 29, 2025

Uh oh!

DaanHoogland commented Jul 29, 2025

Uh oh!

slavkap commented Jul 29, 2025

Uh oh!

rosi-shapeblue left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

JoaoJandre commented Jul 30, 2025

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

slavkap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JoaoJandre commented Jul 31, 2025

Uh oh!

Uh oh!

JoaoJandre commented Jul 10, 2025 •

edited

Loading

codecov bot commented Jul 10, 2025 •

edited

Loading

DaanHoogland commented Jul 14, 2025 •

edited

Loading

rosi-shapeblue left a comment •

edited

Loading