Linked clone migration between file-based storages on KVM

[JoaoJandre]

Description

Currently, when a volume that was created with linked clone is live migrated from an NFS storage to another NFS storage in KVM, it continues to have the template as a backing file on the destination storage. In all other cases, such as NFS to SharedMountPoint, the volume is consolidated with its backing file during migration.

This special case adds unnecessary complexity to ACS; In most cases, performing the migration without consolidating the volume does not significantly optimize resource usage. On the other hand, when consolidating the volume with its backing file, the hypervisor detects empty sectors and removes them from the final volume, therefore, consolidating the volume can reduce storage usage.

Instead of only maintaining linked clone only on NFS to NFS migration, we now maintain it when migrating between NFS, shared mount-point and local storage, as all of these are file based and have no reason to always consolidate the volumes.

In order to fix the situations explained in
#7615 and #8834, we consolidate the volume when its template is deleted.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

I've tested creating a VM and migrating between NFS/SMP/Local storage, all migrations were successful, the template was copied to the storage and the VM was not corrupted.

I've also tested creating a VM, force deleting its template and then migrating it, the migration consolidated the volume, and the VM was not corrupted.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 35 lines in your changes missing coverage. Please review.

Project coverage is 15.07%. Comparing base (f0ba905) to head (f83c449).

Files	Patch %	Lines
...torage/motion/StorageSystemDataMotionStrategy.java	0.00%	33 Missing ⚠️
...cloud/hypervisor/kvm/resource/MigrateKVMAsync.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               4.19    #8911    +/-   ##
==========================================
  Coverage     15.07%   15.07%            
- Complexity    11168    11173     +5     
==========================================
  Files          5406     5406            
  Lines        472795   472807    +12     
  Branches      57834    58265   +431     
==========================================
+ Hits          71282    71299    +17     
+ Misses       393585   393577     -8     
- Partials       7928     7931     +3     

Flag	Coverage Δ
uitests	4.30% <ø> (ø)
unittests	15.79% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9252

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9256

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9262

[JoaoJandre]

@sureshanaparti could we run de CI here?

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-9870)

[blueorangutan]

[SF] Trillian Build Failed (tid-9873)

[blueorangutan]

[SF] Trillian test result (tid-9884)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45095 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8911-t9884-kvm-centos7.zip
Smoke tests completed. 127 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_events_resource	Error	296.55	test_events_resource.py
test_01_events_resource	Error	296.56	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	Failure	85.55	test_network_permissions.py
ContextSuite context=TestNetworkPermissions>:teardown	Error	1.39	test_network_permissions.py

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9536

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 9547

[JoaoJandre]

@blueorangutan package

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[JoaoJandre]

Could someone take a look ?
@mlsorensen, @weizhouapache, @rohityadavcloud, @borisstoyanov, @gpordeus

[weizhouapache]

Could someone take a look ? @mlsorensen, @weizhouapache, @rohityadavcloud, @borisstoyanov, @gpordeus

@JoaoJandre
code lgtm

is it possible to cover it by smoke test ?

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10576

[JoaoJandre]

@weizhouapache I would have to access the storage and check if the volume was linked clone before, and if it stayed after, I'm not sure how to do this in a smoke test. I can cover it with unit tests, those are easy.

Btw, could we run the CI here?

[weizhouapache]

@blueorangutan test alma9 kvm-alma9

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11042)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 45836 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8911-t11042-kvm-alma9.zip
Smoke tests completed. 132 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[rohityadavcloud]

cc @sureshanaparti @DaanHoogland @JoaoJandre @weizhouapache is this considered critical for the release? I think this may be useful for local storage users for example, but not sure what others are considering.

[DaanHoogland]

clgtm

[DaanHoogland]

@JoaoJandre , I will see what I can do in terms of testing this.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11026

[rohityadavcloud]

@blueorangutan test

[blueorangutan]

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11411)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 50262 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8911-t11411-kvm-ol8.zip
Smoke tests completed. 132 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_unsecure_vm_migration	Error	446.79	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	Error	3954.86	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	Error	3955.01	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	Error	142.23	test_vm_life_cycle.py

[DaanHoogland]

• tested by migrating a vm with storage to another host and its local storage,
• Then stopped the VM and migrated the volume to shared storage and started the VM again

no issues

[JoaoJandre]

cc @sureshanaparti @DaanHoogland @JoaoJandre @weizhouapache is this considered critical for the release? I think this may be useful for local storage users for example, but not sure what others are considering.

@rohityadavcloud I would like to have it merged into 4.20. But we are still missing a review/LGTM. It probably will not make it.