Skip to content

CLVM: remove inactive libvirt pool on storage pool delete and stop autostartign CLVM pools - #14379

Open
Pearl1594 wants to merge 3 commits into
mainfrom
clvm-delete-storage
Open

Pearl1594 wants to merge 3 commits into
mainfrom
clvm-delete-storage

Conversation

@Pearl1594

@Pearl1594 Pearl1594 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR fixes: #14204
CLVM/CLVM_NG pools are deliberately kept inactive in libvirt, because LVM is driven directly with vgs/lvs. ClvmStorageAdaptor had no deleteStoragePool of its own, so it used the generic LibvirtStorageAdaptor version. That version calls destroy() on the pool, and libvirt rejects destroy() on an inactive pool. As a result, a CLVM pool could not be deleted, and enabling maintenance on it failed the same way.

With this fix, deleting the pool in CloudStack only removes the CloudStack record and the libvirt pool definition on each host. It does not touch the volume group or the shared storage behind it. The VG, the lvmlockd/sanlock lockspace are left as they are, as CloudStack does not set any of these up. It only checks that the VG exists when the pool is added.
Tearing them down is the operator's responsibility, for example vgchange --lockstop, vgremove, iSCSI logout and releasing the LUNs.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Added a CLVM_NG/ CLVM pool
Put it in maintenance
Deleted the pool - Observed that the records were marked removed in the DB and the virsh pool-list --all no longer showed the volume group.

How did you try to break this feature and the system with this change?

Copilot AI balanced review requested due to automatic review settings October 9, 2026 20:48
@Pearl1594
Pearl1594 requested a review from nvazquez October 9, 2026 20:49
@Pearl1594 Pearl1594 added this to the 24.0 milestone Oct 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Error handling, resource cleanup, and an existing failing test must be addressed.

4 open findings
What changed in this PR

Fixes CLVM/CLVM_NG pool deletion by handling inactive libvirt pools correctly.

Changes:

  • Adds CLVM-specific pool deletion.
  • Disables libvirt autostart for CLVM pools.
File Description
ClvmStorageAdaptor.java Undefines CLVM pools without requiring activation.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@@ -290,7 +326,6 @@ private StoragePool createCLVMStoragePool(Connect conn, String uuid, String host
try {
StoragePool pool = conn.storagePoolDefineXML(poolDef.toString(), 0);
logger.info("Created libvirt pool definition for CLVM/CLVM_NG VG: {} (pool will remain inactive)", volgroupName);
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.66%. Comparing base (68cff97) to head (b880b67).
⚠️ Report is 68 commits behind head on main.

Files with missing lines Patch % Lines
...oud/hypervisor/kvm/storage/ClvmStorageAdaptor.java 87.50% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14379      +/-   ##
============================================
+ Coverage     20.21%   20.66%   +0.45%     
- Complexity    20750    21731     +981     
============================================
  Files          6426     6492      +66     
  Lines        580181   590759   +10578     
  Branches      71033    72453    +1420     
============================================
+ Hits         117287   122089    +4802     
- Misses       450180   455394    +5214     
- Partials      12714    13276     +562     
Flag Coverage Δ
uitests 3.74% <ø> (+0.05%) ⬆️
unittests 21.98% <87.50%> (+0.46%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI balanced review requested due to automatic review settings October 9, 2026 20:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active-pool deletion can deactivate the backing VG, while upgraded pools may retain their existing autostart flag.

4 open findings
2 resolved since last review

🧠 Review effort: Balanced

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

🔴 Test Coverage Grade: D — Marginal

Metric Value
Line coverage 25.76%
Branch coverage 20.07%

Grade Scale

Grade Line Coverage Meaning
🟢 A ≥ 80% Excellent - this code sleeps well at night 😴
🟡 B 60-79% Good - almost there, don't stop now 😉
🟠 C 40-59% Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D 20-39% Marginal - boldly shipping where no test has gone before 🖖
⛔ F < 20% Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

}

@Override
public boolean deleteStoragePool(String uuid) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pearl1594 , sonar has a good point here; this either returns true or throws an exception. I realise it is an override but is the return needed at all? (here or in undefineInactiveClvmPool(..))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the same pattern we follow for most the storageAdaptors too - LibvirtStorageAdaptor

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said the override may not be helped, but that doesn't mean the private method has to as well. Also the pattern might well need an anti-pattern.

Copilot AI balanced review requested due to automatic review settings October 26, 2026 18:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues
2 resolved since last review

Review effort: Lite

try {
undefineInactiveClvmPool(LibvirtConnection.getConnection(), uuid);
} catch (LibvirtException e) {
throw new CloudRuntimeException(e.toString(), e);
Comment on lines +93 to +107
@Override
public boolean deleteStoragePool(String uuid) {
logger.info("Attempting to remove CLVM/CLVM_NG storage pool {} from libvirt", uuid);

if (decStoragePoolRefCount(uuid)) {
logger.info("deleteStoragePool: CLVM/CLVM_NG storage pool {} still in use", uuid);
} else {
try {
undefineInactiveClvmPool(LibvirtConnection.getConnection(), uuid);
} catch (LibvirtException e) {
throw new CloudRuntimeException(e.toString(), e);
}
}
return true;
}
Comment on lines +109 to +127
private void undefineInactiveClvmPool(Connect conn, String uuid) throws LibvirtException {
StoragePool sp = lookupClvmPool(conn, uuid);
if (sp == null) {
logger.warn("CLVM/CLVM_NG storage pool {} doesn't exist in libvirt. Assuming it is already removed", uuid);
return;
}

try {
if (sp.isActive() == 1) {
sp.destroy();
}
if (sp.isPersistent() == 1) {
sp.undefine();
}
} finally {
sp.free();
}
logger.info("CLVM/CLVM_NG storage pool {} was successfully removed from libvirt", uuid);
}
@sonarqubecloud

Copy link
Copy Markdown

@nvazquez

Copy link
Copy Markdown
Contributor

Hi @Pearl1594 are there any pending review comments to be addressed or can we start a new round of packaging/tests?

@RosiKyu

RosiKyu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20084

@RosiKyu RosiKyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I tested the PR packages on a 4-host KVM cluster (Oracle Linux 8, lvmlockd + sanlock over iSCSI), with CLVM and CLVM_NG pools created earlier on the old build.

  • New CLVM and CLVM_NG pools: Autostart: no and inactive in libvirt on every host. Pools created on the old build keep Autostart: yes, as expected.
  • Maintenance + delete of a CLVM pool and of a CLVM_NG pool: both succeed, the libvirt pool is gone from every host, the VG is left alone.
  • The same VGs re-added as new pools: work, a disk is created and attached on the CLVM_NG one.
  • Delete of a pool that still has a disk: refused ("non-destroyed volumes associated to this pool"), nothing removed.
  • libvirt pool removed by hand on one host first: delete still succeeds, with a warning on that host.
  • Host reboot with the old autostart pools: the pools come up but no LV is activated at boot.
  • Delete after reconnecting all hosts twice: the pool is still removed from libvirt on every host.
  • Delete of a CLVM and of a CLVM_NG pool that is running and set to autostart in libvirt (same definition as the pools created on 4.23.0.0 in this env, which was upgraded from 4.23.0.0): removed from every host, the VG and its lockspace stay, other VMs keep running, the same VG can be added again.
  • Maintenance + cancel on a pool in use: the VM with its root disk there is stopped and started again.
  • Delete while one host is disconnected, and while one host is rebooting: works, the other hosts remove the pool.

A minor observation: the host that was disconnected during the delete still has the pool defined in libvirt after it comes back (after a reboot it even shows as running, autostart no); nothing removes it on reconnect. No LV was activated. Harmless here, just mentioning it in case cleanup on reconnect is wanted.

Host, delete of the CLVM pool: the libvirt pool is removed, the VG is still there

[root@ref-trl-12929-k-mol8-rositsa-kyuchukova-kvm1 ~]# grep -E '2026-11-17 (18:5|19:[34])' /var/log/cloudstack/agent/agent.log | grep -E 'still in use|successfully removed from libvirt|Assuming it is already removed|Failed to remove|Attempting to remove CLVM'
2026-11-17 18:54:30,196 INFO  [kvm.storage.ClvmStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:b8662707) Attempting to remove CLVM/CLVM_NG storage pool 18a44068-f219-3c60-b6b7-11ce09483858 from libvirt
2026-11-17 18:54:30,199 INFO  [kvm.storage.ClvmStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:b8662707) CLVM/CLVM_NG storage pool 18a44068-f219-3c60-b6b7-11ce09483858 was successfully removed from libvirt
[root@ref-trl-12929-k-mol8-rositsa-kyuchukova-kvm1 ~]# vgs delclvm
  VG      #PV #LV #SN Attr   VSize VFree
  delclvm   1   0   0 wz--ns 2.99g 2.74g

Host where the libvirt pool was removed by hand: the delete logs a warning and goes on

2026-11-17 18:55:10,629 WARN  [kvm.storage.ClvmStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:f7454807) CLVM/CLVM_NG storage pool 18a44068-f219-3c60-b6b7-11ce09483858 doesn't exist in libvirt. Assuming it is already removed

@RosiKyu

RosiKyu commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@RosiKyu a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-17590)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55826 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14379-t17590-kvm-ol8.zip
Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@DaanHoogland DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@RosiKyu

RosiKyu commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

@Pearl1594 could you please resolve the open threads? Thank you!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not able to delete CLVM_NG primary storage (Failed to delete storage pool on host)

6 participants