You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add Host HA support for CLVM/CLVM_NG pools and retry transient lock refusals on activation - #14450
This PR does two things for CLVM and CLVM_NG primary storage (lvmlockd + sanlock).
Host HA support
CLVM and CLVM_NG pools were not in LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT, so the KVM HA activity check never gave a usable answer for them and a failed host was never resolved.
Both pool types are now HA-capable.
No heartbeat file is written for CLVM pools. The sanlock delta lease is the heartbeat, and wdmd resets a host that stops renewing it.
A neighbour host reads the lease with sanlock client host_status -s lvm_<vg> -D. The host is dead only when last_check - last_live >= 8 * io_timeout + 60 (140s with the default io_timeout). Missing or unparsable state is treated as alive, so a host is never declared dead without evidence.
The agent reports its sanlock host id (lvmconfig local/host_id) as the host detail host.sanlock.id. KVMHostActivityChecker passes it to the checking host in HostTO.sanlockHostId. Sanlock host ids are host-wide and can repeat across clusters, so the lease is looked up by VG name plus host id.
KVMHAMonitor no longer unmounts or checks libvirt for CLVM pools, and deleting a CLVM pool removes it from the monitor.
When recovery needs an exclusive activation target, hosts that did not answer the lock-state query are skipped.
Retry of transient lock refusals
After a holder failed or rebooted, the first exclusive activation (lvchange -aey) can be refused for a short time: either the previous holder's sanlock lease has not expired yet (LV locked by other host), or the lockspace is still being joined on a rebooted host (lock start in progress). Previously the activation was tried once, the lookup returned no holder, and the VM start went ahead with the LV inactive.
The agent then waits for the device node (KVMStoragePoolManager.getPhysicalDisk, 100 tries every 3 seconds). That wait only checks whether /dev/<vg>/<lv> exists. It never requests the lock, and ClvmStorageAdaptor.handleMissingDeviceNode activates only template- volumes. So even after the lease expires nothing activates the LV and the start fails after about 6-7 minutes. HA then retries only after restart.retry.interval (600s by default).
ClvmPoolManager now retries the exclusive activation on the chosen host until sanlock grants it or the budget runs out:
Only locked by other host and lock start in progress are retried. Any other failure or exception is a single attempt, as before.
New settings: clvm.lock.recovery.retry.timeout (default 180s, counted from the start of the lookup; 0 disables the retry) and clvm.lock.recovery.retry.interval (default 10s). The lease expires 8 x io_timeout + 60 seconds after the holder stops renewing, so raise the timeout if lvmlockd runs with a larger --sanlock-timeout.
The budget is shared by all volumes of one VM start, so a VM with several volumes cannot wait several times the timeout.
sanlock still decides who may hold a lock. An LV was never exclusive on two hosts at once in any test.
Behaviour changes to be aware of
Existing CLVM/CLVM_NG clusters become HA-eligible on upgrade. When a host disconnects, KVMInvestigator can now conclude it is down, send the "Host is down" alert, and HA can restart HA-enabled VMs elsewhere. Before this, investigation was skipped for CLVM-only clusters.
Requirements on every host: sanlock, wdmd, lvmlockd and lvmlocks enabled at boot, and a working watchdog device.
Types of changes
Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Major
Minor
Screenshots (if appropriate):
How Has This Been Tested?
HA support
Failover with Host HA enabled.
Failover with VM HA only, Host HA off.
Ablocked the host's connection to management, and HA correctly left the host alone because its storage lease was still alive - no fence
Mixed NFS and CLVM pools, stale NFS heartbeat with a live lease: host stays alive, no fence.
sanlock timing: another host's lvchange -aey is refused until about 140s after the last renewal.
Skipping unreachable hosts as the activation target, in four scenarios:
Host unreachable but still Up in the DB (kvm1's agent port blocked): recovery no longer picked kvm1 and chose kvm1, where sanlock refused at once.
Host already marked Alert: no commands were sent to kvm1 at all, and the StartCommand went out within 2 seconds.
Healthy cluster: a start on the holder took ~10s and the DB lock host was corrected. A start on the other host did a normal lock transfer in ~15s.
Holder fully isolated, VM started on the other host 8 seconds later, DB host still Up: the first query waited 130s, and then the activation succeeded at once because the wait ended about 20 seconds after the lease expired. The VM was Running about 5 minutes after the request.
Alive lock holder, VM start on another host: safe, the start fails.
Retry
Healthy path (stop, start, start on the other host, live migrate): no retry lines, normal times, lock host followed the VM.
Holder rebooted, HA restarts the VM, 2 volumes: root volume retried (11 refusals, lock start in progress) and the data volume activated at once. VM Running in 232s. Without the retry the same scenario failed after 6m47s and the VM stayed stopped.
Holder hard-reset and kept down, VM started on the survivor inside the lease window: 5 refusals (locked by other host) over 48s, activated about 136s after the reset, VM Running 142s after the reset.
Holder alive but unreachable from management, VM started on the other host: the start fails as expected, the LVs never became active on the other host, and the second volume got no extra retry wait.
Retry disabled (clvm.lock.recovery.retry.timeout=0): one refusal per volume, no retry, the start fails as before. Setting back to 180 restored the retry without a restart.
Two HA VMs (3 volumes) failing over together: both Running within 10 seconds of each other, both root volumes retried in parallel, the data volume activated at once, two HA worker threads used.
In all runs LV state was polled on both hosts every 3 seconds: no LV was ever exclusively active on two hosts, and there were no CLVM-related errors in the management server log.
How did you try to break this feature and the system with this change?
❌ Patch coverage is 64.95327% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.47%. Comparing base (68cff97) to head (fdcace1). ⚠️ Report is 40 commits behind head on main.
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes: #14178
This PR does two things for CLVM and CLVM_NG primary storage (lvmlockd + sanlock).
CLVM and CLVM_NG pools were not in
LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT, so the KVM HA activity check never gave a usable answer for them and a failed host was never resolved.sanlock client host_status -s lvm_<vg> -D. The host is dead only whenlast_check - last_live >= 8 * io_timeout + 60(140s with the default io_timeout). Missing or unparsable state is treated as alive, so a host is never declared dead without evidence.lvmconfig local/host_id) as the host detailhost.sanlock.id.KVMHostActivityCheckerpasses it to the checking host inHostTO.sanlockHostId. Sanlock host ids are host-wide and can repeat across clusters, so the lease is looked up by VG name plus host id.KVMHAMonitorno longer unmounts or checks libvirt for CLVM pools, and deleting a CLVM pool removes it from the monitor.After a holder failed or rebooted, the first exclusive activation (
lvchange -aey) can be refused for a short time: either the previous holder's sanlock lease has not expired yet (LV locked by other host), or the lockspace is still being joined on a rebooted host (lock start in progress). Previously the activation was tried once, the lookup returned no holder, and the VM start went ahead with the LV inactive.The agent then waits for the device node (
KVMStoragePoolManager.getPhysicalDisk, 100 tries every 3 seconds). That wait only checks whether/dev/<vg>/<lv>exists. It never requests the lock, andClvmStorageAdaptor.handleMissingDeviceNodeactivates onlytemplate-volumes. So even after the lease expires nothing activates the LV and the start fails after about 6-7 minutes. HA then retries only afterrestart.retry.interval(600s by default).ClvmPoolManagernow retries the exclusive activation on the chosen host until sanlock grants it or the budget runs out:locked by other hostandlock start in progressare retried. Any other failure or exception is a single attempt, as before.clvm.lock.recovery.retry.timeout(default 180s, counted from the start of the lookup; 0 disables the retry) andclvm.lock.recovery.retry.interval(default 10s). The lease expires 8 x io_timeout + 60 seconds after the holder stops renewing, so raise the timeout if lvmlockd runs with a larger--sanlock-timeout.Behaviour changes to be aware of
sanlock,wdmd,lvmlockdandlvmlocksenabled at boot, and a working watchdog device.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Screenshots (if appropriate):
How Has This Been Tested?
HA support
Retry
lock start in progress) and the data volume activated at once. VM Running in 232s. Without the retry the same scenario failed after 6m47s and the VM stayed stopped.locked by other host) over 48s, activated about 136s after the reset, VM Running 142s after the reset.clvm.lock.recovery.retry.timeout=0): one refusal per volume, no retry, the start fails as before. Setting back to 180 restored the retry without a restart.How did you try to break this feature and the system with this change?