[fermosan]

Description

KVM-hosted VM disks have no way to configure libvirt/QEMU's error_policy (rerror/werror).
Every disk defaults to QEMU's report policy, so any I/O error from backing storage — even a
brief, otherwise-recoverable one — is passed straight through to the guest as a real disk
error. This PR adds first-class error_policy support to disk/compute offerings, modeled on
the existing cacheMode offering attribute, end to end: schema, domain model, API, KVM agent,
and the Vue UI (Add/Clone offering forms).

Problem. Tested against a KVM cluster backed by NFS primary storage during a planned,
successful storage-side HA failover (floating IP moved in ~6s, well within normal NFS client
timeouts): a VM with a continuous write workload hit a hard blk_update_request: I/O error on
its data disk within 3-4 seconds — long before any NFS-level recovery had a chance to run. The
guest's ext4 filesystem aborted its journal and remounted read-only. Root cause: the disk's
libvirt XML carries no rerror/werror attribute at all, so QEMU's default report policy
applies unconditionally, independent of how well the storage/HA layer itself is tuned.
Manually patching a live domain's XML with error_policy='stop' and re-running the same
failover eliminated the problem entirely — identical failover, zero guest-visible impact.
Today the only way to get this is manually patching each VM's persisted domain XML outside
CloudStack's orchestration, which gets silently overwritten the next time CloudStack redefines
the domain.

What this adds:

• Schema: disk_offering.error_policy column (default 'report'), propagated to
  disk_offering_view/service_offering_view/volume_view.
• Domain model: DiskOffering.DiskErrorPolicy enum (REPORT, STOP, IGNORE,
  ENOSPACE), getter/setter on DiskOffering/DiskOfferingVO.
• API: new errorpolicy param on createDiskOffering/updateDiskOffering and the
  compute-offering equivalents, surfaced on DiskOfferingResponse/ServiceOfferingResponse.
• Orchestration: value carried DiskOfferingVO -> VolumeObject/VolumeObjectTO ->
  hypervisor command, mirroring the existing cacheMode path.
• KVM agent: LibvirtVMDef.DiskDef renders error_policy='<value>' on the <driver>
  element, set from the volume's offering in KVMStorageProcessor.
• UI: errorpolicy exposed on the Add/Clone Disk Offering and Add/Clone Compute Offering
  forms, defaulting to report.
• Other hypervisors (XenServer/VMware) simply ignore the new offering field — this is a
  libvirt/QEMU-specific concept.

Functional behavior change: none for existing offerings/VMs (default report is the current
implicit behavior). Opt-in per offering going forward.

Default is report (fully backward compatible); supported on both disk and compute offerings,
matching cacheMode's precedent; alerting on a long-paused domain is intentionally left to
operators/monitoring, not part of this feature.

Fixes: #14366

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Extensive manual, end-to-end testing on a real KVM cluster with three storage backends: local
disk, a self-hosted NFS pool, and a real production HA NFS cluster (Pacemaker-managed,
active/standby).

• Core behavior: error_policy=stop pauses the VM on a real I/O error (via both a
  disk-full/ENOSPC test and a real production NFS HA failover) and resumes cleanly with zero
  data loss end-to-end (API -> DB -> KVM agent -> libvirt -> QEMU). Without the fix (default
  report), the same failover caused the guest to remount read-only within 3-4 seconds.
• Confirmed virsh update-device --live rejects changing error_policy on a running domain —
  it only takes effect from the persisted definition at next (re)start; a full stop/redefine is
  required for an already-running VM to pick up a changed offering.
• Mixed-storage topology: tested a 3-way rotation of root/data disks across local storage,
  local-NFS, and production-NFS, mirrored for both Linux and Windows guests, to rule out
  cross-contamination between disks on different pools on the same VM.
• Windows guest validation: confirmed identical pause/resume behavior to Linux once the
  virtio-blk driver is installed (Windows has no built-in virtio drivers, so a virtio-bus data
  disk is otherwise invisible to the guest).
• Combined-outage test (6-VM fleet, both NFS pools down simultaneously): only the VMs whose
  disks spanned both outaged pools actually paused; data integrity verified via a continuous
  per-second write-and-verify loop on every VM throughout, confirmed via the resulting log
  sequence rather than just checking VM state.

How did you try to break this feature and the system with this change?

• Disk-full (ENOSPC) on local storage, and a real HA failover on NFS-backed disks — both
  correctly paused under stop and resumed with zero data loss.
• Mixed root/data topology across three different storage backends simultaneously, on both
  Linux and Windows, specifically to look for cross-contamination between a VM's own disks.
• A simultaneous outage of two separate NFS pools across a 6-VM fleet, checking not just which
  VMs paused but verifying actual data integrity afterward, not just final VM state.
• Deliberately went looking for the feature's limits, not just its success cases: found and
  root-caused a related gap via live, controlled fault injection — a storage backend that goes
  silently unreachable under a hard NFS mount (as opposed to returning a real I/O error)
  bypasses error_policy=stop entirely, since the guest's I/O just hangs rather than erroring,
  with no visible signal to libvirt/CloudStack. Confirmed this is protocol-independent (same
  behavior under NFSv3 and NFSv4) via a dedicated isolated test. Worth documenting as a known
  limitation alongside this feature rather than treating it as a blocker — it's outside what
  error_policy can address by itself.
• Confirmed the config-change-doesn't-apply-live constraint so UX/documentation expectations
  are set correctly.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.74%. Comparing base (fd6b755) to head (c78dd18).
⚠️ Report is 4 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (fd6b755) and HEAD (c78dd18). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (fd6b755)	HEAD (c78dd18)
unittests	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##               main   #14499       +/-   ##
=============================================
- Coverage     20.63%    3.74%   -16.90%     
=============================================
  Files          6489      493     -5996     
  Lines        590136    42573   -547563     
  Branches      72361     8068    -64293     
=============================================
- Hits         121788     1595   -120193     
+ Misses       455100    40749   -414351     
+ Partials      13248      229    -13019     

Flag	Coverage Δ
uitests	3.74% <ø> (-0.01%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 19723

[fermosan]

@blueorangutan package

[blueorangutan]

@fermosan a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19726

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests