Skip to content

mdc/production: encrypted live migration and online snapshots of encrypted RBD volumes - #25

Merged
calvix merged 2 commits into
mdc/productionfrom
review/mdc-production-3.1.0
Nov 12, 2026
Merged

calvix merged 2 commits into
mdc/productionfrom
review/mdc-production-3.1.0

Conversation

@calvix

@calvix calvix commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Adds two topics to mdc/production for the next release, cs-4.23.0.0+mdc-3.1.0. Each topic is one commit.

1. kvm: encrypt KVM live-migration data stream with QEMU-native TLS (183c2f4546)

Live migration encrypts only the libvirt control channel. Guest RAM, and the disk for storage migration, cross the migration network in plaintext. With the new zone setting kvm.migrate.tls=true, CloudStack sets VIR_MIGRATE_TLS, reusing the certificates the CA framework already provisions.

  • It is used only when both hosts advertise host.migrate.tls. Otherwise the migration falls back to the plaintext stream.
  • The default is off, so nothing changes until the setting is enabled.
  • Ported from fork kvm: encrypt the KVM live-migration data stream with QEMU-native TLS (VIR_MIGRATE_TLS) #1 (4.22.1.0), which ran on dev in July. TLS 1.2 and ciphertext on the data ports were seen on the wire, and the URI stays tcp:.
  • Conflicts with the RBD-encryption topic, resolved by keeping both: the host capability line and method in LibvirtComputingResource, and the release notes.
  • To enable it per zone: the hosts need /etc/pki/qemu and migrate_tls_x509_cert_dir in qemu.conf (re-provision the host certificates), then set the setting.
    • Dev hosts already have both.
    • Stable hosts have neither.

2. kvm: allow online snapshots of encrypted RBD volumes (8a2f35899f)

Snapshots of encrypted volumes needed the VM stopped. For RBD kept on primary storage (snapshot.backup.to.secondary=false, as on dev and stable), a running VM now gets a native rbd snap create.

  • The guest filesystems are frozen through the guest agent when it is available. Otherwise the snapshot is crash-consistent.
  • Other encrypted cases are still refused.
  • storage: KVM - enable RBD/Ceph volume encryption support apache/cloudstack#13556, backported as the RBD-encryption topic, does not cover this: the refusal is still in KVMStorageProcessor.
  • Conflicts with the RBD lock-leak topic, resolved:
    • getRbdSnapshotSize stays protected, as the lock-leak topic made it;
    • the freeze and thaw methods and their tests sit next to that topic's tests;
    • the release notes keep all entries.

Testing

Module build with dependencies (api, core, components-api, orchestration, datamotion, server, KVM): 0 checkstyle violations.

test class                                tests
----------------------------------------  -----
LibvirtComputingResourceTest                329
VolumeApiServiceImplTest                    159
VirtualMachineManagerImplTest               100
StorageManagerImplTest                       92
KVMStorageProcessorTest                      55   (3 new)
StorageSystemDataMotionStrategyTest          33
LibvirtMigrateCommandWrapperTest             29
LibvirtStorageAdaptorTest                    29
RbdEncryptionTest                            10
MigrateKVMAsyncTest                           3

Not tested yet: either feature on a cluster with this build. That's planned on dev after the release build.

Topic branches: feature/kvm-migrate-tls-4.23 (on 4.23.0.0) and feature/kvm-online-encrypted-rbd-snapshots-4.23 (on the RBD-encryption topic 66e719ab5e).

Václav Rozsypálek added 2 commits November 12, 2026 11:36
CloudStack's secure KVM live migration only encrypts the libvirt control
channel; the guest RAM (and, for storage migration, non-shared disk) data
stream is transferred in plaintext over TCP. This adds an opt-in QEMU-native
TLS path (VIR_MIGRATE_TLS) for the migration data stream, reusing the
certificates the CA framework already provisions - mirroring the existing
VNC-TLS pattern.

- New zone-scoped global setting 'kvm.migrate.tls' (default false) on
  StorageManager; registered by StorageManagerImpl.
- The agent advertises the 'host.migrate.tls' capability in
  StartupRoutingCommand when the QEMU migration certificates exist under
  /etc/pki/qemu and qemu.conf sets migrate_tls_x509_cert_dir.
- The management server sets MigrateCommand.migrateTls only when the setting
  is enabled for the zone and BOTH the source and destination hosts advertise
  support; otherwise it silently falls back to the plaintext data stream, so
  mixed / partially-upgraded fleets keep migrating (VirtualMachineManagerImpl
  and StorageSystemDataMotionStrategy).
- MigrateKVMAsync ORs in VIR_MIGRATE_TLS; the data URI stays "tcp:" because
  libvirt has no "tls:" migration URI scheme (valid: tcp/rdma/unix/fd) - TLS
  is negotiated over the normal tcp: connection.
- Agent scripts provision the /etc/pki/qemu certificate set (keystore-cert-import)
  and write the qemu.conf migrate_tls_* keys (serviceConfig.py).
- Unit test asserts the data URI stays "tcp:" when migrateTls is enabled.
- PendingReleaseNotes entry.

Verified end-to-end on a 3-node KVM/Ceph dev cluster (libvirt 12): a live
migration between two secured hosts shows a TLS 1.2 handshake and ciphertext
on the data ports (no QEVM magic / no legible guest RAM), with migration
still succeeding.

(cherry picked from commit 93f18f1)

Upstream: none yet (fork #1)
Encrypted volume snapshots previously required the Instance to be stopped,
because the running-VM path uses a qemu external-snapshot overlay that cannot
handle the encryption layer. For Ceph/RBD kept on primary storage
(snapshot.backup.to.secondary=false) we can instead take a native,
encryption-transparent librbd snapshot ('rbd snap create') of the live image:
librbd snapshots operate on the raw image at the Ceph layer, work on a live,
exclusive-locked image and need no passphrase.

- KVMStorageProcessor.createSnapshot: for a running VM with an encrypted RBD
  volume kept on primary, take a native snapshot via the new
  takeRbdVolumeSnapshotOfRunningVm instead of refusing. The guest filesystems
  are frozen via the qemu guest agent for an application-consistent snapshot
  when the agent is available, and always thawed in a finally block; when the
  agent is unavailable a crash-consistent snapshot is taken. Every other
  encrypted case (qemu-native LUKS, or backup-to-secondary) still requires the
  VM to be stopped.
- VolumeApiServiceImpl: relax the management-side guard to match.
- Unit tests for the best-effort freeze decision and thaw safety.
- PendingReleaseNotes entry.

Validated on a KVM/Ceph dev cluster (libvirt 12): a live 'rbd snap create' on
an in-use, exclusive-locked, encrypted image succeeds without disrupting the
VM, guest-fsfreeze/thaw works alongside it, and the snapshot reverts to the
correct data.

(cherry picked from commit ccd915c98f815a524ef256d5eeb299a2233846ae)

Upstream: none yet - depends on apache#13556 (merged to main)
@calvix
calvix merged commit 8a2f358 into mdc/production Nov 12, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant