Skip to content

KMS: fix volume key migration and lifecycle, use authenticated key wrapping for PKCS#11 - #14453

Draft
vishesh92 wants to merge 1 commit into
apache:mainfrom
shapeblue:fix-kms-issues
Draft

vishesh92 wants to merge 1 commit into
apache:mainfrom
shapeblue:fix-kms-issues

Conversation

@vishesh92

@vishesh92 vishesh92 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Description

This PR fixes several problems in the KMS volume encryption that came with 4.23.

Volumes migrated to KMS don't start anymore

migrateVolumesToKMS wrapped the legacy passphrase as it is stored in the passphrase table, which is its base64 text. When the key is read back, VolumeObject.getPassphrase() base64-encodes the unwrapped key again, so the volume gets a different passphrase from the one it was encrypted with. The VM fails on the next start or attach.

Migration now wraps the decoded passphrase. For keys already migrated on 4.23, a background task runs after the upgrade. It unwraps every existing key once and rewraps the ones that are in the old format. If a key is used by a volume created after the migration (e.g. a copy made by a storage migration on 4.23), that volume may have been encrypted with the wrong passphrase, so the task leaves the key alone and logs it.

Migration also removes the old passphrase row once no volume uses it. The upgrade removes rows that were already left behind.

Wrapped keys removed while still in use

VolumeDaoImpl.remove() removed a volume's wrapped key unconditionally. After a volume migration the new volume shares the key with the source, so expunging the source left the migrated volume without a key. Expunging a volume that has snapshots did the same to the snapshots: they could be neither restored nor deleted.

The DAO doesn't touch the key anymore. KMSManager.deleteKMSWrappedKey() and VolumeObject.deletePassphrase() keep the key (KMS or legacy passphrase) while another volume or a snapshot of the volume still needs it. The storage cleanup task releases it afterwards through VolumeService.releaseKeysOfRemovedVolumes(). On upgrade, keys removed this way are restored if their KMS key and KEK still exist. The ones that can't be recovered are logged.

Key wrapping in the PKCS#11 provider

The PKCS#11 provider wrapped keys with AES/CBC/PKCS5Padding, which detects neither a modified blob nor a wrong KEK. unwrapKey() also tried all KEK versions when the recorded one failed. So a wrong key could be returned without any error, and during a rotation it would be rewrapped under the new KEK and the real key lost.

  • The wrap algorithm is now set per HSM profile with the wrap_algorithm detail. The options are AES/GCM/NoPadding (the default), AES/KWP/NoPadding (needs Java 18 or later) and AES/CBC/PKCS5Padding. It can't be changed once the profile exists.
  • The upgrade sets existing PKCS#11 profiles to AES/CBC/PKCS5Padding, so nothing changes for them. To move existing keys over, create a profile with GCM or KWP and rotate the KMS keys onto it.
  • Each wrapped key records the algorithm it was wrapped with, in kms_wrapped_key.wrap_algorithm.
  • unwrapKey() uses the recorded KEK version first. It tries the other versions only for GCM/KWP keys, where a wrong KEK is detected.
  • Padding and tag failures give the same error message.
  • When a KMS key is created or rotated, the new KEK is tested with a wrap/unwrap round trip. An HSM that doesn't support the configured algorithm fails right there, and the KEK is deleted again.
  • KMSProvider got a deleteKek(kekId, hsmProfileId) variant for KEKs that don't have a version row (yet). Rotation now uses it when its DB update fails. Before, that KEK stayed in the HSM.
  • The algorithm shown for a KMS key is now the actual wrap algorithm, instead of always AES/GCM/NoPadding.

Other changes

  • deleteKMSKey no longer deletes the KMS key if the HSM fails to delete one of its KEKs, so the KEK isn't left in the HSM without anything pointing to it. A KEK that is already gone is fine, e.g. one deleted after a rotation. The PKCS#11 provider now reports such a KEK as not found instead of a generic failure.
  • For a removed volume whose key no longer exists, getPassphrase() returns an empty passphrase instead of throwing, so its snapshots can still be deleted.
  • kms.dek.size.bits is now checked to be 128, 192 or 256.
  • The database provider is documented as meant for testing. Its default profile is renamed from "HSM Database Provider" to "Database Provider", and enabling it logs a warning.

Schema changes: kms_wrapped_key gets wrap_algorithm and format_check_pending.

For the release notes: volumes already migrated on 4.23.0.0 are repaired automatically after the upgrade, once their HSM profile is reachable. Existing PKCS#11 profiles keep using CBC until their keys are rotated onto a GCM/KWP profile.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@vishesh92

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 48.23009% with 234 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.48%. Comparing base (c76c01e) to head (743cc40).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...udstack/kms/provider/pkcs11/PKCS11HSMProvider.java 16.16% 82 Missing and 1 partial ⚠️
...java/org/apache/cloudstack/kms/KMSManagerImpl.java 77.21% 46 Missing and 8 partials ⚠️
...main/java/com/cloud/storage/dao/VolumeDaoImpl.java 0.00% 39 Missing ⚠️
...java/com/cloud/upgrade/dao/Upgrade42300to2400.java 0.00% 21 Missing ⚠️
...e/cloudstack/storage/volume/VolumeServiceImpl.java 0.00% 16 Missing ⚠️
...pache/cloudstack/kms/dao/KMSWrappedKeyDaoImpl.java 0.00% 8 Missing ⚠️
...ava/org/apache/cloudstack/kms/KMSWrappedKeyVO.java 50.00% 6 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java 0.00% 5 Missing ⚠️
...apache/cloudstack/storage/volume/VolumeObject.java 92.85% 0 Missing and 1 partial ⚠️
...e/cloudstack/kms/provider/DatabaseKMSProvider.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14453      +/-   ##
============================================
+ Coverage     20.45%   20.48%   +0.03%     
- Complexity    21417    21457      +40     
============================================
  Files          6487     6487              
  Lines        589329   589669     +340     
  Branches      72246    72281      +35     
============================================
+ Hits         120554   120808     +254     
- Misses       455642   455712      +70     
- Partials      13133    13149      +16     
Flag Coverage Δ
uitests 3.70% <ø> (ø)
unittests 21.79% <48.23%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19644

@vishesh92
vishesh92 force-pushed the fix-kms-issues branch 2 times, most recently from 90b18fc to 6abe098 Compare October 21, 2026 15:13
@apache apache deleted a comment from blueorangutan Oct 21, 2026
@vishesh92

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19656

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants